Anonymous

Sydney, Australia

software developer

You should NOT rely on robots.txt for security! All it does is say, in effect, "if you are an automatic process, please don't search in the following spots..." Yes, Google obeys it, so your crappy security will be harder to find if the attackers just use Google. But not all spiders obey it, and web spiders aren't the only way to find vulnerabilities. For example, another way is to read your robots.txt file and assume that anything it excludes is likely to be interesting!

So setting up robots.txt without securing a site is like putting a "Keep Out" sign on an unlocked door. Nice people will obey it, crooks will go "hmm, wonder what's behind ... ooh, it's unlocked!"

If you are sufficiently aware of the problem to configure robots.txt, then do the job properly and actually secure your resources.