By Gemma Simpson
Published: Monday 25 September 2006
Email story to a Friend | Report Abuse
Name
Joe Whitehead
Location
USA
Occupation
PC tech
Comment
Reinventing the wheel? Just ask any experienced website operator what methods can be used to control/ban spiders and their user's. Hint: Most search engines do obey robots.txt.
It's a bit silly that currently available methods,...
Anonymous
Do these guys know about robots.txt? Evidently not...
Anonymous
so putting sensitive content in specific folders a...
Karen Challinor
Reinventing the wheel? Just ask any experienced w...
Joe Whitehead
Agenda Setters 2009
Welcome to the ninth annual Agenda Setters poll – silicon.com's list of the top 50 most influential individuals in the technology and IT industries, from techies and CIOs to entrepreneurs and business leaders. Find out more in our latest special report.
Copyright © 2008 CBS Interactive Limited. All rights reserved. Top of page
RSS Feeds
The Round-Up The Weekly Round-Up: 03.12.09 'Ere guv, you'll never guess who I had in the back of my cab the other day…'
Stuart Roberts Shared services - how to get it right in your business Recession boosts uptake