Unwanted WWW Robots (bots) I recently woke up to the staggering level of abuse occurring on my web site. This is old news to some, but we all wake up at different times. I'm talking, of course, about automated robots and spiders. They come at all hours, they take as much as they can, and they leave me with the (bandwidth) bill. They do so without respecting the Robot Exclusion Standard (robots.txt), now almost 10 years old. Some come to gather email addresses, which are then sold to spammers; some come to steal images or other content, and republish it without my consent; some come to spy on me and sell information to their clients about perceived violations of copyright, trademark, or some nebulous concept of brand identity.