Good bot, bad bot

If you haven't got an inkling what this bot business is about, you're really missing out.

• The Web Robots Pages – http://www.robotstxt.org/faq.html

• RobotsGen – http://robotsgenerator.com/wordpress-robots.html

Bot what?

Essentially, bots are scouts, seeking out information. Often, that's for a mutually-beneficial reason such as to furnish search engines with your latest-greatest. Sometimes these scripts are deployed by corporations, scrutinizing the use of their brand. Frequently they are harvesters, listing e-mail addresses or scraping site content. All too often they are pure evil, searching for site weaknesses. In all cases, they report back to their botmasters.

Take a peek at your server access logs to see how bots operate, for good and bad. If you install a plugin such as WordPress Firewall, scrutinize the alert e-mails to see how evil bots try to penetrate your site, some making dozens of requests within seconds, for example, as they automate directory traversal techniques...