I am a robots.txt

Search engines are our guests. They should be looked after and treated respectfully because when they get grumpy with you they can cause all sorts of problems. For example, stop sending you visitors. Never forget: the search engines are businesses themselves. They are not obliged to feature you in their search results.

Creating a file and placing it conveniently on your website server will let the search engines know they are welcome, but also point out that there are some places that a guest should not go. That's fine they won't be offended. In fact, they will be quite pleased you saved them the bother of searching areas that are not needed. Want to see what robots.txt looks like? Have a look at the following screenshot or download yours from the main top-level folder of your website:

You can see that the robots.txt file is not brain surgery, but there are many things you can do with one. For a full exploration of robots.txt files, have a look at www.robotstxt.org, or to...