robots.txt

Search engines are our guests. They should be looked after and treated respectfully, because when they get grumpy with you they can cause all sorts of problems. For example, they could stop sending you visitors. Never forget that search engines are businesses themselves. They are not obliged to feature you in their search results.

Creating a file and placing it conveniently on your website server will let the search engines know that they are welcome, but you also need to point out that there are some places that a guest should not go. That's fine, they won't be offended. In fact, they will be quite pleased that you saved them the bother of searching areas that are not needed. Want to see what robots.txt looks like? Have a look at the following screenshot or download your copy of the file from the main, top-level folder of your website:

You can see that the robots.txt file is not like brain surgery; moreover, there are many things you can do with one. For a full exploration of robots...