Robots management
A webmaster has the chance to instruct search engines on what they should and should not index on a specific website. These instructions are provided through a simple text file called robots.txt
that should be placed in the website root folder.
This file generally contains a set of rules that allow or disallow indexing specific pages or sections of a website. For example, it is advisable to disable indexing of the backend section of the website; or, in certain circumstances, it can be useful to disable indexing of images or a specific type of file (such as PDFs). Or you may want to index a specific section of the website. All these commands can be provided through the robots.txt
file.
Joomla comes with a default robots.txt
file out of the box that includes the following content:
# If the Joomla site is installed within a folder # eg www.example.com/joomla/ then the robots.txt file # MUST be moved to the site root # eg www.example.com/robots.txt # AND the...