Increasing Search Accuracy by Generating Search Engine Specific Pages

As our menus on our templates display not only the name of the current posting but links to other postings, we are creating additional noise for the search results. The way to overcome this is to determine when a page is requested by a crawler and then hide the listing controls. Because search engines traverse sites from links within the page, we cannot simply remove the navigational controls leaving only the main posting content: we must also create hyperlinks to the channel items (postings and channels) within the current channel.

Interrogating the User Agent

To determine if the site is being crawled, we will create a helper method that checks the Request.Header["User-Agent"] value and compares it to a list of known search agents stored in the web.config file.

Firstly, we have to set up a list of search agents in the web.config file. Under the <configuration> | <appSettings> element, we will insert an ...