Using a robots.txt File

What is a robots.txt file?

A robots.txt file provides restrictions to search engine robots (known as "bots") that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.

Does my site need a robots.txt file?

Only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).

Where should the robots.txt file be located?

The robots.txt file must reside in the root of the domain. A robots.txt file located in a subdirectory isn't valid, as bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location. But, http://www.example.com/mysite/robots.txt is not. If you
don't have access to the root of a domain, you can restrict access using the Robots META tag.

Getting Out of Google Supplemental Results

Getting out of the Google Supplemental Results may be possible by improving your website navigation system. To get more pages fully Google indexed, the prominence of important website pages can often be boosted by linking to them from pages within your domain having the highest Page Rank, such as your homepage. The reason for this being that Page Rank is passed from one page to another by links and the most common cause of Supplemental results is lack of Page Rank.

Start by determining your most important web pages which have been made supplemental - for example those promoting lucrative products and services, and then improve your website internal linking by adding links to these pages from more prominent fully Google indexed pages of your site including your homepage. At the same time, ensure that your website navigation system is search engine friendly using a website link analyser.