Search Engine Optimization: Robots.txt

Get Your Robots.txt File in Shape

Robots.txt files are used for only one reason to tell search engine spiders which pages not to index. A common misconception is that a robots.txt file can somehow be used to encourage search engines to crawl a site. Not true! Pages that meet the guidelines outlined above are readily spidered by search engines without needing additional encouragement. As you may have noticed, an important part of search engine optimization (SEO) is identifying the elements that cause indexing difficulties for the spiders and eliminating those elements.

So why might you want to tell a search engine not to index some of your pages? Because search engine spiders function with limited time and resources when indexing site, your site will be better served by focusing on getting your important customer-development, product, and sales pages indexed.

Case-in Point: Why would you want a search engine to index your shopping cart? Chances are there is no benefit to you when your shopping cart checkout pages show up in the search engine results. Use a robots.txt file to make sure search engines don’t waste time indexing your shipping cart when they could be using their resources indexing your more important sales or informational content pages.

Other pages you will want to keep search engine spiders away from include anything in your cgi-bin folder as well as directories that contain images or otherwise sensitive company data. Basically, if there isn’t any benefit to having a page (or image) shows up in the search results, then you should hide it from the spiders by using a robots.txt file.

That will not only increase the search engine resources spent on your important pages, but will also have the important side benefit of making your site safer from hackers who may otherwise use search engine results to acquire sensitive information about your company or site. Search engine spiders are pretty voracious about indexing anything they can find on the web, including things like password files, so do be careful.

There is one other issue to be aware of when it comes to robots.txt files. A surprising number of sites have accidentally set up their robots.txt files to prevent search engine spiders from crawling their site.

Did you know that adding the following to lines to your robots.txt file is enough to keep all the major search engines from ever crawling your site?
User-agent:*
Disallow: /

Many people don’t and then wonder why they can’t find their site listed in the search engines. Don’t let this happen to you!