It is important to have robots txt on your site, this allows you a degree of control over where the search engine spiders visit and what they report and store in their index about your site, this allows you to keep them away from commercially sensitive material or out of member only information if you have a membership area on the site.

Matt Cutts a leading engineer at Google has produced a video that explains the robots txt in more detail and and answes some questions such as

Why is my url showing up in Google when I blocked it in robots.txt? Did you fetch that url?

How do I make that url disappear from Google?

The video can be seen on Matts Blog at http://www.mattcutts.com/blog/robots-txt-remove-url/

This is going to be be of interest to any one who needs to know how to sculpt the page rank of their sites to ensure they are getting the most from the search engines.

This is an important part of search engine optimisation and implementing these changes on your site in the appropriate manor can help your rankings in the search engines.

One word of advice, do not go too over board with the robots txt file or you might find you have stopped the search engines indexing some important parts of your website and this will see you fall in the rankings rather than rise.