nice SEO FAQ

If you want to allow robots/spiders to index all of the content it finds on your webpage (i.e. all linked content or ocntent listed on your sitemap) then you can use the following in your robots.txt file .

 User-agent: *

 Disallow:

It is prefered for Google that you have either a blank robots.txt file, or you have the content above. If the request for the robots.txt file on your domain returns an error or a redirect status, your domain is at risk of being removed from the index. Take our advice and at least use the                 above.