Robots.txt

ID #1018

How to block all robots from a certain folder

It may happen that you have certain sub directories that you don't want to be listed in the search engines. All major search engines that follow the REP , i.e. Yahoo, MSN and Google, and the promise not to index folders and files that are specifically mentioned on your robots.txt file. Here is an example to excluse subfolders /private/ /photos-saved/ and /images-fullsize/ from the search index,
User-agent: *
Disallow: /private/
Disallow: /photos-saved/
Disallow: /images-fullsize/
By using User-agent: * it referes to the fact that all search engines should obey this request.  

Tags: seo, robots.txt, REP

Related entries:

Last update: 2008-08-18 22:15
Author: Paul Savage
Revision: 1.0

Digg it! Print this record Send to a friend Show this as PDF file
Propose a translation for Propose a translation for
Please rate this entry:

Average rating: 0 out of 5 (0 Votes )

completely useless 1 2 3 4 5 most valuable

You can comment on this entry