Robots.txt
Why do you need this file?

In general, robots are nice, being sent out by search engines whose listings will improve visits by real people who you will want to visit your site.

However, you may not want them going into every corner.

The answer is a little text file called robots.txt which all well behaved robots look for and which tells them where they can go and where they can not go.

If you want to just say welcome, go anywhere in the web files folder, then just an entirely blank file called robots.txt can be placed in the top web files folder, which, on sites hosted by WorsleyWeb is public_html.

If you want to put some folders off limits, like cgi-bin or your images folder then you could create a file like this:

User-agent: *
Disallow: /images/
Disallow: /cgi-bin/ 

You might also want to disallow a temp folder where you have things which will not be around for long.

You can put a # sign at the beginning of a line to make that a comment (maybe so you can remind yourself why you did something). This sign can also be used within a line to say that all the rest of that line is a comment.

Without a robots.txt file you may see a lot of entries in the errors report, because all well behaved robots will look for it.

If you have any problems and you are on a site hosted by us, then just ask and we will put program for site updating and management)

For more detailed information, see
A Standard for Robot Exclusion