Free SEO Tools

Robots Generator

Use this tool to generate a simple robots.txt file for your website, allowing you to hide files or directories that you don’t wish to be spidered. This generator tool is designed to create the text for the file for you and for you to then make changes should you feel necessary to afterwards.
Generating a robot file with this little number is ideal if you wish to block certain directories or files from search engines.

To use the generator tool, enter the required information below and click the button. You will then be shown the text for the file. Copy this to a file called robots.txt and place on the root of your website (same place as your home page).

Imposing Restrictions
You may impose restrictions on which WebPages to disallow indexing. By default, most users will want to allow all directories except their /cgi-bin/ directory which commonly holds scripts and images directory /images/. To enable all WebPages, select Yes to "Enable All WebPages”, enter each webpage or directory path in the exclusion box, one per line.

Example: "http://www.yourdomain.com/cgi-bin/" (Excludes /cgi-bin/ directory)
Example: "http://www.yourdomain.com/images/" (Excludes /images/ directory)
Example: "http://www.yourdomain.com/welcome.html" (Excludes /welcome.html webpage)