Free Robots.txt Generator

Build robots.txt rules with User-agent, Allow, Disallow and sitemap lines, then download the file for your website.

What is a robots.txt generator?

The robots.txt file sits at the root of your website and tells search engine crawlers which areas they should not crawl. Most small websites need only a very simple file that allows everything and points to the sitemap.

How to use this tool

  1. Choose a preset such as allow everything, or block a folder you do not want crawled.
  2. Add any Disallow or Allow rules for the folders you want to control.
  3. Add the full address of your sitemap.
  4. Download robots.txt and upload it to the root of your website so it opens at yourdomain.com/robots.txt.

Tips

Robots.txt is not a way to hide private pages. A blocked address can still appear in search results if other pages link to it. To keep a page out of search results use a noindex tag, and protect private content with a password.

Frequently asked questions

Where does robots.txt go?

In the root folder of your website, so it opens at yourdomain.com/robots.txt.

Does Disallow remove a page from Google?

Not by itself. It only asks crawlers not to crawl the page. Use a noindex tag to keep a page out of search results.

Can a wrong robots.txt hurt my site?

Yes. A rule like Disallow: / blocks your whole site from crawlers, so double check before you upload.

Do I need a robots.txt file?

It is optional. Without one, crawlers can crawl everything they find.

More free SEO tools