Robots.txt Generator

Build a robots.txt file with allow and disallow rules, sitemap links and optional AI-crawler blocking.

About this tool

A robots.txt file tells search engine crawlers which parts of your site to visit. Set your rules, then copy or download the finished file.

How to use Robots.txt Generator

  1. Choose whether to allow all crawlers or block everything.
  2. Add paths to disallow or allow, and your sitemap URL.
  3. Copy or download robots.txt and upload it to your site root.

Frequently asked questions

Where do I put robots.txt?

At the root of your domain, for example https://example.com/robots.txt.

Does Disallow hide a page from Google?

Not reliably. A blocked URL can still be listed. Use a noindex tag to keep a page out of results.