A robots.txt file sits at the root of your domain (for example example.com/robots.txt) and gives crawlers simple rules about which paths they should or should not request. It is a polite request, not a security feature, so never use it to hide private pages.
The four lines you will use most
- User-agent names the crawler a group of rules applies to.
*means every crawler. - Disallow blocks a path prefix, such as
/admin/. - Allow re-opens a path inside a blocked folder.
- Sitemap points crawlers to your XML sitemap with a full URL.
A safe starting point
For most small sites the best file is almost empty: allow everything, block internal search pages and list your sitemap. Blocking CSS or JavaScript folders is a common mistake because search engines need those files to render your pages.
Mistakes to avoid
- Leaving
Disallow: /in place after a staging site goes live. - Using robots.txt to remove a page from search. Use a
noindexmeta tag instead, and leave the page crawlable so the tag can be seen. - Forgetting that paths are case sensitive.
Our robots.txt generator builds the file for you, and the sitemap generator creates the file the Sitemap line points to.