Where do I put the robots.txt file?

Upload it to the root of your domain so it is reachable at https://yourdomain.com/robots.txt. Search engines only read robots.txt from the root directory.

Does Crawl-delay work for all search engines?

Crawl-delay is honored by Bing and some other crawlers but ignored by Google. To control Googlebot's crawl rate, use Google Search Console settings instead.

🤖

Robots.txt Generator

Create a robots.txt file for your site.

Loading tool…

How to use the Robots.txt Generator

1Choose a preset: Allow all, Disallow all, or Custom.
2Add disallow paths, crawl-delay, and your sitemap URL.
3Copy or download the generated robots.txt to your site root.

About the Robots.txt Generator

Free robots.txt generator. Pick allow all, disallow all, or custom disallow paths, set crawl-delay and sitemap URL, then copy or download a valid robots.txt.

Part of our SEO & Marketing collection. Browse all tools.

Frequently asked questions

Where do I put the robots.txt file?: Upload it to the root of your domain so it is reachable at https://yourdomain.com/robots.txt. Search engines only read robots.txt from the root directory.
Does Crawl-delay work for all search engines?: Crawl-delay is honored by Bing and some other crawlers but ignored by Google. To control Googlebot's crawl rate, use Google Search Console settings instead.