Free Robots.txt Generator
Create a valid robots.txt file in seconds – presets for WordPress, WooCommerce and online stores, AI crawler controls (GPTBot, ClaudeBot, Google-Extended) and built-in validation.
What is a Robots.txt Generator?
A robots.txt generator creates the robots.txt file that tells search engine and AI crawlers which parts of your website they may crawl. The file sits at yoursite.com/robots.txt, uses User-agent, Allow, Disallow and Sitemap rules, and controls crawling – not indexing.
One wrong line, such as Disallow: /, can remove a whole website from Google. That is why this generator checks the file for mistakes before you upload it and explains how to fix anything it finds.
What the Robots.txt Generator checks
Every problem comes with plain-English steps to fix it, and every result is added to your downloadable Technical SEO report.
Safe starting rules for WordPress, WooCommerce, Shopify-style stores, blogs and staging sites.
Block AI training bots (GPTBot, Google-Extended, CCBot) while still allowing ChatGPT search, Perplexity and Claude to cite you.
Warns you if Google is blocked, if CSS or JavaScript is blocked, or if a rule is written wrongly.
Adds your XML sitemap so Google, Bing and AI engines find every page.
Optionally stop Ahrefs, Semrush and Majestic bots from using your server resources.
Copy the code or download a ready robots.txt file to upload.
How to use the Robots.txt Generator
Click Load Example above to see a real result first.
Enter your website and pick a preset
Type your domain and choose WordPress, WooCommerce, online store, blog or staging.
Adjust the rules
Add or remove Disallow and Allow paths, choose how AI crawlers are treated and add your sitemap.
Click “Generate robots.txt”
The tool builds the file and checks it for errors instantly.
Upload and test
Paste it into Rank Math or Yoast, or upload it to your root folder, then test it with the Robots.txt Tester.
Common problems and how to fix them
| Problem | Why it matters | How to fix it |
|---|---|---|
Disallow: / on a live site | Blocks every page from Google and other search engines. | Remove the line or change the default policy to “Allow all crawlers”. |
Blocking CSS, JS or /wp-content/ | Google cannot render the page, which can hurt mobile usability and rankings. | Delete Disallow rules for theme, plugin, CSS and JavaScript folders. |
Using Noindex: in robots.txt | Google stopped supporting it in 2019, so pages can still be indexed. | Use a meta robots noindex tag or an X-Robots-Tag header instead. |
| No Sitemap line | Crawlers have to discover pages on their own. | Add Sitemap: https://yoursite.com/sitemap_index.xml. |
| Blocking all AI bots by accident | Your pages cannot be cited in ChatGPT, Perplexity or Claude answers. | Block only training bots (GPTBot, Google-Extended, CCBot) and allow AI search bots. |
Frequently asked questions
Where do I upload robots.txt?
In the root folder of your domain, so it opens at https://yoursite.com/robots.txt. In WordPress you can edit it in Rank Math (General Settings > Edit robots.txt) or Yoast (Tools > File editor).
Does robots.txt stop a page from being indexed?
No. It only stops crawling. A blocked page can still appear in Google if other sites link to it. To keep a page out of search results use a noindex meta tag and do not block it in robots.txt.
Should I block GPTBot and other AI bots?
Block AI training bots (GPTBot, Google-Extended, CCBot) if you do not want your content used to train models. Keep AI search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) allowed if you want to be cited in AI answers.
Does Google support Crawl-delay?
No. Google ignores Crawl-delay. Bing and Yandex follow it. Google adjusts its crawl rate automatically based on how fast your server responds.
How long does Google take to see a new robots.txt?
Google usually caches robots.txt for up to 24 hours. You can ask for a faster refresh in Google Search Console under Settings > robots.txt.
Related free tools
Built and maintained by the AM Digital Marketing Agency SEO team. Last updated: .
Want us to fix it for you?
Our team fixes technical SEO issues, speed and indexing problems – and optimizes your site for Google, AI Overviews and ChatGPT.