Free Robots.txt Tester
Check if Googlebot, Bingbot and AI crawlers can reach your pages. Finds syntax errors, blocked URLs and blocked CSS/JS files – with step-by-step fixes.
What is a Robots.txt Tester?
A robots.txt tester reads your robots.txt file the same way Googlebot does and tells you whether a URL is allowed or blocked for a crawler. Google uses the most specific (longest) matching rule, and when an Allow and a Disallow rule are equally specific, Allow wins.
This tester follows those Google rules, including the * and $ wildcards, and also checks the file itself: server errors, size, wrong directives, missing sitemap, blocked CSS/JS files and which AI crawlers can access your content.
What the Robots.txt Tester checks
Every problem comes with plain-English steps to fix it, and every result is added to your downloadable Technical SEO report.
Fetch your live robots.txt or paste a draft to test it before you upload it.
Longest-match logic with * and $ wildcards, exactly like Googlebot.
Checks the files on your page that Google needs for rendering.
Shows whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended can reach your site.
Flags unknown directives, rules before User-agent, unsupported Noindex lines and paths without a slash.
See the exact rule and line number that allows or blocks each URL.
How to use the Robots.txt Tester
Click Load Example above to see a real result first.
Enter your page URL
Type any URL from your site. The tool fetches robots.txt from the same domain.
Choose the crawler
Test as Googlebot, Bingbot or an AI bot. Add more paths if you like.
Click “Test robots.txt”
See allowed and blocked URLs, the matching rule for each and any file errors.
Fix and re-test
Follow the steps under each problem, update the file and run the test again.
Common problems and how to fix them
| Problem | Why it matters | How to fix it |
|---|---|---|
| robots.txt returns 5xx | Google pauses crawling of the whole site. | Fix the server or firewall error so the file returns 200 or 404. |
| Important page blocked | Google cannot crawl or refresh the page. | Remove or narrow the Disallow rule, or add a longer Allow rule. |
| CSS/JS files blocked | Pages cannot be rendered correctly. | Allow theme, plugin, CSS and JavaScript folders. |
| Unknown directive | Crawlers ignore the line, so the rule you wanted does nothing. | Check spelling: User-agent, Allow, Disallow, Sitemap. |
| Sitemap URL not absolute | Crawlers may not find your sitemap. | Use the full URL starting with https://. |
Frequently asked questions
Did Google remove its robots.txt tester?
Yes. The old tester in Search Console was retired in 2023 and replaced by a robots.txt report (Settings > robots.txt). This free tool lets you test individual URLs and crawlers again.
Which rule wins if Allow and Disallow both match?
The longest (most specific) rule wins. If they are the same length, Allow wins. For example Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/.
Is robots.txt case-sensitive?
Paths are case-sensitive: /Blog/ and /blog/ are different. Directive names like Disallow are not case-sensitive.
What happens if I have no robots.txt?
Crawlers assume everything is allowed. That is fine, but you cannot list your sitemap or block private areas.
Does blocking a page remove it from Google?
No. To remove a page from search results, allow crawling and add a noindex tag, or use the Removals tool in Search Console.
Related free tools
Built and maintained by the AM Digital Marketing Agency SEO team. Last updated: .
Want us to fix it for you?
Our team fixes technical SEO issues, speed and indexing problems – and optimizes your site for Google, AI Overviews and ChatGPT.