Free Robots.txt Tester

Check if Googlebot, Bingbot and AI crawlers can reach your pages. Finds syntax errors, blocked URLs and blocked CSS/JS files – with step-by-step fixes.

100% freeNo sign-upStep-by-step fixesDownloadable report

What is a Robots.txt Tester?

A robots.txt tester reads your robots.txt file the same way Googlebot does and tells you whether a URL is allowed or blocked for a crawler. Google uses the most specific (longest) matching rule, and when an Allow and a Disallow rule are equally specific, Allow wins.

This tester follows those Google rules, including the * and $ wildcards, and also checks the file itself: server errors, size, wrong directives, missing sitemap, blocked CSS/JS files and which AI crawlers can access your content.

What the Robots.txt Tester checks

Every problem comes with plain-English steps to fix it, and every result is added to your downloadable Technical SEO report.

Live file or pasted code

Fetch your live robots.txt or paste a draft to test it before you upload it.

Google matching rules

Longest-match logic with * and $ wildcards, exactly like Googlebot.

Blocked CSS and JS finder

Checks the files on your page that Google needs for rendering.

AI crawler access

Shows whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended can reach your site.

Syntax validation

Flags unknown directives, rules before User-agent, unsupported Noindex lines and paths without a slash.

Line-by-line view

See the exact rule and line number that allows or blocks each URL.

How to use the Robots.txt Tester

Click Load Example above to see a real result first.

Enter your page URL

Type any URL from your site. The tool fetches robots.txt from the same domain.

Choose the crawler

Test as Googlebot, Bingbot or an AI bot. Add more paths if you like.

Click “Test robots.txt”

See allowed and blocked URLs, the matching rule for each and any file errors.

Fix and re-test

Follow the steps under each problem, update the file and run the test again.

Common problems and how to fix them

ProblemWhy it mattersHow to fix it
robots.txt returns 5xxGoogle pauses crawling of the whole site.Fix the server or firewall error so the file returns 200 or 404.
Important page blockedGoogle cannot crawl or refresh the page.Remove or narrow the Disallow rule, or add a longer Allow rule.
CSS/JS files blockedPages cannot be rendered correctly.Allow theme, plugin, CSS and JavaScript folders.
Unknown directiveCrawlers ignore the line, so the rule you wanted does nothing.Check spelling: User-agent, Allow, Disallow, Sitemap.
Sitemap URL not absoluteCrawlers may not find your sitemap.Use the full URL starting with https://.

Frequently asked questions

Did Google remove its robots.txt tester?

Yes. The old tester in Search Console was retired in 2023 and replaced by a robots.txt report (Settings > robots.txt). This free tool lets you test individual URLs and crawlers again.

Which rule wins if Allow and Disallow both match?

The longest (most specific) rule wins. If they are the same length, Allow wins. For example Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/.

Is robots.txt case-sensitive?

Paths are case-sensitive: /Blog/ and /blog/ are different. Directive names like Disallow are not case-sensitive.

What happens if I have no robots.txt?

Crawlers assume everything is allowed. That is fine, but you cannot list your sitemap or block private areas.

Does blocking a page remove it from Google?

No. To remove a page from search results, allow crawling and add a noindex tag, or use the Removals tool in Search Console.

Related free tools

Built and maintained by the AM Digital Marketing Agency SEO team. Last updated: .

Want us to fix it for you?

Our team fixes technical SEO issues, speed and indexing problems – and optimizes your site for Google, AI Overviews and ChatGPT.

View SEO Services