Check crawler access
The path in the URL is what gets tested against the rules. Sitemaps declared in robots.txt are followed; if none is declared, /sitemap.xml is tried.
How it works
Fetch
robots.txt is fetched from the edge. A 404, a 5xx, a redirect, or an HTML page in its place are each reported, since crawlers treat them differently.
Parse
Groups, Allow/Disallow rules, Crawl-delay and Sitemap lines are parsed the way Google's parser does — longest matching rule wins, Allow beats Disallow on a tie.
Test
The path from your URL is tested against Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Applebot, the wildcard group and any agent you add.
Audit sitemaps
Every declared sitemap is fetched, index children followed one level, URL counts, lastmod and hosts checked, and a sample of URLs fetched and tested against the robots rules.
Two files, one question
Robots.txt says what crawlers may fetch; the sitemap says what you want them to fetch. They are edited by different people at different times, and the most common fault is not a broken file but a contradiction: URLs listed in the sitemap that the robots file forbids, or a sitemap declared at a path that returns HTML. This tool reads both in one pass and checks them against each other, because a robots checker that does not open the sitemap is answering half the question.
Which crawler, which rule
Robots rules are matched per user-agent group, and the matching is specific: the group whose agent name is the longest match wins, then the longest path rule inside it, with Allow beating Disallow on a tie. A file with User-agent: * and Disallow: /admin above a User-agent: Googlebot group with no rules allows Googlebot everything — the specific group replaces the wildcard, it does not add to it. The access table shows the group that matched and the rule that fired for each crawler, so surprises like that are visible.
The AI crawlers
GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended are tested separately because blocking them is now a deliberate decision on a lot of sites. The tool reports it without judging it. My own view, for a site that sells expertise: being cited when someone asks an assistant about cold email infrastructure is worth more than the training-data concern, so this site allows all of them. A publisher with content behind a paywall will reasonably decide otherwise. Either way, know which it is.
What a healthy sitemap looks like
Returns 200 with an XML content type, lists only URLs on its own host, none of them redirecting, none of them blocked by robots, none over the 50,000-URL or 50 MB limits, and — if it carries lastmod — the dates are real. The sampled-URL check fetches a handful of entries spread through the file and reports dead ones; two dead out of six usually means the sitemap is generated from a database that includes deleted pages, which is worth fixing at the source rather than in the file.
Frequently asked questions
Does a 404 on robots.txt block crawling?
No — it allows everything. A 5xx is the dangerous one: Google treats it as 'the site may be blocked' and can pause crawling until it resolves.
Why does the tool test the path from my URL?
Because rules are path-specific. The site root being allowed says nothing about /pricing. Paste the page you care about and the table shows the rule that applies to it.
Can I test a crawler that is not in the list?
Yes — type its user-agent token in the box. Matching is by substring, the way real crawlers do it, so 'AhrefsBot' matches a group written for 'ahrefsbot'.
Why is a sitemap index only followed one level?
Budget. Each child is another fetch, and a large index can have hundreds. The first six are followed; run any other child directly to audit it.
Does this submit anything to Google?
No. It reads. Submitting a sitemap is done in Search Console, or by declaring it in robots.txt, which every major crawler reads.
Last reviewed
What to run next
The checks that most often follow this one.
More in this category
Guides that go deeper
When the tools tell you something is wrong
The diagnostics here are free and always will be. When the fix is bigger than a DNS record, this is the work I do.
Deliverability rescue
Mail landing in spam, replies gone quiet, or a domain suddenly blocked. I find the actual cause rather than guessing, and fix it.
- Authentication and alignment failures
- Blocklist delistings and reputation repair
- Gateway and filter-level blocks
- A written report on what broke and why
Email & sending infrastructure
Sending domains, inboxes, authentication and warmup, built to survive volume instead of burning down in a month.
- Domain and inbox fleets at any scale
- SPF, DKIM, DMARC and tracking domains
- Google Workspace — Silver Partner
- Handover documentation you actually own
Domain, DNS & migration
Changing registrar, mail provider or host without a day of downtime or a week of mail silently failing.
- Registrar and nameserver moves
- Workspace and Microsoft 365 migrations
- MX, SSL and subdomain cutover
- Staged rollout with rollback at every step
Monitoring & retainer
Infrastructure drifts. Records get edited, certificates expire, domains get listed. Ongoing eyes on the fleet.
- Scheduled checks across every domain
- Alerts before your clients notice
- Monthly reporting
- Priority response when something breaks
Start with a call
Bring a domain and the symptom. I will tell you what is actually wrong and whether you need me at all — plenty of people leave that call able to fix it themselves.
Thirty minutes, no pitch
We will run the checks together on your actual domains, and you will leave knowing what is broken, what it takes to fix, and what it should cost. If that is a job you can do in-house, I will say so.