Free robots.txt tester & analyzer
One wrong line in robots.txt can hide your whole site from Google. Enter a URL to parse every rule, test whether a path is crawlable for Googlebot or Bingbot, and catch syntax errors and missing sitemaps — with a 0-100 score.
Everything a crawler reads in your robots.txt
Crawl access
- Site not fully blocked
- Googlebot & Bingbot can crawl
- Default User-agent: * group
- Live per-URL tester
Sitemaps
- Sitemap declared
- Absolute sitemap URLs
- Multiple sitemaps parsed
- Host directive
Syntax & health
- Unknown directives flagged
- Malformed lines caught
- Under Google's 500KB limit
- Served as text/plain
Rules & groups
- Every group parsed
- Allow / Disallow rules
- Crawl-delay detected
- Wildcards & $ anchors
Common robots.txt questions
What does a robots.txt tester do?
It fetches your site's /robots.txt, parses every User-agent group and Allow/Disallow rule, then simulates how a crawler like Googlebot decides whether a given URL is allowed — using the same longest-match rule Google uses. It also flags syntax errors, missing sitemaps and accidental site-wide blocks.
Why is my page blocked in robots.txt?
Usually a Disallow rule matches its path. Paste the path into the URL tester above and pick the crawler — MarkPilot shows the exact rule (e.g. "Disallow: /admin/") and which User-agent group it came from, so you know precisely what to change.
Does robots.txt remove a page from Google?
No. robots.txt controls crawling, not indexing — a blocked URL can still appear in search without a snippet. To keep a page out of Google, allow crawling and use a noindex meta tag instead.
Want MarkPilot to write a corrected robots.txt and re-check it as your site changes? Create a free MarkPilot account →