Skip to main content
CiteCheckup logoCiteCheckup

Robots.txt checker

Fetch a public robots.txt file or paste one in. Test the exact path and user-agent you care about, then see which rule wins.

Robots policy result

Enter a URL or paste a file to see the exact matched rule and AI crawler matrix.

This tool evaluates the text you provide or the public robots.txt response. It does not prove that a crawler will fetch or index a page.

What this robots.txt checker checks

The response itself

Checks whether the supplied or fetched response looks like a usable robots.txt file instead of silently treating HTML, an empty body, or an unrecognized response as an open policy.

Groups and directives

Parses user-agent groups, Allow and Disallow rules, and sitemap lines while keeping blank lines and comments inside the current group.

The exact decision

Tests the path and user-agent you enter, shows the rules that matched, and compares common AI crawler identities with their own evidence.

Find the rule that controls the path

  1. Trace the match

    Start with the crawler and path that matters. Read the matched rule and the surrounding group instead of guessing from the file's first line.

  2. Change the scope

    Remove an accidental broad block, add a deliberate exception, or correct the group syntax. Keep the rule short enough to review safely.

  3. Check the response

    Publish the file, run the same path and user-agent again, and confirm the response is valid before treating the result as useful evidence.

What this result cannot prove
  • A robots.txt decision describes the text and response observed by this tool; it does not prove a crawler will fetch, index, or cite a page.
  • Crawler names are user-agent strings, not a live inventory of every provider or infrastructure behavior.
  • Redirects remain subject to the same public-URL and cross-origin safety rules as other fetches.

Robots.txt questions

Does a blank line end a robots group?

No. A blank or comment-only line does not end the group. The next user-agent group or the end of the file does.

Why can the result be Unknown instead of Allowed?

An invalid response or syntax issue is not reliable evidence of permission, so the tool keeps it separate from a confirmed allow decision.

Can this test Google or AI indexing?

It tests the published robots text for a chosen identity and path. It does not observe a provider's index, ranking, or citation behavior.

Related checks and guides

Structured data testing tool

Check JSON-LD structure when the crawler can reach the page but its meaning is unclear.

Test structured data

How to get cited by AI

Separate crawlable access from the content and evidence a source page should provide.

Read the guide

Full page audit

Use the 30-check audit when the problem spans access, structure, content, and publisher details.

Check a URL