Robots.txt and AI crawler access tester
What your robots.txt and your page's headers tell GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot and the other crawlers about one page.
WHAT IT READS
- Your robots.txt, read by the same RFC 9309 rules our own crawler obeys, for each crawler by its own name.
- The page's X-Robots-Tag header and its robots meta tags, including tags addressed to one crawler.
WHAT IT DOES NOT TELL YOU
- This is what your site asks of each crawler, not what the crawler does: whether a crawler obeys robots.txt is its operator's policy.
- We only ever request your site as KutayGO-ScanBot, never under another crawler's name, so whether a firewall lets a given crawler through is not measured.
- If your robots.txt does not permit our crawler to read the page, we answer from the file alone and do not request the page.
- FREE · NO ACCOUNT
Core Web Vitals checker
How fast a page really is for the people who visit it: Google's field data for the page and its site, on phones and on desktops, with one lab run beside it.
- FREE · NO ACCOUNT
Indexability checker
Whether Google can index one page: its status, every redirect, robots.txt, noindex, the canonical, sitemap presence and hreflang.
- FREE · EMAIL CODE
AI Overview and AI Mode citation checker
For one keyword in one market: does Google show an AI Overview or an AI Mode answer, which sites does it cite, and is yours one of them.