🕷️ AI Crawler Access Checker
Paste your robots.txt and see, crawler by crawler, whether ChatGPT, Claude, Perplexity, Gemini and the others may read your pages - and which line decides. Nothing is sent anywhere.
Open yourdomain.com/robots.txt, copy everything and paste it here.
A path like /blog/my-post or a full URL. "/" checks the home page.
| Crawler | Company | Purpose | Access | Why |
|---|
⚠️ = the company says robots.txt may not apply, because a person asked for the page. 🔧 = a control token in robots.txt, not a separate crawler.
Three kinds of AI crawler
AI companies now run separate crawlers for separate jobs, and robots.txt lets you treat them differently. Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent) collect pages that may be used to train future models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index that AI answers cite and link to. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page because someone asked about it in a chat.
Blocking a training crawler keeps your pages out of future training data but does not remove you from AI answers. Blocking a search crawler is different: if OAI-SearchBot or PerplexityBot cannot read a page, those products cannot cite it. Many publishers allow search and block training - the table shows at a glance which way your file leans.
Control tokens
Google-Extended and Applebot-Extended are not crawlers you will see in your logs. They are switches: Googlebot and Applebot still crawl as usual, and these tokens tell Google and Apple whether that content may be used for their AI models. Blocking Google-Extended does not affect your ranking in Google Search.
Content signals
Some sites - including any site on Cloudflare that turned on its managed robots.txt - add a line like Content-Signal: ai-train=no, search=yes, ai-input=yes. It states how the owner wants the content used: not for training, yes for search, yes as input to AI answers. It sits beside the crawl rules rather than replacing them, so a crawler can be allowed to fetch a page while the owner says it may not train on it. The checker shows the signal when it finds one.
Limits
robots.txt is a request, not a lock. Reputable companies follow it, but it cannot stop a crawler that ignores it - that takes a firewall or CDN rule. Crawler names change as companies launch new products, so the list here is dated: it was last checked in October 2026. The matching itself follows Google's published robots.txt rules, the same engine as our robots.txt Tester.
Other tools you might need next
See all SEO & Metadata tools, or browse every free tool on the site.