The AI crawlers we test
AI companies run separate bots for separate jobs. Training crawlers collect data for future models. Search crawlers build the index AI answers draw on. User-triggered fetchers load a page the moment someone asks about it. Blocking training bots is a policy choice; blocking the other two takes you out of AI answers.
AI crawlers by company and purpose
| Company | AI search | User fetch | Training |
|---|
| OpenAI (ChatGPT) | OAI-SearchBot | ChatGPT-User | GPTBot |
|---|
| Anthropic (Claude) | Claude-SearchBot | Claude-User | ClaudeBot |
|---|
| Perplexity | PerplexityBot | Perplexity-User | |
|---|
| Google (Gemini, AI Overviews) | | | Google-Extended |
|---|
| Apple (Apple Intelligence) | | | Applebot-Extended |
|---|
| DuckDuckGo | DuckAssistBot | | |
|---|
| Mistral (Le Chat) | | MistralAI-User | |
|---|
| Meta, Amazon, Common Crawl, ByteDance | | | Meta-ExternalAgent, Amazonbot, CCBot, Bytespider |
|---|
Three reasons AI bots can’t read a site
robots.txt
A Disallow rule for a bot’s user-agent, or a blanket User-agent: * block, stops compliant crawlers. We evaluate your rules for every bot using the same longest-match logic crawlers use (RFC 9309) and show which line decides.
Firewall & CDN
Cloudflare, Vercel, Akamai and WordPress security plugins can return 403s, CAPTCHAs or challenge pages to AI bots even when robots.txt allows them. We request your page as each bot and compare the answer to a normal browser visit.
JavaScript rendering
Most AI crawlers don’t execute JavaScript. If your content is rendered in the browser, they see an empty <div id="root">. We measure how much real text, headings and structured data exist in the raw HTML.
We also check the newer signals AI systems read: noai directives, Content-Signal policies in robots.txt, llms.txt and llms-full.txt, Markdown content negotiation, XML sitemaps and Link headers.
FAQ
What is AI crawlability?
AI crawlability is whether AI systems such as ChatGPT, Claude, Perplexity and Google Gemini can fetch and read your pages. It depends on three layers: your robots.txt rules for each AI user-agent, whether your CDN or firewall lets those bots through, and whether your content exists in the HTML without running JavaScript.
Should I block GPTBot and ClaudeBot?
GPTBot and ClaudeBot collect training data. Blocking them is a legitimate policy choice and does not remove you from live AI answers. The bots that fetch pages when a user asks a question are different: OAI-SearchBot and ChatGPT-User for OpenAI, Claude-SearchBot and Claude-User for Anthropic, PerplexityBot and Perplexity-User for Perplexity. Block those and you disappear from AI search results and citations.
Why would an AI crawler be blocked if my robots.txt allows it?
Many CDNs and firewalls block AI bots on their own. Cloudflare has a one-click Block AI Bots setting and Bot Fight Mode, and Vercel, Akamai, Sucuri and Wordfence have similar rules. The checker requests your page with each bot’s user-agent and compares the response to a normal browser request to spot these blocks. Some CDNs only admit the real bot from its published IP ranges, so confirm a flagged block in your CDN’s settings.
Do AI crawlers run JavaScript?
Mostly no. GPTBot, ClaudeBot, PerplexityBot and most AI crawlers read the raw HTML the server sends. If your site is a client-rendered single-page app, they see an empty shell. Server-side rendering, static generation or prerendering fixes this.
What is llms.txt?
llms.txt is a Markdown file at the root of a website that summarises the site and links to its key pages, so AI tools can understand it in one request. It is a community convention described at llmstxt.org, not a formal standard. See this site’s llms.txt for an example.
Is the AI crawlability checker free?
Yes. It is free with no signup, and every report can be exported as PDF, CSV or JSON. It is built and run by Mecverse. URLs and results are not stored; see the privacy policy.
