Türk SEM · SEO Tool
What AI Sees: AI Crawler Checker
The crawlers behind ChatGPT, Claude and Perplexity don't run JavaScript. They read the raw HTML your server sends and nothing more. Enter a URL and see exactly what they miss, which AI crawlers your robots.txt lets in and how to fix both.
Fetch automatically from your site
We open the page once in a real browser and compare it with the raw HTML that AI crawlers receive. We don't store anything else from your site.
How It Works
- 1
We open the page once
A real browser loads your page. From the same visit we keep the HTML the server sent first and the page as it looks after JavaScript runs.
- 2
We compare the text
Any passage of eight words or more that appears only after JavaScript is text AI crawlers never see. The score is the share of your text already in the raw HTML.
- 3
We read your robots.txt
We check each documented AI crawler separately and sort them by job: training, AI search or fetching a page for a user.
- 4
We test the door
If a firewall or CDN treats AI crawlers differently from visitors, you'll see it here, even when robots.txt lets them in. To find out, we request your page once as a regular browser and once under each of six crawler names from OpenAI, Anthropic and Perplexity.
What Your Result Means
The big number is the share of your page's text that AI crawlers can read. Here's what each range means.
- 95% or more
- AI crawlers see the whole page. Nothing to fix on the rendering side.
- 70 to 94%
- The core content gets through. Some extras, like reviews or tabs, only load with JavaScript.
- 25 to 69%
- AI crawlers miss a large part of the page. Move the important text into the server's HTML.
- Below 25%
- The page is nearly empty without JavaScript. To an AI assistant, it barely exists.
JavaScript SEO: Why AI Crawlers See Less
In December 2024, Vercel and MERJ measured how AI crawlers behave across real sites. None of the major crawlers ran JavaScript. That includes the ones from OpenAI, Anthropic, Meta and Perplexity (Vercel).
Some of them do download JavaScript files: ChatGPT in 11.50% of its requests and Claude in 23.84%. But downloading a file is not the same as running it, so any text your scripts build stays invisible.
Google is the exception. Googlebot renders JavaScript before indexing (Google Search Central), and Google's AI Overviews draw on that index. That is why a page can rank well on Google and still look empty to ChatGPT.
Which Crawler Does What
The big AI companies each run several crawlers, and each one has its own job. OpenAI uses GPTBot to collect training data, OAI-SearchBot for ChatGPT search, and ChatGPT-User when someone asks ChatGPT to open a page. According to OpenAI, sites that block OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigation links (OpenAI).
Anthropic follows the same split with ClaudeBot, Claude-SearchBot and Claude-User (Anthropic). Perplexity has PerplexityBot for its search results and Perplexity-User for live requests, and says Perplexity-User generally ignores robots.txt (Perplexity).
Google-Extended is different: it never visits your site. It's a robots.txt token that controls whether your content is used to train Gemini or to ground its answers. Google says it has no effect on Search inclusion or ranking (Google).
A common and costly mistake is blocking a whole company when you only meant to opt out of training. Block only GPTBot, and ChatGPT search still sees you. Block OAI-SearchBot as well, and you drop out of its answers.
Since August 31, 2026, Search Console has had a site-wide setting that keeps a site out of AI Overviews and AI Mode while leaving its regular rankings alone (Google). At the page level, nosnippet stops content from being used as a direct input for those features, and max-snippet caps how much text they can take (Google).
The tool also lists any preference lines it finds in your robots.txt, such as Cloudflare's Content Signals (Cloudflare), Content-Usage from an IETF draft (IETF) or RSL License lines (RSL). None of them is enforced.
How to Fix It
If text is missing, send it in the server's HTML. Server-side rendering or prerendering does this, and most modern frameworks support it. Start with what answers the searcher's question: headings, the main text, prices and FAQs.
If a crawler is blocked by mistake, use the ready-made rules in the tool. They list every documented crawler for each role, so you can keep AI search open while opting out of training.
Blocked at the door? When the access test flags a crawler, the cause is usually not robots.txt but a firewall, a security plugin or your CDN. Cloudflare, for example, can block AI crawlers from its dashboard, and its pay-per-crawl beta answers them with HTTP 402 and a price. None of this shows up in robots.txt.
Being readable to machines is the first step. Getting quoted by AI assistants is the next, and that's what our GEO (generative engine optimization) and AIO (AI optimization) work is about. For how Google builds its AI answers, see our guide to Google AI Overviews.
Frequently Asked Questions
Does ChatGPT run JavaScript when it reads my site?
No. ChatGPT's crawlers read the raw HTML your server sends. They may download your JavaScript files, but they don't run them, so text that your scripts create is invisible to them.
How do I block GPTBot without dropping out of ChatGPT search?
Block GPTBot in robots.txt and leave OAI-SearchBot and ChatGPT-User allowed. GPTBot only collects training data. OAI-SearchBot is what puts you in ChatGPT search answers.
Will blocking AI crawlers hurt my Google rankings?
No. Google says Google-Extended has no effect on Search, and blocking AI crawlers doesn't touch your rankings either. Only blocking Googlebot itself would hurt them.
Do I need an llms.txt file?
It's optional. llms.txt is a community proposal for giving language models a short guide to your site. No major AI company has said it uses the file, so fix your HTML and robots.txt first.
Why does my page look fine on Google but empty here?
Googlebot renders JavaScript before it indexes a page, so Google sees the finished page. AI crawlers stop at the raw HTML. If your content arrives through JavaScript, only Google gets it.
Does robots.txt actually stop AI crawlers?
It's a request, not a lock. The major AI crawlers say they honor it, but agents that fetch a page on a user's behalf, such as Perplexity-User, may ignore it.
Why does Cloudflare block AI crawlers when my robots.txt allows them?
Most often because AI blocking is switched on in Cloudflare itself. That setting works independently of robots.txt and can block AI crawlers or send them to a challenge page. To see whether it affects you, run our access test: it requests your page under each crawler's name. Keep in mind that Cloudflare recognizes real crawlers by the network they come from, and our test doesn't come from that network. Confirm in your dashboard whether the real ones are blocked too.
Can I stay in Google Search but keep my site out of AI Overviews?
Yes. Page by page, nosnippet keeps content from being used as a direct input for AI Overviews and AI Mode, but it also strips your normal snippets. For the whole site there's a cleaner option: since August 31, 2026, a Search Console setting can exclude your site from AI Overviews, AI Mode and Discover's AI features without affecting regular Search. Google-Extended won't help here; it only decides whether Gemini trains on your content and uses it to back up its answers.
What do Content-Signal, Content-Usage and License lines in robots.txt do?
Content-Signal (Cloudflare's format: search, ai-input, ai-train), Content-Usage (an IETF draft: train-ai, ai-use, search) and License (a link to an RSL licensing file) are all ways to state what you want. They're preferences, so a crawler can still ignore them. If you need to keep a crawler out, that's what Disallow rules are for.
Sources
- Vercel and MERJ: The rise of the AI crawler (December 2024)
- OpenAI: Overview of OpenAI crawlers
- Anthropic: How site owners can block the crawler
- Perplexity: Perplexity crawlers
- Google: Common crawlers and Google-Extended
- Google Search Central: JavaScript SEO basics
- Cloudflare: Content Signals Policy
- llms.txt proposal
- Google: Robots meta tag, data-nosnippet and X-Robots-Tag
- Google Search Console Help: Search generative AI setting
- IETF: Associating AI usage preferences with content (draft)
- RSL: Declaring a license in robots.txt
- Cloudflare: Managed robots.txt