Türk SEM

Türk SEM · SEO Tool

What AI Sees: AI Crawler Checker

The crawlers behind ChatGPT, Claude and Perplexity don't run JavaScript. They read the raw HTML your server sends and nothing more. Enter a URL and see exactly what they miss, which AI crawlers your robots.txt lets in and how to fix both.

Fetch automatically from your site

We open the page once in a real browser and compare it with the raw HTML that AI crawlers receive. We don't store anything else from your site.

How It Works

  1. 1

    We open the page once

    A real browser loads your page. From the same visit we keep the HTML the server sent first and the page as it looks after JavaScript runs.

  2. 2

    We compare the text

    Any passage of eight words or more that appears only after JavaScript is text AI crawlers never see. The score is the share of your text already in the raw HTML.

  3. 3

    We read your robots.txt

    We check each documented AI crawler separately and sort them by job: training, AI search or fetching a page for a user.

What Your Result Means

The big number is the share of your page's text that AI crawlers can read. Here's what each range means.

95% or more
AI crawlers see the whole page. Nothing to fix on the rendering side.
70 to 94%
The core content gets through. Some extras, like reviews or tabs, only load with JavaScript.
25 to 69%
AI crawlers miss a large part of the page. Move the important text into the server's HTML.
Below 25%
The page is nearly empty without JavaScript. To an AI assistant, it barely exists.

JavaScript SEO: Why AI Crawlers See Less

In December 2024, Vercel and MERJ measured how AI crawlers behave across real sites. None of the major crawlers ran JavaScript. That includes the ones from OpenAI, Anthropic, Meta and Perplexity (Vercel).

Some of them do download JavaScript files: ChatGPT in 11.50% of its requests and Claude in 23.84%. But downloading a file is not the same as running it, so any text your scripts build stays invisible.

Google is the exception. Googlebot renders JavaScript before indexing (Google Search Central), and Google's AI Overviews draw on that index. That is why a page can rank well on Google and still look empty to ChatGPT.

Which Crawler Does What

The big AI companies each run several crawlers, and each one has its own job. OpenAI uses GPTBot to collect training data, OAI-SearchBot for ChatGPT search, and ChatGPT-User when someone asks ChatGPT to open a page. According to OpenAI, sites that block OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigation links (OpenAI).

Anthropic follows the same split with ClaudeBot, Claude-SearchBot and Claude-User (Anthropic). Perplexity has PerplexityBot for its search results and Perplexity-User for live requests, and says Perplexity-User generally ignores robots.txt (Perplexity).

Google-Extended is different: it never visits your site. It's a robots.txt token that controls whether your content is used to train Gemini or to ground its answers. Google says it has no effect on Search inclusion or ranking (Google).

A common and costly mistake is blocking a whole company when you only meant to opt out of training. Block only GPTBot, and ChatGPT search still sees you. Block OAI-SearchBot as well, and you drop out of its answers.

How to Fix It

If text is missing, send it in the server's HTML. Server-side rendering or prerendering does this, and most modern frameworks support it. Start with what answers the searcher's question: headings, the main text, prices and FAQs.

If a crawler is blocked by mistake, use the ready-made rules in the tool. They list every documented crawler for each role, so you can keep AI search open while opting out of training.

Being readable to machines is the first step. Getting quoted by AI assistants is the next, and that's what our GEO (generative engine optimization) and AIO (AI optimization) work is about. For how Google builds its AI answers, see our guide to Google AI Overviews.

Frequently Asked Questions

Does ChatGPT run JavaScript when it reads my site?

No. ChatGPT's crawlers read the raw HTML your server sends. They may download your JavaScript files, but they don't run them, so text that your scripts create is invisible to them.

How do I block GPTBot without dropping out of ChatGPT search?

Block GPTBot in robots.txt and leave OAI-SearchBot and ChatGPT-User allowed. GPTBot only collects training data. OAI-SearchBot is what puts you in ChatGPT search answers.

Will blocking AI crawlers hurt my Google rankings?

No. Google says Google-Extended has no effect on Search, and blocking AI crawlers doesn't touch your rankings either. Only blocking Googlebot itself would hurt them.

Do I need an llms.txt file?

It's optional. llms.txt is a community proposal for giving language models a short guide to your site. No major AI company has said it uses the file, so fix your HTML and robots.txt first.

Why does my page look fine on Google but empty here?

Googlebot renders JavaScript before it indexes a page, so Google sees the finished page. AI crawlers stop at the raw HTML. If your content arrives through JavaScript, only Google gets it.

Does robots.txt actually stop AI crawlers?

It's a request, not a lock. The major AI crawlers say they honor it, but agents that fetch a page on a user's behalf, such as Perplexity-User, may ignore it.

Sources