How ChatGPT, Claude and Perplexity Decide Which Websites to Cite

There is a comfortable myth that AI assistants “know” the web and pick the best page for each answer. They do not. When ChatGPT, Claude or Perplexity searches, a model writes a query, a search system returns a short list of candidate pages, and the model reads only those pages before it writes its answer. If your page is not in the candidate list, it does not exist for that answer.

That pipeline has three stages, and each one has a different failure mode. Below I go through them, name the crawlers and indexes each assistant actually documents, and then audit a real site: gscmagic.com, which on October 23, 2026 was close to invisible to AI search for reasons that took ten minutes to find.

AI answers are retrieval first and writing second

Every cited answer starts with a search. The model rewrites your question into one or more queries, a search backend returns candidates, and the model writes from the few pages it was handed.

This design is called retrieval-augmented generation. It exists because a language model’s built-in knowledge is frozen at training time and cannot be quoted with a link. Retrieval fixes both problems, but it moves the competition somewhere else.

Think of it as three gates:

1. Indexed. Your page is in the index the assistant’s search backend uses.
2. Retrieved. For the assistant’s rewritten query, your page lands in the handful of candidates it reads. Assistants read only a few pages per answer, so this is a top-few contest, not a top-ten one.
3. Selected. Among the candidates, your page contains the sentence the model needs, stated plainly enough to quote.

Classic SEO advice mostly works on gate 3. Gate 1 gets the least attention, even though it is the cheapest gate to fix.

Each assistant searches a different index

There is no single “AI index”. ChatGPT, Claude, Perplexity and Google each document their own crawlers, and some also lean on outside search providers.

Here is what each company documents, with the parts that are inference marked as such.

The sources for each row:

• OpenAI. The OpenAI crawler documentation states that “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”, that GPTBot governs training, and that ChatGPT-User handles user-initiated visits where “robots.txt rules may not apply”. OpenAI describes ChatGPT search as using third-party search providers without naming them in its help text. Independent tests, such as Windows Latest’s in 2024, found results that tracked Bing closely, and a September 2026 analysis by Peec AI reports OpenAI’s own index plus several external providers. Bing matters, but it is not the whole story.
• Anthropic. The Anthropic crawler page lists ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for user requests, and says all of them honor robots.txt. Anthropic added Brave Search to its subprocessor list in March 2025, which TechCrunch and Simon Willison tied to Claude’s web search by matching results.
• Perplexity. The Perplexity crawler docs describe PerplexityBot as the crawler that surfaces and links sites in results, and say Perplexity-User “generally ignores robots.txt rules” because a person triggered the fetch.
• Google. Google’s crawler documentation says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. AI Overviews are part of Search, so they draw on the normal index built by Googlebot.

The practical reading: three different companies, at least four different indexes, and a separate decision for training and for search in each case.

robots.txt decides gate 1 for most assistants

Blocking a training crawler does not hide you from AI search, and blocking a search crawler does. Most accidental invisibility comes from treating them as one switch.

A common pattern after 2023 was a blanket block on “AI bots”. If that block names OAI-SearchBot, Claude-SearchBot or PerplexityBot, the site removes itself from those assistants’ search results while thinking it only opted out of training.

If you want to be cited but not trained on, the separation looks like this:

# Allow AI search crawlers (needed to be cited)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of model training (does not affect citations)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Whether to opt out of training is a business decision, and I am not going to make it for you. The point is that it is a separate decision. OpenAI and Perplexity both note that robots.txt changes can take about 24 hours to take effect.

The live-fetch agents behave differently. When a user pastes your URL into a chat or asks about your page specifically, ChatGPT-User and Perplexity-User may fetch it even if robots.txt disallows them, because their documentation treats those as user actions, not crawling. Claude-User, by Anthropic’s description, does honor robots.txt.

Being indexed is not the same as being retrieved

Gate 2 is a ranking problem with a twist: the query is written by the model, not the user, and it tends to be longer and more literal than what people type into Google.

When someone asks an assistant “how do I connect Search Console to Claude?”, the assistant’s search query often looks like a cleaned-up version of that sentence. Pages that state the task in the same plain words, in a heading or the first lines of a section, match better than pages that bury it under a brand-flavored title.

Two habits help here, and both are boring:

• Name the task the way a person asks it. “Connect Google Search Console to Claude” beats a clever headline for retrieval.
• Answer directly under the heading. A one to three sentence answer right below each H2 gives the selection step something it can lift whole. That is the same pattern Google’s AI Overviews tend to quote.

None of this beats gate 1. A perfectly phrased page outside the index is still outside the index.

A real audit: why a new site was invisible to AI search

On October 23, 2026, gscmagic.com had 74 pages in its sitemap, 3 pages in Bing’s index, no sitemap submitted to Bing and zero sessions from AI assistants in GA4. Every one of those numbers had a cause.

I ran this check on our own site with the same MCP tools the product exposes, and the numbers below come straight from Search Console, Bing Webmaster Tools and GA4.

The fix took a few minutes, all through public interfaces: resubmitting the sitemap in Search Console so Google saw all 74 URLs, submitting the sitemap to Bing Webmaster Tools, sending the 74 URLs through Bing’s URL submission API, and pinging IndexNow so Bing, Yandex and other participating engines hear about every new page the moment it is published.

I am not going to promise citations from that. Indexing is necessary and not sufficient. It moves the site from “impossible to cite” to “possible to cite”, and the next weeks of Bing and GA4 data will show how much of gate 2 it wins.

Measuring AI traffic got harder in October 2026

Clicks from AI assistants are increasingly arriving without a referrer, so an empty AI row in GA4 no longer proves that nobody came from ChatGPT.

For most of 2025, ChatGPT added utm_source=chatgpt.com to the links it showed, which made its traffic easy to spot. In mid October 2026, Search Engine Roundtable and others reported that both the tag and the referrer appeared to be missing on many clicks, sending them into Direct. OpenAI has not confirmed this, and the reported sample was small, so treat it as a warning rather than a fact.

What I would do now:

• Keep a GA4 custom channel for AI sources that matches on session source with a regex (chatgpt.com, perplexity.ai, claude.ai, gemini.google.com), placed above Referral. Our guide on measuring ChatGPT and Perplexity traffic in GA4 walks through it.
• Watch Direct traffic that lands on deep pages, such as a specific blog post, rather than the home page. People rarely type a long blog URL by hand.
• Read your server logs for OAI-SearchBot, Claude-SearchBot and PerplexityBot. A crawler that visits is the earliest sign that gate 1 is open.

If you would rather ask than click through dashboards, GSC Magic lets Claude or ChatGPT run these checks on your own Search Console, Bing and GA4 data, and the Bing Webmaster Tools tools include URL submission and crawl statistics.

Questions people ask about AI citations

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the search crawler, and says each robots.txt setting is independent. Sites that opt out of OAI-SearchBot are the ones that will not be shown in ChatGPT search answers.

Does ChatGPT use Bing?

OpenAI says ChatGPT search uses third-party search providers alongside its own systems but does not name them in its help text. Independent tests have repeatedly found heavy reliance on Bing results, so being indexed in Bing is a sensible precaution rather than a confirmed requirement.

Can I submit my site to Claude or Perplexity directly?

There is no public submission console for either. Allow their search crawlers in robots.txt, keep your sitemap current and get indexed by the search engines their answers draw on.

Why does my GA4 show almost no traffic from ChatGPT?

Either the assistant is not citing you yet, or the visits are arriving without a referrer and landing in Direct. Reports from October 2026 suggest ChatGPT links may have stopped carrying utm_source=chatgpt.com, which OpenAI has not confirmed, so check landing-page patterns in Direct as well.