Why ChatGPT cites some pages and ignores yours.

The three things that actually decide whether ChatGPT cites you

ChatGPT's web search has three filters, in order. If you fail any of them, you are out of the citation pool before relevance is even scored. This is the most useful mental model we have found for diagnosing AI visibility — almost every "why am I not cited?" question reduces to one of these three boxes being unchecked.

1. Crawl access. OAI-SearchBot must be able to fetch your page. If your robots.txt blocks it, or your CDN serves a 403, ChatGPT cannot see you [1].
2. Index inclusion. Your page has to be in the index ChatGPT searches against. OpenAI does not document this index publicly, but pages that are well-linked, fresh, and machine-readable get in.
3. Citation worthiness. Among the candidate set, ChatGPT prefers pages that directly answer the user's question, cite primary sources themselves, and are written in a form that LLMs can extract cleanly.

Filter 1: Crawl access

OpenAI runs three distinct crawlers, and they do very different jobs [1]. The mistake almost everyone makes is to treat them as one bot called "AI" and either allow or block all three together. That is a six-figure decision dressed up as a one-line config change.

• GPTBot collects training data. Blocking it removes you from future model training, not from search citations.
• OAI-SearchBot powers ChatGPT search. Blocking it removes you from the citation pool entirely.
• ChatGPT-User fetches pages on behalf of a specific user query. Blocking it breaks the live-fetch path.

What "blocking GPTBot removes you from training" really means

Frontier models are retrained every 6 to 18 months on a refreshed crawl. If GPTBot has been blocked from your domain for two retraining cycles, the model's parametric memory of your brand quietly atrophies — product names, founder bios, pricing tiers, the wording you use to describe your category. None of this affects search citations, but it does affect every non-search answer: the moments when a user asks ChatGPT "what's a good tool for X?" and the model answers from memory without firing a web query.

OAI-SearchBot: the bot you cannot afford to block

OAI-SearchBot is the crawler behind ChatGPT Search [2]. When a user asks a question that triggers the web tool — almost every commercial, local, news, or "latest" query — ChatGPT runs a search against the index OAI-SearchBot built and picks 3 to 8 pages to read and cite. If your robots.txt disallows this user agent, you are not in the candidate set, full stop. No amount of authority, schema, or copywriting compensates.

# Allow ChatGPT Search to read and cite you
User-agent: OAI-SearchBot
Allow: /

# Allow live "summarize this URL" fetches
User-agent: ChatGPT-User
Allow: /

# GPTBot is the training-data crawler — your choice
User-agent: GPTBot
Allow: /

Sitemap: https://example.com/sitemap.xml

ChatGPT-User: the live-fetch bot most teams forget exists

ChatGPT-User is a different beast. It does not crawl ahead of time — it fetches a page in real time when a user pastes your URL and asks ChatGPT to summarize, extract, or compare. Block this user agent and every "what does this page say?" prompt against your domain dead-ends in an error. That is a brutal first impression for a prospect who specifically went to ChatGPT to read about you.

Filter 2: Index inclusion

Allowing the crawler is necessary but not sufficient. OAI-SearchBot has to choose to keep your pages in its index, and like every retrieval system before it, it favors content that is cheap to fetch, easy to parse, and obviously fresh. ChatGPT search uses its own index, not Google's [2] — but the signals it rewards are familiar.

• Fast, server-rendered HTML. Client-only JavaScript apps are read poorly by every LLM crawler we have tested [10].
• Clean URLs and a working sitemap.xml so the crawler can find your pages.
• Quality inbound links from sites that are themselves indexed. Authority still compounds.
• Freshness signals: visible publication and last-updated dates, reflected in Article schema [11].
• llms.txt. The convention [5] gives LLMs a curated map of your highest-value content. Adoption is rising fast and the cost to ship one is small.

Server-render or stay invisible

The single biggest cause of "I'm allowed but I'm not in the index" is a React or Vue SPA that returns an empty <div id="root"></div> to the crawler and hydrates client-side. AI crawlers don't run your JavaScript bundle the way Googlebot eventually does. Static rendering, SSR, or pre-rendered snapshots are not optional in 2026.

A working llms.txt takes ten minutes

Think of llms.txt as a sitemap written for humans-pretending-to-be-LLMs: one Markdown file at your domain root, a clear H1, a one-paragraph elevator pitch, then grouped links to the pages you most want cited [5]. Our own file is live at /llms.txt and is the single highest-leverage artifact we shipped this quarter.

# Acme Analytics

> Open-source product analytics for self-hosted apps.

Acme Analytics is the analytics layer used by 1,200+ engineering teams to track events without sending data to a third party.

## Pages
- [How it works](/how-it-works): The ingestion → storage → query pipeline.
- [Pricing](/pricing): Self-hosted is free; cloud starts at $19/mo.
- [Docs](/docs): API reference and SDKs.

## Optional
- [Changelog](/changelog)
- [Comparison vs Mixpanel](/vs/mixpanel)

Filter 3: Citation worthiness

Now you are in the candidate set of, say, 200 pages the model could cite for the user's query. It will pick 3 to 8. The pages that win are not the longest, the prettiest, or the most authoritative — they are the ones that look most like a good answer when an LLM scans them. There is a craft to this, and most of it is invisible to writers who optimize only for humans.

Answer-first structure

The first 200 words of the page should directly answer the page's primary question. Burying the answer below 600 pixels of brand copy is a citation killer. LLMs extract the most-likely-answer passage and compare it to the user's query embedding. If your answer is not near the top, a competitor's is.

Question-shaped H2s

Pages that use H2s phrased as the actual questions users ask ("What is X?", "How does Y work?", "When should you use Z?") match more user query embeddings and surface in more answers. This is the single cheapest copy edit you can make for AI visibility.

Schema for entities

Person and Organization JSON-LD with sameAs edges [6] to Wikidata, LinkedIn and Crunchbase gives the LLM the entity backbone it needs to attribute information correctly. Without it, the model may cite a competitor by mistake — we have seen this happen at the executive-name level, where a quote from your founder gets attributed to a more-schemaed competitor in the same category.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Acme Analytics",
  "url": "https://acme.com",
  "logo": "https://acme.com/logo.png",
  "sameAs": [
    "https://www.wikidata.org/wiki/Q123456",
    "https://www.linkedin.com/company/acme-analytics",
    "https://www.crunchbase.com/organization/acme-analytics",
    "https://github.com/acme-analytics"
  ],
  "founder": {
    "@type": "Person",
    "name": "Jane Doe",
    "sameAs": ["https://www.linkedin.com/in/janedoe"]
  }
}

Citation density

Pages that cite primary sources themselves are statistically more likely to be cited. LLMs are trained on the structure of academic and journalistic writing: references signal authority. A page with six inline citations to government, standards-body, or vendor-primary documentation will out-cite a page with the same claims and no references — even if the unsourced page is otherwise better written.

Freshness

Show "Published" and "Last updated" dates visibly near the top of the page, and mirror them in Article schema [11]. Stale-looking content is deprioritized for time-sensitive queries. Pair this with a real <lastmod> in your sitemap so the crawler knows to recheck [9].

What about Claude, Perplexity and Gemini?

The same three-filter mental model applies across every AI engine that does retrieval. What changes is the user-agent string and the relative leverage of each engine for your audience.

Claude (ClaudeBot)

Claude is the most-used LLM in serious B2B research workflows — legal, finance, consulting. Anthropic's crawler is ClaudeBot [4], and the same allow-list logic applies. Claude tends to favor long-form, well-structured content (briefs, methodology pages, glossaries) over marketing copy. If you sell to professional buyers, this is the engine to prioritize after OpenAI.

Perplexity (PerplexityBot)

Perplexity is the most generous citer in the industry — it surfaces a list of sources next to every answer [7]. That makes it the highest-leverage engine to optimize for if you are starting from zero. The "Get cited" loop is also tightest here: ship a question-shaped page, see whether Perplexity picks it up within a week.

Gemini (Google-Extended)

Gemini uses Google's existing crawl infrastructure plus a separate opt-out token, Google-Extended[8]. Blocking Google-Extended opts you out of Gemini training without affecting classic Search ranking. Most sites should leave it allowed — Gemini citations in AI Overviews compound with your existing Search presence.

The 10-minute citation checklist

1. Open /robots.txt. Confirm OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot are not disallowed.
2. View-source your top landing page. Is the answer in the HTML, or does it need JS to appear? If the latter, server-render it.
3. Add a one-paragraph answer in the first 200 words of every page you want cited.
4. Rewrite at least three H2s on that page as the actual questions a user would ask.
5. Ship an Organization + Person JSON-LD block with sameAs to Wikidata, LinkedIn, Crunchbase.
6. Add visible "Published" and "Last updated" dates, and mirror them in Article schema.
7. Cite at least three primary sources inline on every long-form page.
8. Publish /llms.txt at your domain root with links to your highest-value pages.
9. Add or refresh <lastmod> in your sitemap; submit it to Bing Webmaster (which feeds Copilot and ChatGPT Search).
10. Run the free audit below to confirm none of the above silently regressed.

Want this rolled into a one-page report for your site? Run the free audit and we will grade all ten of these checks with the evidence and the fix.