Measure
June 20, 2026 · Updated August 19, 2026 · 11 min read
By Marcus Bransbury · Founder, Robot Visible
Why ChatGPT cannot find your website
Why ChatGPT and search engines overlook websites, with practical fixes for access, readable content, entity clarity, trust, and citations.
Quick answers
Why can ChatGPT not find my website?
Usually one of four things, and they are worth checking in this order: the crawler cannot fetch the page, the fetched page is not eligible to be indexed and served, the delivered HTML does not state the facts an answer would need, or the page is fetchable and clear but loses to better-corroborated sources for that question. Rewriting the copy first is the common mistake, because copy cannot repair an access or eligibility failure.
Can ChatGPT crawl my website?
Only if your robots policy and your edge both allow the specific agent involved. OpenAI documents them separately: OAI-SearchBot for ChatGPT search results, GPTBot for model training, and ChatGPT-User for fetches a person or a Custom GPT triggers. They are controlled independently, so allowing one tells you nothing about the others. Request your own production URL as each agent and read the response code rather than assuming.
Does robots.txt block GPTBot or OAI-SearchBot?
It does if a group names them, and it can also do so through a wildcard group that disallows what they need. Check both, then check what your CDN does, because a managed bot rule can return 403 to an agent your robots.txt allows. Blocking GPTBot does not block OAI-SearchBot: training access and ChatGPT search access are separate decisions, and they are worth making separately.
How do I confirm an AI crawler actually fetched my site?
Find the request in server or CDN logs, then verify it instead of trusting the user-agent string, which anyone can copy. OpenAI, Google, Perplexity, and Anthropic each publish address ranges or a reverse-DNS method for confirming that a request genuinely came from them. A verified fetch is proof of access. It is not proof of indexing, of selection, or of a citation.
Why are competitors cited instead of me?
Because for that question, on that provider, at that moment, their page was the better source: closer to the wording asked, more specific, more current, or corroborated by places the answer already trusts. Compare the exact cited pages with the single page you have assigned to that question, then close the evidence gap the comparison shows. Matching the competitor's page length is not the fix.
What are the most common issues that prevent AI from reading a website?
Common problems include crawler blocks in robots.txt or a firewall, noindex directives, incorrect canonicals, broken responses, missing sitemap and internal links, essential content that is unavailable in the delivered HTML, ambiguous page purpose, inconsistent entity information, thin content, and unsupported or stale claims.
Free diagnostic · no sign-up
Check the exact page AI systems receive
Enter a public production URL. The scan checks access, the delivered and rendered page, machine-readable signals, and one real buyer question. It keeps readiness evidence separate from what the live answer actually mentioned or cited.
The discovery problem has changed
For a decade, getting found meant ranking in Google. Today a growing share of buyers ask an AI assistant instead: "What's a good tool for X?" If AI assistants and search engines can't read and understand your site, you are invisible to those buyers — no matter how good the product is.
AI and search crawlers do not all render or revisit JavaScript in the same way. If essential information is absent from the initial HTML, hidden behind interaction, or slow to appear, some retrieval systems may receive an incomplete page. Clear text, metadata, matching structured data, and useful supporting pages reduce that uncertainty.1,2
Why websites get overlooked
- Thin site architecture: one homepage can only answer one intent. Buyers ask dozens of specific questions you have no page for.
- Generic copy: vague value statements provide few concrete facts or direct answers for a retrieval system to use.
- Ambiguous entities: inconsistent names, descriptions, prices, and organization details make basic facts harder to reconcile.
- Weak provenance: missing authors, dates, sources, methodology, or contact details can make important claims difficult to verify.
- Rendering dependencies: content that appears only after JavaScript, consent, login, or interaction may be missing from some fetches.
Where the failure sits, stage by stage
Discovery fails in stages, and the stage decides the fix. A page has to be fetchable, then eligible to be indexed and served, then understandable as a specific answer, then selected in competition with everything else that could answer the same question. Each stage has its own symptom and its own test. Skipping to the last one is how teams end up rewriting a page that was never fetched. The maintained crawler reference lists which named bot does what, and the Cloudflare guide covers the case where the block sits at the edge and not in your own robots.txt.3,8
| Stage | What has to be true | Symptom you see | Test that settles it | First fix |
|---|---|---|---|---|
| Access | The named crawler can request the production URL and receives a 200 | Nothing you publish ever appears, and logs show no fetches or non-200 responses for those agents | Request the production URL as each agent you intend to allow, then verify a claimed visit against the provider's published address ranges or reverse DNS | Repair the block, redirect loop, or edge rule before touching a word of the copy |
| Indexation | The fetched page is eligible to be indexed and served with a snippet | The page is crawlable but missing from index inspection tools | Inspect the URL in Search Console and Bing Webmaster Tools, then compare canonical, noindex, and duplicate signals | Resolve the eligibility conflict and consolidate duplicates before expanding the page |
| Understanding | The delivered HTML states what the page is, who it is for, and the facts an answer would need | The brand is described vaguely or incorrectly, or only the homepage ever appears | Read the initial HTML with JavaScript disabled and check whether those facts are present as text | Move the missing facts into server-rendered text and make structured data match them |
| Selection | The page is a better source for this question than the pages currently cited | Competitors are named for questions you already have a page for | Compare the exact cited pages with the one page you have assigned to that question, on specificity, evidence, freshness, and corroboration | Close the specific evidence gap on the assigned page instead of matching competitor page length |
| Lost citation | A previously captured citation is still present on comparable repeat runs | A citation you recorded stops appearing | Repeat the same question on the same provider and surface several times, and check crawl and deployment history for a regression | Confirm the loss persists before changing anything, and record a provider error as unavailable rather than as a loss |
Keep the three kinds of evidence apart
The stages above are also where the evidence separates. Access and indexation produce Eligible evidence: they say a page can be used. Understanding and selection are Competitive: they say the page could credibly answer the question. Only a captured answer is Observed, and it is Observed only for the question, provider, surface, and moment it was captured. Collapsing the three into one number is how a healthy technical score turns into a promise nobody can keep.
Engine behaviour differs enough that the surface you care about is worth reading about directly. The ChatGPT guide covers OpenAI's separate search, training, and user-fetch agents; the Perplexity guide covers PerplexityBot and the user-triggered fetcher; and the Google guide covers indexation, AI Overviews, and AI Mode as distinct Search surfaces rather than one ranking position.1,3,4
The fixes that move the needle
Start with access and index eligibility: make sure robots.txt permits the crawlers you intend to allow, publish a current sitemap, remove accidental noindex directives, use correct canonicals, and keep essential content available as text. Then create useful pages for the distinct questions customers ask.
Use structured data where it accurately describes visible content, state clearly what you do and who it is for, and replace generic claims with concrete use cases and appropriate proof. These changes improve eligibility and clarity, but no individual change guarantees selection or citation.
Diagnose the symptom before choosing a fix
| Symptom | Likely causes | Test | First fix |
|---|---|---|---|
| URL cannot be fetched | Robots block, firewall, redirect loop, timeout, or server error | Fetch production URL as the intended crawler and inspect logs | Restore a stable response and deliberate crawler policy |
| URL is crawlable but not indexed | Noindex, canonical conflict, duplicate or low-value page, weak discovery | Use engine inspection tools and compare canonical signals | Resolve eligibility and consolidation before expanding copy |
| Brand is described incorrectly | Ambiguous entity, inconsistent facts, stale third-party sources | Compare about, product, pricing, schema, and cited sources | Publish one consistent canonical description and correct important sources |
| Competitors are cited instead | Weaker intent match, evidence, freshness, authority, or structure | Compare the exact cited pages with the best customer page | Close the specific evidence gap rather than copying page length |
| A citation disappeared | Answer variation, source update, recrawl, competitor change, or page regression | Repeat comparable runs and inspect technical history | Confirm persistence before changing the page |
| Only the homepage appears | Supporting pages are weak, new, orphaned, or canonicalized incorrectly | Inspect index status and internal links for target pages | Strengthen page ownership and contextual discovery |
Two constructed examples, and how to read them
Both examples below are written, not captured. They are not redactions of a real log or a real scan: the sites are fictional and use the reserved .example top-level domain, the client addresses come from the ranges reserved for documentation, and no customer, person, key, or token appears in either because none was ever there. What is real is the shape. These are the two outputs a discovery investigation actually produces, and most of the value is in knowing which line to read first. Treat them as illustrations of a format, not as evidence of anything that happened.
One rule applies to both. A user-agent string is self-declared, so it is a claim rather than an identification. OpenAI, Perplexity, Anthropic, and Google each publish address ranges or a reverse-DNS method for confirming that a request came from them, and a row you have not verified that way should not be treated as a genuine visit.3,4,5,7
# CONSTRUCTED EXAMPLE, not a captured log. Fictional site, RFC 5737 addresses.
203.0.113.7 - - [11/Aug/2026:09:14:22 +0000] "GET /pricing HTTP/1.1" 200 18422
"-" "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)"
198.51.100.24 - - [11/Aug/2026:11:02:47 +0000] "GET /robots.txt HTTP/1.1" 200 412
"-" "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
198.51.100.24 - - [11/Aug/2026:11:02:49 +0000] "GET /guides/setup HTTP/1.1" 403 0
"-" "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"Read the third request first. robots.txt was served happily, and then the same client was refused the page it asked for next, which is an edge or firewall rule rather than a robots policy. That is an access failure, and no amount of rewriting /guides/setup will change it. Note also what the first request does and does not say: OAI-SearchBot fetched /pricing, which is evidence of access and nothing more. A user-agent string is self-declared, so confirm the source address against the provider's published ranges before you treat any of these rows as a genuine visit.
CONSTRUCTED EXAMPLE - not a captured scan; fictional .example site
URL https://northloop.example/pricing
Fetched at 2026-08-11T09:20:04Z
Status 200
Canonical https://northloop.example/pricing (self)
Page robots index, follow
robots.txt OAI-SearchBot: allowed | PerplexityBot: allowed | GPTBot: disallowed
Initial HTML 1,204 characters of text; no plan name, no price, no currency
After JS render 4,910 characters of text; 3 plans, 3 prices, 1 currency
Structured data Organization only; no Offer, no Product
Assigned question "how much does northloop cost"Access and indexation both pass here, so the temptation is to call the page healthy and go looking for a content idea. The finding is the two HTML lines: every fact the assigned question needs arrives only after JavaScript runs, so a retrieval fetch that does not render sees a pricing page with no prices on it. The GPTBot line is a deliberate policy choice, not a defect, and it is worth stating separately because blocking training access is independent of ChatGPT search access.
When to wait instead of rewriting
- The page was published or materially updated recently and has not been recrawled or indexed yet.
- The apparent loss occurred in only one answer run and other comparable runs still cite the page.
- The recommendation compares a detailed competitor guide with an unrelated homepage rather than the page assigned to the question.
- The cited competitor evidence does not actually support the requested change.
- A rewrite would duplicate another canonical page or introduce claims that cannot be substantiated.
This page is itself being measured
From 17 August 2026 this page is one of the pages inside a public, dated measurement programme: a fixed set of buyer questions, one page assigned to each, and a record published whether a result moves, stays flat, or goes backwards. The Zero to Cited evidence log holds the question panel, the baseline, the change ledger, and what each figure can and cannot support. Its first production answer check is now published: Robot Visible appeared only when its name was already in the question, and in none of the readable non-branded answers. The stricter weekly record remains empty until the full answer excerpts, source captures and timestamps can be published. Nothing on this page is written to make that log look better, which is the only reason the log is worth reading.
Where to start
You don't have to guess which gaps matter most. Run a free Robot Visible scan on your live URL and you'll get a six-category AI visibility score and the three highest-impact fixes to make first.
Sources and further reading
- AI features and your website — Google Search Central. Documents technical eligibility, textual content, crawl controls, internal links, and structured-data consistency.
- Publishers and Developers FAQ — OpenAI. Explains the role of OAI-SearchBot access in ChatGPT summaries and snippets.
- Overview of OpenAI Crawlers — OpenAI Developers. First-party roles, robots controls, user-agent strings, and published address ranges for OAI-SearchBot, GPTBot, and ChatGPT-User.
- Perplexity Crawlers — Perplexity. First-party distinction between PerplexityBot and the user-triggered Perplexity-User fetcher, with their robots behaviour and address endpoints.
- Does Anthropic crawl data from the web? — Claude Help Center. First-party roles and robots controls for ClaudeBot, Claude-User, and Claude-SearchBot, plus Anthropic's published crawler address list.
- Google's common crawlers — Google Crawling Infrastructure. First-party documentation for Googlebot and Google-Extended, the robots tokens each honours, and the products they affect.
- Verify requests from Google crawlers and fetchers — Google Crawling Infrastructure. Google's method for confirming genuine crawler traffic using published address ranges or forward-confirmed reverse DNS.
- Managed robots.txt and Content Signals — Cloudflare. Documents the edge-level bot controls that can refuse a request independently of the origin's own robots.txt policy.
Continue learning
See where your website stands
Run a free scan and get your AI readiness score across all six categories, with the gaps to fix first.