A common claim is that a crawlable page is visible to AI. That claim is wrong, and most of the work now sits in the difference.

Three things have to happen, in order, and each one can stop you on its own:

  1. Crawled. A crawler fetched the page and stored the bytes.
  2. Trained. Those bytes survived curation and made it into a model's training corpus, or into a live retrieval index.
  3. Retrieved. The model surfaced your page in an answer, with a citation.

If you treat these as one thing, you optimize the wrong layer. If you treat them as three, you can identify where you disappear.

Gate oneCrawledA crawler fetched the page and stored the bytes.Fails at: robots.txt, a WAF rule, a bot-management default, a CDN setting.
Gate twoKeptThose bytes survived curation and made it into the corpus or the index.Fails at: extraction, quality filtering, deduplication, data mixing.
Gate threeRetrievedThe model surfaced your page in an answer, with a citation.Fails at: web-graph distance, weak entity association, a shrinking citation surface.
Three gates, three different failure modes. A single "AI visibility score" averages across all of them and tells you nothing about which one you are losing.

Gate one: are you in the data?

Common Crawl is the largest public source of AI training data, and CCBot is the crawler that collects it. If CCBot cannot reach you, you are absent from the dataset that a large share of every model was built on. You can also be absent when your edge returns a 403 status to CCBot while serving human visitors a 200. On-page work does not fix an empty Web ARChive (WARC) record.

Most teams skip this gate, because the tools they already use do not report it. Google Search Console does not report what Common Crawl stored. Neither does your rank tracker. The cause is usually not robots.txt, which is the one file most teams check. The cause is usually a web application firewall (WAF) rule, a bot-management default, or a content delivery network (CDN) setting that someone enabled in an earlier year for a valid reason. Ranking well in Google tells you nothing about whether a training crawler can reach you. Different client, different rules, different outcome.

To answer this with evidence instead of inference, query the index directly. The audit tool I maintain does that against Common Crawl.

Gate two: crawled does not mean kept

Say the crawler got in. The bytes still have to survive a pipeline that nobody outside the lab can observe.

Between a stored WARC record and a training run, text passes through extraction, language detection, quality filtering, safety filtering, deduplication, tokenization, and data mixing. A page can be discarded at any one of those steps, and each step discards a different kind of page. Jérôme Salomon calls this the training-readiness layer, and almost nobody audits it.

Stored WARC→Extraction→Language ID→Quality filter→Safety filter→Deduplication→Tokenization→Data mixing→Training
Everything between the first step and the last is invisible from outside the lab. A log line proves the fetch happened. It says nothing about which of these steps your page passed.

Extraction keeps the main content and discards navigation, footers, banners, and ads. If your homepage assembles itself in the browser with JavaScript, extraction receives an almost-empty shell. The page is then present in the corpus and carries no usable text. If your main content is not structurally distinct from your template, extraction can keep the wrong half.

Quality filtering is where the current labs do most of their work. FineWeb built a 15-trillion-token dataset from 96 Common Crawl snapshots, and the paper's contribution was an ablation of the filtering and deduplication choices, showing how much those choices change downstream model quality. Its educational subset, FineWeb-Edu, is 1.3 trillion tokens, under a tenth the size, and it performs substantially better on knowledge and reasoning benchmarks. The trend is toward training on a curated slice rather than the whole crawl. Being in Common Crawl puts you in the raw pile. It does not put you in the slice.

Deduplication collapses repetition, and the web repeats constantly. Lee et al. showed that deduplication reduces memorization and improves training efficiency, which is why every lab does it. If your content is templated across hundreds of URLs, syndicated to partners, or mostly restated boilerplate, deduplication is the step that removes you.

Data mixing decides how much any of it counts. LLaMA's paper remains the clearest public example: Common Crawl, C4, GitHub, Wikipedia, books, arXiv, and Stack Exchange, each weighted separately, with the curated sources sampled more than once and the raw web sampled less.

"We are in Common Crawl" is a claim about raw material.

Training data is the selected result. The distance between the two is where the fix stops being technical and becomes editorial: be extractable, be distinctive, be corroborated somewhere other than your own site, and describe yourself the same way everywhere you appear. Publishing more pages is not on that list. At this layer, more pages is the wrong instinct, and deduplication is the reason.

Gate three: retrieval is a popularity-and-proximity problem

You can be fully crawled and cleanly kept and still lose the last gate. Retrieval favors sources the system already trusts, and that trust follows from how central you are in the web's link graph, not only from how many links point at you. Distance from the well-connected core matters as much as volume. The mechanism behind that distance is documented, and I have set it out in why Common Crawl skipped your site.

The gates are also less independent than the diagnosis implies. Retrieval is not neutral search. The model decides what to query, how to rewrite the question, and which domains are worth reading, all before it fetches a single result. A brand the model already associates with a topic becomes a candidate. A brand it has never encountered may never become a candidate query at all. Failing gate two makes gate three harder, and no amount of on-page work recovers that.

The citation surface can also move under you, and a model release is enough to move it. When ChatGPT's default switched to GPT-5.3 on March 4, 2026, RESONEO measured the average number of unique domains cited per response falling from 19.1 to 15.2, across 27,000 responses to 400 prompts. The ratio of URLs to domains held steady, so the model was not reading less. It read the same amount from fewer sources.

Read that as a dated measurement rather than the current state. GPT-5.3 stopped being the default on May 5, 2026, when GPT-5.5 Instant replaced it, and again on August 6, 2026, when GPT-5.6 Luna took over on free accounts. No comparable before-and-after study has been published for either change, and there is reason to expect the number moved back: RESONEO notes that GPT-5.4, which shipped the day after the switch, runs more than ten fan-out queries per answer. What holds is the mechanism, not the figure. One default change cut a fifth of the cited domains in four weeks, nobody outside the lab saw it coming, and no amount of work on your own site had any bearing on it.

Unique domains cited per response, before and after the GPT-5.3 switch of March 4, 2026 19.1 15.2 Before GPT-5.3 After GPT-5.3 0 5 10 15 20 Unique domains cited per response
A 20.5% drop in cited domains over the four weeks after the March 4, 2026 switch, measured across 27,000 responses to 400 prompts. Source: RESONEO, data from Meteoria. Scale starts at zero. GPT-5.3 has not been the default since May 5, 2026.

This is the least controllable gate and the most misunderstood. Structured data does not produce a citation. Proximity takes years to build.

Timing: when your work can reach a model

One more thing gets conflated with the three gates. It is a timing problem rather than a layer problem.

Once a model is trained and shipped, its parametric memory does not change. Nothing you publish today changes what it already knows. New content can influence a live retrieval answer, where the model searches and reads at the time of the request, but it does not enter the model's memory until a future pretraining run or a new release.

Two consequences belong in front of anyone who signs off on this work.

The first: you cannot correlate a crawler hit with a change in model behavior. The timescales do not line up. The crawl that matters happened months ago, and no lab's release cycle was ever aligned to your publishing calendar.

The second: what you ship this quarter is graded against a model trained before you shipped it. Publishing and being known can be two quarters apart.

That gap is where most AI visibility reporting goes wrong. A team ships content, probes a model a week later, sees no change, and concludes the work failed. Only the retrieval layer could have moved in a week. The memory layer was never in play.

What to do

  • Run the five checks. I wrote them up as a field guide for Common Crawl, The AI Visibility Audit: CCBot access, index coverage, harmonic centrality, structured data, and server-side rendering. Allow about 90 minutes per site.
  • Measure all three gates separately. A single "AI visibility score" hides which gate you failed. To avoid guessing, diagnose per gate.
  • Fix the cheapest gate first. Unblocking a crawler and server-rendering your words costs one deploy. Building web-graph proximity takes years. Do the deploy today.
  • Optimize for survivability, not volume. Extractable, distinctive, corroborated, and consistent. Those four words are the whole training-readiness brief.
  • Probe with search on and with search off. Search off reports what the model knows. Search on reports what your customer is served. Reporting either one as the other is how teams end up confidently wrong.
  • Do not report "crawled" as "cited." It is the most common way teams convince themselves the work is done when it has not started.

Crawled is not cited. Measure the two separately, and treat the difference between them as the work that remains.

Sources