If you are missing from Common Crawl and your robots.txt is clean, the most likely cause is that CCBot gave your domain a budget too small to reach you.

That mechanism is documented, mostly in a conference talk by Common Crawl's Sebastian Nagel that few people in search have read. It explains a category of absence that no amount of on-page work will fix.

How CCBot fills its queues

Most SEOs carry Googlebot's model: a crawler follows links continuously, discovers pages as it goes, queues them, and comes back. Discovery and fetching happen in one rolling process.

Common Crawl works differently. The queues are filled shortly before a crawl rather than continuously during it, from three sources: links sampled from the preceding crawl, URLs discovered through sitemaps, and a shallow crawl of sampled home pages.

Domain-level harmonic centrality ranks then define a budget for each domain, meaning how many of your URLs get sampled or fetched at all. That domain score is projected down to individual pages by OPIC or inlink counts, which decides which of your pages make the cut.

The distance between budgets is larger than most people assume. In summer 2021, top domains were allowed 25 million URLs, 150,000 per host, and 500,000 subdomains. Domains below rank 24 million were allowed 3,000 URLs, 2,500 per host, and 12 subdomains. The distribution between those two ends is logarithmic.

Rank also governs discovery. Sitemaps and home pages are sampled for URL discovery always for top-ranking domains, and only sometimes for lower-ranked ones. A weak position therefore means more than fewer pages fetched. Your sitemap may never be read, so new pages are never found, so the next crawl has less of you to sample from.

The discovery loop: low harmonic centrality gives a smaller budget, so the sitemap is only sometimes sampled, so fewer URLs are found, so there is less to sample from next crawl.

What harmonic centrality measures

Harmonic centrality measures how close your domain sits to the structural core of the web, computed across the whole hyperlink graph. It is not a count of the links pointing at you. It is how short the paths are between you and everything else.

Common Crawl uses it in preference to PageRank because it resists manipulation better. PageRank can be inflated with link farms and dense clusters of interlinked pages. Nagel's talk states that harmonic centrality is more robust against link spam than PageRank. The published ranks come from harmonic centrality with PageRank alongside for comparison, computed with the WebGraph and LAW packages from the University of Milan.

The scale of the spam problem is on the record. In October 2017 the crawler hit a spam cluster and fetched 56 million pages, 1.5 percent of that crawl, across 70,000 domains. The link graph showed 320,000 domains and 2.5 billion subdomains in the same cluster. Common Crawl responded by capping the number of crawled subdomains per domain and by detecting the worst clusters directly.

CCBot has prioritized this way since 2017, documented in infrastructure talks rather than SEO blogs, which is why so few people in search have run into it.

The practical translation: link volume is the wrong thing to optimize. Distance from the well-connected core is the thing that moves. A hundred links from sites that are themselves peripheral move you very little. A handful from domains sitting near the core move you a lot.

How to check yours

Metehan Yeşilyurt built a CC Rank Checker that looks up a domain's harmonic centrality and PageRank position. It is the fastest way to see where you sit.

Common Crawl publishes the underlying data too. The web graph statistics cover domain and host rankings across releases, and the raw rank files live under data.commoncrawl.org/projects/hyperlinkgraph at domain or host level, though those run to multiple gigabytes each.

To check presence rather than rank, query the index server directly. It reports whether the domain is there, when it was last captured, and how many pages. That tells you the outcome. The rank tells you why.

I fold both into the AI Visibility Audit, because reading them apart is misleading. A domain absent from the index with a strong rank has a blocking problem. A domain absent with a weak rank has a budget problem. Those need completely different work.

Be careful about the inference

A chain of reasoning circulates in AI-search commentary: harmonic centrality drives crawl priority, more crawling means more presence in the archive, the archive trains the models, therefore a better web-graph position means better representation in AI.

The first link is documented. The rest are plausible and unproven. Presence in the corpus is filtered, deduplicated, and weighted before it becomes training data, which I have written about in Crawled Is Not Cited, and none of that pipeline is visible from outside. Treat the full chain as a correlation worth tracking rather than a mechanism you can pull.

Some commentary also confuses the two directions. A lower centrality score means less crawling, not more. Rank 1 is the top of the list, not the bottom, and the domains at the bottom are the ones on the 3,000-URL budget. Use the Common Crawl and academic sources rather than the secondary write-ups.

Who this leaves out

Mozilla has made the equity criticism and it is a fair one. Domains with lower centrality scores, including those serving digitally marginalized communities, are less likely to be included. A crawl budget allocated by structural web position will systematically under-collect the sites that were already under-linked.

That is not a defect in harmonic centrality. It is what centrality means. It does mean that the corpus training these models is not a sample of the web. It is a sample weighted toward the web's existing core, and the sites furthest from that core are the least equipped to notice or complain.

If you work with organizations in that position, such as community groups, regional publishers, and small institutions in smaller-language markets, run the rank check before anyone spends a budget on content. Being invisible for structural reasons and being invisible for content reasons look identical on a dashboard and need entirely different work.

Sources