Once a month, somewhere in a server farm, a crawler named CCBot wakes up with a long list and a budget. Your site is on the list or it isn't, and where it lands on that list comes down to a number almost nobody in SEO has heard of. It's called harmonic centrality. Common Crawl runs on it, and Common Crawl is where a big chunk of AI training data starts.
Common Crawl publishes a monthly web graph that shows how sites link to each other at the domain and at the host level. It's used to decide which sites get crawled first. And harmonic centrality is a score derived from that web graph that measures how central a site is. How few clicks it sits from the well-connected core of the web and a rank closer to number one means a higher crawl priority.
PageRank, which we've been optimising for over 25 years, is how many important sites link to your site, or how many roads lead to your house. Harmonic centrality is about proximity, your distance from the web's core. It's how few hops it takes to get from the core to you, or how few turns you are from the center of the city.
I bring this up at every SEO conference I speak at on behalf of the Common Crawl Foundation, and almost every single time someone comes up afterward and asks where it came from. It's a great question. The answer is a 76-year relay race, and the people running it had no idea they were in one. A social psychologist. A Norwegian peace researcher with an airline map. A mathematician who was mad at search engines before Google was a company. A nuclear physicist. A guy who went on to study Rousseau's paranoia. Two professors in Milan. None of them was trying to help you rank. They were chasing other things, and what they built ended up deciding a good part of what today's machines get to read.
Five people, a wheel, and MIT
Alex Bavelas was one of Kurt Lewin's first grad students. He followed Lewin from Iowa to MIT, got his PhD in 1948 and started the Group Networks Laboratory the same year.
The experiments that came out of that lab, a lot of them run by his student Harold Leavitt, were simple. Five people sat in separate cubicles. They couldn't see or talk to each other; the only way to communicate was to push written notes through slots in the cubicle walls. Each person got a card with five of six symbols on it, and the group had to figure out the one symbol that was on everybody's card.
The trick was which slots were open. Open them one way and everyone could only send notes to one person in the middle, a wheel. Open them another way and the notes had to pass down a line, a chain. There was also a Y, and a circle where each person could reach only the two people beside them.
The wheel won. In Leavitt's 1951 study the wheel groups solved the puzzle in about 32 seconds on average and the chains took 53. The person in the middle collected everyone's symbols, spotted the answer and sent it back out. The circle was slow, but the people in it enjoyed themselves the most. Later studies with harder, messier problems found the decentralized groups did better. Anyone who has survived a reorg could have told them that for free.
Bavelas needed a way to say who sat in the middle, so in 1950 he wrote one down in "Communication Patterns in Task-Oriented Groups". Add up how far a person is from everyone else and flip it. Short total, high score. That's closeness centrality, the grandparent of everything that follows. In 1951 he took the experiments to the Macy Conferences, the postwar meetings that launched cybernetics. His talk, "Communication Patterns in Problem-Solving Groups", sits in that year's transactions a few talks ahead of Claude Shannon showing off a maze-solving machine.
Sixteen years later in 1966, Gert Sabidussi, a mathematician from Graz, Austria, then at McMaster University (coincidentally where, about two decades after he left, I attended the fine art program and too many fraternity parties), cleaned it up in "The centrality index of a graph" and laid down rules a centrality measure ought to follow. Remember the rules. They come back in 2013 wearing a nicer suit.
Here's the catch. Closeness has one fatal habit. If one node can't reach another, the distance is infinite, the sum is infinite, and the score drops dead at zero. In a room with five people that never happens; on the web it happens every second of every day. The web is a vast, humming, ungoverned sprawl of orphan pages, dead ends, sites that link to nothing and sites nothing links to. Closeness can't handle that; one missing path and every score drops to zero.
1968, Oslo: a peace researcher and an airline map
Two years after Sabidussi, a young sociologist named Nils Petter Gleditsch wrote his thesis at the University of Oslo: The Structure of the International Airline Network. Countries were the nodes, airline connections were the links. And his data had the web's problem decades before there was a web: some countries had no connections to anyone. In 1951, nine countries sat on his map as isolates. By 1965 it was down to one.
He tried it the Bavelas way first, adding up distances, and then wrote down exactly where that breaks: "the indices derived so far only apply to strongly connected structures. How can we derive measures which apply generally?" Translation for SEOs: one orphan page and the math falls over.
His answer, two typewritten pages later, starts with a better question: "what contribution does each cell in the distance matrix make to the centrality of its row and column units?" Then: "Clearly, if the distance is small, the contribution is great, if it is large the contribution is small." He called each contribution "closeness," one over the distance, and added them up for each country. A country with no connections at all scores (n−1)·(1/∞) = 0. Not broken, just zero. His verdict was that the new indices "are applicable both to strongly connected structures and to non-connected structures."
That's harmonic centrality, normalized and everything, 41 years before anyone called it that. It's the earliest written version anyone has found. In SEO terms, the original page: page 2.B.9 of a typewritten thesis in Oslo, with zero backlinks.
In 1970 Gleditsch and Tord Høivik, who was finishing his own thesis in Oslo, put it in a journal, "Structural parameters of graphs" in Quality and Quantity, revised from a talk at the Second Nordic Peace Research Conference in Örenäs, Sweden, in May 1968. They didn't claim to have invented it. "A common solution has been to work with inverse distances," they wrote, and moved on. They also flagged the trade-off that matters most to anyone reading this: flipping the distances makes the score "influenced almost solely by the small distances in the distance matrix." Your near neighbours decide your number. The sites a hop or two from you count for a lot; the far side of the web barely moves it.
Then the idea went quiet. Gleditsch spent his career on war and peace, directed the Peace Research Institute Oslo and edited the Journal of Peace Research from 1983 to 2010. Everyone after this point in the story found the idea again on their own.
The thesis only resurfaced in October 2026, and I got a front-row seat. When this post first went up, I wrote to Sebastiano Vigna, who you'll meet below, to thank him for the math Common Crawl runs on. That turned into an overnight email conversation about who really wrote this down first. He went back through the old literature, and by morning he had it: Gleditsch, 1968, all of it, on one typewritten page. A couple of hours later he'd fixed Wikipedia's centrality pages to credit him, from his own account, "vigna", fifty-eight years after Gleditsch typed it up. Thank you, Sebastiano. This section exists because you went looking.
I wrote to Gleditsch too. He's 84 and still a research professor at PRIO, and he wrote back within the day. "I am greatly touched by this instance of belated fame," he said, and his only regret was that he and Høivik never followed it up. "Looking back, I think I had more fun working on this than anything I have worked on since, except possibly the democratic peace in the 1990s."
The Italians fix it again (twice, sort of)
It's 1997. Massimo Marchiori, a mathematician at the University of Padova, walks into the WWW6 conference held in Santa Clara, California with "The quest for correct information on the Web: hyper search engines". His system, HyperSearch, rode on top of Excite, HotBot, Lycos, WebCrawler and OpenText and re-ranked their results by giving each page credit for the pages it pointed to. The average result quality went from 63.2 to 76.7.
He wasn't the first to rank by what a page links to. In July 1988, Mark Frisse, an internist at Washington University School of Medicine in St. Louis who had just finished a master's in medical information science at Stanford, published "Searching for information in a hypertext medical handbook" in Communications of the ACM. It opens in an emergency room, early evening: a diabetic man in a coma, pneumonia on the chest x-ray, allergic to penicillin, and a doctor who can't remember the dosing details. Frisse's fix was to chop the Washington University Manual of Medical Therapeutics, one of the most popular books in medicine, into hypertext cards and rank them. Each card got two scores: an "intrinsic" one for the search terms on the card, and an "extrinsic" one passed up from the cards it linked to. Search "tracheostomy" and seven cards come back; one scores higher because of the weight passed up from the cards beneath it. Note the direction: the credit flows from the pages you link to, not the ones linking to you. That's backwards from the way PageRank or harmonic centrality would do it a decade later.
Marchiori's version added distance. HyperSearch faded each linked page's text exponentially with depth, which his paper defines as "how many clicks" away it sits. The further away the page, the less it counts. Hold that thought.
Marchiori also named the enemy. "Search engines persuasion," he called it: tuning pages to cheat a search engine. That's SEO spam, diagnosed by a mathematician in 1997, back when the web was a few years old and people were already rigging it. I've been fighting spam since my blekko days, and it hasn't gotten any less weird.
Google had started at Stanford as a project called BackRub (here's what it looked like in 1996, and yes, that was the name). When Sergey Brin and Larry Page published the paper that introduced Google in 1998, they cited Marchiori twice. Once among the earlier projects using links to improve search. And once, tucked into an appendix about advertising, for reporting that OpenText was selling companies the right to sit at the top of its results. Search engines selling the top spots. Imagine.
Then Marchiori teamed up with Vito Latora, a theoretical nuclear physicist from Catania. Everybody in the field was losing their minds over Duncan Watts and Steven Strogatz's 1998 Nature paper on "small-world" networks (a worm's nervous system, the western US power grid, the Kevin Bacon graph). Marchiori and Latora noticed the path-length measure everyone was borrowing only worked if every node could reach every other node. The web makes no such promise. The web makes no promises at all.
Their fix, "Harmony in the Small-World" in 2000, was the one Gleditsch had typed up in Oslo 32 years earlier, though their paper doesn't cite him: use "not the arithmetic mean but the harmonic mean of all the distances." It sounds complicated; the fix is a single change. No path? Distance is still infinite, but now you add one over it, and one over infinity is zero. A site with no path to you simply counts for nothing, and the rest of the score still works. The paper even traces the harmonic mean back to Pythagoras and Plato.
Marchiori went on to the W3C at MIT, edited the P3P privacy standard, and in 2004 made MIT Technology Review's TR100 list of top innovators under 35. Latora ended up holding the Chair of Complex Systems at Queen Mary University of London. Their 2001 follow-up, "Efficient Behavior of Small-World Networks", called the idea network efficiency.
Nine years later, Yannick Rochat invents it again
2009, Zürich. Yannick Rochat, a researcher at the University of Lausanne, presents "Closeness centrality extended to unconnected graphs: the harmonic centrality index". His version: the sum of the inverted distances instead of the inverted sum of the distances.
Same idea Gleditsch typed up in 1968 and Marchiori and Latora published in 2000; his paper cites neither. In SEO terms, a duplicate of a duplicate, and nobody set a canonical.
He tested it on the classics, a pod of dolphins, a karate club and the 16 Florentine families of the Renaissance, which his paper notes, with an exclamation mark, "were especially chosen among more than a hundred!" He later wrote a PhD mapping the characters in Rousseau's Confessions, including "a period of deep paranoia," and now teaches game studies in Lausanne. Dolphins, karate, the Medici, Rousseau, video games. That's a career.
Milan proves it, then makes it run on the whole web
Paolo Boldi and Sebastiano Vigna run the Laboratory for Web Algorithmics at the University of Milan. They built WebGraph (software that crushes web-sized link graphs down to something you can work with) and UbiCrawler, an early distributed crawler.
In 2013 they dragged Sabidussi's rules back out. In "Axioms for Centrality" they picked three, size, density and score monotonicity, and ran the usual suspects through them. Closeness failed all three; PageRank failed size. Harmonic centrality walked out clean, which they admitted "was surprising for us to discover."
A metric you can't compute on the whole web is a nice paper and nothing more, so they built HyperBall. They ran it on 4.8 billion nodes on one 40-core machine and figured it could handle a hundred billion with "just" 2 TiB of RAM. Their scare quotes, not mine. Not too shabby.
Side note, because it's too good to leave out. Vigna also designed xorshift128+, a random number generator. In 2015 Chrome, Firefox and Safari all switched Math.random to it. So for years, every time a web page asked your browser for a random number, an algorithm from a professor in Milan picked it.
Where Common Crawl (and I) come in
Gil Elbaz, our founder, started Common Crawl in 2007. Before that, Google bought his company and turned it into AdSense. When he was done at Google, he realized that nobody had a copy of the web that the public could freely access. So he built this nonprofit, the Common Crawl Foundation, and later on it became the quiet infrastructure for the AI era. He had no idea that was going to happen. He wanted to make a public data set available. Academics started using the data to do research, and we're cited in over 10,000 research papers. And then around 2020 the AI boom started, and LLMs started downloading the data set and using it to train their models.
Nobody else has this data. We've been crawling since 2008, and you can't go back in time to recrawl the web; it's a history of mankind on a disk. It's free, it's on AWS, and we keep no logs of who downloads it.
From 2012 to 2015, blekko donated ranking data and metadata for 140 million sites and 22 billion pages, and Common Crawl used it to avoid "webspam, porn, and the influence of excessive SEO," according to Sebastian Nagel's 2021 talk. I was at blekko then, working with Rich Skrenta and Greg Lindahl. All three of us work at Common Crawl now. It's a small valley.
Harmonic centrality wasn't there from the start. From 2008 to 2012, Common Crawl's own crawler ranked pages with its own PageRank. Then came blekko's data, and in 2016 rankings from Alexa and Common Search. Sebastian Nagel, Common Crawl's crawl engineer, was fighting spam, and he moved the crawl onto Common Crawl's own web graph, ranked by harmonic centrality. His talk gives the reason in one line: harmonic centrality is "more robust against link spam than page rank."
May 22, 2017. Sebastian Nagel publishes Common Crawl's first in-house web graph: 385 million hosts, 2.5 billion links, loaded into Boldi and Vigna's WebGraph, harmonic centrality computed by HyperBall. PageRank came along "for comparison." The post lists what came out of it, including a ranked list of hosts to expand the crawl frontier and pages ranked by harmonic centrality "with less influence from spam." Five months later the crawler hit a spam cluster anyway: 56 million pages from 70,000 domains, 1.5% of that crawl.
Common Crawl still uses these ranks today. A 2026 paper from Common Crawl Foundation researchers Michael Paris, Hande Çelikkanat and Luca Foppiano, "Measuring What the Crawler Sees", describes each monthly fetch list as built from the previous crawl and ranked by harmonic centrality, with host budgets and rank thresholds deciding how far coverage reaches. They don't publish the actual numbers.
How harmonic centrality works
Your score is the sum, over every other site, of one divided by how many link hops it takes to get from that site to yours.
H(x) = Σ 1 / d(y, x), summed over every site y other than x.
A direct link is worth 1. Two hops, 1/2. Three hops, 1/3. No path at all, zero, courtesy of the Italians.
Common Crawl computes it per host and per domain, never per page, so a link from any page on a central site counts as a link from that site.
This is why where a link comes from matters more than how many you've got. Site A has 3 direct links and 4 more sites two hops out: 3 + 2 = 5. Site B has one lonely direct link, but it's from a hub that 10 other sites link to, so all 10 reach B in two hops: 1 + 5 = 6. One good link beats a pile of junk from the edges. Every time.
Well, almost every time. In Boldi and Vigna's own retrieval test, PageRank variants and SALSA (a hub-and-authority cousin of PageRank) beat it when only links between different hosts were counted.
What does this have to do with AI?
If Common Crawl doesn't fetch your page, your page isn't in anything built from Common Crawl. Filtered Common Crawl supplied about 82% of GPT-3's training tokens (Brown et al., 2020), and at least 64% of 47 large language models released through 2023 used it (Mozilla).
This is parametric memory, the model's long-term memory, separate from the RAG search that looks up what the price of these shoes is today. It's your whole brand and your entities inside the LLM. You want to be remembered versus recalled.
No AI lab says it uses these ranks as a signal, every lab filters Common Crawl its own way, and being in training data isn't the same as being cited (Crawled is not cited covers that gap).
To get into the crawl, your site has to pass three gates:
- Permission. Your robots.txt lets CCBot in.
- Edge. Your CDN or Cloudflare isn't blocking AI bots with a preset filter.
- Rendering. Your words are in the HTML. CCBot doesn't run JavaScript.
My free AI Visibility Audit checks all three, reports your harmonic centrality, and shows your rank history across Common Crawl's web graph releases.
Once the gates pass, work on your harmonic centrality. One high harmonic centrality link could be more powerful than 50 high PageRank links. Earn links from sites that lots of other sites already link to. For more on how the ranks shape what CCBot finds, see Why Common Crawl skipped your site.
Bavelas worked all of this out with five people, five booths and a stack of handwritten notes. The rest of us needed 76 years and a few billion nodes.
Sources
Nagel, From Web Graphs to Prioritizing Web Crawls, Open Search Symposium, 2021
TensorFlow Datasets, C4
Bavelas, Communication Patterns in Task-Oriented Groups, Journal of the Acoustical Society of America 22(6), 1950
Alex Bavelas, Wikipedia
Bavelas, Communication Patterns in Problem-Solving Groups, in Pias (ed.), Cybernetics: The Macy Conferences 1946–1953, The Complete Transactions, diaphanes, pp. 349–381 (1951 conference)
Sabidussi, The centrality index of a graph, Psychometrika 31, 1966
Gleditsch, The Structure of the International Airline Network, Mag.art. thesis, University of Oslo, 1968, pp. 2.B.8–2.B.9
Høivik and Gleditsch, Structural parameters of graphs. A theoretical investigation, Quality and Quantity 4, 1970
Nils Petter Gleditsch, Wikipedia
Frisse, Searching for information in a hypertext medical handbook, Communications of the ACM 31(7), 1988
Marchiori, The quest for correct information on the Web: hyper search engines, WWW6, 1997
Brin and Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine, 1998
Massimo Marchiori, W3C
Watts and Strogatz, Collective dynamics of 'small-world' networks, Nature 393, 1998
Marchiori and Latora, Harmony in the Small-World, Physica A 285, 2000
Latora and Marchiori, Efficient Behavior of Small-World Networks, Physical Review Letters 87, 2001
Vito Latora, Wikipedia
Rochat, Closeness centrality extended to unconnected graphs: the harmonic centrality index, ASNA 2009
Rochat, Character Networks and Centrality, PhD thesis, University of Lausanne, 2014
Boldi and Vigna, Axioms for Centrality, Internet Mathematics 10, 2014
Boldi and Vigna, In-Core Computation of Geometric Centralities with HyperBall, 2013
Laboratory for Web Algorithmics, University of Milan
V8, There's Math.random(), and then there's Math.random(), 2015
Sebastiano Vigna, Wikipedia
Common Crawl, About
Nagel, Common Crawl's First In-House Web Graph, 2017
Paris, Çelikkanat and Foppiano, Measuring What the Crawler Sees, 2026
Brown et al., Language Models are Few-Shot Learners, 2020
Mozilla Foundation, How Common Crawl's Data Infrastructure Shaped the Battle Royale over Generative AI, 2024