Almost 800 sites seen by an AI, and a ruler changed three times

On the first of July we opened Pharos, the tool that measures how readable and reachable a site is for an AI (what it is and why it matters we covered in If an AI can't read you, it won't cite you). In four days almost 800 real sites ran through it, all different. This article tells you what they saw, but also something we had not planned: in three of those four days we changed the measuring stick three times. And that, in the end, is the most interesting part.

Because the average score of those sites climbed in three days without the web improving: what changed was the way we measured it. A report that hides the movements of its own instrument is lying by omission. This one puts them on the front page.

The ruler that moves

Pharos gives each site a score from 0 to 100. But not all sites were measured with the same yardstick, because we recalibrated it as we used it. Three versions in three days, each with its own average score:

• Version 1.0.0 (1 July): 571 sites, average score 83. One site in four 'Excellent'.
• Version 1.0.1 (2 July): same scaffolding, corrected verdicts. Average score 87.
• Version 1.1.0 (3 July): seven new checks, weights rebalanced. Average score 90, almost three sites in four 'Excellent'.

Same web, three different rulers, seven points of difference. The bare average would suggest the sites got better; what got better was only the way we measure them. To understand what these numbers really say, you have to separate what changed with the instrument from what stayed put.

What did not change: the door is open

Let us start with where every version agrees, because it is the part to trust most: if a figure survives three recalibrations, it is not an artifact of the ruler. The recurring fear ('AIs will block me', 'I have to defend myself from crawlers') finds no support in any of the three.

• About 93% of sites let AI crawlers through in the robots.txt. Across 1.0.0, 1.0.1 and 1.1.0 the number does not move.
• About 90% already have the content in the HTML served by the server: an AI that does not run the JavaScript still finds the text.
• About 94% are reachable by CCBot, Common Crawl's crawler, the source many AIs draw on.

Opening the door, in short, is almost free, and almost everyone has done it. Often without even deciding to: it is the default setting of most sites. Whoever fears being cut off from AI, in the vast majority of cases, is not. It is the most solid result in the whole report, precisely because it held up across every version of the instrument.

What did change: the 'house in disorder' was the ruler

In the first version one figure jumped out: one site in two had 'critical' HTML semantics. Semantics is the way the code tells a machine what is a heading, what is an article, what is a menu, instead of leaving it to guess from a wall of anonymous divs. It looked like the most widespread flaw on the web.

Then we went and looked at the verdicts one by one, and it was the ruler that was crooked. Version 1.0.0 counted the ratio of semantic tags to total elements: a perfectly readable site built with lots of divs got failed all the same. On 2 July, in 1.0.1, we switched to looking at the presence of landmarks (is there a main, a header, a nav?), which is what a machine actually looks for. Same sites, verdict flipped: 'critical' semantics goes from 50% to 13%, and with the next recalibration to 7%. The house was not in disorder, the yardstick was bent.

Same story for reachability. 1.0.0 tried to knock on the door with the user-agent of about a dozen crawlers, and counted as 'blocked' every server that refused the probe: one site in four came out that way. But a site behind Cloudflare that refuses our probe is not necessarily blocking AI, it simply does not let us verify. From 1.0.1 that outcome became 'Not verifiable' and no longer weighs on the score: 'critical' drops from 25% to 5%.

What stays genuinely weak

There is one area that stayed low across all three versions, and so is weak for real: the signals dedicated to AI agents. These are recent files, meant to tell an agent what it can do on a site and where to find things (skill.md, the files under .well-known and the like). Only 13-17% of sites have at least one. More than four in five have none, in every version.

Here the sites are not getting it wrong: they simply have not started yet. It is the empty frontier, the point where today you stand out with little because the comparison almost always starts from zero. With a caveat that counts as much as the others: in 1.1.0 we removed llms.txt from the score, because an Ahrefs study shows that about 97% of those files are never read by AI bots. A fashionable signal that moves nothing does not deserve to weigh.

The new version then uncovered more prosaic gaps, of plumbing. Two sites in three do not expose an ETag or a Last-Modified, the two 'validators' that let a crawler ask 'has anything changed?' and get a clean 304 instead of the whole page. Without them, every pass re-downloads everything: wasted work for whoever passes by often. And the SEO fundamentals stay approximate on about one site in three (title and meta too long, too short or missing), a figure stable across all versions.

Why we are telling you this way

A tool that measures readability for AI is as young as the problem it measures. It might have looked more solid to publish a single number and stay quiet about the corrections. But every recalibration came from the feedback of those using Pharos and from primary sources (the documentation of OpenAI, Google, Bing, Cloudflare, the Ahrefs and Vercel studies): telling them is the honest part, not the embarrassing one.

Pharos's numbers stay deterministic: rules, not opinions, and the same site analyzed twice with the same version gives the same score. But deterministic does not mean infallible. It means that when you fix the rule, every past verdict updates with it, in the same direction, for everyone. That is exactly what happened over these three days.

How to read these numbers

Two things, to read them right. These sites are not a random sample of the web: whoever tries Pharos already thinks about the topic, and starts with an advantage. So the gaps that remain here are unlikely to be smaller on the less attentive web. And these are aggregated numbers: what counts are the distributions and patterns, not the individual site.

Then there is the ruler, which the whole article is about. The figures that do not depend on the version (who opens the door, who has the files for agents) we read across all sites; those the ruler changed (semantics, reachability) only inside the version that measured them. Mixing them would give a fake average.

Where to start

If you are wondering where you stand, the way to know is to measure it, with today's ruler. The map from these days suggests the right order of moves anyway:

• Check that you are actually reachable by AI crawlers: it is the check that depends least on fashions and most on your server. An open robots.txt is not enough if an over-zealous firewall leaves the crawler outside.
• Cover the SEO fundamentals: clear title and meta, a coherent heading structure. About one site in three still has room here, and it did not change from one version to the next.
• Look at the semantics, but without panic: on today's ruler most sites are fine. What counts is that the code says what is a heading, an article, a section, and that depends on how the site is built.
• The agent signals come last: they are the frontier, but on a house already in order, not before.

You find out your score in a minute: try Pharos on your address, with the latest version of the ruler, the report is open and email-free. And if you want the wider picture of how to get found inside an AI's answers, not just whether you reach them, we covered it in The right answer, before they leave your site.