How to Build an AI Visibility Index for Your Own Niche
We published an index of 54 therapy practice websites. This is the method behind it, written so you can run it on your own niche without asking us. It costs an afternoon, and the hardest part is not the measuring. It is catching the four times your own instrument lies to you.
Why the method is worth more than the numbers
An index of a niche is a piece of content anyone can produce once. What makes it defensible is whether a stranger can rebuild it and land close to your figures. If they cannot, you have published an opinion with decimal places.
So here is the whole thing. The sampling frame, the exclusions, the bias we know about, and the four bugs that our own tooling introduced before the numbers were fit to print. We wrote it as a sequence because that is how it runs, and because the order matters more than any single step. Skipping the fifth one is how most published indexes go wrong.
We measured on three axes, which is how this practice thinks about the problem generally: whether a machine can reach you, whether it can trust what it reaches, and whether a person can decide from it. The method page covers the axes. This piece covers the measuring.
First, pick a niche you can define in one sentence
Ours was private therapy practices in the United States that see patients. That sentence has to survive contact with edge cases, because you will hit them within twenty minutes. Directories, marketplaces, national chains, office rental businesses and hospital systems all show up in the same search results and none of them is the thing you said you were measuring.
A niche you cannot define in a sentence produces an index nobody can repeat. If you find yourself writing "and also", stop and narrow it.
Second, use one identical query per city, and write down what you excluded
We ran the same query in ten metros: Denver, Seattle, Austin, Atlanta, Chicago, Boston, Phoenix, Portland, Nashville and Minneapolis. One query, no variations, no hand picking. Fifty eight sites came out of it.
Then the part almost nobody publishes: every exclusion is written down with the reason next to it. Not "we removed directories", but the specific domain and the specific reason. A reader who disagrees with an exclusion can see exactly which one and argue with it. A reader who cannot see the exclusions has to take the whole index on faith.
This is also the honest place to declare a conflict. We sell to this niche. Anyone reading our numbers about therapists should know that before they read the numbers, not after.
Third, sample the pages the site actually has
Here is where our instrument lied to us first, and it is the mistake most likely to be sitting in yours right now.
To score a site you read a handful of its pages. Ours took the sitemap, and only looked at the homepage links if the sitemap returned nothing. That sounds sensible and it is a cascade, which means a stale sitemap silently wins over the live site, every time, leaving no trace in the output.
We caught it because we ran the tool on ourselves. Our own sitemap has not regenerated since late August, so the instrument was scoring us on our old slugs and never once read our pricing page, our FAQ or our method page. The three pages that decide two of the three axes.
Changing the cascade to a union of sitemap and homepage links moved our own score from 84 to 88 without touching a line of the website. Across the cohort it changed which pages got read on 24 of 54 domains. What came in was fees, rates, FAQ, contact and services pages. The whole niche was being undervalued by our own shortcut.
Fourth, read robots.txt by user agent group, not with a regex
A robots.txt file is a set of groups. Each group names its agents and then states its rules, and a rule means nothing until you know which group it sits in. Read the file as one blob and a rule aimed at one crawler looks like a rule aimed at everyone.
Our first pass reported that ten sites in the cohort blocked everything. Read by group, the number of sites blocking everything was zero. Most of those ten were blocking a single Huawei crawler that feeds no assistant at all.
That is a 100 percent error rate on the most quotable finding in the study, and it is the difference between "this niche has locked the door" and "this niche left it open and nobody noticed". If you publish only one thing from this article, publish this one.
Fifth, fix the bug where the number is produced, not where you found it
We fixed that robots.txt bug. We fixed it in the new file that parses robots.txt properly. The script that actually produced the published cohort numbers kept its old loose regex, and nobody noticed for a full sprint.
So the article said "zero sites block everything" while the score above it had been computed treating ten of them as if they did. Ten practices were losing 20 points each for rules that never pointed at them. Correcting it moved the cohort average from 72.4 to 74.3.
The rule we wrote for ourselves afterwards: fixing a bug is not fixing it where you noticed it. It is fixing it where the number comes out. Two implementations of the same logic is what let this through, so now there is one, and the other file imports it.
Sixth, assume the failure is yours before you report it
Three sites returned 429 to our crawler and we nearly published them as broken. They were not. We were fetching eight pages in a burst and tripping their rate limits. Even with a slow single thread they kept refusing us while a plain command line fetch got 200 from the same address in the same second, which points at the TLS fingerprint rather than anything the site did wrong.
Eleven sites looked like they served a single crawlable page, which would have made a spectacular headline. They did not. Our sitemap reader was not following a sitemap index and its pattern did not match CDATA. One of those eleven publishes 187 pages.
Every one of these was found by asking "is this the site or is this me" before writing anything down. Every one of them would have been a confident, quotable, wrong sentence in a published study.
The four times the instrument lied, and what each one cost
| What broke | What it would have said | What was true |
|---|---|---|
| robots.txt read as one blob | 10 of 54 block every crawler | Zero block every crawler |
| Same fix never reached the scoring script | Cohort FOUND of 72.4 | 74.3, and 10 sites wrongly penalised |
| Sitemap read as a cascade, not a union | Pricing and FAQ pages never read | Sample changed on 24 of 54 sites |
| Rate limits and CDATA read as site defects | 3 sites broken, 11 sites one page deep | All readable, one publishes 187 pages |
Seventh, publish the bias in the same breath as the number
Our sample comes from sites that already rank in a public search. That inflates one axis by construction, because we selected on the thing we then measured. It cannot be fixed by wording, only by declaring it.
What the sample can say honestly is narrower and still useful: for practices that have already won the search, here is how they do on the two axes nobody optimises for. That sentence is worth more than a bigger claim we would have to defend badly.
Four sites out of 58 were unreadable to us, and they are excluded from every average rather than scored as zero. Scoring them zero would have quietly moved every number we published, in the direction that flatters our own argument.
What the finished index looked like
| Signal | Cohort average | What it measures |
|---|---|---|
| Retrievability | 97.0 | Whether a machine can fetch and read the pages |
| Citability | 50.5 | Whether there is anything specific enough to quote |
| Verifiable identity | 57.8 | Whether a named human is in the structured data |
| Decision served | 60.5 | Whether price, scope and fit are on the site |
Retrievability at 97 against citability at 50.5 is the entire story of the niche, and it only becomes visible because the two are measured separately. A single blended score would have averaged the two into a number that means nothing. The full results are in the index itself.
What we chose not to publish, and why
We published aggregates only. No practice is named, no per domain table exists in public, and that is the settled form rather than a first step. The method stays copyable without turning the work into a public assessment of 54 businesses that never agreed to be measured.
If you run this on your own niche, that decision is yours to make again. It is easier to make before you have the table than after.
Questions
How long does this actually take?
An afternoon for the measuring once the tooling exists, and considerably longer the first time, because most of the work is the four corrections above rather than the fetching. Budget more time for checking your instrument than for running it.
Do I need a large sample for this to be worth publishing?
No, but you need an honest one. Fifty four sites with a published frame and written exclusions is more defensible than five hundred scraped without either. Size is what makes a claim precise. The frame is what makes it true.
Can I just use an existing SEO audit tool instead?
For retrievability, mostly yes. For citability and for whether a decision is served, no, because those are judgements about content that crawl tools do not make. That gap is the reason the index says anything the tools do not.
What if my niche is one where sites block crawlers?
Then you have found something worth publishing, provided you read robots.txt by group first. We expected to find that and found the opposite, which is why the finding was worth writing up at all.
Should I name the sites I measured?
Decide before you collect, not after you see the table. We chose aggregates because we sell into the niche we measured, and a ranking would have read as a sales tool rather than a study.
The part that transfers
Almost none of the value here is in the therapy niche. It is in the sequence: define, exclude in writing, sample the live site, parse by group, fix at the source, suspect yourself, then declare the bias. Run that on accountants, on architects, on wedding photographers. The numbers will differ and the four failure modes will not.
We publish our own corrections because an index that never reports an error is either lucky or unexamined, and readers are right to assume the second. If you want a second read on what a machine can currently retrieve, trust and quote from your own site, that is a reasonable thing to bring to a call. What the practice does and what gets reported is on the services page, with more detail on the FAQ page. The call is thirty minutes.
