How the Agent Visibility Index is measured

Every number we publish is designed to survive scrutiny. This page describes the full pipeline; it is the contract behind every score.

What we measure

How often AI assistants recommend each brand when asked realistic buyer questions — the Share of Recommendation — plus first-mention rate, sentiment (clean / caveat / recommended-against), and the reasons assistants give for choosing or rejecting each brand.

The task corpus

• 60 tasks per cycle (current pilot scale: a 60-task stride sample of the 448-task corpus), expanded deterministically from 40 hand-written templates × 8 buyer personas × 8 constraints, spanning five intents: discovery, comparison, displacement, constraint-based, validation.
• Templates are mined from real buyer phrasing on forums and review sites, not LLM-generated.
• Discovery tasks never name a brand — no question-begging.
• The corpus is frozen and versioned (currently v1.0); it changes only at version boundaries, so week-over-week comparisons always compare like with like.

Sampling

Each task runs 3× per model surface per weekly cycle (models are stochastic; single answers are anecdotes). Raw responses are stored verbatim, forever — every historical score can be re-derived and re-audited.

Statistics

• Every proportion carries a 95% Wilson score interval — scores wear their error bars.
• Week-over-week movement is gated by a two-proportion z-test with Benjamini–Hochberg false-discovery control across the brand family. Only movement that survives is labelled significant; everything else renders as "within noise". With ~18 brands tested weekly, uncorrected p<0.05 alone would produce roughly one false "mover" per cycle — we correct for that.
• The weekly test treats cycles as independent samples although the task set is fixed (a paired design); this makes the test conservative — we under-claim rather than over-claim.

Model-update detection

A ~24-task control set of stable scenarios runs every cycle. If its brand distribution shifts beyond a total-variation threshold, the cycle is flagged and the surface's epoch increments. Deltas across an epoch boundary are annotated "model update" and are never labelled significant — you should never mistake a model change for a market change.

What "the model said" means

We query each provider's API with an identical, neutral system prompt across all surfaces. This is a controlled proxy for consumer chat products, which layer their own prompts, memory, and retrieval on the same models. It maximizes comparability across surfaces; it is not a screenshot of any one app. Grounded (web-search-enabled) surfaces are tracked separately as they ship.

Measured surfaces

Model versions are pinned, never aliases — epochs rotate only at cycle boundaries, and the full history is below.

The judge

Structured extraction (which brands, what position, what sentiment, what reasons) is performed by an LLM judge with a strict JSON schema and alias normalization against the corpus brand dictionary. The judge-agreement rate on a 200-response hand-labelled sample is pending publication — it will appear here before the public index launches.