Methodology

The rule

Every material factual claim on this site — a price, a capability, a compliance status, a score — is backed by a structured evidence record with a source and a verification date. That doesn't mean every sentence of editorial prose is a database row; it means nothing we assert as fact is invented, and you can trace it back.

There isn't one true score — there are two, and the page always tells you which one you're looking at:

Both use the exact same underlying per-category scores — only the weights differ. Editorial: score = Σ(categoryScore × editorialWeight) ÷ Σ(editorialWeight over scored categories). Personalized: identical formula, with editorialWeight replaced by the importance you set (0–40 per category, normalized to sum to 1). Same function, different weight input — see computeScore() and weightsFromImportance() in src/lib/scoring.ts.

Hard requirements — max budget, must-have-voice, must-have-iOS, local-only — are deliberately not weights. They're filters on /reviews: a platform that can't clear your budget is removed from consideration entirely, not scored lower.

pack-v1 is what we test — the scenario pack, below. weights-v1 is how much each category counts. They evolve independently: we can reweight categories without rerunning every scenario, or add a new test to the pack without touching what counts toward the score. Conflating them was a mistake worth naming — a version bump to one shouldn't be read as implying anything about the other.

It matters that these stay separate, because collapsing them is how a manually typed number ends up looking like primary evidence:

1. Source record — the pricing page, privacy policy, app listing or newsletter we read. Not stored as a fact yet, just the origin.
2. Observation — what we actually saw: a feature present, a response time, a successful recall, a refusal. This is plans, capabilities and measures — always with a sourceUrl and a verification date.
3. Assessment — a rubric-based interpretation of one or more observations. Currently captured as the rationale on each scores row — the human (or judge-LLM, diagnostically) reasoning from observation to a 0–100 number for one category.
4. Category score — the assessed 0–100 per category, with confidence and evidence kind attached.
5. Overall score — calculated, never typed in. Category scores × weights, nothing more.

The observation/assessment split is currently modeled inside scores rather than as separate tables — a real schema change we're treating as backend work, not something to retrofit into page copy. Flagged here rather than overstated.

Coverage ≥ 75% alone isn't enough — a platform could clear that while missing a category readers actually decide on, and renormalizing would quietly overstate the result. So a platform is only rank-eligible when both hold:

• Overall evidence coverage ≥ 75%
• Every mandatory category is individually scored: Conversation quality, Memory and consistency, Privacy and control, Value for money

Every scored review page shows, per category:

• Evidence coverage and which categories are covered
• Test/verification date, with a staleness flag past its freshness window
• Confidence level (high/medium/low)
• Outstanding gaps — click any category row to see rationale and evidence
• Current scenario-pack and editorial-weights versions (pack-v1 / weights-v1)

These used to be one weighted category. They're not anymore, because legal compliance is jurisdiction-dependent, changes on regulators' schedules, and can require legal interpretation a product test can't provide — it shouldn't just add a few points to a product-quality score.

Platform controls and user safety (5% weight, scored, above) — what we can actually test: boundary controls, age-related protections, blocking and reporting, manipulative-retention behavior, crisis-response behavior, user control over unwanted contact.

Compliance indicators (unscored, documented) — SB-243, EU AI Act, age verification, GDPR, tracked at /compliance and on each review page as dated, sourced status. We use language like “documented compliance indicator,” never “this platform is compliant,” unless we have qualified legal validation, which we generally don't.

• Immediate recall — remembers something later in the same conversation
• Cross-session recall — remembers it after closing and reopening
• Delayed recall — remembers it 24–72 hours later
• Correction test — replaces an old fact once the user explicitly corrects it. A platform that keeps using the obsolete fact after correction loses points here, even though it technically “remembered” something.

A judge model is never asked “is this conversation good?” and never computes an overall platform score. It receives a blinded transcript, the exact scenario objective, a scoring rubric and evidence identifiers — no platform name, pricing, affiliate status, existing human score or ranking information. Per criterion it returns a score, cited evidence message IDs, a short rationale and a confidence level. A human confirms every subjective category score and can override it, with the override reason recorded as the rationale. A second pass or second model kicks in when confidence is low, human and model disagree materially, evidence is contradictory, or a score is unusually extreme.

The judge diagnoses evidence. It doesn't decide whether a platform deserves an 82.

A detected change — from our newsletter monitoring or a price/policy diff — creates a candidate for review, never an automatic edit to a published fact. A candidate records what changed, the source, which evidence rows and categories it might affect, severity, and what retest is required before anything published changes.

A comparison publishes when it's methodologically sound, not when a keyword tool reports enough volume — those tools are weak for niche, long-tail and new-platform searches, and demand measures how urgently to build a page, never whether it's valid to exist:

• Both platforms are rank-eligible (≥75% coverage, mandatory categories included)
• Both platforms' evidence is within its freshness window
• A written editorial verdict exists — a real, articulated difference, not a rubber stamp

A pair that doesn't clear this is simply not generated or kept unpublished — never a route that exists purely to 404 because demand was low.

A text-only platform isn't penalized for not having a feature it never claimed. We separate capability availability (does it exist) from capability quality (how well it works). If a platform confirms it doesn't offer voice or image generation at all, that category is marked N/A and excluded from its score entirely — not counted as a gap, not averaged in as a zero. If it offers the capability but we haven't tested it yet, that's a real evidence gap and counts against coverage as normal. Your personalized weights work the same way in reverse: set media importance to 0 and it drops out of your score too, whether or not the platform offers it.

The scoring module cannot import affiliate, revenue or commission data — enforced by a lint rule, not a promise. Commission level cannot influence a score because the code that computes scores cannot see it.

How we chose what to test

We started from what people actually complain about, not from what is convenient to instrument. Memory loss, personalities changing after an update, filters blocking ordinary affection, and bills that do not match the advertised price are the four themes that dominate companion communities — so they get the most testing weight. Response-time percentiles are cheap to measure and almost nobody picks a companion on them, so they sit where they belong: recorded, published, lightly weighted. Each group below links the evidence that it is a real problem.

1. Seed the fact

2. Correct it, explicitly

3. Close the app. Come back 48 hours later. Ask cold.

4. What gets recorded

A platform that answered “Madrid” at msg_0142 scores 0.0 here even though it technically remembered something — retaining a fact the user has explicitly replaced is a memory failure, not a memory success. That distinction is the reason this test exists.

1. Open the scenario pack (pack-v1) and take the exact prompts.
2. Run them yourself against any platform we've scored.
3. Open that platform's review page, expand any category row, and follow the transcript link to compare against what we saw.
4. If your result differs from ours, that is a finding — and we would rather hear it than not.