Evidence disclosure. No answer service was queried. All prompts, outputs, service names and numbers on this page are synthetic fixtures for explaining the calculation.
Decision summary
Report three separate observations: whether a brand appeared, whether the answer displayed a citation, and whether a cited source supported a selected factual claim. Always show the denominator, sampling frame, collection period and failure count.
Three measures, three questions
| Measure | Question | Denominator | Does not mean |
|---|---|---|---|
| Appearance rate | Was the coded brand present in the assessed response? | Valid assessed responses in the declared sample | Market share or user preference |
| Citation presence | Did the response display at least one source URL? | Valid responses assessed for citations | That a source supports every claim |
| Claim support | Did the inspected source support the selected claim at collection time? | Claims with an available source and completed assessment | Overall answer accuracy |
Build a prompt register
Give each prompt an immutable ID. Store the exact wording, category, service, displayed model or version when available, mode, locale and collection timestamp. A later wording change creates a new version rather than silently replacing history.
Synthetic prompt fixture
“Which types of provider should a small example retailer compare when planning a fictional returns service?”
No service call occurred. “Synthetic service A” and “Example brand A” are placeholders, not real entities or observations.
A worked calculation that stays inside its sample
Suppose a synthetic fixture contains three valid assessed responses and Example brand A appears in two. The fixture appearance rate is 2/3. It should be reported as “2 of 3 synthetic responses” - not as 67% of a market. Another brand can appear in the same responses, so brand rates need not sum to 100%.
Failures are records
A blocked, empty or technically failed run remains in the execution log. It is excluded from the valid-response denominator under a declared rule; it must not be converted into a non-appearance.
Methods
Freeze the prompt version and brand-panel version for each wave. Run the declared schedule, retain permitted raw evidence privately, code the same panel against each response, review an annotation subset and publish aggregate counts with the protocol version.
Before collection, define what counts as a brand appearance, a displayed citation and a factual claim selected for source review. Keep automated extraction separate from human verification. Recalculate every published rate from immutable run and annotation records so a reviewer can reproduce both numerator and denominator.
Limitations
Answers may vary with time, account state, location, mode and unreported service changes. A small declared sample is not a census. Citation presence is not claim support, and a supported selected claim is not a general accuracy judgement.
Sources
Evidence scope. This methods article uses no live service response, citation or brand record as a source. A future measured release must link each aggregate claim to a versioned prompt register, execution log and permitted evidence record.
- Disclose exact prompts and permitted outputs for any measured release.
- Record service and model information only as displayed at collection.
- Review calculation logic and keep appearances, citations and verified statements separate.
Update status
No material updates have been recorded since publication. If a run, annotation or denominator later changes, the note will identify the affected wave and asset version, preserve lineage and explain whether the conclusion changed.