Planned protocol; pilot not completed. The proposed pilot is 12 non-branded prompts x two answer services x three repetitions = 72 planned response runs. The category, 20-brand panel, prompts and services will be frozen and versioned before collection. The 72 planned response runs have not been run and no pilot result exists.
Research question
Within a declared UK category and collection window, how often do brands in a fixed panel appear in valid responses to a fixed set of non-branded questions, and what citation and claim-support evidence accompanies those appearances?
Methods
The protocol uses a fixed hierarchy: programme, wave, prompt run, annotation and public aggregate release. Before collection, approve the category rationale, immutable brand IDs, editable aliases, exact prompt versions, service contexts, repetition schedule, run states, evidence permissions and reviewer roles. Each completed run must be traceable to one frozen wave configuration.
Execute prompt-service pairs according to the declared schedule rather than stopping when a desired answer appears. Record UTC timestamps, technical outcomes and displayed model or version only when the service exposes it. Code the same 20-brand panel against each valid response, then review a declared annotation subset independently.
Unit of collection
A run is one exact prompt submitted once to one declared service context at one timestamp. The 20-brand panel is coded against the same response; it does not multiply the service-call count.
| Component | Planned value | Required record |
|---|---|---|
| Prompts | 12 non-branded questions | Exact text, prompt ID, version and category rationale |
| Services | 2 answer services | Service, displayed model/version if available, mode and locale |
| Repetitions | 3 per prompt/service pair | Run ID, UTC timestamp, status and permitted evidence reference |
| Brand panel | 20 brands | Immutable brand ID, editable public name and alias record |
| Planned calls | 72 | Execution log that retains technical failures |
Sampling and version control
Document how the category and panel are selected. Freeze the panel version and prompt version for a wave. A prompt wording or service-mode change creates a new version. Store collection timestamps in UTC and state the reader-facing London date.
Run states and exclusions
Allowed states are valid, technical failure, blocked, empty, interrupted and excluded-with-reason. Failed runs remain visible in the execution log. They are never coded as zero brand appearances and never silently replaced. Any replacement is a new linked run.
Annotation
For each valid response, code brand IDs, mention positions, displayed citation URLs and extraction notes. Claim-support review is a separate record linked to a selected factual claim and its source. Use supported, unsupported, ambiguous and unavailable states; preserve disagreement and correction history.
Denominators
- Appearance rate: valid responses assessed for the brand.
- Citation presence: valid responses assessed for displayed citations.
- Claim support: selected claims with an available source and completed assessment.
Multiple brands can appear in one response, so rates need not sum to 100%. Publish counts with rates and show missing, failed and unavailable records separately.
Quality checks
Independently review a declared annotation subset. Check prompt/run lineage, duplicate runs, panel consistency, citation extraction and denominator calculations. Record disagreements rather than overwriting them without trace.
Rights, privacy and release
Keep raw outputs and sensitive evidence in restricted storage. Release only material permitted by platform terms, rights, privacy review and the documented editorial purpose. Public files require a version, data dictionary, limitations, licensing note and safe filename.
Limitations
The sample cannot represent every user, prompt, mode, location or time. Service behaviour can change without a stable displayed version. Appearance is not preference, citation is not support, and claim support for a selected statement is not a general accuracy score.
Sources
Evidence scope. This published protocol cites no live answer, brand observation or completed study. A measured wave must publish its approved protocol version and data dictionary, while permitted raw evidence remains in restricted storage. External methodological and platform rules must be rechecked at collection time.
Update status
No protocol amendments have been recorded since publication. A later material amendment will receive a new version, state what changed and why, identify any affected runs and prevent unmatched waves from being combined without explanation.