# Interpret A/B test results correctly

Reads out a test result: significance, practical impact, caveats, and a decision.

- Topic: Data & Analysis
- Framework: RCTFVV
- Level: Advanced
- Tags: Data & Analytics, Analyze, Fill-in Template
- Source: https://www.thinkwithprompt.com/prompts/interpret-a-b-test-results-correctly

## Prompt

```text
Role: Act as a data analyst reading out an A/B test.

Context: The experiment - hypothesis, primary metric, guardrail metrics, unit of randomisation, design (fixed horizon / sequential), planned sample and duration: [DESIGN]. Results - by variant: users, primary metric, guardrails, and any pre-registered segments: [RESULTS]. How long it ran and any issues (sample ratio mismatch, bugs, seasonality, novelty): [RUN_NOTES]. The decision rule agreed up front: [DECISION_RULE].

Goal: Give a correct read on whether the change worked, how confident we should be, and what to do - without p-hacking, peeking bias, or overclaiming.

Task: Produce: (1) the primary result - effect size with a confidence interval, and the p-value or posterior, interpreted correctly (what it does and does not say); (2) practical significance - is the effect big enough to matter given the cost of the change; (3) guardrail metrics - any harm; (4) validity checks - sample ratio mismatch, was the test stopped per the pre-registered rule or peeked, novelty/seasonality, segment cherry-picking; (5) a decision (ship / do not ship / iterate / extend) against the agreed rule, with the reasoning and the residual risk.

Constraints: Interpret significance correctly - a non-significant result is not proof of no effect; a significant result on a secondary metric after the fact is not a finding. Respect the pre-registered decision rule; call out any deviation. Report the confidence interval, not just the point estimate. No claims beyond the primary metric unless pre-registered.

Examples: Result line: 'Primary metric (signup rate): +0.8pp, 95% CI [-0.1pp, +1.7pp], p=0.09. Interpretation: consistent with anything from a small negative to a meaningful positive; not conclusive at the agreed bar.'

Format: Sections: Primary result & interpretation | Practical significance | Guardrails | Validity checks | Decision & residual risk. ~550 words.

Verification: Recompute the effect and interval from the raw counts; check the sample ratio; flag any analysis choice made after seeing the data.

Validation: Challenge as a sceptical statistician: is any part of this overclaiming, and would the decision survive if the true effect were the bottom of the confidence interval? Adjust the read.
```

## What to fill in

- DESIGN
- RESULTS
- RUN_NOTES
- DECISION_RULE

Free to copy and use. From ThinkWithPrompt (https://www.thinkwithprompt.com/), a library of 2,663 curated AI prompts.