Interpret A/B test results correctly

Role: Act as a data analyst reading out an A/B test. Context: The experiment - hypothesis, primary metric, guardrail metrics, unit of randomisation, design (fixed horizon / sequential), planned sample and duration: [DESIGN]. Results - by variant: users, primary metric, guardrails, and any pre-registered segments: [RESULTS]. How long it ran and any issues (sample ratio mismatch, bugs, seasonality, novelty): [RUN_NOTES]. The decision rule agreed up front: [DECISION_RULE]. Goal: Give a correct read on whether the change worked, how confident we should be, and what to do - without p-hacking, peeking bias, or overclaiming. Task: Produce: (1) the primary result - effect size with a confidence interval, and the p-value or posterior, interpreted correctly (what it does and does not say); (2) practical significance - is the effect big enough to matter given the cost of the change; (3) guardrail metrics - any harm; (4) validity checks - sample ratio mismatch, was the test stopped per the pre-registered rule or peeked, novelty/seasonality, segment cherry-picking; (5) a decision (ship / do not ship / iterate / extend) against the agreed rule, with the reasoning and the residual risk. Constraints: Interpret significance correctly - a non-significant result is not proof of no effect; a significant result on a secondary metric after the fact is not a finding. Respect the pre-registered decision rule; call out any deviation. Report the confidence interval, not just the point estimate. No claims beyond the primary metric unless pre-registered. Examples: Result line: 'Primary metric (signup rate): +0.8pp, 95% CI [-0.1pp, +1.7pp], p=0.09. Interpretation: consistent with anything from a small negative to a meaningful positive; not conclusive at the agreed bar.' Format: Sections: Primary result & interpretation | Practical significance | Guardrails | Validity checks | Decision & residual risk. ~550 words. Verification: Recompute the effect and interval from the raw counts; check the sample ratio; flag any analysis choice made after seeing the data. Validation: Challenge as a sceptical statistician: is any part of this overclaiming, and would the decision survive if the true effect were the bottom of the confidence interval? Adjust the read.