Experiments
A/B tests on top of flags, with sequential statistics that are safe to peek at, CUPED and SRM checks.
An experiment is a multivariate flag plus metrics. Users are assigned by the flag, exposure comes from
$feature_flag_called, and results are computed from sufficient statistics per variant, so no individual
events leave the region to compute them.
Lifecycle
- Plan. The power calculator gives the sample size you need for a minimum detectable effect, or the
effect you can detect with the traffic you have (
POST …/experiments/plan). - Draft. Pick the flag, variants and weights, a primary metric, secondary metrics and guardrails.
- Launch turns the flag on for the experiment's audience.
- Stop ends assignment. Ship rolls the winning variant out; it needs the named
experiments.shipscope.
Metrics
| Kind | Example |
|---|---|
| Proportion | share of exposed users who purchased |
| Mean | average order value per user |
| Sum | total revenue per user |
| Ratio | revenue per session (delta method) |
Statistics
The engine (crates/stats, MIT) was checked against scipy, statsmodels and GrowthBook's gbstats to within
3e-10. In a 2,000-run A/A simulation the fixed-horizon false-positive rate was 4.95%, and the sequential test
stayed under 5% even with continuous peeking.
| Method | When to use it |
|---|---|
| Sequential (default) | Always-valid confidence sequences. Look at results whenever you like; peeking does not inflate false positives. |
| Fixed horizon | Welch's t-test / z-test. Decide the sample size up front and read the result once. |
| Bayesian | Chance to beat control and expected loss. Exact Beta-Binomial for proportions, Normal otherwise. |
- CUPED is on by default: a pre-exposure covariate reduces variance, so the same traffic detects smaller effects.
- Sample-ratio mismatch is checked with a chi-square test (alpha 0.001). A mismatch blocks the verdict instead of reporting a result that is probably wrong.
- Multiple comparisons: Holm correction across primary metrics, Benjamini–Hochberg across secondary ones.
- Guardrails are one-sided non-inferiority tests with a relative or absolute margin. Each reports
pass,fail,inconclusive,insufficient_dataorblocked_by_srm. - Users exposed to more than one variant are excluded and counted; a warning appears above 1%.
Defaults: alpha 0.05, power 0.8, at least 100 units per variant, Bayesian threshold 0.95.
Verdicts are one of better, worse, inconclusive, insufficient_data or blocked_by_srm.
Through the API
Experiments are served on /v1/projects/{project_id}/experiments with a token that has experiments:read or
experiments:write: list, create, get, update, archive, launch, stop, ship, results and plan. See
the API overview.