← All writing

Experimentation in the Real World Part 3 of 3

Can we roll out early and keep learning?

Can we make an informed commercial decision halfway through an experiment without necessarily giving up the ability to learn from it?

Shan Islam

In Part 1, I discussed the pressure to act on promising results in a planned six-week e-commerce experiment. In Part 2, I explored fixed-power design as an alternative to committing to a sample size based on historical variance.

Here I want to bring those two ideas together. Can we make an informed commercial decision halfway through an experiment without necessarily giving up the ability to learn from it?

Protecting the original comparison

One approach I find particularly interesting is Spotify’s separation of sequential safety monitoring from final product evaluation. I would extend that idea to positive early commercial signals as well.

Spotify’s documented approach runs sequential deterioration checks regardless of the final evaluation strategy. The positive early commercial signal and separate rollout population discussed below are my proposed extension, not a description of Spotify’s existing methodology.

Consider a protected experimental population, randomly assigned 50:50 and left unchanged throughout the experiment. A separate eligible population initially receives control. If an early efficacy checkpoint and acceptable guardrails justify expansion, that separate population can receive the treatment while the protected comparison continues.

This requires planning before launch, sufficient traffic, persistent assignments and an assumption that expansion does not affect the protected population. In a capacity-constrained business, for example, extra orders might change availability for both experimental groups. A protected holdout would not automatically resolve that interference.

I simulated the combined design using 250 new customers per arm per day, seven days of observation before each cohort enters analysis, and a planned interim check after 2,500 mature customers per arm. The interim criterion was a one-sided p-value below 0.005. Another 250 visitors per day could receive the treatment after an early decision. This means the protected control, protected treatment and rollout population each account for a third of eligible new traffic. The protected experiment continued to its fixed-power endpoint irrespective of the efficacy check.

Traffic allocation before and after expansionEach bar totals 750 new customers per day. The protected 50:50 comparison stays unchanged.
Traffic allocation before and after expansionBefore expansion750 / day250250250After expansion750 / day250250250Protected controlProtected treatmentRollout: controlRollout: treatmentTraffic allocation before and after expansionBefore expansion750 / day250250250After expansion750 / day250250250Protected controlProtected treatmentRollout: controlRollout: treatment
Chart data
ScenarioProtected controlProtected treatmentRollout on controlRollout on treatment
Before expansion250 per day250 per day250 per day0 per day
After expansion250 per day250 per day0 per day250 per day
True relative upliftEarly expansion rateMean incremental seven-day conversions attributable to pre-readout expansion
0%0.5%0
5%5.6%2
10%26.6%18
20%90.0%126
Probability of early expansionSimulated expansion rate at the planned one-sided interim efficacy check.
Probability of early expansion0%25%50%75%100%0.5%0%5.6%5%26.6%10%90%20%True relative upliftProbability of early expansion0%25%50%75%100%0.5%0%5.6%5%26.6%10%90%20%True relative uplift
Chart data
ScenarioEarly expansion rate
0% true uplift0.5%
5% true uplift5.6%
10% true uplift26.6%
20% true uplift90%

Across the three simulated baseline rates, the final two-sided false positive rate was between 4.9% and 5.0%, and power at the absolute MDE was approximately 81%. Early expansion and the final conclusion are separate decisions. If either can declare a definitive success, their joint error rate must instead be accounted for.

There are important limitations. The simulation assumes independent customers, constant effects and no interference or rollout costs. A treatment that initially increases conversion could subsequently lose its effect or even become harmful. In a stress test where the first ten recruitment cohorts had a 20% uplift but subsequent cohorts had a 10% reduction, expansion triggered in about 90% of simulations and cost an average of 58 conversions. This is why the rollback plan matters.

Conversions gained or lost before final readoutMean incremental seven-day conversions. Effect reversal is a separate stress test.
Conversions gained or lost before final readout-140-700701400% uplift05% uplift210% uplift1820% uplift126Effect reversal-58Incremental seven-day conversionsConversions gained or lost before final readout-140-700701400% uplift05% uplift210% uplift1820% uplift126Effect reversal-58Incremental seven-day conversions
Chart data
ScenarioIncremental conversions
0% uplift0 conversions
5% uplift2 conversions
10% uplift18 conversions
20% uplift126 conversions
Effect reversal-58 conversions

Guardrails introduce their own information requirements. In another stress test, a checkout failure rate of 4% with a tolerable increase of one percentage point was used as a non-inferiority gate. Requiring the upper one-sided 95% confidence bound to fall below that threshold reduced early expansion for a genuinely harmful treatment from about 90% to 0.05%. It also reduced expansion for a safe, effective treatment to around 50%. This illustrates a trade-off, not a universal threshold recommendation.

A protected cohort is only one option. A full commercial rollout may sometimes be justified when the opportunity is time-sensitive and reversibility is high. In that case we should be clear that we have chosen to forgo some further randomised evidence, rather than presenting an early estimate as a definitive final result. If we change the allocation inside an ongoing experiment, we need an analysis designed for that adaptation, particularly when outcomes vary over time.

Designing the decisions before the experiment

Experiment lifecycle and decision points
  1. LaunchImplementation and assignment checks
  2. Early monitoringHarm and technical alerts
  3. Interim evaluationCommercial action and guardrails
  4. Information assessmentSample, duration and maturity
  5. Final evaluationEffect and commercial relevance

The article table below gives the full question and approach at each stage.

StageQuestionApproach
LaunchIs it working correctly?Production checks, SRM, assignment and integrity monitoring
Early monitoringIs it causing harm?Technical alerts and calibrated sequential harm checks
Interim evaluationShould we act commercially?Planned efficacy checks, mature outcomes, guardrails and commercial judgement
Information assessmentDo we have enough data?Fixed-power or planned fixed-sample requirement, minimum duration and maturity
Final evaluationWhat have we learnt?Effect estimate, appropriate confidence interval, guardrails and commercial relevance

Before launch I would want the team to agree on the primary metric and observation window, MDE, power, significance level, guardrail margins, interim criteria, rollout options, maximum duration and rollback conditions. If the information target cannot be reached within the available time, the result may simply be inconclusive. Where early harm can terminate the study or where multiple interim efficacy decisions are possible, the inferential plan must account for those rules; the simulations here do not establish guarantees for all such combinations.

There will always be an element of commercial judgement. The role of the data scientist is to help the business understand the uncertainty and make an informed decision. I don’t think the objective should be to eliminate early commercial rollout. It should be to design experiments that support the decisions we may need to make throughout their lifecycle.

Further reading

The ideas and perspectives in this article are my own, informed by practical experience. I used an LLM to assist with research, simulations, visualisations and editing. All numerical examples are fictional or simulated.