Experiments: how you know the change improved something

Why not before and after

Seasonality, a running campaign, a shift in the user mix — all have an effect. A simultaneous random split neutralizes them, which is why it's the only tool that gives a reliable answer.

How to run it correctly

Decide in advance on the primary metric, the sample size and the duration, and don't stop the moment an improvement appears — early stopping is the most common way to fool yourself.

Keep at least one balancing metric, to detect an improvement achieved at the expense of something else.

When there isn't enough traffic

In internal systems there isn't always a statistical sample. Then compare on a fixed evaluation set, and add a qualitative check: twenty real cases, two evaluators, a reasoned decision.