A/B Testing and Experimentation
Most product changes that a team feels sure about make no measurable difference. Ron Kohavi and colleagues reported in 2009 that at Microsoft, only about a third of tested ideas improved the metric they were meant to improve.
Learn on the move: press play and turn your commute, walk or drive into time that grows your design career.

That's why teams run experiments before they trust a change. An A/B test shows two versions to comparable groups at the same time, so a difference in results can be traced to the change itself.
One Change in the Invoicing App
The lessons follow one experiment in the invoicing app used by about 20,000 small businesses. Product analytics showed that new accounts stop most often at adding a client, and interviews suggested that people expect to pick the client first.
So the team tests a new invoice editor that starts with the client. Today 41% of new accounts send their first invoice within seven days, and the test asks whether the new editor raises that share.
Anatomy of an Experiment
A before-and-after comparison can't show what a change did. Imagine a team that ships a new editor in June, sees first-week sends rise, and misses that June signups came from a tax-season campaign that brought more motivated accounts.

So an experiment runs both versions at the same time, built from a few named parts.

The Control and the Variant
The control is the current version, and the variant is the changed one. In the invoicing test, the control is today's editor and the variant starts with choosing the client.
That's why the variant changes one idea at a time. A variant that moves the client step and also rewrites every label can win, and nobody will know which change did it.
The Unit of Assignment
The unit is what gets assigned to a version: a user, an account or a session. The invoicing test assigns accounts, so everyone on one business account sees the same editor.
So the unit matches how people use the product. Assigning users would let a bakery's owner and its bookkeeper see two different editors on the same invoices.

The Split and the Exposure
Accounts are split at random, often half and half. Random assignment spreads motivated and casual accounts evenly across both versions, which a hand-picked split can't do.
Thus only accounts that reach the editor enter the test. Counting accounts that never open the editor adds people the change can't affect and hides its effect.
The Duration
Decide the length before the test starts, in whole weeks. Small businesses invoice more at the start of a month and less on weekends, so a test that runs for nine days overweights some days.
That's because behavior changes across the week and the month. A test that covers whole cycles compares like with like.

Pro tip: Before launch, run the test with both groups seeing the control for a few days. If the two groups already differ, the assignment is broken.



