UX glossary
A/B Testing
An A/B test is an experiment that shows two versions of a page or element at the same time to randomly split groups of visitors. It lets you decide with real data which version works better.
Imagine an online garden supply store debating whether the button in the cart should say “Checkout” or “Pay securely now.” Both options have passionate supporters on the team, but nobody has data. Instead of arguing in circles, the team lets visitors decide, with an A/B test.
Explained simply
In an A/B test, you show two versions of the same page to different people at the same time. The current version is called the control (A), and the modified version is the variant (B). Each visitor is randomly assigned to one group and only ever sees that one version. At the end, you compare a metric you chose up front, such as completed purchases.
You may know the principle from medicine: one group gets the new drug, another doesn’t, and who ends up where is decided at random. That randomness is what makes the comparison fair, because on average both groups are made up of the same kinds of people. That’s why A/B tests are also called online controlled experiments or split tests.
What makes a good test
An A/B test needs a clear hypothesis, a testable assumption such as: “If the button emphasizes security, more people will complete their purchase.” Ideally, you change only one element. If you change several things at once, you won’t know afterward which one made the difference.
You also need enough traffic. To tell a real difference from random noise, you look at statistical significance. It tells you how unlikely a difference this big would be if both versions actually performed the same. A confidence level of 95 percent is common, which often takes thousands of visits. The Nielsen Norman Group recommends running tests for at least one to two weeks and not stopping early just because one version is briefly ahead.
An example
Back to the garden store. Before: the button says “Checkout,” and the team wonders why so many carts get abandoned. After: for two weeks, half of all visitors see “Pay securely now.” Version B leads to noticeably more purchases, and the difference is significant. At the same time, the team keeps an eye on a guardrail metric, such as the number of returns, to make sure the win doesn’t cause damage somewhere else. You can learn more about purchase decisions in Why Do Visitors Convert into Buyers?.
Common misconceptions
An A/B test tells you what works better, but not why. To understand why people hesitate, usability testing is the better tool, because you get to watch over their shoulder. And a statistically significant result isn’t automatically a meaningful one: a tiny difference can be measurable and still not be worth the effort.
In practice
- Write down your hypothesis before you start. Note what you’re changing, which metric should move and why. The best hypotheses come from observations in user research.
- Change one element per test. That way you can clearly attribute the effect. Multivariate tests can handle several changes at once, but they need even more traffic.
- Set a duration and stick to it. Plan for at least one to two full weeks so that weekdays and weekends are equally represented.
Related terms
Usability Testing, User Research, Microcopy, Dark Patterns