A/B testing
A/B testing is a method of comparing two versions of a web page, email, or other digital element to find out which one performs better. Randomly selected groups of users see version A (the control) or version B (the variant), and the winner is chosen based on statistically significant data. It's one of the most reliable ways to make design and marketing decisions based on how users actually behave rather than on guesswork.
A/B testing delivers measurable business results. Its value is backed by research and by the practices of the world's largest tech companies:
- Startups that A/B test grow 30–100% faster. A Harvard Business School study (Koning, Hasan, Chatterji) analyzed 35,262 high-tech startups founded between 2008 and 2013 and found that companies that adopted A/B testing improved their performance by 30–100% within a year. Interestingly, experimentation helps companies not only grow but also fail faster—founders with bad ideas move on to their next project sooner, thanks to the data (HBS – Experimentation and Start-up Performance, 2020).
- A small headline change at Bing brought in $100 million a year. One employee suggested changing how headlines were displayed in Bing search results. The idea sat on the back burner for months as a low priority, but a simple A/B test showed a 12% revenue increase—more than $100 million a year in the US alone. It became the most profitable change in Bing's history (Kohavi, Thomke – HBR, 2017).
- Obama's campaign raised an extra $60 million thanks to A/B tests. In 2008, the campaign team tested variants of its sign-up page—combinations of images and button copy. The winning variant increased the sign-up rate by more than 3 percentage points, resulting in 2.88 million additional email addresses and an estimated $60 million in additional donations. The 2012 campaign ran more than 500 A/B tests over 20 months, increasing donation conversions by 29% and sign-ups by 161% (Optimizely – Obama's $60 Million Experiment).
- The world's biggest companies run more than 10,000 experiments a year. Microsoft, Amazon, Booking.com, Facebook, and Google each run more than 10,000 online controlled experiments a year, many of them involving millions of users. Booking.com runs more than 1,000 concurrent experiments across 75 countries and 43 languages (Kohavi, Thomke – HBR, 2017; Kaufman et al. – KDD, 2017).
FAQ
How long should an A/B test run?
At least 2 weeks, ideally 3–4 weeks. A test should always run for full weeks (7, 14, 21, or 28 days), because conversion rates vary significantly by day of the week. CXL Institute recommends a minimum of 250–400 conversions per variant, with the standard 80% statistical power and a 95% confidence level. Stopping a test before reaching the predetermined sample size leads to unreliable results (CXL – A/B Testing Statistics).
What should you test first?
Start with the elements that have the biggest impact on conversions: the page's main headline, the CTA button (copy and color), and forms. A VWO analysis showed, for example, that removing a single field from a checkout form increased conversions by 5.8%, and a better display of color options on a product page led to a 17.63% lift. The key is to test only one change at a time so it's clear what drove the result (VWO – A/B Testing Examples).
Can a small website with low traffic do A/B testing?
With low traffic, reaching statistical significance takes longer, but testing is still possible. It's important to calculate the required sample size up front with a calculator and focus on testing bigger changes (a higher minimum detectable effect), where the difference between variants is easier to measure. For sites with fewer than 1,000 conversions a month, CXL recommends testing bolder redesigns rather than small tweaks (CXL – Statistical Power).
Sources:
- HBS – Experimentation and Start-up Performance: Evidence from A/B Testing (2020) – a longitudinal study of 35,262 startups (2008–2013) using instrumental variables and synthetic controls
- Kohavi, Thomke – The Surprising Power of Online Experiments, HBR (2017) – an overview article by authors from Microsoft and HBS, with data from Bing and other platforms running 10,000+ experiments a year
- Optimizely – How Obama Raised $60 Million (2010) – a case study of the 2008 presidential campaign's A/B tests of sign-up pages
- Kaufman et al. – Democratizing Online Controlled Experiments at Booking.com, KDD (2017) – an academic paper: 1,000+ concurrent experiments, 75 countries, 43 languages
- CXL Institute – A/B Testing Statistics – a methodology guide to statistical significance, sample size, and test duration
- VWO – A/B Testing Examples & Case Studies (2026) – case studies with specific conversion lift metrics
%201.webp)