What is

A/B testing?

What is A/B testing?

A/B testing is a method where you compare two versions of a web page, ad or email by showing them to two randomly assigned groups at the same time. Version A is the original, and version B has one change. By measuring which version produces the most conversions, you get a data-driven answer about what works best.

The method is also called split testing. When several elements are tested at once in different combinations, it is called multivariate testing, but that requires far more traffic.

How to run an A/B test

• Start with a hypothesis: for example that a shorter quote form will generate more enquiries because many people drop off at the phone number field.

• Choose one primary goal: usually a conversion, not clicks or time on page.

• Calculate the sample size you need: work out how many visits you need to detect a realistic improvement.

• Run the test long enough: ideally at least one to two full weeks, so you capture differences between weekdays.

• Analyse and document: look at statistical significance and share what you learned, even when B loses.

An A/B testing example

An online store tests new text on the buy button. Version A, “Add to cart”, gets 10,000 visits and 300 purchases, or 3.0 percent. Version B, “Buy now, delivered in 2 days”, gets 10,000 visits and 345 purchases, or 3.45 percent. With an average order of NOK 700, the difference equals NOK 31,500 in extra revenue for the test period, provided the result is statistically reliable.

What should you A/B test?

• Headlines and value propositions on landing pages.

• Offers, prices and how shipping costs are presented.

• Form length and field order.

• Placement of reviews and other trust signals.

• Newsletter subject lines and ad copy.

Start with the elements that influence the decision most, not the details that are easiest to change.

Common mistakes

• Stopping the test as soon as B seems to be winning.

• Testing too many things at once, so you don't know what made the difference.

• Testing on pages with too little traffic to give a reliable answer.

• Testing small cosmetic details instead of messages, offers and structure.

A/B testing is a central part of CRO, but not always the right tool. If you have little traffic, user testing and qualitative insight often give you more. Read more about how we work with conversion.

Related terms

• CRO: A/B testing is one of the main methods in conversion rate optimisation.

• Conversion rate: the most common metric for picking a winner in the test.

• Landing page: one of the most profitable page types to A/B test.

• CTA: button text and placement are classic elements to test.

See all the terms in the glossary.