What is an A/B test?
An A/B Test is a controlled experiment that has two variants called A and B.
The goal of an A/B test is to improve a specified metric. The idea is that A and B are identical in every way except one. For example A is a web page where the button is green and B is exactly the same web page but where the button is blue. Thus, by subjecting some users to variant A and some other users to variant B, we can measure and see which variant performs better. As our A and B variant are identical apart from one difference, our desired goal is to determine if this difference has improved the metric we are measuring.
You can apply an A/B approach on your landing pages, in your product or website or emails by varying elements like:
Buttons
Page layouts
Images
Colors
Pricing
Below, we can see two landing page variants (A & B). They are identical apart from one thing (the buttons on A and B are a different color). We are measuring how many users actually click this button and if an orange button substantially more clicks than the blue button variant, we will conclude that users prefer and click more on an orange button.
We have isolated button color because everything else is identical between variant A and variant B. Therefore, the only factor that can "explain" substantially more clicks is the button color. This is the nature of A/B testing.
Split testing (Bucket testing)
An A/B test is a type of split test (also called a bucket test).
The idea behind split testing is that you divide your audience: any given user gets either experience A or experience B. You can think of this as breaking your audience up into 'buckets'. For example, 61% of your traffic (called Bucket A) experience the 'A' variant (say an orange button on a landing page) and 40% of your traffic (called Bucket B) experience the 'B' variant (say a blue button on the landing page).
So in this example, we split our audience into two buckets and each bucket of users will see the same product / have the same experience as each other (but a different experience to users in other buckets!).
Statistical Significance
Let's imagine that you run an A/B test with 200 users where 100 users are in bucket A and 100 are in bucket B. The test is for a landing page with different color buttons on each variant. The A variant has an orange button and a click rate of 19% and the B variant has a blue button and a click rate of 21%.
The B variant performs better (21% vs 19%) in this test (which is a sample of 200 users).
However, we cannot be sure that B variant is scientifically better because this is a small sample size. It is possible that if we ran a bigger test with a larger sample size (say ten thousand users) that we might see different results.
A good way to think about our test above is that the B variant is probably better but we can't be sure.
The good news is that you don't need to understand all the mathematics behind statistical significance. There are many tools that will do the calculations for you such as this significance test calculator from Kissmetrics.
If we enter the data from our A/B test example above, we will see that it is 64% certain that variation A is better than variation B. Typically, we have to achieve at least 95% certainty to claim statistical significance!
However, if we had a a much bigger sample size of say ten thousand users where five thousand were in bucket A and experienced variant A and five thousand were in bucket B and experienced bucket B - and we had the same click rates (21% and 19% respectively), then we would be 99% sure that A is better (and this is statistically significant).
A/B Tests and Landing Pages
A landing page is a single page that:
Describes your product or service
Illustrates some advantages of using your product or service (your "unique value proposition")
Contains a button that lets interested visitors click to read more, join a mailing list, buy now, or some other action.
Applications of Landing Pages
Online marketing - typically when you click on an online ad you are brought to a page which extends the message of that ad in a longer form. You can click to visit a site or purchase a product. This type of application is considered to be a "lead capture" where some visitors read the page and go further into the product or website and some visitors do not. The percentage of users that click the call-to-action button in the landing page represent the 'conversion rate' for the advertiser.
Testing a value proposition - before you even build your product or site, you could decide to experiment with different landing page messages to see which benefits (as described in the page text) cause the highest conversion. This is also known as a Landing Page MVP
Landing Page tools
Professional-looking landing pages can be made in half a day. Using a platform like Unbounce or Instapage provides templates for quick creation of professional-looking landing pages. Some benefits of using these tools are that are that:
You can choose a pre-existing template to create your landing page. It will look professional and you can easily edit the text and images in the tool. You don't need to be a developer or a designer.
Once a landing page is created, you can create a variant in one click and then change the one thing you wish to test (e.g. change the color of a button or the text in the heading).
You can choose how many variants you want and how much visitor traffic should be sent to each (i.e. you can set the size of each bucket's percentage of traffic).
At a glance, you can see the data - the conversion rate of each variant is displayed and you can compare variants performance.
Any email addresses collected are saved and can be downloaded as a file.
Each landing page has a unique URL so you can post this URL on facebook (for example) to get some visitors to your landing page.
The great thing about using a tool like this is that you can have your first A/B test up and running and capturing data in a couple of hours!
Additional Resources
If you want to send Google Ads to your Unbounce landing page, you will need a custom domain. You can read more about that here
Optimizely explain statistical significance (warning: lots of math and the math is not strictly necessary to run A/B test)
Kissmetrics significance calculator
Unbounce - note you need a credit card to get a 30 day free trial
Instapage - you can try for 14 days (without requiring a credit card)
