A/B And Multivariate Testing

Many companies worry so much about how much traffic they can drive to their website.  The real focus should be how can you convert more website visitors? Effective website testing can help increase site performance, usability and conversions.

By using A/B and multivariate testing techniques, you can experiment with different design elements to find a winning combination.

Lets take a look at the fundamentals of both A/B and multivariate testing.

A/B Testing

By definition A/B testing or split testing is testing the effectiveness of one landing page over another. Normally the current landing page will be used as the control, and a second page with some changes to the original will be used as the experiment.

There are a number of different elements that can be tested including colors, fonts, layouts, graphics, icons, headlines, offers, etc.

For this example, I will be using the Google Website Optimizer, a free testing tool that allows you to perform and track A/B and multivariate tests.

1.  Identify the Page to Test

Decide on a page you would like to test.  It’s recommended that you choose a page with high traffic so you can gather data faster and make a conclusion.

Once you have chosen a page, decide on one element that you would like to use as your testing element. Be bold when it comes to your testing element. If you simply change one word in a headline or change a color from black to grey, your chances of seeing noticeable differences are minimal.

2.  Choose the Conversion Page

Decide on a desired goal that you want to track. That can be a contact form submission, a download, a purchase, a sign-up, a time-on-site goal, etc.

If you are tracking a form submission, purchase or a sign-up…you will want to have a unique “thank you” url that you can use as your completion page.  This is the url you will add in the goals.

URL Goal in Google Analytics:  Analytics Settings >> Edit >> Add Goal >>

Name the Goal, Choose “URL Destination”, then insert the unique URL under “Goal URL”

If you are looking to track an engagement on your site like Time On Site or # of Pages Visited:

URL Goal in Google Analytics:  Analytics Settings >> Edit >> Add Goal >>

Name the Goal, Choose “Time on Site” or “Pages/Visit”, then enter the length of engagement.

If you are looking to track engagement goals, every site is different, so there will not be a benchmark to go off of….except your own.  Monitor your current time on site and pages visited, then watch your progress over time to see your metrics have improved or declined.

3.  Set up Tracking Scripts

In order to track your experiment properly you will need to add tracking scripts to your control, test, and goal pages.  Depending on which testing software you use, it may very slightly, however they are very similar when setting up the tracking.  Usually it will be a small snippet of java-script code.  For Website Optimizer you can see the full installation guide here.

There are two types of script that needs to installed on your pages. Although it looks complicated, each set of script serves a purpose:

  • First, there’s the control script. Among other things, the control script makes sure that the experiment variations are switched randomly and that all variations are displayed an equal number of times. For this experiment (and in most cases), place the control script immediately after the <head> tag. You’ll need to install the control script just on your original test page. For this experiment (and in most cases), place the tracking script immediately before the closing </body> tag in each page.
  • The second set of script is the tracking script. It ensures that visits to both the test page and the conversion page are tracked by Website Optimizer for the experiment. For this experiment (and in most cases), place the tracking script immediately before the closing </body> tag in each page. You’ll need to add the tracking script to your original test page, each of the alternate variation pages you’ve created, and your conversion page.

4.  Decide on A/B Distribution, then Start!

Depending on the number of tests you are running, you will have to decide on what percentage of your traffic will be displayed the control page and version A, version B, etc. If you’re testing 2 pages, splitting the traffic up with 50% for each page is the simplest way to do it.

I would recommend starting with only one variable (i.e. your normal page, plus a page with one thing changed), unless you have a large amount of traffic to send to multiple test pages.

5.  Analyze Results

Your testing efforts mean nothing if you don’t analyze the results and implement changes based on them. Google Website Optimizer has great reporting features that allow you to see which variation was more successful.

In this screen shot you can see the different variations that were tested, the estimated conversion rate, chances to beat the original page, and the actual improvements.  The green percentages are improvements, while the red are variations that perform worse than the original.  To the far right you can see the number of conversions and impressions that each variation received.  Again, its important to state that unless you have a high number of conversions, you may want to start with one variation.

Multivariate Testing

A more complex test, the multivariate test allows you to test multiple page variables at one time.  Unlike A/B testing, multivariate testing can essentially test endless variable combinations.

The only limitations are the amount of time it will take to gain sufficient data to come to a reasonable conclusion.  The more components you add to a test, the longer and more data you will need to complete a test.

The process of designing a multivariate experiment is very similar to setting up an A/B testing experiment, however, what to test is slightly more involved.  This graphic is a great representation of how each user is shown different elements on a page.

Website Testing Tools and Resources

Testing Tools:

Further Resources & Readings:

Recommended Books:

It should be said that you can perform these testing techniques on not only web pages, but email marketing campaigns, banner ads, and paid placement campaigns.

It should also be noted that you should always be testing. Just because variation A beat out your original page, doesn’t mean you should stick with that page. Try testing variation A with variation B to see if you can continue to improve.

Website testing is becoming a main component in more and more company marketing strategies. Marketers are beginning to realize that improving the conversion rate for existing traffic can be much more effective than trying to drive more traffic and convert less.

Have you experimented with A/B or Multivariate testing before? I’d love to hear your experiences and what parts of your site you tested!

Enjoy this post? You should follow me on Twitter!