Fundamentals of A/B and Multivariate Testing
45Many companies worry so much about how much traffic they can drive to their website. The real focus should be how can you convert more website visitors? Effective website testing can help increase site performance, usability and conversions.
By using A/B and multivariate testing techniques, you can experiment with different design elements to find a winning combination.
Lets take a look at the fundamentals of both A/B and multivariate testing.
A/B Testing
By definition A/B testing or split testing is testing the effectiveness of one landing page over another. Normally the current landing page will be used as the control, and a second page with some changes to the original will be used as the experiment.
There are a number of different elements that can be tested including colors, fonts, layouts, graphics, icons, headlines, offers, etc.
For this example, I will be using the Google Website Optimizer, a free testing tool that allows you to perform and track A/B and multivariate tests.
1. Identify the Page to Test
Decide on a page you would like to test. It’s recommended that you choose a page with high traffic so you can gather data faster and make a conclusion.
Once you have chosen a page, decide on one element that you would like to use as your testing element. Be bold when it comes to your testing element. If you simply change one word in a headline or change a color from black to grey, your chances of seeing noticeable differences are minimal.
2. Choose the Conversion Page
Decide on a desired goal that you want to track. That can be a contact form submission, a download, a purchase, a sign-up, a time-on-site goal, etc.
If you are tracking a form submission, purchase or a sign-up…you will want to have a unique “thank you” url that you can use as your completion page. This is the url you will add in the goals.
URL Goal in Google Analytics: Analytics Settings >> Edit >> Add Goal >>
Name the Goal, Choose “URL Destination”, then insert the unique URL under “Goal URL”
If you are looking to track an engagement on your site like Time On Site or # of Pages Visited:
URL Goal in Google Analytics: Analytics Settings >> Edit >> Add Goal >>
Name the Goal, Choose “Time on Site” or “Pages/Visit”, then enter the length of engagement.
If you are looking to track engagement goals, every site is different, so there will not be a benchmark to go off of….except your own. Monitor your current time on site and pages visited, then watch your progress over time to see your metrics have improved or declined.
3. Set up Tracking Scripts
In order to track your experiment properly you will need to add tracking scripts to your control, test, and goal pages. Depending on which testing software you use, it may very slightly, however they are very similar when setting up the tracking. Usually it will be a small snippet of java-script code. For Website Optimizer you can see the full installation guide here.
There are two types of script that needs to installed on your pages. Although it looks complicated, each set of script serves a purpose:
- First, there’s the control script. Among other things, the control script makes sure that the experiment variations are switched randomly and that all variations are displayed an equal number of times. For this experiment (and in most cases), place the control script immediately after the <head> tag. You’ll need to install the control script just on your original test page. For this experiment (and in most cases), place the tracking script immediately before the closing </body> tag in each page.
- The second set of script is the tracking script. It ensures that visits to both the test page and the conversion page are tracked by Website Optimizer for the experiment. For this experiment (and in most cases), place the tracking script immediately before the closing </body> tag in each page. You’ll need to add the tracking script to your original test page, each of the alternate variation pages you’ve created, and your conversion page.
4. Decide on A/B Distribution, then Start!
Depending on the number of tests you are running, you will have to decide on what percentage of your traffic will be displayed the control page and version A, version B, etc. If you’re testing 2 pages, splitting the traffic up with 50% for each page is the simplest way to do it.
I would recommend starting with only one variable (i.e. your normal page, plus a page with one thing changed), unless you have a large amount of traffic to send to multiple test pages.
5. Analyze Results
Your testing efforts mean nothing if you don’t analyze the results and implement changes based on them. Google Website Optimizer has great reporting features that allow you to see which variation was more successful.
In this screen shot you can see the different variations that were tested, the estimated conversion rate, chances to beat the original page, and the actual improvements. The green percentages are improvements, while the red are variations that perform worse than the original. To the far right you can see the number of conversions and impressions that each variation received. Again, its important to state that unless you have a high number of conversions, you may want to start with one variation.
Multivariate Testing
A more complex test, the multivariate test allows you to test multiple page variables at one time. Unlike A/B testing, multivariate testing can essentially test endless variable combinations.
The only limitations are the amount of time it will take to gain sufficient data to come to a reasonable conclusion. The more components you add to a test, the longer and more data you will need to complete a test.
The process of designing a multivariate experiment is very similar to setting up an A/B testing experiment, however, what to test is slightly more involved. This graphic is a great representation of how each user is shown different elements on a page.
Website Testing Tools and Resources
Testing Tools:
Further Resources & Readings:
- Website Optimizer User Guide (Video Tutorial)
- Advanced Website Optimizer Tricks
- How to Increase Site Performance Through A/B Split Testing
- How to Improve A/B Testing
Recommended Books:
- Always Be Testing: The Complete Guide to Google Website Optimizer
- Advance Web Metrics with Google Analytics
- Landing Page Optimization: The Definite Guide to Testing and Tuning for Conversions
- Web Design for ROI: Turning Browsers into Buyers & Prospects into Leads
It should be said that you can perform these testing techniques on not only web pages, but email marketing campaigns, banner ads, and paid placement campaigns.
It should also be noted that you should always be testing. Just because variation A beat out your original page, doesn’t mean you should stick with that page. Try testing variation A with variation B to see if you can continue to improve.
Website testing is becoming a main component in more and more company marketing strategies. Marketers are beginning to realize that improving the conversion rate for existing traffic can be much more effective than trying to drive more traffic and convert less.
Have you experimented with A/B or Multivariate testing before? I’d love to hear your experiences and what parts of your site you tested!
Enjoy this post? You should follow me on Twitter!
Hey Mark,
You may also want to check out (and include in your list) Visual Website Optimizer http://visualwebsiteoptimizer.com ; aim is to create world’s easiest to use A/B split and multivariate testing tool.
Our blog has useful articles and case studies, http://visualwebsiteoptimizer.com/split-testing-blog/
The tool is in beta so if you’d like to try it out, use the invite code “wingify-private” (without quotes) while signing up.
Regards
Paras Chopra
Looks like a nice tool that yous are working on! Will drop in again and see how it is once yous leave the beta :)
Hey Michael, thanks for checking out the tool. It will be out of beta soon – though already lots of people are using it in real world setting and improving their conversion rates. Have a look at some of the case studies – http://visualwebsiteoptimizer.com/split-testing-blog/category/case-studies/
We also had an article on Carsonified http://carsonified.com/blog/business/the-business-case-for-ab-testing/
Hope you get some time to check out the tool in more detail sometime in near future.
Great Article! I’ll have to try A/B testing sometime in the future
You can also test processes with split testing and multivariate testing. I recently wrote a blog post on testing sending the visitor to a squeeze page or to a sales page… you can split test that by pitting a page with redirect code to the squeeze against a page with redirect code to the sales page.
That’s a great idea for testing usage. Instead of optimizing just one page, you can optimize the wholes sales funnel. Nice tip! :)
Interesting article! A/B testing is something I would really like to try sometime, it looks like it can get some good results.
It really does. I think the names just make it sound like more work than it is, and even small improvements can have a big impact over time! :)
Hi Michael,
You seem to have stolen some graphics from our website:
http://www.widerfunnel.com/our-process/ab-split-testing
This is copyright protected content owned by WiderFunnel Marketing, Inc. Please remove the images immediately.
Chris Goward
CEO
WiderFunnel Marketing, Inc.
Hi Chris,
I’m really sorry about that, I had no idea. I check all author writing we receive, but images are much harder to ensure haven’t been copied!
I’ll take the picture down right away.
LOL. You should hire some legal team for these purposes. Its not a task of CEO.:P
Thats a great write up, nice work. I’ve been meaning to dive into GWO myself for some projects.
Great write up.
A/B testing is so crucial nowadays. We have used it extensively on page.ly to improve results. I say results and not funnel conversions because it can be used to test any scenario.
As an example we tested background colors of an adwords landing page and measured the CTR.. not the ultimate conversion, but just the single click to the next page (start of conversion funnel).
When you first start you may not have the traffic volume to effectively measure funnel conversions… but we used to A/B just to improve bounce rates for a while as the conversions were infrequent.
They say if you measure it, you can improve it.
Thanks for the great examples Strebel. You make a great point, it can be used to improve any aspect of a website, not just for making sales.
Feel free to share the link, would be interesting to see your results! :)
Do you think that my site – TurboWhiteTrash could use some testing?
Experiments are being successful because of testing and testing. You can get better results when you have tested a thing and find out what is effective. This is a great information to put in mind.
Love the way to getting better results with the Google optimizer and methods of Multivariate Testing. Experiments and testing brings the information needed to consistently keep getting better results.
GWO is awesome. Your graphics really show the way conversions can be tracked. A/B testing is a virtual way of analyzing the marketing network.
Appreciate this great supportive article for testing and analytical work.
Hi Michael/Mark,
Nice breakdown of the differences between A/B and MVT. Optimization really seems to be taking off in 2010 which is great to see.
Another tool you might want to consider adding to your list is Unbounce.com, which is a self-serve landing page platform which includes a simple yet powerful A/B testing framework.
Something for your readers to consider is that if you are sending inbound traffic (PPC, Email, Banner, Social Media) to your homepage that you can achieve significant conversion improvements by A/B testing a standalone landing page as it help maintain the focus of your upstream ad.
This is really nice. If you doubt about your page rank, you can use this to determine if you really deserve less or more of the rank that you have.
This multi variant testing is quite difficult to understand. Good thing this post was able to make very reliable information about this.
uch traffic they can drive to their website. The real focus should be how can you convert more website visitors? Effective website testing can help increase site performa
In the past, I have found that RAM is the single most important thing that limits your computer- so I would always go for the larger RAM no matter what- it might be just 1GB, but that might really mean alot in the programs you can run and the internet- it actually is expressed as much more than just 1GB when the computer is using it.
Hotels in Bucharest
Hi
I wrote an article on this a while back: http://www.matizmo.co.uk/landing-pages-using-multivariate-testing-to-improve-your-conversion-rate – it is amazing how many A/B testing companies are springing up offering this kind of service. I particularly like the idea of tailoring content for the visitor – whether they have returned or first time, and using multivariate testing to maybe tell them about info they had not seen the last time they came. Systems are getting easier to use and more sophisticated at the same time.. it is an interesting space to watch.
Cheers
Jake
I had the same issue. I thought it was just too much work until I gave it a shot. I’ve had great results so far.
When you first start you may not have the traffic volume to effectively measure funnel conversions, But You can get better results when you have tested a thing and find out what is effective.
Interesting article! A/B testing is something I tenant Screening would really like to try sometime, it looks like it can get some good results.
I think the names just make it sound like more tenant Screening work than it is, and even small improvements can have a big impact over time!
Experiments are being successful because of testing and testing. You can get better results when you have tested a thing and find tenant Screening out what is effective. This is a great information to put in mind
Through trial and error I have managed to come up with a
ovarian cysts treatment blog that has a good conversion.
The tips mentioned here may have saved me a lot of time and effort olo !!
This is a good idea to see which web page format gives the bettter conversion.
Like many online, I have overlooked the visits that convert to a sale.
I have concentrated on SEO to increase traffic but have not considered conversions.
mike walden acne no more
This is a very scientific method.
i am very impressed with your experimentation, and
will use it to improve my own conversion rate.
maximising conversion is often overlooked as it can be time consuming
Some really interesting results here. Have you tried the Taguchi method?
Through trial and error, i have come up with a
blog page that has what I think is a good conversion rate.
using your method would probably have saved a lot of time.
Will try your method thanks before committing changes
to my fairly successful blog layout
elena peterson
Note that this kind of testing can not tell how the different variants perform as far as the search result position is concerned. So unless one tests here exclusively with regard to AdWords conversions, any changes of the page design on the basis of such testing could actually backfire.
Thomas
The A-B test is always a good method to find the most effective method to help you in some moments. I think this can be done with good effects in this way.
I love what you guys are up too. Such clever work and exposure! Keep up the very good works guys I’ve incorporated you guys to my own blogroll.
I am only commenting to let you know of the remarkable experience our girl encountered reading the blog. She noticed many pieces, which included how it is like to possess an amazing giving nature to get certain people really easily learn certain multifaceted things. You undoubtedly exceeded visitors’ desires. I appreciate you for rendering the important, healthy, informative as well as easy tips about the topic to Kate.
download hollywood movies free
This is a good summary of how to pick a blog design for better conversion.
You are correct.
It is best to increase traffic and also look at the proportion of visitors that convert ie buy the product.
I have always trial and error to do this but i like this scientific approach.
Great of you to share the idea !
Many webmasters throw up websites thinking that once they have content on their sites their work is done. Unfortunately, it isn’t. To make the most of the site you have you need to continuously test and see what pages, headlines, formats, fonts, colors, sales copy etc are converting the best for you.
Good article!
Another tool you might want to consider adding to your list is Unbounce.com, which is a self-serve landing page platform which includes a simple yet powerful A/B testing framework.
Oferim cazare litoral mamaia constanta la particular in vile,pensiuni si apartamente in regim hotelier
I would like to consider the ability of thanking you for the professional instruction I have always enjoyed going to your site. I’m looking forward to the commencement of my school research and the whole planning would never have been complete without surfing your site. If I might be of any assistance to others, I would be ready to help via what I have learned from here.
Great post I will reach out to you Mark to see if you are doing more of there somewhere. Love to invite you to http//www.convert.com A/B testing software for ecommerce
Dennis