How To Increase Site Performance Through A/B Split Testing

Ever had the desire to change something on your website, but you were afraid the change might have a negative impact on performance? Welcome to A/B Split testing, the practice of testing multiple variations of the same site to see which works better.

Do your viewers react better to Apples or Oranges?

Ever had the desire to change something on your website, but you were afraid the change might have a negative impact on performance? Welcome to A/B Split testing, the practice of testing multiple variations of the same site to see which works better.

What is split testing?

A/B split testing is a technique to find which changes really improve your website and which changes don’t.

Let me give you an easy example: You want to try a much bigger “add-to-cart” button on a product page.

You create 2 versions of the page—you call the old version “A” and the new version “B”.

You then use special software to randomly show site visitors the 2 different versions, and you measure which version works best (ie: which results in more conversions or higher performance). You are testing the new version with real people to see if it works in the real world. You then stop the test and go ahead with just the winning version.

Why not test A for a while then B?

Just look at any graph of your conversion rates over time—it’s all over the place. Some months can be 20% better than the previous month—then it gets worse again—then better, etc…

Conversion rates are affected by season of the year, sources of traffic, news events, the state of the economy, competitor activity… You’ll see big differences and big swings even with no changes at all to your own site.

So if you tried A in a good month then tried B in a bad month you could make an incorrect decision. And you don’t want to freeze all other changes while you try the A and B buttons.

Split testing aims to eliminate these other differences and see if button B really is better than button A. You’re testing the two versions at the same time in the same season with similar visitors.

How long do I run the test for?

This depends on how much traffic your site is getting, and also on how big the improvement is.

It’s very important not to make any premature decisions. On a low traffic site a rule of thumb is to wait until you have over 30 conversions from A plus B. On a high traffic site you should wait at least 24 hours and try to include some weekend time as well.

You’ve really got to keep an open mind about the results—so often I see “obvious” or “common sense” changes that just make no difference or actually make conversion rates worse.

If you still have no measurable effects after 6 weeks then just stop the tests and go with what you like best.

Is it just buttons?

You can test any element on your site: different photos, different headlines, different copy. Even simply moving different elements around can have drastic effects on performance! Test a contact form on the right of the page against the same form on the left and you might get double the number of messages sent from it.

You can also test different checkout sequences: maybe a 3 page sequence versus a 2-step checkout process.

Or a different offer: Or describing one offer in 2 different ways.

One client was discounting $33 off a $100 dollar product. She found “save $33″ was way way better than “save 33%”. Your mileage may vary, of course. You need to do your own test with your own visitors.

And it’s not just 2 variants A and B—you can create as many as you want and do an A/B/C/D split test.

You can even test different prices: Visitor A does not know what price visitor B is seeing. Amazon got into trouble in September 2000 for doing this—but a lower profile e-tailer could probably pull it off and find their own sweet spot for maximum profit (not to encourage bad practice. Always check to make sure you’re not breaking any laws obviously)

I have even split tested the idea of hiding the product price.

Does it have to be 50/50?

You can vary the proportions. It could be 90 / 10 or any other ratio. Splitting the traffic 50/50 is the quickest way to the end point.

Some changes may be too high-risk to try out on half your visitors. I’m thinking here about wacky stuff like having music or a voice-over when the visitor arrives on your site. Or something that doesn’t scale very well, like a personal chat between the site visitor and a doctor. Try on a smaller sample first.

And don’t forget that you can test different elements of your site at the same time—you don’t need to finish one test before you start the next. This is one reason why the splitting is done randomly and not just showing alternate visitors the different versions.

Who is doing split testing?

The big websites all do A/B split testing: Amazon, eBay, Lands-End, Boden in the UK. If you look at a Google results page, the pale yellow background on the sponsored (PPC) results was chosen by a split test of dozens of colours. Pale yellow got more clicks.

Can a small company do split testing?

For a small business you need to balance the effort and cost of setting up a test against the rewards from improved conversions. Most small business sites can get huge improvements just by applying best-practice across the board.

After doing that you can start testing. Start with things that could have maximum impact and lots of traffic – for the minimum effort. One of the easiest ways for a small business to test variations is with Google AdWords. You set up 3 variants of an ad. Rotate all 3 for some time then suspend the worst – and then create a new contender. I’ve blogged about improving AdWords this way. You will also learn what words work for your visitors and you can then apply this knowledge on your site itself.

How do I get started?

You need to choose – or make – a tool. The tool will randomly split new traffic, and remember who has seen which version so that people get a consistent view as they move around your site. And it needs to track the visitors through to a goal – like your checkout page or a thanks page that she sees after a form submission.

One tool that might do the job for you is the Google Website Optimizer.

What should I test first?

Choose an element of your site that’s high in impact and also controversial.

A good example is product viewing widgets on an e-commerce site. These widgets are nice for some people—the site visitor can spin the product round, zoom in and out, really experience your stuff.

But they can also be slow to download and difficult to use for other visitors. Rather than debating the pros and cons or listening to the widget’s vendor, why not test the widget in action with your own real visitors.

Set it up on your 10 highest-traffic products. Half of your visitors will see the widget and the other half will see just a plain old photo of the same product.

Let the people decide!

Have you tried split testing in the past? What tools do you use to get the job done, and what is factors play into your decision making process for implementing changes? Share your ideas below!

Useful Split Testing Links

About the Author

John Hyde

John Hyde of Site Doublers has been split testing for 4 years. John improves website conversion rates by applying best practice, running usability tests, and doing split tests.

http://www.sitedoublers.com/

About the Author

John Hyde

John Hyde works closely with clients to get better results. He uses analytics, A/B split testing, and usability to improve - and improve again. Check his own about Site Doublers page.

Related Articles

23 Comments

  • Daniel Sevitt Reply

    This is a great introduction to A/B testing, thanks. We are big believers and practitioners of this methodology with regards to implementing video on websites. We have sen remarkable results and been able to fine tune conversion rates in ways that would not be possible without side-by-side, simultaneous testing. Great stuff.

  • David Reply

    I never actually understood how this was possible but now i do. I had seen Google Website Optmizer but thought it would be alot of extra work and time for something that was just a guessing game.

    This post puts the process in an easy to understand and gives me easy ideas on how to do it.

    Nice one!

  • Nicholas Z. Cardot Reply

    Great information on this type of testing. I’ve never actually done anything like that before but you definitely showed me the importance of it.

  • Sarah Reply

    Good introduction to split testing John. I have used Google web optimizer to conduct A/B split testing on clients websites with great success – up to 40% increase in conversions just based on a different home page layout – same content. I highly recommend trying it. A lot can be learnt from testing on your real customers.

  • Matt Dempsey Reply

    Really great introductory post on the subject. I haven’t seen a lot of info on it on the blogs that I follow, the only other post that springs to mind is one from 37Signals which I really enjoyed: http://www.37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page .

    Hopefully I’ll have a reason to have a play with the Google Website Optimizer some time in the near future. A/B testing is something that looks like it could be a lot of fun and really rewarding, there must be a ton of usability lessons to learn from it too.

    Thanks for the post guys!

  • Jake Rocheleau Reply

    I never really understood what this phrase meant until after reading this, a great article thanks for the post

  • Paras Chopra Reply

    Though simple A/B tests are better than not doing any tests at all, the true power of testing is realized when you are doing targeted tests. In most of the solutions currently in the market, you can only do blanket tests: that is, all visitors see the same test. Much better approach would be to do tests for a segment of visitors because it is quite likely that your organic traffic would have different optima than your direct traffic. Similarly, repeat visitors would respond differently than first time visitors.

    Also, websites usually have multiple goals. For example, while your A/B test may tell you that you have reached optima as far as clicks on the banner is concerned, you won’t know if you have compromised on other goals such as visitor engagement, newsletter signups, etc. The point here is that you must measure performance on multiple goals to see the correct tradeoff.
    My team has been developing a powerful testing solution called Wingify (http://www.wingify.com/). We have limited number of private beta accounts available. If anyone is interested, mail me at paras@wingify.com

    Oh, and BTW, there is a screencast of using Wingify for setting up an experiment as well. You might find it helpful for an introduction to A/B testing – http://www.wingify.com/video-demos/create_experiment/

  • Mike Darnell Reply

    Great article.
    I’ve seen loads of stuff about A/B testing over the years but this post tops them all. Why?
    It manages to say everything necessary but avoids being long winded, overly technical or boring…

    Kudos!
    Mike
    @pop_art

  • Rob Reply

    Good tips. Also, make sure to write your changes down. I like to do what I call “Journal Analytics” for any changes I make to a site logging everything.

  • Badal Reply

    This is a very helpful and interesting article. Thanks!

  • Sanket Nadhani Reply

    Very helpful article. Didn’t know that the different background colors that I saw for the Adwords Ads were a part of A/B Testing. Did not know about Amazon A/B testing either.

    Had been thinking about A/B testing for a while and now with examples like this, it gives me a greater push.

  • Justin Hunter Reply

    Good piece John.

    These kinds of tests (and more complex multi-variate tests) are easy to run and often deliver very powerful results.

    I’ve written some of my own thoughts relating to this same theme at: http://hexawise.wordpress.com/2009/08/25/what-else-can-software-development-and-testing-learn-from-manufacturing-dont-forget-design-of-experiments/

    and also

    http://hexawise.wordpress.com/2009/08/18/learning-using-controlled-experiments-for-software-solutions/

    - Justin Hunter
    Founder of Hexawise

  • Crazy Surfer Reply

    Interesting, also read Brian cray – how I used a/b testing ( a large collection of a/b resources)
    http://briancray.com/2009/ultimate-ab-testing-resources/
    Interesting that you are saying ‘google found pale yellow got the more clicks’, may be they tested this with a limited number of testing people, but they had always been yellow since the time I began to see them.

    but twe cant copy from it blindly because our site may be different background color and the color scheme may be different.

  • Mike Reply

    Great post – I was happy to tweet this : )

    We’ve taken split testing a step further by automatically generating slightly different videos promoting the same product and continuously optimizing the video to be displayed based on how well each version preforms in terms of sales generated. The results speak for themselves:

    “Since launching online videos earlier this year with Treepodia we’ve experienced a 30% increase in product page conversion rates”
    Roy Hessel – CEO – EyeBuyDirect.com

    Original article here: http://bit.ly/18um4X

    Cheers,
    Mike
    http://Treepodia.com
    @treepodia

  • Sal Reply

    Very interesting piece about split testing, easy to understand and it’s given me some ideas to really get started. I have been looking into split testing lately myself and have reviewed several software packages, including Logaholic Web Analytics [logaholic.com]. It’s easy to use and seems to do everything I need.

  • Carly Reply

    This is a fab resource, we do a lot of split testing at work, and we’ve found HUGE differences with our A/B variants in some places.

    It’s fun to watch small changes have drastic effects!

  • Carmen Reply

    I love this post. I’m going to bookmark this A/B Split testing guide for future references.

  • Типография Reply

    Good tips. Also, make sure to write your changes down. I like to do what I call “Journal Analytics” for any changes I make to a site logging everything.

Leave a Comment on This Article