40 Do’s & Dont’s of A/B Testing

When it comes to A/B testing the benefits can be out of this world. There are tons of articles out there telling you how to test and what to test and the thing is, they’re all pretty great. But we don’t want to rehash all that. We wanted to keep it simple. So here you go, 40 Do’s and Don’ts of A/B testing:

Do

  1. Use a testing tool. There are a lot of great tools out there that make it really easy to run an A/B test. More importantly, they make it easy to get and measure real results. Tools to check out include Visual Website OptimizerOptimizely & Unbounce.
  2. Make sure tracking is set up properly. When it comes to testing, you want to be making decisions based on accurate data. Most tools let you know if tracking is set up correctly but do a few tests yourself to make sure.
  3. Create a testing plan. Do you know what you want to test? The requirements needed? A testing plan can make things run a whole lot smoother.
  4. Test both versions simultaneously. You may think it doesn’t matter if the tests run at the same time or not but the fact is, it does. To be sure that all factors are equal, be sure to run your variations at the same time.
  5. Know how long to run a test. One mistake companies often make is calling a test too early. Before you start, know how long the test needs to run. How many visitors do you need to get through the recipes? Does your buying cycle work in hours, days, weeks or months?
  6. Show repeat visitors the same recipe. You don’t want to confuse your visitors so make sure your tests are set up to show repeat visitors the same test they saw on their first visit.
  7. Know your goals for success. Before you start running a test ask yourself what’s the goal of the test is? If it’s an increased conversion rate, what is the improvement needed? If it’s a higher CTR, know how much higher. Set goals make it easier to determine test results.
  8. Keep your tests consistent across the site. When running a test, make sure that messaging is consistent across the site for that test. For example, if you are testing a new price point, all pages with a price point should show the new price to users in that test.
  9. Monitor your test. With a lot of the testing tools, it’s easy to set it and forget it for a week. It’s important that you don’t do that. No one knows your site metrics better than you so you need to be watching to make sure everything is running smoothly.
  10. Prioritize what you test. You probably have a ton of awesome testing ideas ready to go. Do you know which is the most important? Which can be done easily or which will take development work? Lay out your tests and then you can determine the order.
  11. Be open-minded. The great part about A/B testing is that it can often surprise the hell out of you. Something you think couldn’t possibly work, may result in vast improvements. Sometimes it won’t work at all. What’s important is that you keep an open mind.
  12. Know how to understand the results. Once you know what the goals are for the test, make sure you understand the results. Most tools make it easy but if you aren’t sure, ask.
  13. Use it to validate a hypothesis. Have a hunch that an image will be better than a headline? Go test it out! The great thing about A/B testing, is you can validate ideas quickly and easily.
  14. Remember your target market. While it’s a good idea to test, it’s important you don’t forget who your market is. That headline you want to test; what colors and fonts resonate with your audience?
  15. Be ready to answer questions from visitors. When you are running a test, make sure everyone on your team knows about it and is ready to answer questions. Perhaps a customer is already aware of you but comes to your site during a test only to see different messaging and pricing? Your customer service and social teams need to be aware.
  16. Learn from othersAs we mentioned earlier, the great thing about A/B testing is so many people are doing it and seeing killer results. Get out there, do some research and find out what you can do to make your business better.
  17. Establish statistical significanceThis goes along with knowing how long to run a test. Establishing statistical significance will ensure you are getting enough people into your test to validate the results.
  18. Remember you can’t test everything. Have you ever been on a testing high? Your latest test just killed and you are psyched at the possibilities that lie ahead. Remember though, you can’t test everything and you can’t test it all at once.
  19. Test the stuff that matters. Does the copy in your footer really matter? Probably not. Test the things that really matter to users and potential customers.
  20. Be aware of timing. Testing during your busy season probably isn’t the best idea. Be aware of seasonality, events or any other times when testing may not be the best idea.

Don’t

  1. Call it too early. It’s easy to see great results after three days and think that the test can’t possibly fail. The truth is, it can. Results can change in a day so make sure you are sticking to the rules laid out above.
  2. Test too many variables. If you test more than one thing at a time, how will you know which factor was the differentiator? Test one variable at a time.
  3. Ignore the results. You took the time to run a test, now use the results. If they were positive, make sure changes happen immediately. If the results were negative, what can you do with that information?
  4. Only focus on big changes. We all want that big win but the little things can make a difference too. Don’t be afraid to test out small changes.
  5. Run more than one test at a time. Just like testing multiple variables, multiple tests can make results extremely confusing. A testing plan will make sure you only run one test at a time.
  6. Change more than one element at a time. Just like testing multiple variables, you don’t want to change more than one element at a time. For example, if you are testing a new CTA, you don’t want to change the color AND the text.
  7. Forget about your other campaigns. If your PPC campaign touts a $25 price point but your test shows a $35 price point, you are going to confuse your visitors and lose money as well. Make sure your campaigns are in sync. Tools like Visual Website Optimizer will also let you exclude traffic from sources such as PPC.
  8. Assume something will work. Just like keeping an open mind, you can’t assume something will work. Why? Because it may make you look at the results differently or not test it long enough. A/B tests can have some surprising results.
  9. Run your test for too long. Once you’ve hit the statistical significance and have the results, end the test. If the test is positive and you don’t roll it out, you miss out on the benefits of all your visitors seeing the better recipe. Adversely, if the result is negative, you’re basically losing customers.
  10. Let your ego get in the way. Problems can arise when a test you want to work, doesn’t. You may want to give it a bit longer or try it again. Remember, by setting goals, you alleviate these types of issues.
  11. Ignore your gut. While it’s important to trust your tests and your testing tools, if something feels off, you shouldn’t ignore your gut. As we said earlier, no one knows your metrics like you so if traffic is weird or conversions are down dramatically, take a closer look.
  12. Be afraid to ask for help from professionals. As much as we want to be, we can’t all be experts in everything. Thankfully there are people who do this for a living. Don’t hesitate to reach out to them for help.
  13. Focus entirely on conversion rate. While setting goals is important, you need to look beyond them. Let’s say your goal was to increase conversion rate by 1% but it remained almost the same.  However, upon looking more closely, your average order value increased 5%. The test may not have given you the results you set out with but could still be considered successful.
  14. Do it half-assed. You hear about how awesome A/B testing is and you want to do it. So you go out, you set up a test and you don’t do any of the things we’ve outlined above. You probably aren’t going to get much out of it. Do it right.
  15. Ignore SEO implications. If you’ve spent any amount of time on SEO, you know that on page changes can make a difference in results. When you are running your tests and rolling them out, don’t ignore the SEO values.
  16. Test new contentSimilar to testing multiple variables, you don’t want to test something you can’t attribute results to. If you test new content, you won’t know what affected results. Test using an existing page.
  17. Be afraid to fail. The bottom line is that every test will not be successful. You have to be ok with it.
  18. Stop testing after one success. You did it! You tested something and it went awesome! Guess what? You need to keep testing. Take your winning recipe and test a new element on that page. You never know what you might find.
  19. Change code without backing up your current site. While most tools don’t require you to change any code behind the scenes, some do. Make sure you that you have your current site backed up.
  20. Wait to test. The good news is there’s not ‘right’ time to test. Launching a new business? Why not use A/B testing to try out different pitches and messaging? You’ll find the right messaging without spending a ton of money on marketing.

Well what do you think? Did we miss any? Any tool suggestions?

 

 

 
Close

We wrote the book on shopping cart abandonment.

The Cart Abandonment Equation

Join our private newsletter and download The Cart Abandonment Equation ebook instantly. The book contains nothing but actionable advice to increase your ecommerce conversion rates.