A-B testingIn the marketing world there is a myth that A/B testing answers all questions about your website. Unfortunately, this is not the case A/B testing only works when a page is adjusted based on insights that are gathered through market research. When A/B testing is used in that way it is the way to test new ideas and insights.

In a recent webinar by Larry Kim (CTO Wordstream) and Tim Ash (CEO Siteturners) they explain when to use A/B testing, but mainly when not to use it.

From the webinar it became clear that a lot a people don’t exactly know what A/B testing actually is. Often people think that when you change the spacing, colors or the images you have a completely new page. To test this “new page” is not an A/B test because the page is not an actual new page, it is the same page with a new color.

When is A/B testing useful? To test insights gathered by using Google Analytics, market research or to test behavioral targeting. To test these ideas, it is often necessary to completely redesign the page. Once you redesigned the page then A/B testing becomes extremely useful. So its essential to first know why your page does not work, and then change the page.

When is A/B testing useless?

  • When the problem is obvious, simply fix it you don’t need an A/B test.
  • When it is about trivial parts of a page. For instance, Google once did a test where they measured which color blue worked better. These test are a waste of time and money as they will never increase conversion.
  • When the problem does not only occur on the landing page but on the website as a whole. In this case it is necessary to change the whole website. There are examples where a company fully optimized their landing page but not the rest of the website. Result: lower conversion rate because the website could not fulfil the expectations set by the landing page.
  • When you don’t have enough traffic, in this case the results wont be significant.
  • When traffic is season bound. For instance, visitors of a flower shop during mother’s day. The visitors a week before mother’s day will mainly be interested in price and the variety of products while visitors a day before mother’s day will mainly be interested in delivery speed.
  • When you don’t understand statistical information. An A/B test is a type of hypothesis testing, therefore it is crucial that you understand the results.

In the end A/B testing is the perfect way to test ideas based on insight about your visitors. However, to do this you often need to redesign the page.

This and more was presented in the webinar: The A/B testing Delusion and how to REALLY get more conversions by Larry Kim and Tim Ash.

Watch the presentation