The Science of Split And Multivariant Testing - Karl Blanks at Conversion Conference 2011

Posted by Mark Slocock on 01 December 2011

Dr Karl Blanks is Chairman of Conversion Rate Experts (CRE) and began by outlining how he and Ben Jesson first started in conversion rate optimisation and how they tripled the sales of Modal, a telecoms company in less than 12 months. They apply this same process to all their client projects.

They find that split testing gives you surprise results and things that you weren't expecting to see and these unexpected results can have a big effect on conversion.

Before and after tests stink! Because external variables will change this is where A/B and Multivarient testing come in to their own because they allow you to implement continuous improvements.

If you pick an early goal (e.g. CTR or Bounce Rate) these are not good as the main goals for split or multivarient testing – you need tangible goals (e.g. Orders) - you can't take Click Through Rates to the bank.

Writing good copy is important and a lot can be learnt form direct marketing, were a lot of research has been done. For example, Martin Conroy wrote an advert for the Wall Street Journal which is the most successful advert ever.

Multi-variant allows you to run lots of split tests at once. Useful if you have several split tests that could be carried out simultaneously.

They tend to run A/B testing to start with – i.e. to test completely different routes, use multi-variant test to test the finer elements of the design.

CRE have published a website that compares all the multi-variant testing tools on the market - look at – find which testing platform is best for you.

The main point here is that tools will only get you so far! If you put garbage in you get garbage out! It's not about the tool, it's about what you do with it.

How long will will your A/B split test take – if you test the small things it will take a long time, big bold tests will get results much more quickly.

Here is how you should approach testing:

  • Know the Rules of the Game – know your business KPIs, USPs and can you afford to outbid your competitors on PPC?
  • Understanding and tuning your existing traffic sources
  • Understand your visitors – particularly your non-converting ones – know why they are not ordering.
  • Advanced Marketing Intelligentence – know your market
  • Spot the hidden wealth in your company
  • Create your experimental strategy
  • Design your webpages
  • Carry out your experiments
  • Transfer your results to other media
KISSmetrics have had made a great infographic of the conversion optimisation process

First know why your visitors aren't converting

Where do you start? Look at Google's visitor flow and unblock the blocked arteries – work on your high traffic pages first. Segment by visits with conversions.

How do you know what's blocked? Look at exits and look at the page – does it make sense? If not, you have a blockage. A lot of what CRE do is taken from process engineering.

Secondly create the missing links

Create M&M trails (in ET Elliot uses a trail of M&M's to get ET into the house), lead your users down the path to conversion don't expect people to do what you want straight away.

Three column approach – create a good sales funnel

Thirdly create an ideas list in Excel

Use three columns:

  • The idea
  • How likely is it to double the business
  • How much effort is needed?
Then create a score based on likelness to double the business and how easy is it to implement and start with the top ideas.

Karl talked about persuasive marketing and, in particular, underhand ways to get people to buy ( and recommended against these method as it doesn't build an honest long term business.

Method Marketing

This was my favourite part of the talk, something CRE do is method marketing, like method acting – when Robert De Niro get the part for Taxi Driver he went to New York and got a job as a taxi driver.

So do the same when working on improving a website's conversion rate.

  1. Order the product
  2. Use Vira Screen flow and video yourself researching and ordering the product
  3. Karl spent 2.5 hours buying an iphone
  4. Go into store and experience the offline process
  5. Use the product (Karl joined weight watchers)
  6. Ben ordered a shed, had to get it through his house it didn't fit and the company realised this was a common problem so re-designed their shed so it was easier to carry through a house

Other Tips

  • Use objection / counter objection
  • SEO Moz example – placing a video of the sales person selling SEO moz pro increased conversion by 53%
  • Sunshine – explaining why there was no phone number helped grow sales by $14million
  • Ask your converted customers what the main motivation for ordering today was. Then integrate the findings into landing pages.
  • Have a look at the million dollar homepage
Next Talk: Where's My Conversion by Craig Sullivan


No one has commented on this page yet.

Post your comment