An MVT buyer’s guide – part 2

Reading Time: 3 minutes

hippo Welcome to the second installment of my MVT “buyer’s guide”. In the first installment, I provided a short introduction to MVT and talked about one of the biggest challenges – that of getting statistically significant results in a reasonable amount of time. Now let’s move onto two more key things to look for: analytical power, and segmented optimization.


2. Analytical power and segmented optimization

Once you have your experimental results, the answers should just pop out, right? Well, if you’re working in an ideal environment, where you can deliver the same optimized page (or ad, or e-mail) to your entire audience, and you have no external constraints and a simple, universally agreed-upon goal, then maybe yes, you can just pick the winner and bunk off work early. But as you surely know, the real world doesn’t work like that.

Heading for a goal

Say you’re looking to optimize a key landing page that you’re driving traffic to via (say) search marketing. The goal that you could optimize around could be one of the following:

  • Time spent on the page
  • Clicks through from the page
  • Clicks through to specified other pages (e.g. “add to cart”)
  • Conversions (purchases) during the session, or subsequently
  • Progress through a defined path (e.g. a checkout process)

So at the very least, you need an MVT tool which allows you to choose whichever one of the above success criteria suits you the best. An even better tool would allow you to pick more than one, and to score or weight them against each other, so that you can achieve a blended optimization across multiple goals. This is essential for organizations that measure the success of their website in more than one way – for example, one part of the organization may be focused on acquisition (and hence be more interested in clicks and conversions) and another more focused on engagement (where perhaps time on site might be important).

Tweaking things

The second thing to look for is the ability to tweak the model once it’s run. For example, the MVT engine might report that the purple banner, combined with the lime green “apply now” button, delivers the best results. But the creative director at your agency decrees that this color combination is more than he can bear, and threatens to throw himself off a cliff if you use it. What to do? Well, a smart MVT solution will allow you to say, “what would the effect be if we switched out the lime green button for a dusky pink button?” You may not get absolutely as much uplift as if you’d gone with the computer’s first choice, but everything in life is a compromise, and sometimes it’s extremely helpful to keep key people on-side during this kind of process, notwithstanding the need to avoid the HiPPO (Highest-paid person’s opinion) effect.

One size doesn’t fit all

If everybody were the same, we wouldn’t need testing or MVT – by now, there’d be a universally agreed-on set of website templates for common online apps (“selling shoes? Make sure your website is blue with yellow text”) and that would be it. But everyone is different, of course, both in terms of innate characteristics (male vs female, old vs young), but also in their context – people who arrive at your website from search may have very different needs from those who arrive from a CRM e-mail campaign.

What this means for MVT is that you need to be able to generate optimized results for different segments of your audience base. It may be that by optimizing for a “one size fits all” approach, you’ll generate a 15% lift in click-throughs from your landing page; but if you optimize for (e.g.) men and women separately, you see an average 35% increase.

A smart MVT solution will not only be able to optimize on a segmented basis, but should be able to advise you about how to segment your audience for the maximum lift. This involves building optimization models for all the user segment variables that you throw at the system and then seeing which ones generate the best average uplift. It may be that it’s not age or gender that matters most in creating the perfect page, but instead the number of previous visits to the website for that user. You shouldn’t have to work this out yourself; your MVT solution should do it for you.


That’s it for this installment. Check back for more on this topic, including:

  • Results automation
  • People
  • Getting started

3 thoughts on “An MVT buyer’s guide – part 2”

  1. Michael,
    You might say that, but I couldn’t possibly comment 😉
    Actually, I know that a lot of sites derive a lot of value from the tool you are referencing – Google’s Website Optimizer. I also know that every one of the features I’ve described in this post and the previous one add complexity to the set-up and interpretation phases of an MVT project – sometimes (quite a lot of the time, in fact) simpler is better.
    My next post on this topic will address the questions of how to get value from MVT, and one important element will be how much support to expect from your MVT provider. If you want to do some of these more advanced things, you need decent support from someone who really understands the space.
    So as with Analytics, there’s room in the market for enterprise tools which demand (and reward) a high degree of set-up effort, and lower-end tools which aren’t as powerful, but are consequently easier to get going with. I still think WSO needs better experimental design, though…

  2. Thanks for these informative posts.
    Where are parts 3 to 37? In part 1 you indicated there are parts 2 – 37 (!)

Comments are closed.