After his breathless article last year, proclaiming Google Analytics to be something like a cross between the second coming and Barack Obama, Brandt Dainow seems to have soured on the big G, proclaiming this week that GA contains ‘disturbing inaccuracies’:
Google Analytics is different from other products in that it has been intentionally designed by Google to be inaccurate over and above the normal inaccuracies that are inevitable. These inaccuracies are so glaring that most people are getting a very false picture of what is happening in their sites.
Dainow’s main beef with GA is two-fold:
- It treats single-page visit as valid visits (i.e. it doesn’t remove them from visit counts or other related measures)
- It includes single-page visits in average visit duration calculations
He also remarks that Google did in fact change the way that GA calculated average visit duration last year, but then changed the calculation back in the face of user pressure:
Google intentionally rolled Google Analytics back so that it produced an incorrect average duration…It’s been that way ever since — Google is intentionally and knowingly providing inaccurate numbers because a few people preferred neatness to truth.
Brandt then proposes two alternative measures – ‘retained visits’ (the count of visits with more than one page impression) and ‘true average duration’ (the average duration of retained visits). These metrics are not without some merit – it’s useful to know how many visits contained more than one page view, and the average duration of these visits. But Brandt goes on to assert that these two metrics should replace the standard measurements of visits and average duration in GA and (presumably) other tools. This suggestion is ridiculous, for the following reasons:
- Contrary to Brandt’s assertions, there are a host of scenarios where a single-page visit is a perfectly valid visit, including, for example, this blog, for crying out loud, which has a high proportion of single-page visits because readers either just read the homepage and leave, or click through to an article from their RSS reader. So chucking all these kinds of visits out is crazy.
- Whilst the inaccuracy of including single-page visits in average visit duration calculations is known to be a problem, removing these visits from the calculation doesn’t yield a magically ‘accurate’ number, it just yields one that is inaccurate in a different way. You still have no idea how long people looked at the final page of their visit for, and with a two-page visit this can introduce a huge potential inaccuracy.
- Such standard metrics as exist in the web analytics industry are the result of long and arduous wrangling. There are no sacred cows, but you need a really good reason to exchange a simple and easy-to-understand metric for one which is more complex and offers no discernible benefit.
Whilst I can understand Brandt’s motivations for posting these ideas (which, I imagine, lie somewhere on a spectrum between a genuine desire to spark debate and a desire to generate a lot of traffic to his blog, in which regard I am obliging him), his remarks do irk me a bit (can you tell?), principally because he commits the unpardonable sin of absolutism when talking about web analytics, bandying about words like “truth” and “wrong” when really he is just presenting his own preferences.
When, as an industry, we can’t even agree what constitutes a visit, it’s pretty rich to start decrying one tool or another as ‘inaccurate’ simply because it takes an approach to data that you don’t believe in. And besides, as Brandt surely knows, Google Analytics now has the capability (via its custom segmentation) to calculate the metrics he seeks.
Finally, as every half-experienced web practitioner (of whom Brandt seems to have a low opinion also) knows, the key to success in web analytics is to pick your metrics, stick to them, and measure them continuously as you make changes to your site and your marketing, to see what is working. If you’re looking to increase engagement, and have decided that visit duration is a good measure of this (a debatable point, as it happens), then it doesn’t matter whether you include single-page visits in your duration calculation – if your visit durations are going up, you’re happy. And if your visit durations suddenly jump because your web analytics vendor has changed the way they calculate the metric, this could in fact cause more pain than benefit, perhaps causing you to go to said vendor and say, “Oi! Change it back to how it was!”.
So feel free to read the article, but be warned: it’s not very accurate.
I don’t condone name calling, but I got a full list here I could throw at that guy. How come I don’t get those iMedia gigs?!?
You’re right to be upset, and thanks for your “accurate” criticism of his fallacies.
Very interesting discussion!
Some of Brandt arguments could have some merit… if applied “in context”. Your answer is right on the spot! My belief is any metric is a good metric if you can explain it, and if it makes sense for your specific situation.
I totally agree the tone of his article is condescending. I do have my darker moments with Google, but Brandt post is, as you say, “unpardonable sin of absolutism”…
Saying “how limited the understanding of web analytics is among practitioners” is nothing to make friends in the small community of web analytics professionals!
Stéphane
immeria.net
I have no problem with disagreement of my position, but don’t cheapen your arguments by mis-representing me. I did NOT suggest replacing any Google Analytic’s metrics, in fact I made a point of arguing for keeping the existing metrics and adding the extra ones, just as you do. My exact words are: “I like to aim for minimal disruption when I correct them. I therefore leave the existing metrics in place and add a few”
Since your entire blog is based on your assertion that I said “replace” when I actually said “add” I think an apology, or at least an admission of error, is in order.
Brandt,
Nice try. But the whole second page of your post is concerned with how you feel that Google should have retained the change to its calculation method for average visit duration that it made during the summer of 2007. And even if you don’t actually propose removing GA’s visit count metric and replacing it with your own version, it’s clear that you consider your own metric to be unequivocally superior to GA’s (and the industry’s) version.
The point I make in my post is firstly that you make some unsupportable assertions (such as that single-page visits are not visits at all), and secondly that, when it comes to web analytics metrics, we are all in glass-houses, and that chucking stones at one another is an especially foolish thing to do.
You could have written a genuinely useful post about how the presence of single-page visits makes it difficult to determine how long people really are looking at pages and the site for, and included your useful information about how to calculate alternative measures, and in which circumstances these measures would be useful. But instead you chose to sling mud at Google (a company that I have absolutely no reason to defend), and, by extension, the rest of the industry; so no, no apology.
Regards,
Ian