Lies, damn lies and not so Unique Users…
Recently, while working on Ceros, I’ve had to become very involved with the auditing process of web sites - everything from how statistical data is recorded, the ways to measure it and the vocabulary used in describing the results. While it may seem obvious, there are very strict rules and regulations about what you can and can’t do… the reason for this is because these numbers are the life blood of the commercial web - after all, if no one’s looking at your site, why bother?
In the UK if you want a third party certificate for the statistics of your web site then the minimum mandatory metric you have to declare is your total Unique Users for the period being audited. But this metric is under increasing scrutiny by the industry as many consider it unreliable and misleading.
The metric is misleading because the best it can actually measure is unique web browsers. Most sites use cookies to ‘tag’ browsers as being unique, but this is no indication of how many Users that browser has or conversely if those Users have multiple browsers (IE and Fire Fox) or multiple devices (Desktop PC and Mobile Device).
Interestingly it’s ‘illegal’ to rely on cookies alone to measure Unique Users, which is why a lot of tools like Google Analytics are un-auditable. The non cookie based approach to counting Unique Users is to concatenate the device’s IP Address and User Agent to form a unique id, this is something sites must fall back on for Users who reject cookies. This is because users who set their browsers to reject cookies will appear to be a different Unique User every time they make a request to a site that relies on cookies alone, which will obviously over state the statistics (both Bounce Rate and Unique Users).
The biggest problem with the Unique User metric though has been widely reported and it’s a group of people who have been christened Serial Resetters.
Who are they? They’re a group of users who for what ever reason (mostly concerns about privacy) delete all their cookies on a regular basis which, for analytical purposes, is worse than people who out right reject cookies because they can’t be detected and catered for with User Agent/IP address. But there’s an increasing problem with even that…
Microsoft, in their infinite wisdom, have allowed plug-ins and add-ons to show up in Internet Explore’s User Agent. Which would be fine but people like to fiddle with things which means of the course of an audit period it is entirely possible for a browser’s User Agent to change as well as any cookies installed! Which means that even ignoring the problems counting cookied users there is still potential for the Unique Users of a site to be overstated!
So, what is to be done? Either we’ve all got to except a permanent unique identifier in our browsers (which isn’t going to happen) or the industry has to put more emphasis on session based metrics like Visits or Page Impressions.
Comments
3 Responses to “Lies, damn lies and not so Unique Users…”
Leave a Reply
A great post, very interesting. Especially the bit about google analytics being un-auditable commercially.
So we find a bit of a deadlock. Who will yield? Will google change they way analytics measures? Or will the commerical powers interested in metrics change their perception of what’s acceptable?
I know that in a few years time when I sell the websites I’m building up, a full set of analytics stats that go back to the birth of the site are going to be worth their weight in gold.
Question is, how much will they weigh?
It’s all relative. Unique users should just be called X! Then the definition can changed but there has to be a base calculation to go on, that is until IPV6 makes it a lot easier
[…] Web Analytics industry is in the midst of a debate about how to identify and count Unique Users. Some people are starting to suggest that we should […]