Vanity Metrics Need to Die

Unless you’re working for a Goliath of a company like Amazon or Google, chances are you don’t necessarily have access to massive data sets that intelligently report user experience insights. This is a huge problem in Silicon Valley and the startup world, and today I want to complain about the problem of so-called vanity metrics, and why they’re not doing anyone any favors.

I spent a couple years in the heart of Silicon Valley, and one thing became very clear: everyone wants data. Developers want it. Marketers really want it. Investors really really want it. And it makes sense, right? In order to see if a product or a service is working, it helps to see how people are using it, and what works for “conversions” — AKA getting what you want out of people.

At the heart of many-a product design meetings beats the drum of metrics, metrics, metrics. How can we measure user “engagement”? How can we A/B test for better conversion? How can we prove to our investors that people actually need our product?

And this is where things go horribly, horribly wrong. Introducing vanity metrics: the thing everyone denies, but so many are guilty of.

You’re so vain…

Data is great. User engagement is great. Adoption tracking is great. But at the end of the day, if you’re using your data only to convince someone in a position of power that people do, in fact, use and love your product…well you’re not only failing your users; you’re failing yourself. If this is how you’re (and I’m not pointing any fingers here; this is the proverbial “you”— most of you are awesome) using your data, congratulations: you’re festering in vanity metrics.

Data should absolutely be used to track success for both users and for the business, but again, no one benefits from hiding certain truths. Here are a few ways I’ve seen genuine data be used for, well, frankly, lying to investors:

• Click-through rates are really high on our email campaigns! Except what we fail to mention is that we bought this email list for a very costly price, and the recipients are essentially paid to click through on emails. But look at that 50 percent rate though!
• Our new push notification strategy has upped user engagement by nearly 30 percent! Except by “user engagement” we just mean that users opened the app after receiving a notification. What we fail to mention, oddly enough, is that nearly 95 percent of said users leave the app immediately after. Oh well!
• We’ve grown our Twitter and Facebook presence by 500 percent in only 3 months! Here’s a great quote from one of our followers…Except we paid for all of those followers, and they’re incredibly irrelevant to our actual goals, and provide no sort of actual engagement. (Oh and those nice sentiment quotes are from our senior developer’s wife.)

I could go on for hours about these ridiculous examples, because they quite literally pop up in discussion every day. To be sure, not all startups and small businesses need to, or want to, lie to themselves or others; this is a worst-case scenario. But vanity metrics are out there, and once you’ve seen how they manifest, perhaps you can be a warrior for truth and speak up.

Using data the right way

Of course, tracking clicks, opens, design changes, etc, is a very good thing that everyone in design should be doing. Just make sure you’re asking the right questions: does this provide value for my users? Does this provide value to my product or business, without subtracting from the user experience?

There are a bunch of tools out there, too. Google Analytics is the most obvious choice for web design metrics. MailChimp has excellent metrics for newsletter campaigns (and I just like them, because they’re awesome). And then there are tools like NomNom, which aim to be an all-in-one kinda place to track both customer feedback and user research, combining them into one location for product insights. And since they’re friends of UX Booth, here’s a bit about what their product allows you to do:

• Upload user research files from Google Drive, Dropbox, Box, and email
• Powerful search engine for all your qualitative data
• Learn from users faster by looking at your data in different ways
• Track and group user feedback and research files based on keywords, tags, and any metadata available
• User profiles allow you to get a single view of your users and their feedback across multiple sources
• Validate hypotheses by building powerful and collaborative research projects
• And a whole bunch more

I haven’t gotten a chance to use tools like NomNom since leaving Silicon Valley / the startup world, but would love to. If I ever go back (and I probably will not, ever, for lots of reasons), I would love the opportunity to use tools like this to generate genuine insights from my design and product decisions. And, perhaps, put up a fight or two when a higher-up decides we need to jimmy the numbers a bit.

And I get it. It’s scary. A plummeting user engagement metric might tell a real hard truth: people don’t need what you’re selling. Not acknowledging that truth doesn’t make it go away though. The truth will prevail one way or another, no matter how hard vanity metrics cover up the stench. Let’s all do ourselves a favor: fight the good fight against vanity. Fight for truth in data.