Kill Your KPIs

A good product has good numbers. You’ll track the usage of your product and see how “it performs”. Usually, we measure key performance indicators or KPIs. For digital products, a typical KPI can be something like conversion rate, usage numbers, retention numbers. Digital companies love these quantitive KPIs: They are easy to collect, easy to compare and easy to design for.

The rise of big data and data-driven companies like Facebook and Google has put KPIs in the limelight: Almost no tech company can work without measuring and staging their KPIs today. We have dedicated business intelligence teams, omnipresent performance dashboards, and split-test based optimization. Data Porn.

There is only one issue with data-driven design: It can be misleading and eventually be bad for your business.

You Choose the Wrong KPIs

A typical mistake when analyzing business performance is picking the wrong indicators. “Time spent” on a marketing landing page is not important, the same as conversion rate on a support page is not the KPI you are looking for.

Often, this mistake happens when you pick a KPI without looking for the purpose of your design, just because you might have heard it is important. Some KPIs also poorly translate between teams and departments – a sales team doesn’t care about “cases closed”, while a support team won’t think about “deals sold”. What is really important for your team and business – and what is important for your users?

Be careful to not confuse KPIs with success factors – this is like driving to a sign on the road that says “Beach 50 miles” instead of going to the beach.

Your Data Basis is not big Enough

Playing like the big guys can make you think you can test and optimize like them. We have been told that running split tests and looking at user stats is the right way to go – but what, if you don’t have enough users to make significant statements of these tests?

Most designers will have never heard of significance calculation and error probability – I also had to learn this the hard way. Some testing tools work around the low numbers by presenting a significance of 90% or 95% - which no real statistician would dare to do. Just imagine: you have a 90% probability of being the father – you would not trust this result. Why should you in a business context?

As a consequence of this, split testing tool Optimizely canceled it’s a small business plan last year, making their services only available for bigger clients.

The typical online experiment will need more than 25,000 visitors to reach significance, meaning that everything you test with lesser visitor is destined to fail because of insufficient data.

Optimizing the Numbers Kills Your Brand

This is a tricky one: When you design for your product, you usually do so with values, ideas, and a mission. All of these are easily forgotten when optimizing your KPIs – in worst case generating a mediocre or generic outcome.

I had a project at my old company where we tested several new icons for a game. The old icon had been done by one of the game artists and never been tested against other variants – it was about time!

We ran a number of tests, comparing our current icon against successful competitors and completely new ideas. Icons that drove more attention, icons that showcased gameplay better. We knew what KPIs we were looking for, and our data set was big enough. Almost all of the icons outperformed the current version, yet we didn’t change it. Why not?

While all the new versions seemed to work better, we forgot one very important thing: This asset had become an ambassador for the whole game. It featured the main character and was a big part of the brand. Mindlessly changing this to increase new installs would have altered the brand integrity, alienating those who used the icon to identify with the game.

Affection to a brand is hard to measure, but it is not impossible it’s only that most of the current KPI testing and dashboard solutions do not offer a way to do so.

How to Design With Data

Don’t fall for the data porn trap. Designing with data requires a very good understanding of your business model and the jobs to be done. An exercise to get started is the HEART framework – a technique developed at Google, designed to help identify quantitive metrics that are relevant for your product.

More important: Make sure that you have enough data points before you make decisions based on quantitive data. Also, don’t forget about qualitative KPIs: Is my product on brand? Is my product safe? Do people enjoy using my product? Those are harder to measure, for sure – but once you get them, they can tell you so much more than simple conversion or retention numbers could.

Look beyond KPIs and ask more questions about the greater context: Why do people want to use my product? Which purpose will it have in their life? How can it turn them into a better version of themselves? Answering those question will make for a much better product than optimising alone.