Startup Series Part 3 You Make What You Measure

Merely measuring something has an uncanny tendency to improve it. If you want to make your user numbers go up, put a big piece of paper on your wall and every day plot the number of users. You’ll be delighted when it goes up and disappointed when it goes down. Pretty soon you’ll start noticing what makes the number go up, and you’ll start to do more of that. Corollary: be careful what you measure.

- Paul Graham

Metrics are one of the most powerful and most dangerous tools at a startup’s disposal. They can help focus a team around a common goal or push you off-course for months.

Start by tracking one thing

We realised pretty early at GoCardless that we needed to track something to work out if we were improving. As we are a payments processor, the first metric we chose was our total daily transaction value - how much money was going through the system. We chose this metric because, in the very early days, total daily transaction fees (ie our revenue) was embarrassingly low, whereas our total transactions were roughly 100x higher. This worked well for a few months - we hit milestones in encouragingly quick succession, moving from £1k-a-month to £1m-a-month to £1m-a-day. We’d display the daily number on a large board in the office, and send weekly and monthly summaries to everyone in the company, with the change over the previous period. It worked well both to motivate the team and act as a Pole Star when making decisions; “which of these options will maximise our monthly transaction value?”. When we hit 10% week-on-week growth, we were happy. When we didn’t, we redoubled our efforts.

Choose a proxy for long-term value

However, a year or so down the line, we realised that we’d encouraged some pretty perverse behaviour. A large minority of our transaction value was coming from a few merchants putting through very high-value transactions. Because of a quirk in our pricing model (1% capped at £2 per transaction), we were actually making relatively little revenue and assuming way more risk than we’d like on these customers. For a short time, we were actively pursuing more of these customers in order to boost transaction value. Yet, when we came to search for a solution, it was incredible how ingrained our hunt for transaction volume had become - any change we made to combat the problem would inevitably result in lower transaction values, although dramatically higher revenue - and yet, emotionally, we struggled to make the change. Once we swapped the key metric to daily transaction fees, the problem went away.

Since it sometimes seems fashionable to run companies without revenue, you need to pick the best proxy for long-term value of your company, be that profit, revenue, installs, MAU or page-views. But this raises the next problem we encountered. Even if you could track company value on a day-to-day basis (and you can if you’ve gone public), it’s not an actionable metric. This is usually a problem for B2B companies since the enterprise sales cycle can often be several months - the work you do today won’t impact revenue for some time.

Break out KPIs

One solution is to have each team within the company break down that key number into its component drivers until you get to something actionable. A really neat example is an outbound telesales team. Ultimately, you want your sales team to be closing recurring deals, which will drive revenue in a predictable fashion. But, as any sales director will tell you, if you’re not closing the deals, you need to focus on the activities you’re performing. The stage before a closed deal might be some form of proposal, outlining the key SLAs you will deliver in return for committed fees. To deliver 5 closed deals this month, you might need to make 10 proposals. To make those 10 proposals, you need to attend pitch meetings with 20 companies. To get those pitch meetings, you’ll need to have spoken to 200 companies. Since only 1-in-3 cold-calls results in a conversation with a decision-maker, your telesales teams need to call 600 companies this month. Suddenly, you have some pretty actionable KPIs for your sales team. These are the numbers that you track in daily stand-ups, and measure conversion between each stage. Since they are actionable metrics, the entire process becomes an optimisation problem.

As far as possible, each part of your business should have 2 or 3 actionable metrics that they examine on a daily or weekly basis. They’re the numbers that tell you whether you’re improving or not. An example might be website up-time for the devops team, or average customer wait-time for your customer support team. Whatever numbers you choose need to be objective - if people can massage up a bad month, it undermines everyone’s faith in the metric. These metrics shouldn’t be sticks to beat people with - it’s actually incredibly motivating to work very hard and see that you’re making progress. They’re also useful to highlight strategies or areas of the business that just aren’t working out, rather than allocating personal blame.

Eric Ries spends a lot of time in the Lean Startup talking about actionable vs vanity metrics - it’s well worth a read.

Implement your strategy with KPIs

The choice of KPIs is also a very powerful tool for a founder or management team to set the direction of a company. There are many different ways to reach £1m in monthly revenue - the way you set up your KPIs and associated targets will have a huge influence over the behaviour of people in your company. As a simple example, are you aiming for 10,000 customers to pay you £100 each, or 100 who’ll pay £10,000 each?

When metrics go wrong

Even these actionable KPIs can produce perverse behaviour. I worked at a large national utilities supplier whose (very large) email customer service team committed to responding to 90% of emails within 24 hours. On the face of it, a great actionable metric - and the team were consistently replying to 89 or 90% of emails within the allotted time. But they’d also developed a practice of simply ignoring any email which exceed the 24-hour cutoff. Since these emails didn’t impact their targets (or their bonus), 10% of customers simply never got a response.

A final issue is that, in some areas of your business, KPIs can be really tough or even impossible to pin down. While a front-end team working on your landing pages might measure conversion to signup, a PR team running a 6-month campaign may struggle to find an objective indicator of success. You can try to count the number of articles (and the readership of each), but you’d also need to assess the relevancy of the target audience. You might try to establish a baseline of page-views and count increases or spikes, but there’s so much noise that it quickly becomes subjective. While people have tried to to pin down the value of a “like” or a “re-tweet”, it’s only usually relevant for consumer-facing companies. We’ve never really managed to find a solution to this problem - I’d love to hear what other companies have done.

In summary

• Pick one number that’s your “Pole star” - the best proxy for long-term company value. Re-examine it periodically to see if it’s producing perverse outcomes.
• Have each team break down the key metric into its component drivers or KPIs that individuals can influence. Measure team & individual performance & improvement on this basis.
• Make sure the numbers are objective.
• Make them visible - put the key metric on a big dashboard, and give each team a dashboard that lets them see how they’re doing. Send regular automated emails around your teams with a breakdown of their numbers.

Follow the discussion over on Hacker News