The Metrics Meta-gameAs the stewards of good data practice within a company, we data professionals are often asked to help set metrics, there’s a whole meta-game around the process that we’re playing whether we like it or not.
Randy AuBlockedUnblockFollowFollowingMar 8It’s all games and moody lighting until someone loses an eyeMetric setting is hard work.
While there’s tons of writing out there with advice on how to choose and pick metrics and KPIs (key performance indicators), they all require you to be intimately familiar with how a business functions and understand what its underlying goals are.
You’ve got to deal with making sure things are measured correctly, that it is causally connected to, or at least strongly correlated with your main goals.
It’s got to be relatively easy to understand, something you can affect and change, and on top of it all, you can only pick a small number of them.
Balancing all these competing factors qualifies as an art.
And then, once you do all that work you’ve got a final boss to deal with — Goodhart’s Law.
“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
”Essentially, any metric you pick and use, people will find a way to game it and it stops being useful for the purpose of what we chose it for.
This gaming behavior can be very dangerous because it subverts the original reason for creating the metric in the first place and can even work against the original intent.
They might do it because of extrinsic motivation, like if their yearly bonus depends on it, or they might do it for intrinsic reasons, like they just want a high score.
Humans are weird but clever creatures.
What’s a metric setter supposed to do?Shoot down obviously self-serving metricsThe stereotypical case is when a sales team wants their bonuses to be based on the number of leads brought in, ignoring the number (and size) of deals actually closed.
Obviously it’s easier to rack up a ton of leads than it is to close a deal, and the gaming of this metric would obviously result in a ton of wasted time generating leads that will never close to the detriment of actual closed deals.
Most real world examples won’t be nearly as obvious, but they often fit the pattern of being a relatively easy metric for a person to manipulate, while not being completely in line with the goals of the company.
They’re levers to be pulled, not the outcomes of the system after levers are pulled.
Often there’s a focus on short term gains while ignoring long term effects.
When setting a metric, you should always ask yourself what sorts of actions would it take to game that metric.
Put on your black hat for a bit and see if you can come up with insidious ways to “win” the game.
Accept gaming happens, work with it as best you canJust about any metric can be gamed and you alone won’t be able to come up with every loophole and back door for a given metric, not against the many clever humans who will be affected by the metric.
So just admit to yourself that the metagame exists and ask “if this metric were to be abused and it goes up like crazy, what happens, can it be a net positive?”An example would be something like the number of Friends someone has on Facebook, especially early on.
Sure, unscrupulous people can and do buy armies of robotic friends on Facebook, but on average, users having more connections serves to make the social network stronger and more sticky.
The robots can be dealt with separately as a distorting force, but normal humans who try to maximize the number for their own reasons could be seen as positive from Facebook’s perspective.
Disclaimer: I’ve never worked with or at Facebook nor heard about their metrics, I’m just making up illustrative examples.
Overall, the more aligned the metric is with the core mission of a company, the more likely it will have a similar positive relationship.
Control the distorting factors as they come upAs in the previous example, a metric can be good for various reasons, but still have obvious ways where things can be gamed.
Despite that setback, you might still think it is the correct metric to use because it’s very important to a number of other outcomes.
In that case, you can erect guard rails against gaming behavior, either by explicitly exclusively excluding certain types of behavior from the metric (e.
exclude all internal users), or root out the source of the distortion (e.
ban all the bots).
While it’s an unsatisfying reactive stance, the cost of catching cheaters might be acceptable to you.
Just be aware that it spawns a new set of metrics discussions about how do you detect and catch cheaters.
Its metrics meetings all the way down.
Whatever you pick, be vigilantThe price of liberty might be eternal vigilance, but sadly the same applies to core company metrics.
People need to always be on the lookout for improper behavior.
It’s really tricky to do because it’s so easy to become complacent after things have been running smoothly for a year.
At the least, every so often (every maybe 1–3 years), it is advisable to do a fresh analysis to see if the things that made the metric attractive to begin with — alignment with company goals, simplicity, ability to change — are still valid or not.
Businesses and people change, and your metrics need to follow along.
Similarly, if one metric is telling a great story of growth, other metrics should be telling variations on that story, ideally ones that are measuring completely different systems or processes.
Be on the lookout for when one part says great things and the other parts don’t.
Don’t be paralyzedIt’s easy to be overwhelmed by all these issues.
My advice is to do your homework on what moves your system, get feedback, then spend a day or two thinking about utility and exploits.
After that, just pick the ones that look best given what you know.
You’re unlikely to get your core metrics perfect the first time (or, ever).
Luckily metrics are a tool made by us, for ourselves.
We can change them when they don’t work out.
Other referencesI only focused on the gaming/abuse aspect of metric setting because I think it’s the least explored part.
As usual I draw a lot from my personal experience, and there’ll be various gaps.
For some more serious treatments about metric setting, from top to bottom, take a look at these alternate sources:“Building Less Flawed Metrics “— David Manheim — Academic paper on metrics and Goodhart’s LawAnd an interesting framework for working through defining a new metric:4 Steps to Defining GREAT Metrics for ANY ProductA Mental Model & Framework You Can Use For New Products, Existing Products, and Even PM Interviewshackernoon.