For a while ago, I wrote about the characteristics of the self-optimizing agile organization.
In this post I’ll take a look at the role of metrics and measurements in general, and in a subsequent post on how the self-optimizing agile organization can (and must!) take advantage of a sound (asop to today’s typically ignorant/naive “bean-counter flavor” ) usage of metrics and measurements.
In fact, without metrics and measurements it is impossible to optimize anything.
Let me start by stating that I’m by no means a proponent of today’s typical, naive and clueless usage of metrics and measurements in business (or society in general), where management (or politicians) measure “everything” and practices “management by spreadsheets”, a.k.a. “management by KPI’s”, (or “management by objectives”, which was the previous fad word). The belief of many “modern” managers seems to be that “the more data, the better”, without taking into account the relevance nor the quality of the data. And now, with all the buzz about “Big Data“, I’m sure this trend will explode! The main problem with the “Big Data” approach is that the number of “false positives” will radically increase, due to the radically increasing number of spurious (i.e. random) correlations in any massive data set. These spurious correlations result from the risk for confusing noise for signal whenever you are working with massive data sets, and as we all know by looking at e.g. the web or the media, the amount of noise increases much faster than the level of signal in modern society. So, in the era of Big Data, extreme care must be taken to avoid the trap of believing that correlation always implies causalisation!
Unfortunately, unless you as a manager have a solid background in your business domain, you are pretty much forced to practice mgmt-by-spreadsheets, since you really don’t have the prerequisites to operate in a more effective and productive way. A “manager” who does not possess any knowledge of – or interest in – the domain in which he or his customers operate, is to me a complete joke, an extremely well paid administrator instead of a true manager or leader. Unfortunately, in much of today’s business, these “empty suit” type of managers are much more frequent than those who practice “Go See”, i.e management by actively participating in “the trenches”.
Anyways, I’m getting carried away by my frustration with all the “empty suit”-mgmt types I frequently encounter, let’s try to get back to the topic:
The problems with metrics – Gaming the numbers
First, as we all know, metrics and measurements can be severely misleading (“lies, damn lies, and statistics”), either by ignorance or mistake, or by those involved deliberately gaming the numbers. A few examples:
- I recently saw an article in Swedish newspapers that the number of train delays in Sweden has been declining over the past year. It sounded as progress, until I learned that the reason for the improved statistics was that the definition of a “delay” had been changed from 5 minutes late to 15 minutes late – in other words, by changing the definition of the metric by increasing the range, those responsible were able to demonstrate progress.
- Swedish Police, famous for their inability to resolve (or prevent) crime, can regardless of their high level of ineffectiveness demonstrate a high level of “success” by measuring activity (effort) by focusing their efforts on catching motorists speeding ever so little on empty freeways, or performing alco-tests outside the teetotaler HQ.
- A large multinational in “technology, information & knowledge” business assesses the skills and knowledge by their staff by measuring the number of powerpoint presentations the staff has consumed during an annual “competence building” period.
- A very successful football (soccer) player, known for his spectacular and high precision long range passes, gets a new coach who employs the “scientific method” by measuring the success/failure rate of each pass the players make on the field. All of sudden the star player stops making any passes at all, to the great detriment of his and his team’s overall performance, because he got serious heat from the new coach for “his pass success rate being below bar”.
There are many more examples of misleading metrics, but those listed above should suffice to bring forward the message that extreme care must be taken when it comes to what is measured and why.
We can perhaps bring some clarity into the mist of metrics and measurements by an attempt to categorize metrics into a few groups:
- metrics for activity ( e.g. “busyness”, efficiency)
- metrics for process (e.g. compliance)
- metrics for execution (e.g. skills, techniques)
- metrics for performance (e.g. utilization)
- metrics for results (e.g. financial targets)
- metrics for business outcomes (e.g. customer value)
In a follow-up post, I’ll explain how the self-optimizing agile organization can apply the various types of metrics to improve its overall performance.