Metrics are the problem with Metrics based Performance Monitoring
Yesterday Netflix delivered Law Abiding Citizen DVD. After celebrating our daughters first month milestone we watched it. If you haven't this movie its a about a guy who is upset with the law system that did not do justice for murder of his family. The core of the issue being the prosecutor caring more for his record 95%+ conviction rate than caring for actual justice served. This caught my eye, I saw a pattern, I could relate to things that happen in software world.
In the movie the prosecutor chose to make a deal with the bad guys to get the conviction, however, that deal was not a justice, the deal in fact put the wrong guy to death penalty but the prosecutor chose to ignore all that to get the case closed as soon as possible. Why because he is measured by the conviction rate and the number of cases, so its in his benefit to expedite the cases and make deals to close the cases as soon as possible. End result? Justice is not served. Huh, isn't that the role of the law system? If the prosecutor's and DA office performance is measured by surveying the victims, the system would have gotten a much better metric than by measuring with conviction rate. Summary, conviction rate is a bad metric.
In software I have seen many a time where Quality Engineering folks care for opening more and more bugs with extremely stretch cases and forgetting the customer and testing and thinking like a customer. Why? Their performance is measured by the number of bugs that they file and that get fixed. Sure we get some silly bugs fixed but we miss many functional use cases, many big picture use cases that end up being found at a customer install. This is not the individual fault, its the fault of the system that measures performance based on such a number. Number of bugs filed is a bad metric, number of bugs that are reported by customers and not found by QE is a better metric.
Same thing with developers. In many companies I have seen managers touting how their team fixed 100's of bugs in a week, I have seen many higher managements comparing teams by the number of bugs fixed. 100's of bug fixes in a week, awesome right ? No, its not. Why? Its not important how many bugs a team fixed, its more important how few bugs are reported against a product. Less bugs means well designed, well implemented and well tested product. Again number of bugs fixed is a bad metric, number of bugs filed against that product is a better metric.
Using metrics for measuring performance is nothing new, every software company does that either formally or informally but be very careful on what metrics you chose to monitor it directly effects the actions of the people involved.
