Volume 7, #6 October 2009
Metrics Madness
Why Many R&D Metrics Reduce ROI, Gobble Resources, and Lead You Astray -- and Three Steps to Fix Them
By Wayne Mackey
As chairman of the upcoming 13th Annual Conference on Product Development and R&D Metrics produced by Management Roundtable (MRT), I spent the last few months reflecting on the current condition of R&D metrics. And it's not good. (MRT titled their interview with me as follows: "Expert warns, 'Current state of R&D metrics...is bad.'")
Dire headlines aside, we haven't made enough progress since that first metrics conference 13 years ago, and too many companies have simply given up trying to improve metrics. Lest you throw up your hands and forget about measuring anything, here are a few ideas for turning metrics from a resource-draining burden to a useful tool for increasing competitiveness -- whether you're just starting out or seeking to improve a metrics program that's already in place.
Don't measure the "metric du jour;" instead measure what matters
To understand where metrics have gone astray, let's look at a few examples of popular metrics that would seem to be useful measurements but which actually give rise to unwanted behavior and steer your R&D efforts in directions that undermine competitiveness.
PERCENTAGE OF SALES FROM NEW PRODUCTS
On the surface, it sounds as if measuring percentage of sales from new products would increase innovation activity and lead to the commercialization of more new product ideas. However, when instructed to track this percentage, managers quickly learn to game the system by lowering the bar for defining what a new product is. Suddenly, the line extension with a few new features becomes "new." Product portfolios thus appear to contain lots of new (but low risk/low return) products that are not actually innovative -- and the pseudo-new products lose ground to competitors and end up eroding rather than enhancing a company's market position. Why is your return from R&D investments going down? You’ve joined a crowd of companies seeing the same effect -- blame the preponderance of companies relying on the percentage-of-sales-from-new-products metric. So what’s better? Measure how well-balanced the portfolio is in terms of having projects spread across different technologies and market risks, and also the cash flow generated by the portfolio over time.
NUMBER OF PATENTS AWARDED
Having your researchers receive a lot of patents would seem to be a good indicator of innovation and a good thing to track. Not really. The number of patents says nothing about whether your intellectual property yields value. Companies should instead measure patent diffusion, or the rate at which intellectual property is absorbed into the product portfolio to provide value.
RESOLUTION OF CUSTOMER PROBLEMS
The number of customer issues satisfactorily resolved through the CRM system would appear to be a good indicator of customer satisfaction. But do you really want to be fixing lots of problems? Instead, you want to encourage interactions with customers that prevent issues from ever arising in the first place, which you can do by measuring percentage of development staff time spent in the customer's environment.
Sometimes it's not a specific metric that guides behavior in the wrong direction, but the approach to metrics itself. For example, many companies simply measure too many things. They track dozens of metrics, but try to only pay attention to the few that aren't working. The consequence of this approach is that the company literally drowns in metrics, wasting resources collecting and reporting data that nobody can follow up on. Then there are no resources available to dig under the surface of the few critical metrics, so problems brew unnoticed until they erupt as crises.
If you can do only three things to improve your metrics, you should:
1. Track only the critical few metrics per project or team/individual. A large company may have hundreds of metrics for different divisions and projects, but individuals should be responsible for no more than 4 or 5 that relate most directly to their jobs.
2. Measure what you're not good at. Even without metrics, most companies with competent staffs manage to get through the day being relatively productive. Don't waste time measuring things that already work. Pick the top few things that don't work, and measure them. That gives your decision makers the information they need, early and often, to fix the root issues.
3. Customize, don't standardize. Applying the same metrics up, down and across the entire company or even an entire division doesn't work. Managers should measure management activities. Engineers should measure design activities. Financial planners should measure financial activities. And all of them should be measuring only those items that make a difference to their success or failure at getting their job done.
Are you already doing the basics? Then you might be ready to move to the metrics frontier, where new and more complex ideas are taking shape. For example, measuring information turns applies the concepts of manufacturing inventory to information movement. And heads-up design metrics give product designers immediate information on the impact of their design decisions on the gross margin of the final product. (See the PDC paper on heads-up design.)
No matter where you are on the metrics continuum, I would bet there's room for improvement.
Wayne Mackey is a principal at Product Development Consulting, Inc. and the chairman of the thirteenth annual Management Roundtable Product Development and R&D Metrics conference being held on October 19-21, 2009 in Orlando.
