Metrics Misuse

Flackery ain't what it used to be.

I promised to publish Deloitte's response to last week's critique of its Center for the Edge's Shift Index.

Once upon a time, public relations professionals made sure no potential image challenge went unaddressed. And yet, even though I contacted Deloitte, no response has been forthcoming.

From Deloitte, that is. Steve Denning, author of Forbes' excellent Radical Management blog, posted a critique of my critique (and others as well; he addresses points I didn't and wouldn't have made).

(Endorsement: While Steve and I don't completely agree on this particular topic, his thoughts on business leadership and management are innovative and interesting ... well worth your time and attention.)

Denning's defends the use of declining ROA as an indicator of economic decline, and points out that in Deloitte's 2010 report it considered alternative metrics, like return on equity and return on invested capital. They all reported equivalent trends.

But that three metrics report similar trends simply means they're correlated -- unsurprising given how computationally similar they are.

It's how Deloitte misused them that matters to you, because understanding this misuse can help you as you sort through your own information overload, trying to make sense of things.

Misuse #1 -- starting with the metric: This is a fundamental fallacy that's distressingly common in the world of business. When someone starts by asking, "What metrics should we use?" nothing good will come of it.

The better question is, "What are we trying to accomplish?" It very well might be that as a matter of macroeconomic policy, the United States should be trying to maximize the return on aggregate business investment across all industries (ROA, ROE and ROIC all measure return on overall investment).

If that's the question, say so. Deloitte does not. It simply takes a commonly reported financial ratio that some but not all professional investors consider to be a gauge of management performance, and that all professional investors understand should not be compared across industries, and aggregates it across all industries.

My guess: Deloitte chose its metrics based on what data were readily available that had some association with business performance.

Misuse #2 -- working backward from a pre-determined conclusion. Deloitte's Center for the Edge is committed to its Big Shift view of the world. That's a problem.

To summarize and oversimplify, the Big Shift is increased competitive intensity, knowledge flow mattering more than knowledge assets, and failure to redefine business practices in response to these two changes.

Imagine ROA is a perfect measure of business management performance. Its 47-year decline might mean what Deloitte says it means ... that the caliber of management throughout the U.S. economy has deteriorated in some way, in particular by failing to adapt to the "Big Shift."

Or, it might mean something completely different. As I pointed out last week, the U.S. shift in industry emphasis ... from manufacturing to finance, services and entertainment ... also accounts for the ROA decline.

In private correspondence, Denning made the point that my explanation doesn't change anything, because even if this is the case, it still points to increasing economic weakness.

But it does matter, because there's a huge difference between mismanaging the economy and mismanaging the individual corporations that comprise the economy.

Just because someone finds data that supports their preferred interpretation doesn't mean they're right. And if they get the root cause wrong, they'll only improve the situation by accident.

Root cause analysis is a scientific practice, not a search for ammunition. What Deloitte failed to do was to clearly formulate the different hypotheses that might explain the decline in ROA, identify the evidence that would disprove each of them, and then collect that evidence.

As it happens, I agree that Deloitte's "Big Shift" trends matter. It's a truism that the Internet has made agility more important than size, increasing competition. As for "knowledge flow," I'm pretty sure it's important as well, although I prefer a more prosaic formulation ... that a business's success is tied to how well employees collaborate, both with each other and with employees in its network of suppliers, partners, and customers.

These trends matter to you, because IT lives in the middle of both of them. If you agree, ask yourself what you're doing to help your company exploit them.

And, ask what evidence you have that supports your opinion. It might not be very good, but that's okay. In business, you don't always have conclusive evidence in time to support the decisions you have to make.

Often, evidence arrives just too late to do any good. All I ask is that you don't have more confidence in your conclusions than the evidence allows.

Here's where to find this week's ManagementSpeak.