Have You Misinterpreted The Data?

(By Roy Wiliams) “The data is conclusive,” he told me. “Our close rate is much higher when customers call us on the telephone instead of going to our website. Therefore, you need to write ads that drive customers to the telephone.”

“I agree that the data is conclusive,” I told him, “and it says you need to fix your half-assed website.”

The research community has embraced a new buzzword. They take great delight in demanding that everything be “evidence-based.” It’s a little like listening to a parrot: “Evidence-based.” “Evidence-based.” “Evidence-based.” “Evidence-based.”

By themselves, these two words seem harmless. After all, every new idea is based on evidence. But the smug and devilish side of this trend toward “evidence-based” methodology is that the phrase has come to mean “scientific, conclusive, and therefore above debate.” In other words, if you want to shut everyone up and make them swallow your recommendation, all you have to do is announce that it is “evidence-based.”

I think they learned this trick from online marketers. (Lest the broad brush of that statement paint innocent people with a fault that is not their own, allow me to say that I know several brilliant online marketers who gather data responsibly and examine it from every perspective. They agree that numbers can whisper opposite statements when viewed from different angles.)

I’ve never seen anyone make a decision that wasn’t based on evidence.

So the question isn’t whether you’re basing your decisions on evidence. Of course you are. The question is whether you’re interpreting that evidence correctly.

Here’s how I explain the cognitive bias that has become so alarmingly evident: “The intellect can always find logic to justify what the heart has already decided. Consequently, data is often used in the same way that a drunk man uses a lamppost: for support, not for illumination.”

Let’s examine the facts.

FACT: The Japanese eat very little fat and suffer fewer heart attacks than the British or Americans.
FACT: The French eat a lot of fat and suffer fewer heart attacks than the British or Americans.
FACT: The Japanese drink no red wine and suffer fewer heart attacks than the British or Americans.
FACT: The Italians drink lots of red wine and suffer fewer heart attacks than the British or Americans.
FACT: The Germans eat sausages with beer and suffer fewer heart attacks than the British or Americans.
EVIDENCE-BASED CONCLUSION: Eat and drink what you like. It’s speaking English that kills you.

The misinterpretation of data is as old as humanity. “Post hoc, ergo propter hoc” is the ancient Latin term for the most common of all fallacies of logic. It means “the second thing followed the first thing, therefore the first thing caused the second thing.”

But correlation rarely indicates causation.

A second limitation of data is that it cannot tell you the right thing to do. It can only tell you the result of what you have already done.

Am I against data? Of course not. Data is information, and information is powerful.

But like all powerful things, it can hurt you if you mishandle it.

Five safeguards you should use when evaluating data.

Ask:

1. What were the methods of data collection?
2. Could those methods have influenced the outcome?
3. Is there any other way to look at the numbers? (That is, are they saying “drive customers to the telephones,” or are they saying “fix the website”?)
4. Is there a chance the persons who prepared this information have a bias or an agenda?
5. If the data reveals a surprise, is that surprise supported by indicators outside the data?

You’ve heard it said that “numbers don’t lie.” I’ve heard that, too.

But I also remember my grandfather Roy looking at me after a data-quoting salesman had walked away. He said, “Little Roy, never forget: figures lie when liars figure.”

Granddad, it’s been 50 years.

I never forgot.

Roy H. Williams is president of Wizard of Ads Inc.