Lies, Damned Lies, and Statistics

8 minute read
Share this:
By Patrick Nord
Vice President of Professional Services

I went to a private, Christian university where vices such as swearing were certainly discouraged, if not outright banned. After registering for my first econometrics1 class and quickly skimming my reading list, I was surprised (and secretly thrilled) to see Joel Best’s Damned Lies and Statistics as required reading. I fell in love; numbers, modeling, and the power of statistics to paint a picture fascinated me. More recently, I have come to love the data that is behind decisions. Bad data, or more precisely, bad interpretations of data, leads to bad decisions. Good data, or more precisely, good interpretations of data, leads to good decisions. This post focuses on bad decisions, next month I will highlight good decisions.

New Coke

In April, Coke lovers like me celebrated (or at least commemorated) the 30 year anniversary of the infamous “New Coke” decision with Bill Cosby, among others, pitching it.2 The Real Coke, The Real Story, by Thomas Oliver, gives a great history of the brand and the decision3. Coke was losing ground, he accurately argues. The now Pepsi challenge, in which consumers were challenged to pick their preferred beverage in a blind taste test decidedly was in Pepsi’s favor. Even in Houston, a city with a 25% market share advantage to Coke, taste test data indicated a very slim preference for Coke (p. 34). Coke needed a change, or so they assumed.

To ensure the success of the new formula, extensive market testing was done. Robert Schindler notes that coke spent more than $4 million dollars “and included interviews with almost 200,000 consumers” in their research4. The result was clear—the new formula was preferred “61% to 39%.” The decision seemed clear as well—discontinue old coke in favor of new. So why did it ultimately flop, being labeled the “marketing blunder of the century”5. Data was used incorrectly.

In any use of data, it is imperative to identify extraneous or lurking variables6 and to attempt to control for them. In Coke’s attempt to deal with the issue, the company failed to quantify “the bond consumers felt with their Coca-Cola,”7 the company states in a retrospective on their website. Emotions, preferences, and tastes are notoriously difficult to quantify and tabulate, but an attempt to control for them is important.

Avoiding a New Coke Sized Mistake

Bad interpretations and skewed data are two simple reasons Twain classically grouped statistics with the two other types of lies—lies and damned lies. As you implement analytics and data driven decisions, be careful of the implications of bad data practices or misinterpretations.

First, make sure your data is clean. Are you seeing exactly the result you want? Data is easy to aggregate and cherry pick to give rosy results. Question anything that seems just a bit too good.

Second, ask the simple question “what else may have an impact on this result.” A colleague of mine taught me the rule of 5 whys; when a decision is made, especially one that may not make a lot of sense, ask “why” five times. If there are five good answers, it probably is a good decision. If not, it will become obvious. I would suggest the 5 “what elses” rule. Keep searching for the what else until you have built a really good model. Generally, I believe that the bigger the import of the outcome, the more time should be spent creating and refining the model. Your final algorithm is likely to be beautifully simple—but the process to get there likely was not.

Third, don’t be afraid to ask for help. It’s not always popular or easy, but even experienced analysts and data scientists get stuck or have their own inherent biases. It can be invaluable to ask a friend or colleague not working on the project to take a look at your numbers and conclusions. There can be great power in a “sanity check,” even if it comes from someone other than a traditional expert!

Fourth, admit when you get it wrong. The conclusion of New Coke’s story, with the re-introduction of Coke classic and phase out of New Coke is a fitting conclusion to these tips. Recognizing a mistake can be painful, but when data leads you down the wrong path, turn around as quickly as possible. Evaluate and re-evaluate the data that lead you to make your decision, and rectify errors. Coke did this, and their “blunder” turned into an amazing opportunity and the company exited the situation stronger than ever—to the point that some call “New Coke” a marketing conspiracy. If your errors can be turned on their head like this, you’re doing pretty well!

  1. Econometrics is the study and application of statistics as applied to economic data
  2. Bill Cosby pitches new coke: https://www.youtube.com/watch?v=yJoocpy7UBc
  3. Oliver, Thomas. The Real Coke, the Real Story. New York: Random House, 1986.
  4. Schindler, Robert. "The Real Lesson of New Coke: The Value of Focus Groups for Predicting the Effects of Social Influence." Marketing Research, December 1, 1992, 22-27. https://search.proquest.com/openview/4c9167b55ca610abd43435db371a1b70/1?pq-origsite=gscholar&cbl=31079
  5. "The Real Story of New Coke." The Coca-Cola Company. November 12, 2012. Accessed June 1, 2015. http://www.coca-colacompany.com/history/the-real-story-of-new-coke
  6. Extraneous or lurking variables are those unidentified or un-quantified variables that have an impact on a study’s dependent variable. Excellent research attempts to control for them.
  7. "The Real Story of New Coke." The Coca-Cola Company. November 12, 2012. Accessed June 1, 2015.
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram