Skip to content

Are You Using Data To Analyze The Past Or To Create A New Future?

2014 April 23
by Greg
Wally Cleaver

In Nate Silver’s Five Thirty Eight manifesto, he argues that the plural of anecdote is data.  It is through compiling and analyzing observations that we transform ordinary experiences into scientific conclusions.

Yet the concept of data is still relatively new.  According to Google’s Ngram, the use of the word only began to be heat up in the 1960’s.  So, although data is a term that gets thrown around a lot, it’s not something we’re really accustomed to using in a meaningful way.

That’s probably why many feel that data is in conflict with personal experience.  Yet in reality, data is most often used to codify experience.  Historical observations are aggregated and then analyzed to identify salient trends.  And that’s the real problem with how we use data.  History can be a trap.  It’s not the past that we need to worry about, but the future.

The Lure Of Historical Trends

My first real job out of college was on a trading desk, where I encountered the concept of “technical” market analysis.  Often known in the profession as “chartists,” practitioners scour historical data looking for patterns that they expect will give them clues to how things will play out in the future.

I was always amazed at the nearly religious fervor that many of the chartists had about their work, especially the Elliott wave theorists.  It was as if they were not merely performing analysis, but communing with nature and unlocking its secrets.  Chart analysis was a strange blend of the technical and the traditional.

I suspect that is part of what made the technique so popular.  Tradition has a romantic quality to it, as if we’re not merely forging our own path, but honoring the wisdom of those who came before us.  There has always been conflict between the medicine men, who put their faith in ageless insights and the visionaries who wish to break old molds.

The Poverty Of Historicism

In the 1930’s, Karl Popper began to speak out against excessive reverence of the past in a paper called The Poverty of Historicism, which was later published as a book.  He argued that, in fact, there was no reason to believe that future will look like what came before it.

 

World GDP per capita

 
Take a look at the chart above and you can see what he means.  For most of history, little changed, but when the Industrial Revolution came along the world hit an inflection point. It’s hard to argue that many trends in force before the shift remained valid afterwards. Everything had to be called into question.

One of Popper’s students, George Soros, became one of the worlds richest men by identifying similar, albeit smaller shifts.  People like Henry Ford, Bill Gates and Steve Jobs defined entire new industries by breaking with established norms.  It is the visionaries we admire—not the trend followers—because they’re the ones who make an impact.

Why Big Data Matters

One of Popper’s key concepts was that any assertion, in order to be scientifically viable, has to be falsifiable.  Unless it can be tested and proven wrong it was basically nonsense.  We admire great scientists like Darwin and Einstein not simply but because they had an idea, but because they stated it in a way that others could verify or disprove it.

For most of the 20th century, the testing process was deeply influenced by the methods of Ronald Fisher who set out rules for the design of experimentsconfidence intervals and statistical significance, among other things.  Underlying his ideas was an emphasis on controls.  Put good data in and you would get good answers out.

Yet for all of their charms, Fisher’s techniques were still essentially historical.  You collected data over a period of time and determined mathematically whether they were meaningful or not.  You were still counting on the future looking like the past and, if it didn’t, you could find yourself wildly off the mark.

Today’s big data techniques focus less on the past and more on the present.  No longer are we trying to divine the rhythms of history to uncover eternal truths, but using a vast array of sensors and servers to gobble up data and analyze it in real time.  Truth is no longer measured in eons or even in months, but in milliseconds.  That’s why big data matters.

Managing For Uncertainty

In the old days, when executives began their careers they could be reasonably sure that the the rules that guided their industry would stay fairly constant, their skills would stay relevant and their business models would remain intact.  That’s no longer true.

It’s now becoming clear that business models don’t last and skills need to adapt.  We need to start taking a more Bayesian approach to strategy in which we no longer expect to uncover timeless truths, but instead value adaptation over steadfastness, learning over experience and preparation over planning.

History, even if properly construed, can blind us to important possibilities.  After all, it is not the rhythms of the past that define us, but our dreams for the future.

- Greg

 

6 Responses leave one →
  1. Michael von Gonten permalink
    April 23, 2014

    History can not only blind you, it can mislead you. Most of the Big Data analyses are based on correlation and not causality. Without true causality you cannot change the relationships you saw in the past. If change is what you want, you damn well better have causality in hand.

    Barry Richmond, the MIT Systems Thinking Guru says it this way:
    “Correlation is good enough if your purpose is forecasting … and you’re lucky (meaning that the relationships that existed in the past and from which the correlation have been defined, persist). But when your purpose is to change performance, you are explicitly seeking to alter relationships that prevailed in the past, and to create new relationships in their place. For this you must understand causality.”

    [Reply]

    Greg Reply:

    Very true. I hear that a lot as a knock on big data, but it is really true with any data analysis, which is only a small part of the hypothesis forming/testing process.

    - Greg

    [Reply]

    Winfield Gatash Reply:

    “Truth is no longer measured in eons or even in months, but in milliseconds. That’s why big data matters.” Yes, big data matters more now because of its real-time capture. ‘Recency’ is one of the most powerful variables in identifying a target, and its intent. What you want now…is valuable data to a marketer. (And marketers wear coats of many colors these days.)

    Adding exponential value is context, because intent out-of-context is like pissing in the wind. Or put another way, context + recency equals more sales. Or whatever you want. And big data can deliver.

    But the real point here is missed altogether. “Data is most often used to codify experience.” Codify means “to arrange in a systematic code.” That’s pretty good diction, but there’s much more to the story. Big data, this plural form of anecdotes, is effectively creating a proxy for each of us. Big Brother doesn’t have it, all the brothers do. Meet Inc. Meet Co. And so on.

    As these proxies for individuals’ (consumer) behavior become more and more robust, two things happen. One, we get violated. Both en masse, and individually. And two, we get replaced. Play along, or miss the boat, that’s the proposition.

    Woody Allen’s “Sleeper” comes to mind. Soon enough we’ll inhabit avatar-like proxies in the ‘matrix’ or whatever people call it. Japanese birth rates are declining because Japanese men are finding more willing sex partners in these nascent spaces. And porn always enables the future of tech. But, even as this second machine age envelops us, as in “Sleeper,” some will opt out or be opted out. Good luck.

    [Reply]

    Greg Reply:

    Winfield,

    You make some interesting points, but you lost me at the end. While there are dangers of every new technology, the benefits almost always outweigh the costs. For example, big data techniques are being used to optimize our use of resources—energy especially—and will help save our planet. You have to look at both sides of the equation.

    As for Japanese birth rates, they’ve been falling long before big data came along. So have those of other developed countries.

    - Greg

    [Reply]

  2. Pankaj permalink
    April 28, 2014

    Analysing the past is as important.. *Most of the times* you can predict the future just by that, like people committing fraud, going bankrupt etc etc..Instead of predicting something new where you are 99% (say) wrong and maybe are able to predict 1% rare event, you rather predict 99% correct things (with reasonable accuracy) and work hard on 1%..

    At any level (individual or organization) you have to make sure there are no low hanging fruits :-)

    [Reply]

    Greg Reply:

    True, but agility is becoming increasingly important as well.

    - Greg

    [Reply]

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS

CommentLuv badge