Big Data’s Big Divide: the existential threat faced by analytics adoption laggards as the technology takes off.

By Tom Wright, Incisive Media Enterprise IT Publisher


Big Data Photo

“The greatest shortcoming of the human race is our inability to understand the exponential function”

…or so said Albert Allen Bartlett, emeritus professor of physics at the University of Colorado.

Bartlett was talking about population growth, resource depletion, and sustainability, and as long ago as 1969, but it is a truism that is widely applicable: human beings are hardwired to expect change to happen at a constant rate, with our experience of the past shaping our expectations of the future.

When change is exponential, when it accelerates by an order of magnitude as it does with Big Data, the hardwired human model breaks, leaving many organisations ill-prepared as change happens more rapidly than expected. In today’s world of data driven decision making and fast changing business models, this natural human flaw is an existential threat for technology adoption laggards – as the Chief Data Scientist of a UK bank told us in a research interview ‘“Everything is driven by technology and technology’s changing exponentially, so if your thinking’s not exponential, then you’ll go out of business…”

Is this hyperbole? Is change really happening that fast? Sadly for the laggards, our research bears this out: leading enterprises at the cutting edge of data analytics are already making better business decisions and they can prove it, they have ring-fenced budgets, they are far more likely to be using external data sources as well as internal for predictive analytics, and they are even working on different platforms, leaping from Hadoop to Spark for real-time streamed analytics. Analytics is paying off for them with better business decision making and better business models: its working, and its working now.

Laggards think they have time to play with, but progress at the cutting edge is advancing so much faster than mainstream adoption that they may never catch up. So fast in fact, that Geoffrey Moore’s famous technology adoption diffusion model – the model we’ve all lived by for decades – is at breaking point.

Moore 1

This is tech marketing 101, part of our collective understanding of how technology changes. First you have the innovators and early adopters, those who take on a new technology knowing it is unproven and risky, but seeing the competitive advantage that comes with being a ‘first mover’ – the chance to do something better than the competition and seize markets. If that technology succeeds and competitive advantage is realised, then the more pragmatic buyers of the early majority follow, and the technology crosses ‘the chasm’ into mainstream adoption.

In this model, the laggards actually catch up. That’s all well and good when you’re talking about consumer technology like cellphones, tablets and smart TVs where your worst case scenario is being the last person able to get Netflix on their watch. With big data analytics it is not so good: its possible – even likely – that rather than the whole market crossing the chasm, the chasm will get wider and harder to cross and for some will not make it at all. As harsh as it may sound, laggards genuinely do risk being put out of business, because, as Bartlett observed 47 years ago, they can’t imagine exponential change.

moore 2

Last week I had the privilege of presenting the findings of Computing’s research to a packed conference on Big Data Analytics – with 300 people in attendance. If you’d like to learn more about what sets big data leaders apart, or what Computing Research can do for your content marketing, reach out to me.

You can download the full report here.

Thanks to our event sponsors, Darktrace, Exasol, IBM, Splunk, Condusiv, Experian, and in particular to our research sponsors, Michael Page, MTi and Splunk, who made our research possible.