Did Big Data Over Promise and Under Deliver?

Here’s the truth about big data: it’s a means to an end, not the end in itself.

Big data is nothing more than a huge collection of bits and bytes. It is only when you are able to analyse it, and gain some value from it, that it is actually of any use to you in your organization.

It is the insights businesses can derive from these bits and bytes of information, such as reducing customer churn to increase loyalty in the telco space, identifying buying trends in retail stores to maximize profitability or product placement, or even running algorithms to predict and mitigate pipeline failures in the oil and gas sector, that hold the promise of significant value.

Before we talk about whether the concept of big data has over-promised or under-delivered, I believe it is critical to remember this one point: Big data is not about the information being collected, it’s about how businesses can transform that information to speed decision making.

RELATED: The Big Data Promise

One of the first decisions a company looking to start a big data initiative will inevitably face revolves around what data exists and how much data to keep. Applying too many filters and narrowing down your raw data too early may be detrimental down the road. 

Let’s say a few years from today, you need to investigate a very specific buying pattern. You then discover the information you were wanting to investigate isn’t there because it was thrown away (or filtered out) years ago. That lost data could have helped you derive important insights.

In reality, you can solve data storage concerns in an economical manner, but it’s equally important to contrast the storage costs against the value you could realize from some of this data.  And since businesses are continually evolving, it is very difficult to pinpoint what will or won’t be relevant into the future. Investing in more storage will be less expensive in the long run than missing out on an opportunity that could be uncovered through future data analysis.

With companies starting to collect this big lake of data, how do you jump into the deep end without drowning in random details?

The short answer is partnership. Bringing together the right people can help you take that lake of data and extract the droplets of critical information that can help you to identify trends or reshape aspects of your business.  If you don’t have the internal expertise to derive the insights you need from the information you are collecting, seek help.

I believe partnerships are a business reality of the future and a business model we will see play out in many different forms as companies strive to be more competitive, bring new services to market and respond dynamically to changing customer demands. Businesses, particularly SMBs who form a large percentage of the Canadian economy, may struggle to develop internal skills in all the areas that are becoming mission critical including security, big data analytics, cloud computing and the Internet of Things.

Best in class partnerships can give smaller companies a leg up, give them access to much-needed expert resources and help them progress more rapidly along the path of extracting meaningful information from a lake of data.


If there’s one downfall to big data initiatives in many organizations it is that they’re too small. Quite often, big data projects are launched by the IT department as a bit of an experiment; a trial of how data can be collected and used. This narrow view is precisely why they will fail. 

To be a truly useful process, lines of business, sales, marketing, and departmental leadership teams  need to brainstorm with IT on areas where having deeper insights could help radically benefit the business. For example, knowing why customers behave in specific ways, purchasing patterns that secure or lose the sale or even what communications methods keep customers engaged with us.  There should be a clear business proposition, established at the outset, for big data project that goes beyond an IT-driven initiative. 

You need to think big about the opportunity this could present to transform your business by leveraging the insights locked away in your data.

It might be a scary proposition for some. There is an enormous amount of information out there and the prospect of trying to analyse and understand it all could be daunting, but I would argue we’re reaching a stage where to stay progressive, businesses don’t have a choice.

Businesses today have three options: You can sit on your hands and enjoy the fruits of your labour changing nothing around your business (and hope nothing new appears to challenge your stagnant offer), you can drive change yourself, or you can wait for someone else to come along and change your business for you (which may leave you on your back foot trying to catch up or going out of business).

I’ve always believed in controlling the future so I guess you could say I like to be in the driver’s seat where change is concerned. The best time to drive change in your business is when you are successful as opposed to when you are under competitive threats and your business model is being challenged. 

When you are talking about big data analytics, the information revealed can be truly transformative for the company willing and able to gain those insights.  The technology is available; the only barrier is a willingness to begin. Investing in big data is not a prohibitively expensive activity as long as you couple the initiative with clear objectives and an eye on areas where your future business success could derive benefit from deeper insights.

Imagine as a telecommunication company, you can see the patterns leading up to customer churn and can take action to identify then intervene before customers make the decision to move to another carrier.  That could be very valuable to a business.  Or if in the pipeline industry, they can analyse sensor readings and identify the patterns that signal pipeline failures so they could initiate maintenance or shut down processes sooner to reduce spill risks. In the financial sector, credit card companies have done a lot of work to identify fraud but refining purchase pattern recognition could further mitigate the huge losses facing this industry.

We have a unique made-in-Canada example of big data at work, and it’s in the agricultural sector. GrowSafe Systems has developed a solution that helps producers track and analyse livestock data to maximize growth, but also identify potential health issues sooner so they can treat the animal faster.

All these examples have one thing in common, it’s not about what or how much information is being collected; it is about the insights derived from that data.

We are really at the tip of the iceberg. The amount of data we will be gaining access to is growing exponentially through the increased adoption of wearable devices, internet of things and the evolution of technology that is enabling us to collect more raw data.

The opportunities are boundless for where this data, if we can tap into its insights, can lead us.