Panning for ‘right data’ gold in the Big Data torrent


Enterprises must learn to filter the Big Data torrent and present business users with a golden subset of actionable ‘right data’ writes Infor’s Kevin Price

Gold panning with a sluice box in a river © hildeanna - Fotolia.comWe all know what ’Big Data’ means – data sets that are so large or complex, they can’t be managed with traditional data processing applications. Management of Big Data highlights issues around data capture, analysis, search, sharing, querying, curation, storage, transfer, visualization, updating, and privacy

We also all know the importance of Big Data. McKinsey predicted five years ago:

Big Data … will become a key basis of competition, underpinning new waves of productivity, growth, innovation, and consumer surplus.

McKinsey also noted the implications of Big Data would reach far beyond data managers – the specialists responsible for data care and management – to affect the most senior leaders in every business sector. This proliferation would be thanks to the rise of multimedia, social media, and the Internet of Things (IoT).

The prediction got it right. A Google search today of the phrase ’big data’ produces 2.1 billion results that run the gamut from news reports to product pitches.

With technological advances and greater affordability in alert management and monitoring, we can now collect every kind of available data before making decisions. Yet the challenge of making sense of Big Data in a timely fashion is still a hurdle. More does not necessarily equal better …

NPR reported recently that biomedical researchers can’t keep up with the information flow being generated by scores of megaprojects. The data is costing billions of dollars to collect, but the investment in analyzing the data is nearly nonexistent. ’Drinking from the firehose’ is a phrase you probably know all too well – the sheer volume of information now available can be overwhelming.

Moving from Big Data to right data

One way to get to the right data is to bring it to the surface – in time to make a difference. It’s the digital equivalent of panning for gold. Truly useful data nuggets are buried within the torrential stream of data you collect. Finding those nuggets quickly and making them accessible to the people who need to act on them is the goal of ’right data.’

Right data has been defined as ’the golden subset’ of information that is left after extraneous data, bias, noise, and spam have been eliminated. Right data shows whether completed projects have been successful as compared to their goals, determines if there is a need to course-correct on current initiatives, and uncovers new opportunities. In short, it guides business success. So how do we find this gold?

Consider the case of CERN, the European Organization for Nuclear Research. CERN oversees the world’s largest and most complex machine, the Large Hadron Collider (LHC), and its 1.8 million critical assets—buildings, tunnels, caverns, roads, car parks, electricity, water, cooling and ventilation, access control, machine tools, lifting equipment, its accelerator complex, supra-conducting magnets, cryogenics, controls equipment, electronics, and radiation monitoring. The amount of physics data stored daily is 100 terabytes.

CERN’s asset management software is the central hub for both technical and financial management of physical assets. It serves as the filter that enables critical decision-makers to get to the golden subset of data they need. CERN integrated its asset management system with more than 20 other systems, including document management, control room, radioactive tracing, manufacturing, ERP, SCADA, GIS, reporting, and IoT-networked equipment.

Making right data actionable

We have to go a step beyond even the right data, to the idea that data is only useful if people will use it. Yes, that seems obvious. But to belabor the point, once you’ve winnowed the Big Data down to the right data, it further has to be presented in a format that is digestible and actionable.

For example, I recently got an email from a friend who was promoting a worthy cause. I agreed with everything in the email and wanted to lend my support, but there was no immediate way to do that, short of forwarding the email to others, who would likely end up with my same dilemma: What’s an easy way for me to help? The key word here is ’easy.’ There was no form letter I could send to my state representatives, no crowd-fund link I could click, not even any social media sharing buttons. That simply won’t fly in today’s digital environment. If you want participation, you have to make it easy for people to participate.

The same is true in getting people to embrace new software. It has to have a simple interface, be easy and intuitive to use – something that helps us do our jobs faster or better instead of adding to our tasks. That is the promise of great software, and it is the expectation of customers – data made useful, and in a way that ‘data mining’ becomes ‘data visibility.’

A tall order? Yes. But today’s digital world requires nothing less if we are to find – and use – the hidden gold within our data.

Image credit - Gold panning with a sluice box in a river © hildeanna -