Shining a light on dark data

Shining a light on dark data

In today’s society, it is simply impossible to ignore a rapid and seismic transformation occurring that’s due to change the way all industries and sectors work. This new wave of transformation is being driven by technology and digital data that some might describe as the industrial revolution 4.0. That is, in short, the entire combination of digital, technological, and repeatable methods of producing work through machines or robotics.


This is, of course, a subject that I am not only vested in but passionate about. But I have two major considerations around this subject;

  1. what will the future look like?
  2. how do we get there?


In healthcare, the upside to this revolution is the potential to deliver huge value to society and further improve the day-to-day running of the healthcare systems like the NHS. The downside is the uncertainty this provides around jobs and the reskilling needed to manage a new working environment, issues around security and noncompliance. With the influx of data being generated, the NHS is particularly susceptible to change, especially if the upside is an overall better outcome for the health service.

In my mind, one of the key aspects of embracing this transformation in the digital age is how an organisation can harness their data in a meaningful way for deeper insight and the provisioning of new methods to streamline doctors, or reduce A&E waiting times for example. I recently spent some time with a few NHS COOs and CEOs about machine learning and AI. The focus was how these trends could address A&E waiting times and go a long way in improving patient access and flow. Machine Learning or Artificial Intelligence (AI) will undoubtedly be one of those ‘once in a lifetime opportunities’ that I would rate alongside the advent of the personal computer. I guess it isn’t every day that you get a ‘twice in a lifetime opportunity’ and all business leaders should be ready to work in this new environment.

The genius behind machine learning is the potential that it will be able to tell me something I didn’t know or even consider because it has continuously scanned for patterns, correlations and meaning from the data it has available. This means it looks at outcomes from its own actions to improve its capabilities. The most remarkable feature is its ability to take action on its own through automation.  This means a machine can process huge sets of data and provides prescription and prediction through smart algorithms. But this doesn’t happen by magic, but why isn’t it more mainstream today? This in part is because of the data available to any firm, too much is dark.

Dark data is described by Gartner as “the information assets organizations collect, process and store during regular business activities, but generally, fail to use for other purposes”. And as a CEO of a technology firm, this is a real frustration to me. To harness data in a meaningful way could deliver new economic opportunities and save lives.

In the US, a McKinsey report entitled The age of analytics: competing in a data-driven world stated that in 2011 a report “outlined $300 billion worth of value that big data analytics could unlock in the US health-care sector. To date, only 10 to 20 percent of this value has been realized.” This shows the challenges faced in galvanizing workforces to change models and processes”. Gartner also predicts that through 2021, more than 80% of organizations will fail to develop a consolidated data security policy across silos, leading to potential noncompliance, security breaches and financial liabilities.


In order to take advantage of the benefits the industrial revolution 4.0, it will be those organisations that are most data-centric that will have a greater chance of success. It is important for organisations to find trusted technology partners, or data insight experts to guide them through this challenging time. To make the most of this dark data, or perhaps to shine a light on it, will be the ultimate proofing needed for enabling machine learning, deep insights and new business processes. The future will look different, and we will get there through necessity and for visionary leaders who realise its potential.

Read here about the predictions on AI and machine learning for 2018.