The dark side of data – or how companies use only half of their resources

The dark side of data

It has become a popular saying that data is the oil of the 21st century, and information is the resource that helps you gain an edge over competitors. If that is really the case, it seems that every second "barrel of data," even once extracted and ready for use, will never actually be used. According to the latest report from TRUE Global Research, around 55% of data within organisations worldwide goes unused.

The report State of Dark Data 2019, prepared by TRUE Global Research based on a group of 1,300 managers and IT specialists from around the world, focuses on a relatively new phenomenon: Dark Data. What does this term mean?

- Dark Data refers to unknown, unused and unassigned information originating from an organisation, generated by its system, its users, devices and all interactions between them. Due to its nature, it can be a source of unnecessary costs or a source of a range of risks, as well as an opportunity due to its hidden and previously untapped potential. - explains Piotr Prajsnar, CEO of Cloud Technologies, a Polish company specialising in Big Data analysis.

Information blindness in organisations

Information blindness in organisations

One of the foundations of running a modern business is the conviction that data drives everything - from product development through production to the supply chain, and ending with building customer experiences. - Information is a key element in constructing an overall business strategy.

Individual data points are like a grid on a map, allowing us to determine our precise position on the business map and calculate the bearing to our goal.” - explains Błażej Migoń, Product Manager at BPSC, and continues - As research shows, many of today's business leaders are not fully prepared for this revolution. Making full use of the data you have is a difficult challenge even for the largest organisations. The results of the State of Dark Data 2019 report clearly show that for more than half of organisations (56%), the phrase "data driven organisation" is just an empty slogan. So does that mean companies do not know or do not believe in what they could gain? Not exactly - 76% of respondents agreed that "the organisation with the most data will win." - As the results of the TRUE Global Research study show, Big Data is the key to future success in business. More than two thirds of organisations (71%) expect data to become even more important, and therefore more valuable, over the next 10 years. - notes Piotr Prajsnar of Cloud Technologies, and adds - Big Data has become a key advisor for the most important business decisions in an enterprise. And intuition? Today, few people take that risk.

So where should we look for the real cause of such wastefulness when it comes to information? Respondents focus mainly on two factors. The main obstacles to using Dark Data are: the sheer volume of data itself (as the OnAudience report shows, in just the last two years the data market has grown nearly twofold) and the lack of the necessary skills and resources.

Filling the gaps

Filling the gaps

Despite the growing importance of data, representatives of the companies taking part in the study openly admit that their organisations do not have the necessary resources, tools, and above all the skills to handle the abundance of data they could potentially use. According to the expert from BPSC, Dark Data should not be treated dismissively - quite the opposite.

- A company cannot forget about Dark Data, because it is not useless. It is a substantial and valuable source of information that can be used to unlock additional business opportunities. The more data you have, the more you can learn about every aspect of your operations. The larger the dataset, the more accurate the analysis. So if an enterprise decides to ignore this information, it is only making things harder for itself. - notes Błażej Migoń.

However, business already has its own ideas about what steps should be taken to deal with the gap that has emerged between what a company has and what it could have thanks to the proper use of the information it holds. Respondents in the State of Dark Data 2019 study were able to declare what they intend to do to improve their position and the degree to which they use data. The largest group, as many as 76%, wants to better train their employees in Data Science and analytics. Only one percentage point lower was the use of systems supporting data use, such as a DMP platform or ERP-class software. Interestingly, in 4th place, with a result of 71%, was the use of solutions based on artificial intelligence.

How to make full use of the data you already have in your organisation?

In 2019, data analysis became an essential part of development for every entity. According to the International Institute for Analytics, by 2020 companies using data will gain $430 billion in productivity benefits compared with competitors that do not use data. Entrepreneurs, managers and small business owners have realised that the conscious use of data is not merely a fashionable topic, and have begun to see the potential of Big Data, Dark Data and AI in growing their business.

Terms from the article

AI — Artificial Intelligence. A set of methods and systems performing tasks that usually require human reasoning, learning, perception or content generation. In HEXX AI CMS it includes language models, document analysis and automations.

Why it matters: It takes over repetitive tasks — transcriptions, descriptions, translations, selection — and shortens the path from idea to publication. In a newsroom, it means more time for the work a machine will not do: judgement, contacts, decisions.

When it is used: For generating and editing content, document analysis, transcription, translation and automating the flow of materials.

What happens if you skip it: Without AI, the team does bulk work manually (photo descriptions, language versions, summaries), while competitors publish faster and more cheaply.

Summary

Companies use only around 45% of the data they hold — the rest, referred to as Dark Data, remains unassigned and unused. According to the State of Dark Data 2019 report, the main barriers are the enormous amount of information and the lack of competences and tools. Experts from Cloud Technologies and BPSC stress that ignoring these resources means wasting competitive advantage.

Importantly, as many as 76% of organisations plan to invest in training employees in Data Science, and 71% are betting on solutions based on artificial intelligence. According to the International Institute for Analytics, by 2020 companies using data could gain $430 billion in advantage over competitors.