Use DataOps to shine a light on dark data gold

Engineers love data.  Across all sectors, data is an important part of the job that enables effective design and decision making.  It helps determine how products are designed for a given use case or loading, how a solution should evolve in its next iteration through customer and performance data, as well as help assess how an industrial or manufacturing process can be improved to bring about cost savings or improved performance.  And these examples are really just the tip of the iceberg.

For a long time, one of the big challenges with getting this critical data has been connecting IT and OT systems in a way that gets the data in one place. Innovations such as the Industrial Internet of Things (IIoT) and its related technologies help address some of the complexities of collecting data, but contributed to another problem along the way.  We now have more data at our disposal than ever before, from more systems, and this is leading to what we call ‘dark’ unused data.  Engineers and analysts literally have more data than they know what to do with, and it is sprawled across the organisation, and indeed beyond it if they use cloud-based services.

Data is not being catalogued and made discoverable

This mass availability of data creates a host of problems for engineers and the wider organisation from where to store it and for how long, right up to meeting governance and regulatory requirements.  In the middle of all of this, data is not being catalogued and made discoverable.  For data analysts and engineers, that means lost opportunity, incomplete analysis and poor data sets that lack context, and potentially ‘game changing’ data left festering in a dark digital corner.

Engineers are not simply faced with huge amounts of data, they lack the tools to use it effectively, resorting to manually combing a data pile that is constantly growing for context and correlations. But help is on the way – AI and ML-powered data engines are taking on these tedious manual tasks and empowering organizations to make leverage their data to its full potential.

Turning the table on data

All of the areas we’ve talked about so far, make data a burden on the organisation, when it should be exactly the opposite.  The systems are creating pain and friction for the teams that need to be empowered to get the most from the data produced.

The answer to turning the tables on data, so that it is working for the organisation, and not the other way around is to embrace a DataOps mindset as a new approach to data management.  DataOps tackles the wider problem of making data useful through a methodology that embraces technological and cultural change to improve the use of data through better collaboration and automation.  When DataOps is used well it can have an impact across the whole data stack by reducing the costs of data and data management, improving data quality, implementing governance and compliance, as well as enabling analytics solutions that take the heavy lifting out of analysis for engineers by putting extremely powerful self-service tools in their hands.

MORE FROM ELECTRONICS & COMMUNICATIONS

Automation and the use of new technologies such as artificial intelligence (AI) are a crucial part of a DataOps deployment. Why? Because AI and machine learning have shown time and time again that can outperform our human ability to trawl through data and relationships and patterns that would otherwise be missed or take months to find.  Together with techniques such as Robot Process Automation, they can take care of the heavy lifting of finding, categorising, and analysing data sets so that engineers can focus on the impact of the analysis.

Broadly there are five main steps to undergoing a DataOps transformation in your organisation:

1. Assess your technology.  Review and tune your technology portfolio and processes to remove redundancy and improve turnaround time.
2. Consolidate teams.  Encourage teams to share and look for the inconsistencies in their processes that hamper collaboration.
3. Integrate DataOps practices. Implement DataOps practices across your teams and data pipelines.
4. Automate your processes. Automation makes your data pipelines more efficient and your data operations more effective.
5. Empower your engineers. Make sure your engineers have the self-service tools to take data and turn it into information and insight.

Methodology not a task

DataOps can play a pivotal role in helping engineers achieve competitive advantage for their organisations through insights that previously had stood in darkness.  The key to success is to accept that DataOps is not a task, one technology or process.  It is a methodology that needs to be embraced across the organisation, reviewed and refined aiming for continuous improvement.  Approached this way, it will shine a light on the data gold you have hidden away.

Radhika Krishnan, chief product officer, Hitachi Vantara