In recent years we have been leading and supporting the journey of several clients towards the adoption of Big Data, materialising its famous 3 (4?, 5?.. 7?) V’s in a variety of use cases such as Real Time Analytics, Regulatory Reporting, Complex Events Processing or Data Linking. I can proudly say that at everis we have honoured a lot of Big Data promises.
But sometimes this journey came with a small price. In the past, the market has been flooded with hundreds of distributed and scalable data solutions, increasing the complexity of enterprise data platforms, and requiring highly skilled data teams to understand the technology, manage the platforms and build the data applications. The industry has not invested much time and effort into understanding and mastering the technology, and probably have not paid the same attention and care for data and how we manage it.
Well… the reality is that data has always been a little mistreated in our Enterprise platforms. First, with “classical” Enterprise Datawarehouse ecosystems, we have different business areas owning their own platform and central BI teams building limited use cases on rigid data models, ending up with silos of data which cannot be easily shared through the organisation. Then, with “modern” Data Lakes built on top of Big Data, we have huge monolithic platforms filled with loads of new and flexible data sets coming from diverse business areas and managed by central technical engineers.
In any case, data is owned and managed by centralised engineering teams external to the core applications, with no specialised knowledge in the data domain, undertaking complex data modelling, transformation and analytics projects with strong dependencies on the business teams’ data knowledge. This reality demands heavy waterfall projects with a strong initial definition phase or ineffective Agile initiatives where most of the delivery effort is assumed or lead by the central data platform team.
So this does not feel like the best way to take advantage of an organisation data knowledge. Instead, we should think of a new approach based on bringing back the control of the data to the business areas. Provide them with a distributed, collaborative and governed environment to build, manage and share their own data products and applications autonomously, minimising their dependencies with centralised platform teams, whose role should be focused in providing and running Data and Platform Services and support.
However, this is not a new concept at all. For many years we have built Enterprise Architectures to allow business areas deliver their own Core and Operational solutions, providing methodologies and tools to reuse code and speed up development. These frameworks have been enhanced and optimised during the last decade using DevOps practices, eventually enabling business areas run their own multi-disciplinary Agile teams to deliver their operational solutions more and more autonomously.
Therefore, let’s apply this successful philosophy to also build data products. An approach based on decentralisation (Data Mesh) versus centralisation (Datawarehouse, Data Lake), adapting the data management to the global and distributed nature of the organisations and, at the end, enabling Agile delivery of data products and applications lead by business. A data management strategy is based on the following pillars:
- A distributed Data Mesh, modelling the data platform as a collection of distributed data domains, each belonging to different areas in the organisation, but able to speak with each other and share larger and richer data products.
- A modular, flexible, distributed and Scalable Architecture, capable to implement any required and future use case using the most appropriate technologies for each product.
- A strong DevOps framework, providing Self Service Data and Data Platform to enable distributed teams build their data products autonomously, minimising Development and Operations activities by the Data Platform Team.
- A Centralised Governance framework, involving a strong Data Governance to control the distributed ecosystem of data and an effective Platform Management to guarantee the proper functioning of the Services provided.
The journey towards a true data driven organisation is not an easy one, and it is definitely not just about technology, but also about new and better ways to work.
At everis, we really think this distributed approach is the best way to go, and this is why we have been working on it for quite some time, investing passion and effort to get the best from Cloud, DevOps and Big Data to build new assets in order to implement it. Solutions to provide our clients with the appropriate technical atmosphere to implement this strategy, which will definitely enable and accelerate the Agile delivery of better data products and applications.