Read these data mesh use cases, to learn how a novel data product approach unifies globally dispersed data, while fulfilling data residency requirements.
Table of Contents
Managing the Data Deluge with Data Mesh
Defining the Requirements
Operationalizing Data Mesh with Data Products
Defining the Data Mesh Architecture
Optimizing Data Mesh with K2View Data Product Platform
Managing the Data Deluge with Data Mesh
One of the world's largest account firms, with hundreds of offices and affiliates around the world, hundreds of thousands of employees, expertise in multiple industries, and a range of service offerings, is nothing less than a global powerhouse, which servers thousands of global customers. But in the digital age, this kind of scale creates significant data compliance and governance challenges.
The rising rate of data generation, combined with an increasingly stringent data regulatory environment, make managing large volumes of globally dispersed data a major issue.
The enterprise needed to:
-
Unify, and visualize, customer data from disparate source systems, around the world. Many of its customers are served by multiple offices, located in different geographic regions, each with its own data platforms and technologies.
-
Comply with all applicable global, national, and regional data privacy and security regulations, while allowing for data sharing – conditional to privileges policies governing each affiliate.
After deciding to implement data mesh, and evaluating the top data mesh vendors, the company took a data product approach.
Defining the Requirements
Within the scope of global data unification and data residency compliance, the enterprise identified 8 key requirements that its data mesh solution had to address:
-
Unification of fragmented data
A main imperative for the company was gaining the ability to aggregate customer data across a wide range of data sources, from different geographies, technologies, and formats. The company required a solution that could support data unification on a global scale. -
Data residency
Data residency refers to the geographic location where a company’s data is physically stored. The organization’s objective was to ensure the sovereignty of the company’s data according to each country and region in which it operates. -
Anonymization and masking
The solution needed to be able to provide on-the-fly data tokenization and data masking, at the field level, to support security needs and profiles. -
360-degree customer view
Customer 360 functionality – providing a single, complete, and trusted view of each of the organization’s customers – enriched with operational real-time insights, needed to be accessible to authorized data consumers in milliseconds. -
Data privacy compliance
Data privacy compliance relates to the regulatory requirements imposed on customer data, based on the relevant laws of the country or region in which the data resides. For example, GDPR maintains that any data collected in Europe, be stored in Europe, while US laws prevent workers abroad from viewing the social security numbers of American citizens. -
Disparate systems and technologies
Every one of the company’s offices and affiliates used different technology to store and manage customer data, each with different implementation processes in place to support its particular business needs. The company would need a single platform that could aggregate data from a wide range of source systems. -
Data lineage and profiling
Since offices and affiliates in different countries and regions would have data stored in different data sources and data formats, the enterprise required a data catalog that could support lineage and profiling of the data elements. -
Data governance
Although each of the company offices and affiliates operates independently of one another, global data governance – in the form of a centralized set of policies and rules – needed to be enforced as well.
Operationalizing Data Mesh with Data Products
A data-as-a-product approach covers a variety of data mesh use cases, among them data unification, federated governance (via data governance tools), self-service ownership, and data residency.
To address these data mesh use cases, the company implemented data products to:
-
Integrate and expose customer data at the geo level – by office, and by country – for federated data management
-
Govern data access and movement
-
Aggregate the data products, from the individual locations, leveraging data virtualization, to provide a global view of the data to headquarters for consolidated financial reporting
To ensure compliance with regional and national data regulations, the local domain data products cannot share data with each another. Only the central data mesh can aggregate and expose the data, according to predefined roles and permissions.
In data mesh uses cases, data sources sync with the data products in real time.
Now, the enterprise has the ability to integrate data – despite differences in source system technologies, formats, and taxonomies – while adhering to regulations for data privacy management.
Data Mesh Implementation
Here’s a step-by-step look at how the data mesh was implemented:
-
Data product definition
Each regional or national domain defines a local set of data products, corresponding to business entities, such as customers or employees.Data discovery capabilities enable business users to build the data products quickly, and with minimal technical assistance. -
Integration and masking
Domain experts employ data integration tools and data masking tools to comply with regional privacy regulations. Data virtualization tools are used by HQ to gain a consolidated view of the data products, without physically moving the data, to conform with data residency regulations. -
Role designation
Each domain defines the roles that are permitted to access its data products. Centralized administrators at headquarters determine who in the company, across its various offices, should be assigned to which roles. -
Global governance
Regulatory authorities require that personal employee information never leave its country of origin.
Putting data mesh into action empowered the accounting firm to (1) provide accurate, unified, and complete data to authorized data consumers, worldwide, and (2) comply with data residency and data privacy requirements, at the same time.
Optimizing Data Mesh with Data Product Platform
The need to unify decentralized data across a global enterprise, while remaining compliant with local data residency requirements is not unique. Every global company, that collects, stores, and uses data in different regions, faces the same challenges.
An agile and low-risk approach to fulfill these needs is with a Data Product Platform.
Many organizations, though, are not yet mature enough – in terms of their federated data management skillsets, tools, and best practices – to deploy an operational data mesh. For such organizations, a Data Product Platform could first be implemented as a centralized data fabric architecture or data hub architecture, and systematically phased into a federated data mesh architecture. It delivers data as a product – a fundamental principle of the data mesh concept – in which all data for a given business entity is organized and managed in a patented Micro-Database™.
The result is governed data integration and delivery, at global scale – with built-in support for data residency and sovereignty requirements. As the stakes of a data breach or non-compliance rise, multinational companies need a solution that can automatically enforce regional data requirements. With the ability to configure data anonymization and access permissions, Data Product Platform allows enterprises to ensure security and privacy compliance according to each locality’s rules – rapidly and confidently.
Equally important to the company were the platform’s rapid implementation, and support for on-premise, cloud (iPaaS), and hybrid deployment modes.
