To overcome the disconnect between business and IT, enterprises need to employ Data Product Managers – and a data management platform based on real-time data products.
Table of Contents
Understanding the Data Dilemma
The Data-as-a-Product Approach
Introducing the Data Product Manager
Why You Need a Data Product Manager
Adopt a Platform Flexible Enough to Support Any Architecture
Understanding the Data Dilemma
Today, nearly every company considers itself to be “data-driven.” However, most companies’ data architectures don’t enable the level of democratization and scalability required to actually leverage their data assets.
3 main challenges continue to stifle organizations’ attempts to become data-driven:
-
Data consumers lack access to the data they need
-
As a result, business units must rely on data teams, data engineers, and IT to deliver the data critical to generating business insights
-
The proliferation and complexity of big data slows down data delivery and hinders business success
To overcome these obstacles, enterprises should make 2 key changes:
-
Employ data product managers, to maximize the business value of their data and close the gap between business data consumers and IT
-
Adopt a data management framework based on data products, that improves data trust, availability, and accessibility to improve agility and speed in running the business
In this article, we will examine both of these concepts, and how tackling them together empowers enterprises to become truly data-driven.
The Data-as-a-Product Approach
Before discussing the data product manager's role, let's first look at what a data product is.
A data product is a ready-to-use data asset that is created, managed, maintained, and delivered for consumption by authorized data consumers, in serving a particular purpose.
A data-as-a-product approach enables data teams – in any business domain of the enterprise to easily access and publish data – to ultimately make timely and better-informed business decisions.
A data product contains all the data that a user needs in order to support an operational or analytical workload, and is typically related to one or more business entities (such as a customer, vendor, credit card, campaign, lead, or order).
The data product integrates, unifies, enriches, and synchronizes the relevant data from the underlying source systems – regardless of their technologies, formats, and structure – and delivers it securely to data consumers, in real time or in batch.
The data product lifecycle
A data product approach assumes that data is managed in a lifecycle, similar to that of agile software delivery. It is comprehensive and iterative, with the objective of delivering rapid, incremental value to data consumers. It includes 5 steps, that are closely managed throughout the process.
-
Definition and design
Requirements are defined according to business objectives, data privacy and governance constraints, and existing data asset inventories. Design corresponds to how the data will be structured and componentized as a product. -
Engineering
The engineering process involves identifying, integrating, and collating the data from its sources, and then tokenizing or masking it as needed. Web services APIs provide consuming applications permission to access the data product, and pipelines deliver data to data consumers. -
Quality assurance
The data undergoes testing and validation to ensure that it’s complete, compliant, and fresh, and can be securely consumed by applications at massive scale. -
Support and maintenance
Local authorities and data engineers continually monitor data usage, pipeline performance, and reliability, so they can address issues as soon as they arise. -
Continuous management
Data product managers oversee the data product lifecycle to ensure data consumers’ needs are being met, while maximizing business value and ROI.
Introducing the Data Product Manager
Data product managers are responsible for the entire lifecycle of data products.
They have 2 key roles.
First, they are responsible for defining how the enterprise can utilize its data in a way that maximizes value, aligns with the broader business strategy, and provides RODI (return on data investment). They build strategies to bring data initiatives to life, establish performance metrics, and champion data literacy across the organization.
Second, they serve as liaisons between business stakeholders and IT. Data product managers are responsible for communicating the needs of data consumers across business domains, and working with data engineers to improve the accessibility of data across the enterprise.
Just as a software product manager is responsible for defining user needs, prioritizing them, and working alongside R&D to ensure delivery, data product managers clarify the needs of data consumers, break them down into well-defined tasks, and work alongside data engineers and scientists to deliver on them.
As the name suggests, data product managers are the ultimate data definers. They’re also the primary champions of data products, which enable real-time, self-service data delivery to data consumers.
Why You Need a Data Product Manager
Data product managers help you close the widening gap between your company’s desire to be data-driven, and the ever-growing amounts of dark data being produced.
They facilitate collaboration between data consumers and IT, by capturing data consumers’ needs, setting priorities between them, boosting data literacy across the enterprise, assuring data security and compliance with data masking (via data masking tools) when needed, and alleviating the burden on IT to handle an unending stream of data queries and requests for business insights.
Data product managers also play a critical role in guiding the transition to sophisticated data management platforms that enable the democratization, scalability, and easy accessibility of data.
Adopt a Flexible Data Product Platform to Support Any Architecture
A data fabric architecture is modular, and integrates with a company's existing data and analytics tools. Although it’s centralized by design, a data fabric may be deployed as a distributed network of data fabric nodes, to support high scale and high availability.
A data mesh architecture shifts data strategy to real-time data solutions via a federated data architecture. It provides business domains with autonomy and tools to model data according to their business needs – to simplify and accelerate data delivery – and creates a common framework for creating, and scaling, data-driven solutions, in real time.
There are pros and cons to consider when evaluating data mesh vs. data fabric. And both of them require different skill sets of data product managers. More on that in another article.
With a Data Product Platform, the role of the data product manager becomes instantly more productive, and less tedious, thanks to a combination of real-time, self-aware data products and automation. It also requires less technical expertise, since the platform itself is designed to deal with the underlying data connections and system complexities.
