What is a Data Product?

Today, data is produced at an unprecedented rate, due to the staggering amount of digital services and offerings, combined with ubiquitous Internet connectivity. At the same time, data is a company’s most important asset, and critical to business success.

McKinsey: Data-driven companies are

23x more likely to acquire customers
19x more likely to be profitable

Even though 90% of the world’s data was generated in the past 2 years, enterprise data continues to be managed in data silos, and liberating data from these systems is now the biggest obstacle to delivering data-driven outcomes. This is because enterprise data is:

• Fragmented in 100s of systems

• Captive in vendor-owned applications that lack a rich API set for data access

• Locked within in-house legacy systems, with little or no knowhow of the underlying data models

• Variably structured (or unstructured) in dozens of technologies and formats

• Non-compliant, containing sensitive personal information, which must be anonymized to adhere with regulations (GDPR, CPRA, HIPAA, etc.)

Dark data

Over 80% of enterprise data is “in the dark”, in the sense that it's inaccessible and not being used – to drive business decisions or to improve customer experiences or operational efficiencies. It's only weighing companies down.

A data product is a reusable data asset, engineered to deliver a trusted dataset for a specific purpose. It integrates data from relevant source systems, processes the data, ensures that it’s compliant, and makes it instantly accessible to anyone with the right credentials.

A data product shields data consumers from the underlying complexities of the data sources – by decoupling the dataset from its systems, and making it discoverable and accessible as an asset.

A data product generally corresponds to one or more business entities (customers, suppliers, devices, orders, etc.) and is made up of metadata and dataset instances:

Data product metadata

• Static metadata, including the tables and fields used to capture the data product's dataset

• Data connectors, for ingesting and delivering the required dataset from source systems to data consumers (via Kafka, JDBC, CDC, data services, ETL, messaging, or virtualization)

• Sync rules, defining when and how the data product syncs its dataset with its sources

• Business logic, to process, mask, and enrich the raw dataset, prior to delivering it

• Data governance, to ensure the dataset’s quality and privacy are enforced according to internal and external regulations

• Active metadata logs, for capturing data product performance and usage statistics

• Access controls, including authentication and credential validation

Data product dataset

• Managed as a unit, simplifying data processing and access

• Complete, fresh, clean, and compliant – integrated, cleansed, masked, and enriched

• Stored, cached, or virtualized

• Audited automatically, to record every entry and change to the dataset

• Accessible by any authorized data consumer

Data products are built, versioned, tested, deployed, and monitored, to ensure their ongoing value to the people and systems that use them.

Data products are engineered to drive specific business outcomes, through operational and analytical workloads. Here are examples of some common data product use cases:

• Predicting a customer’s propensity to churn, in real time, immediately before a customer service interaction
• Pipelining inventory data from chain stores into a cloud data warehouse for BI analysis
• Preparing a masked test dataset with data masking tools, and integrating it with a CI/CD pipeline, before launching a new version of a wealth management software system
• Tokenizing sensitive customer data prior to AI/ML analysis
• Delivering a consolidated, real-time, and holistic customer dataset to a CRM application, including customer transactions, interactions, and master data.
• Publishing the latest updates on the spread of COVID-19 to an HMO’s patients in high-risk areas
• Moving a legacy application’s data into a new cloud computing environment safely and quickly, while ensuring business continuity

While data products are often associated with analytical workloads, they are just as important to operational workloads.

Data-driven enterprises have one thing in common: they build data products, as opposed to one-off data projects. Data products are reusable assets focused on business outcomes.

Every data product follows a lifecycle, similar to that of a software product, to iterate and assure that it delivers the desired business outcomes. It looks something like this:

Define
A data product is defined by its business objectives, governance constraints (security and privacy), and data asset inventories. Its design is a function of how the data is to be productized, for consumption via services.

Engineer
A data product is engineered by locating, accessing, and integrating the needed source data, and then processing it as required. Data services are created to provide consuming applications with access to the data, while data pipelines are engineered to deliver the data to authorized analytical data consumers. The data product is versioned and designed to comply with performance SLAs.

Test
Data products only add value once they’re run in production. But, before that can happen, they must be tested to ensure that the datasets they deliver perform as expected, and are fresh, cleansed, complete, compliant, and ready for high-scale consumption.

Deploy
The data product is deployed, monitored (for usage, performance, and reliability), maintained, and supported – to quickly address any issues that may arise.

Enter the data product manager
Similar to the software product manager, a data product manager is responsible for delivering business value and ROI from the data products – defining their goals and priorities (together with data engineers and consumers), and continuously working to ensure that the promised value is attained.

Why the cycle?

Data teams are constantly experimenting – implementing new services, deploying them, and monitoring the results. The quicker they go through the cycle, the quicker they learn, and the quicker they deliver incremental value to their customers.

Traditionally, most companies are project-driven when it comes to data.

For example, if a business domain requires a particular dataset to address a particular need, it typically raises a request with the central data engineering team. That request represents a project to identify, collect, prepare, and deliver the relevant dataset to the business domain. This same pattern is followed every time a new use case emerges, from any domain in the organization.

This “data as a project” approach has some major drawbacks, including slow time-to-delivery, lack of reuse, rigidity, and risk of delivering wrong, and/or incomplete data.

On the other hand, a product-driven approach keeps the entire enterprise’s data needs in mind.

Data products can be reused to support any number of use cases, serving any number of domains

A project-driven approach to data drives greater complexity and minimal reuse,
compared to the simpler, more agile product-driven approach to data.