What is Test Data Management (TDM)?

Test data preparation has always been a challenge, especially with the advent of agile development, CI/CD, and ephemeral data environments enabling parallelization and related resource and cost savings.

And, with the growing fragmentation of enterprise data between application silos, provisioning valid test data, with referential integrity, has become even more complex.

For a test cycle to be effective, whether it's manual or automated, availability of test data that enables 100% test coverage is critical. Fresh, precise test data should be available for running functional and non-functional tests, whether executed manually or using a test automation tool. The test data should enable the entire scope of the software to be tested.

Test data automation is the process of automatically delivering test data to lower environments, as requested by software and quality engineering teams. It integrates test data into an organization's DevOps CI/CD pipelines, ensuring that the test data is complete, precise, and current.

Automated procedures that provision test data by connecting directly to the source systems might create an overload that impacts performance. To prevent this, instead of building one automation flow that extracts data from the source systems, and delivers it directly to the testing environment, the process should be done in 3 separate steps:

• Connect to all your data sources to synchronize data extraction.

• Integrate, mask, transform, subset, and generate your test datasets.

• Provision the test data, from your TDM tools to your testing environments, on demand.

This process, with TDM playing a central role, ensures that testing environments are provisioned with the needed compliant test data, in a timely manner.

When provisioning production-grade test data, the data should be sourced from a centralized test data repository, and not the production source systems, to minimize load on the sources. The 3-step approach to test data automation is also more secure, because it minimizes direct access to production data.

Automation serves to:

• Reduce infrastructure costs, by decreasing storage, and sharing data products across domains.
• Eliminate release cycle friction, with the quick provisioning of high-quality test data.

Software delivery acceleration and quality improvements are enabled by a shift-left testing approach – where testing is performed in the early stages of software development. Testing as early as possible in the software delivery lifecycle enables testers to identify bugs and fix them more quickly and at lower cost. However, earlier testing requires earlier availability of quality test data.

Catalyzed by the rapid growth in applications and the need for much faster delivery, software development has shifted gears to agile development methodologies, releasing smaller software deliverables in quick sprints.

Fig. 1 Agile software delivery allows for continuous design,
development, testing, and deployment, in short sprints.

In support of delivering software in short, iterative sprints, DevOps test data management has emerged as the practice of provisioning precise test data on demand in support of smaller-scope deliveries.

Agile development and shift-left testing must be combined with a test data management strategy that includes data security integration as well as compliance with regulations like CPRA, GDPR, HIPAA, and PCI.

Otherwise – even with test automation processes – testing would be complex, error-prone, and may include unprotected sensitive data.

Data masking tools protect test data by replacing Personally Identifiable Information (PII) with scrambled, yet statistically similar, data. Masked test data can’t be identified or reverse-engineered, but remains functional for testing environments. The use of anonymized data, instead of original production data, safeguards sensitive information in the event of a mass data breach – shielding your company from financial, legal, and brand liability.

Synthetic data generation tools do the same for testing, but have no sensitive data to deal with. By creating a statistically equivalent synthetic dataset, testers can test new software quickly, and without security and non-compliance risk.

In the test data masking vs synthetic test data showdown, DevOps and testing teams must decide which approach, or combination of approaches, is most suitable for their particular needs.

Implementing an efficient and effective TDM process poses several key challenges for enterprises to overcome. These include:

1. Sourcing the test data
   Enterprise data is typically siloed and dispersed across many different data sources – including legacy systems, such as mainframes and SAP, but also new systems, such as NoSQL and cloud. It’s also stored in various formats, making it hard for software teams to get the data they need, when they need it. QA and software engineers spend too much time waiting for test data.

2. Subsetting the test data
   Subsetting larger, diverse test datasets into smaller, more focused ones, enables QA teams to achieve full test coverage. Test data subsetting is particularly critical for recreating and fixing production issues. It also allows data teams to minimize the quantity of test data and its associated hardware and software costs.

3. Protecting sensitive data
   Data privacy regulations, such as CPRA, GDPR, and HIPAA, require that sensitive data and Personally Identifiable Information (PII) – such as names, Social Security Numbers, driver's licenses, and email addresses – be anonymized in the test environment. Discovering and de-identifying sensitive data and PII, while assuring referential integrity of the masked data, can be complex, time-consuming, and labor-intensive for data teams.

4. Enforcing referential integrity
   Referential integrity refers to data and schema consistency across databases and tables. Assuring the referential integrity of masked test data is critical to the validity of the data.

5. Extending test coverage
   Test coverage is a measure of how much of an application's processes has been tested. Defining the necessary test cases is crucial, but making sure you have all the test data needed to fully operate the test cases is just as important. Low test coverage is directly related to high defect density.

6. Reducing false positives and negatives
   When test data is poorly designed, it often causes false positive errors, leading to valuable time and effort wasted in dealing with non-existent software bugs. When test data is insufficient, it leads to false negatives, which can affect the quality and reliability of the software.

7. Reusing the test data
   The ability to reuse test data is crucial when re-running test cases to verify software fixes. Versioning datasets for reuse enables teams to perform regression testing (re-testing to verify that the bugs discovered in previous tests have been resolved).

8. Preventing test data overrides
   A common challenge for QA teams arises when testers accidentally override each other's test data, leading to corrupted test data and wasted time and effort. When this happens, the test data must be re-provisioned, and the tests re-run.

Coupling agile software development with high-performance test data environments, saves enterprises millions of dollars (as explained in the next section). Test data management tools increase test coverage, accelerate software delivery, reduce testing costs, and enhance the end-user experience.

The challenge is to find the most suitable test data management tools for your organization. A test data management approach based on business entities is ideal for overcoming enterprise complexities.

An entity-based approach ingests data from the source environments; masks, compresses, and versions it in flight; and then organizes it by business entities (e.g., customers, orders, loans, or products) into a test data store.

Fig 2. Test data management approach based on business entities

Entity-based test data management tools enable testing teams to instantly:

• Subset test data using business parameters.
• Generate synthetic test data.
• Transform the data so that it suits the needs of the software being tested.
• Reserve test data to segregate it between testers.
• Snapshot and roll back to reuse data across testing cycles.

Entity-based tools simplify and streamline test data management, while enabling complete control of the process. The success of this approach is evidenced in Gartner Peer Reviews.

Discover K2view Test Data Management, the AI-powered test data management tools for the enterprise.

There are several test data management benefits derived from implementing the right TDM solution:

1. Agility and speed to market
   Providing development and testing teams with the right data, at the right time, enhances agility and accelerates time to market for software applications.
2. Software quality
   Test data management increases test coverage and shifts testing to the left, both of which improve the quality of delivered software by reducing defect density.
3. Cost efficiencies
   When done well, TDM improves cost efficiencies by reducing hardware and software costs, accelerating test data provisioning, preventing data duplication, better balancing the use of resources, and providing self-service capabilities to improve productivity. See the next section for more details.
4. Compliance
   Your TDM solution should provide both test data generation and data masking functionality to ensure that only authorized personnel have access to sensitive data, enabling companies to comply with data protection regulations like CPRA, GDPR, HIPAA, and PCI.
5. Employee experience
   For data engineers, copying production databases into staging environments, by manually scrubbing, masking, and formatting data, is a long, tedious, repetitive process. For development and QA teams, a lot of time and effort are wasted on waiting for the data, using the wrong data, dealing with problems related to the data (e.g., reporting false positives, lacking sufficient test coverage, or overriding each other’s test data). The right TDM solution improves job satisfaction for all test data stakeholders.
6. Tester/developer productivity
   TDM empowers data teams to provision test data independently, without requiring data engineering or SQL expertise. So, instead of testers having to wait days and weeks for data teams to provision their test data, they’ll be able to access the test data subsets they need in minutes – with “do-it-yourself” test data provisioning software.

Return on investment (ROI) for a test data management tool can be quantified in 4 dimensions:

1. Reduction in test data provisioning costs, automating up to 70% of conventional manual tasks such as scripting, scrubbing and masking.

2. Improvement in test data delivery speeds, team productivity, and time to market, reducing application delivery cycle times by as much as 25% and test environment refresh times from 3 days to 3 minutes.

3. Savings derived from shifting testing to the left and expanding test coverage,  enabling the detection and correction of errors earlier in the lifecycle.

4. Optimization of test data storage and database costs, by subsetting, generating synthetic data, and compressing test data storage, for more compact test datasets.

Fig 3. Test data management ROI:
Over time, the benefits greatly outweigh the costs.

The global enterprises that rely on test data management hail from a wide range of industries, including telecommunications and media, financial services, healthcare, retail, and more.

The proof is in the numbers. For example, check out the percentage improvements achieved at this TDM Fortune 500 bank.

For more details on calculating the ROI of test data management, read the test data management ROI report.