Data Migration Considerations

After managing many complex digital transformations and migrations, for some of the world’s largest companies, I’d love to share some key data migration considerations with you – as I see them.

Table of Contents

Credentials of a Data Pro
Insights from the Field
Data Migration Considerations Summary

Credentials of a Data Pro

I’ve been immersed in data management technology and platforms for more than 30 years, primarily with large enterprises in the United States. During this time, I’ve participated in, and/or managed, 16 different data migrations in massive-scale operations, in a number of industries.

Insights from the Field

I’ve learned a lot from every enterprise implementation I’ve been involved with. Below is a list of some of the most important data migration considerations, along with my recommendations. I hope you’ll find them useful!

1. Accounting for multiple source systems without all the knowledge
In large enterprises, customer data is fragmented, and sourced from many different big data stores. Some systems are so outdated, that no one in the company really knows how their data sources are structured and organized.

Recommendation:

Make sure your data migration toolkit includes the following:

Data catalog, to collect the metadata related to data formats, semantics, and quality across all systems (including legacy), so you can make informed decision-making related to mapping and transformation logic.
Auto-discovery, to automatically discover and visualize the meta data structure, and the relationships between business entities.
Ready-made connectors to the most common source and target technologies.

DataCatalog

A data catalog automatically discovers and visualizes the meta data
structure, and the relationships between its entities, for all of data assets.

2. Adopting a phased approach to assure business continuity
Dividing the migration into a series of phases can reduce complexity and risk. However, a phased migration also might increase cost and complexity because both old and new environments must run in parallel, until all phases are complete.

Recommendation:
Migrate only the data subsets you need, when you need them. For example, an enterprise with a million customers would be able to migrate the data of one customer segment at a time. Often these are segmented into customers with specific products, in a particular geographical area, consumer, SMB, or enterprise.

If retiring legacy systems to cut costs is your priority, migration can also be performed by customers in a particular system. These customers can then be subjected to one or more of the segmentations within a particular system – by product, geographical area, or market segment.

Select a system that enables you to change your migration strategy – all the way from flash card, to phase, to on-demand – without delay.

3. Focusing on data quality
Bad data causes migration projects to take up to twice as long as originally planned. That’s because there are often serious quality issues in legacy data, and extra time and resources aren’t allocated to identify and address them. Imperfect migrations may result in incorrect bills, or irrelevant correspondence being sent to end-users, impacting customer satisfaction.

Recommendation:
Build a data model centered around the business entities you wish to migrate, and then migrate that data according to those entities. This modeling ensures the completeness and integrity of the migrated data.

In addition, automatically test converted data for integrity and make sure that only 100% fully-tested data is migrated. Ensure your migration solution is geared towards identifying and handling fallout from data mismatches.

Choose a tool that can not only identify imperfect data, but also be able to correct it, in-flight.

4. Minimizing the scope to cut costs
Not all legacy data needs to be migrated. By reducing the migration’s scope – whether by deleting redundant data sources, updating outdated business processes, or archiving data that no longer has value – the scale and complexity of the project can be significantly scaled down, reducing risk, time, and cost.

Recommendation:
Identify only the relevant data, while discarding redundant, old, and “look-alike” information. Don’t migrate everything, just the data required to meet business goals and legal requirements. The rest can be archived. Evaluate if you still need the application, even if you want to keep the data. Turning off applications and keeping the data helps the business case. In short, liberate the data.

5. Defining and tracking metrics
The ability to measure and track the progress of data migration efforts is crucial for informed decisions and process improvement. This requires metrics of the quality of the data being migrated, as well as updates on the pace, progress, and performance of the overall data migration effort.

Recommendation:
Deliver metrics on data quality aspects, determine whether quality-level expectations are being met, and show the status of the data workstream (e.g., # data instances migrated, # iterations, time and resources spent on execution and testing, etc.).

A dashboard showing data migration results allows for informed decision-making.

6. Exceeding parity
It's amazing how many digital transformations bring the same old bad habits in the new environment. If we've learned anything over the pandemic, it's that we can change rapidly if we want to. Why move all the problems to the cloud? Sometimes, the functionality of the new system is initially less than that of the old one. New functionality can be incrementally added over time, to bridge the gaps.

Recommendation:
Rethink the “that’s-the-way-we-do-things” mindset:

Plan that the new system (and approaches to development) be focused on, and integrated with, real-time data pipelines. This will completely change the way the organization operates.
Uncouple the data that does not conform to the new system’s data model, and store it separately. Don’t overextend the new system’s data model to reflect the old one's.
Adopt “bridging”, which allows for legacy systems to serve the missing functionality, alongside the new environment.