Table of contents
Data de-identification removes personal identifiers so teams can use data for testing, analytics, and AI. Learn the techniques, the rules, and the risks.
Table of Contents
What is Data De-Identification?
How Does Data De-Identification Work?
Regulatory Demands for Data De-Identification
Data De-Identification vs Sanitization, Anonymization, and Tokenization
Data De-Identification Based on Business Entities
Key takeaways
-
Data de-identification removes or transforms personal identifiers so records can’t reasonably be linked to a person.
- Masking, pseudonymization, generalization, tokenization, and synthetic data are all ways to de-identify data.
- HIPAA, GDPR, and CCPA define de-identified data differently, so one method rarely meets every rule.
- AI adds new exposure points, including training data, RAG sources, prompts, and evaluation logs.
- De-identifying data by business entity keeps it consistent across systems, so it stays useful for testing and AI.
What is data de-identification?
Data de-identification is the process of removing or transforming personal identifiers in a dataset so the data can’t reasonably be traced to a specific person. Teams de-identify data so they can use it for software testing, analytics, data sharing, and AI without exposing Personally Identifiable Information (PII) or breaking privacy laws.
De-identification is the goal. Data masking, pseudonymization, generalization, and tokenization are some of the techniques that get you there.
Every de-identification project deals with two kinds of identifiers:
- Direct identifiers
These values point to one person on their own, such as a name, Social Security Number, email address, or medical record number. - Quasi-identifiers
These values don’t identify anyone alone, but they can in combination. A study of 2000 Census data by Philippe Golle found that ZIP code, birth date, and sex together were enough to uniquely identify 63% of the US population (Revisiting the Uniqueness of Simple Demographics in the US Population, 2006).
Removing names is the easy part. The real work is handling quasi-identifiers without stripping out the detail that makes the data useful.
How does data de-identification work?
Data de-identification follows six steps. The order matters, because a technique chosen before you know where the PII lives will leave gaps.
-
Discover and classify PII across databases, files, and unstructured text.
-
Define the purpose of the data and the standard it must meet, such as HIPAA, GDPR, or CCPA.
-
Choose a technique for each field, based on how the data will be used.
-
Keep replacement values consistent across systems, so joins and end-to-end flows still work.
-
Measure re-identification risk before anyone uses the data.
-
Automate the process in data pipelines and CI/CD, and keep an audit trail.
Step 4 is where most projects break. A customer who appears as Nancy Wilson in the CRM and N. Wilson in billing has to be masked to the same new name in both places. If not, test cases fail and analytics drift.
What are the main data de-identification techniques?
Most programs combine several techniques, since no single one fits every field or use case.
|
Technique |
What it does |
Reversible |
Best for |
|
Substitution |
Replaces real values with realistic fake ones |
No |
Test and dev data |
|
Shuffling |
Moves values between records in the same column |
No |
Analytics that need real distributions |
|
Redaction and suppression |
Removes values or blanks them out |
No |
Documents, free text, call recordings |
|
Generalization |
Replaces exact values with ranges, like age 34 to 30–39 |
No |
Research and reporting |
|
Date shifting |
Moves dates by a fixed offset per person |
No |
Healthcare and time-series data |
|
Noise addition |
Adds small random changes to numbers |
No |
Statistical analysis |
|
Pseudonymization |
Swaps identifiers for consistent codes, with a separate key |
Yes |
Internal analytics and research |
|
Tokenization |
Swaps values for tokens stored in a secure vault |
Yes |
Payments and LLM prompts |
|
Hashing |
Applies a one-way function to a value |
No, but weak if unsalted |
Matching records without revealing them |
|
Synthetic data |
Generates new records that mirror real patterns |
Not applicable |
Testing and AI training |
Privacy models such as k-anonymity and l-diversity sit on top of these techniques. They measure whether each person blends in with enough others to stay hidden.
How do privacy laws define de-identified data?
Each law sets its own bar, so data that counts as de-identified under one may still be personal data under another.
-
HIPAA
HIPAA offers two paths. Safe Harbor requires removing the 18 identifier types listed below. Expert Determination lets a qualified statistician certify that the risk of re-identification is very small, using methods like those in NIST SP 800-188. Data de-identified by either method is no longer protected health information (PHI). A limited data set, which keeps some dates and location details, still counts as PHI. HHS guidance explains both methods in detail. -
GDPR
Anonymous data falls outside GDPR, but pseudonymized data stays in scope. In EDPS v. SRB (2025), the EU Court of Justice added nuance. Pseudonymized data may not be personal data for a recipient with no reasonable way to re-identify it, but it stays personal data for the company that holds the key. -
CCPA
California treats data as deidentified only if the business takes reasonable technical measures, publicly commits not to re-identify it, and binds recipients to the same rule by contract.
These are the 18 identifiers that HIPAA Safe Harbor requires you to remove:
|
Identifiers 1–9 |
Identifiers 10–18 |
|
1. Names |
10. Account numbers |
|
2. Geographic units smaller than a state |
11. Certificate and license numbers |
|
3. All dates except the year |
12. Vehicle IDs, including license plates |
|
4. Phone numbers |
13. Device IDs and serial numbers |
|
5. Fax numbers |
14. Web URLs |
|
6. Email addresses |
15. IP addresses |
|
7. Social Security numbers |
16. Biometric identifiers, such as fingerprints |
|
8. Medical record numbers |
17. Full-face photos and comparable images |
|
9. Health plan beneficiary numbers |
18. Any other unique number, characteristic, or code |
The rules are still moving. The European Commission’s Digital Omnibus proposal would narrow the definition of personal data, and EU regulators have pushed back. Teams that build to the strictest standard won’t need to redo the work when the rules change.
How is de-identification different from anonymization, pseudonymization, masking, and tokenization?
De-identification is the umbrella term. The other four are specific ways to reach it, and they differ mainly in whether the data can be traced back. Pseudonymization alone usually doesn’t de-identify data, because it replaces direct identifiers and leaves quasi-identifiers in place.
|
Method |
Reversible? |
Personal data under GDPR? |
Typical use |
|
No |
No, if re-identification isn’t reasonably likely |
Public data sets and open research |
|
|
Yes, with a key |
Yes, for whoever holds the key |
Clinical trials and internal analytics |
|
|
Usually no |
Depends on the technique |
Test, dev, and training environments |
|
|
Yes, through a token vault |
Yes, for the vault owner |
Payments and LLM prompts |
|
|
Not applicable |
Generally no |
Testing and AI model training |
What are the top use cases for data de-identification?
Any time production data leaves the place it was collected, it’s a candidate for de-identification.
- Software testing and development
Dev and test teams need realistic data, and they often get full production copies. In our 2026 State of Enterprise Data Compliance survey, only 4% of enterprises said their dev and test environments are fully compliant. - Analytics and BI
Analysts can study trends and behavior on de-identified data without seeing who’s behind each record. - Sharing with vendors and partners
Consultants, offshore teams, and research partners get the data they need without the identities attached. - HR and payroll data
Employee data from systems like Workday often gets copied into lower environments with salaries and Social Security Numbers intact. - Mainframe and cloud migration
Data moving from mainframe to cloud platforms is de-identified on the way, so no sensitive information lands unprotected in the new environment. - AI and LLM projects
Training data, knowledge bases, prompts, and evaluation logs all carry PII. Each one needs a different control.
How do you de-identify data for AI and LLMs?
PII can enter an AI system at four points, and each one needs its own control. Most enterprises aren’t there yet. In our 2026 State of Enterprise Data Compliance report, only 13% said they have enforced technical controls that keep sensitive data out of LLMs.
-
Training and fine-tuning data
Models can memorize rare strings and repeat them later. De-identify the data before training, or use synthetic data so real people never enter the training set. -
RAG knowledge bases
Documents should be de-identified before they’re split into chunks and embedded. Once PII lands in a vector store, it’s hard to find and remove. -
Prompts and agent context
Operational AI agents work with live customer data. Sensitive values should be tokenized before they reach the LLM and restored on output only for users who are authorized to see them. Reversible tokenization matters here, because an agent that updates an account or emails a customer needs the real values to finish the job. -
AI evaluation data
Teams replay production interactions in lower environments to find the root cause of a bad answer. Those snapshots carry the same PII as production, so they need the same protection.
AI also changes the math on re-identification. An LLM can connect quasi-identifiers that look harmless one by one, and free text hides PII that field-level rules never see. privacy.
How is AI changing data de-identification?
AI is making de-identification faster to set up and better at finding PII. That matters, because discovery is the hardest step. In the same survey, 82% of respondents named discovering and classifying PII as their top challenge in scaling data protection.
AI now helps in three places:
-
Discovery
Language models find PII in free text, PDFs, and transcripts, where pattern matching misses names and addresses written in a sentence. -
Configuration
AI suggests the right masking function for each field and builds rules from plain-language instructions. -
Implementation
AI automates the setup work that once took months of scripting.
AI classifiers still miss things. Sample the output and review it on a regular schedule, especially for unstructured data.
How does K2view make data de-identification work across the enterprise?
K2view Data Masking is a standalone solution that’s quick to implement, thanks to AI. Teams can deploy it on its own and add other K2view products later, if and when they need them.
-
Fast implementation with AI
K2skills (prebuilt AI skills for common setup tasks) and K2assistant (an AI assistant built into the product) automate implementation. Time to value is short, training is minimal, and non-technical users can configure it. -
Business entity approach
K2view organizes data by business entity, such as a customer, account, or claim. Masked data keeps its referential integrity and semantic consistency across every system. -
In-flight masking
Data is masked as it moves, so PII is never exposed at rest. -
Dozens of masking functions
Customizable, out-of-the-box functions make setup fast and stay flexible as requirements change. -
Integration with any source
Coverage is complete across databases, mainframes, packaged apps, and files, so compliance holds as your stack changes. -
A foundation for more
The same platform supports test data management and synthetic data generation, so the first investment keeps paying off.
For AI, K2view AI Runtime Governance is also a standalone solution. It tokenizes sensitive values before they reach the LLM and restores them only for authorized users.
Conclusion
Data de-identification lets teams use real data for testing, analytics, sharing, and AI without exposing the people behind it. It works best when you find PII first, pick a technique for each field, and keep values consistent across systems. HIPAA, GDPR, and CCPA each set a different bar, and AI adds new exposure points in training data, RAG sources, prompts, and evaluation logs. A business entity approach keeps de-identified data accurate and complete, so it stays useful wherever it goes.
Take the K2view Data Masking product tour
to see entity-based de-identification in action.







