🎉 K2view named a Visionary in Gartner’s latest Magic Quadrant for Data Integration

Read More
Book a Demo
Start Free
New! 2025 State of Test Data Management Survey 📊
Get the Survey Results arrow--cta

Data de-identification: Techniques, use cases, and the impact of AI

Amitai Richman

Amitai Richman,Product Marketing Director

Get Gartner Report
Gartner data masking report

Gartner® Market Guide
for Data Masking

Learn how to mask data for regulatory compliance.

Get Gartner Report
Data de-identification: Techniques, use cases, and the impact of AI
13:43

Table of contents

    Data de-identification removes personal identifiers so teams can use data for testing, analytics, and AI. Learn the techniques, the rules, and the risks.

    Table of Contents


    What is Data De-Identification?  
    How Does Data De-Identification Work? 
    Regulatory Demands for Data De-Identification  
    Data De-Identification vs Sanitization, Anonymization, and Tokenization 
    Data De-Identification Based on Business Entities  

    Key takeaways

    • Data de-identification removes or transforms personal identifiers so records can’t reasonably be linked to a person.

    • Masking, pseudonymization, generalization, tokenization, and synthetic data are all ways to de-identify data.
    • HIPAA, GDPR, and CCPA define de-identified data differently, so one method rarely meets every rule.
    • AI adds new exposure points, including training data, RAG sources, prompts, and evaluation logs.
    • De-identifying data by business entity keeps it consistent across systems, so it stays useful for testing and AI.

    What is data de-identification? 

    Data de-identification is the process of removing or transforming personal identifiers in a dataset so the data can’t reasonably be traced to a specific person. Teams de-identify data so they can use it for software testing, analytics, data sharing, and AI without exposing Personally Identifiable Information (PII) or breaking privacy laws.

    De-identification is the goal. Data masking, pseudonymization, generalization, and tokenization are some of the techniques that get you there.

    Every de-identification project deals with two kinds of identifiers:

    • Direct identifiers
      These values point to one person on their own, such as a name, Social Security Number, email address, or medical record number.
    • Quasi-identifiers
      These values don’t identify anyone alone, but they can in combination. A study of 2000 Census data by Philippe Golle found that ZIP code, birth date, and sex together were enough to uniquely identify 63% of the US population (Revisiting the Uniqueness of Simple Demographics in the US Population, 2006).

    Removing names is the easy part. The real work is handling quasi-identifiers without stripping out the detail that makes the data useful.

    How does data de-identification work? 

    Data de-identification follows six steps. The order matters, because a technique chosen before you know where the PII lives will leave gaps.

    1. Discover and classify PII across databases, files, and unstructured text.

    2. Define the purpose of the data and the standard it must meet, such as HIPAA, GDPR, or CCPA.

    3. Choose a technique for each field, based on how the data will be used.

    4. Keep replacement values consistent across systems, so joins and end-to-end flows still work.

    5. Measure re-identification risk before anyone uses the data.

    6. Automate the process in data pipelines and CI/CD, and keep an audit trail.

    Step 4 is where most projects break. A customer who appears as Nancy Wilson in the CRM and N. Wilson in billing has to be masked to the same new name in both places. If not, test cases fail and analytics drift. 

    What are the main data de-identification techniques?  

    Most programs combine several techniques, since no single one fits every field or use case.

    Technique

    What it does

    Reversible

    Best for

    Substitution

    Replaces real values with realistic fake ones

    No

    Test and dev data

    Shuffling

    Moves values between records in the same column

    No

    Analytics that need real distributions

    Redaction and suppression

    Removes values or blanks them out

    No

    Documents, free text, call recordings

    Generalization

    Replaces exact values with ranges, like age 34 to 30–39

    No

    Research and reporting

    Date shifting

    Moves dates by a fixed offset per person

    No

    Healthcare and time-series data

    Noise addition

    Adds small random changes to numbers

    No

    Statistical analysis

    Pseudonymization

    Swaps identifiers for consistent codes, with a separate key

    Yes

    Internal analytics and research

    Tokenization

    Swaps values for tokens stored in a secure vault

    Yes

    Payments and LLM prompts

    Hashing

    Applies a one-way function to a value

    No, but weak if unsalted

    Matching records without revealing them

    Synthetic data

    Generates new records that mirror real patterns

    Not applicable

    Testing and AI training

    Privacy models such as k-anonymity and l-diversity sit on top of these techniques. They measure whether each person blends in with enough others to stay hidden.

    How do privacy laws define de-identified data? 

    Each law sets its own bar, so data that counts as de-identified under one may still be personal data under another.

    • HIPAA
      HIPAA offers two paths. Safe Harbor requires removing the 18 identifier types listed below. Expert Determination lets a qualified statistician certify that the risk of re-identification is very small, using methods like those in NIST SP 800-188. Data de-identified by either method is no longer protected health information (PHI). A limited data set, which keeps some dates and location details, still counts as PHI. HHS guidance explains both methods in detail.

    • GDPR
      Anonymous data falls outside GDPR, but pseudonymized data stays in scope. In EDPS v. SRB (2025), the EU Court of Justice added nuance. Pseudonymized data may not be personal data for a recipient with no reasonable way to re-identify it, but it stays personal data for the company that holds the key.

    • CCPA
      California treats data as deidentified only if the business takes reasonable technical measures, publicly commits not to re-identify it, and binds recipients to the same rule by contract.

    These are the 18 identifiers that HIPAA Safe Harbor requires you to remove:

    Identifiers 1–9

    Identifiers 10–18

    1. Names

    10. Account numbers

    2. Geographic units smaller than a state

    11. Certificate and license numbers

    3. All dates except the year

    12. Vehicle IDs, including license plates

    4. Phone numbers

    13. Device IDs and serial numbers

    5. Fax numbers

    14. Web URLs

    6. Email addresses

    15. IP addresses

    7. Social Security numbers

    16. Biometric identifiers, such as fingerprints

    8. Medical record numbers

    17. Full-face photos and comparable images

    9. Health plan beneficiary numbers

    18. Any other unique number, characteristic, or code

    The rules are still moving. The European Commission’s Digital Omnibus proposal would narrow the definition of personal data, and EU regulators have pushed back. Teams that build to the strictest standard won’t need to redo the work when the rules change.

    How is de-identification different from anonymization, pseudonymization, masking, and tokenization?

    De-identification is the umbrella term. The other four are specific ways to reach it, and they differ mainly in whether the data can be traced back. Pseudonymization alone usually doesn’t de-identify data, because it replaces direct identifiers and leaves quasi-identifiers in place.

    Method

    Reversible?

    Personal data under GDPR?

    Typical use

    Anonymization

    No

    No, if re-identification isn’t reasonably likely

    Public data sets and open research

    Pseudonymization

    Yes, with a key

    Yes, for whoever holds the key

    Clinical trials and internal analytics

    Data masking

    Usually no

    Depends on the technique

    Test, dev, and training environments

    Tokenization

    Yes, through a token vault

    Yes, for the vault owner

    Payments and LLM prompts

    Synthetic data

    Not applicable

    Generally no

    Testing and AI model training

    What are the top use cases for data de-identification?

    Any time production data leaves the place it was collected, it’s a candidate for de-identification.

    • Software testing and development
      Dev and test teams need realistic data, and they often get full production copies. In our 2026 State of Enterprise Data Compliance survey, only 4% of enterprises said their dev and test environments are fully compliant.
    • Analytics and BI
      Analysts can study trends and behavior on de-identified data without seeing who’s behind each record.
    • Sharing with vendors and partners
      Consultants, offshore teams, and research partners get the data they need without the identities attached.
    • HR and payroll data
      Employee data from systems like Workday often gets copied into lower environments with salaries and Social Security Numbers intact.
    • Mainframe and cloud migration
      Data moving from mainframe to cloud platforms is de-identified on the way, so no sensitive information lands unprotected in the new environment.
    • AI and LLM projects
      Training data, knowledge bases, prompts, and evaluation logs all carry PII. Each one needs a different control.

    How do you de-identify data for AI and LLMs?

    PII can enter an AI system at four points, and each one needs its own control. Most enterprises aren’t there yet. In our 2026 State of Enterprise Data Compliance report, only 13% said they have enforced technical controls that keep sensitive data out of LLMs.

    • Training and fine-tuning data
      Models can memorize rare strings and repeat them later. De-identify the data before training, or use synthetic data so real people never enter the training set.

    • RAG knowledge bases
      Documents should be de-identified before they’re split into chunks and embedded. Once PII lands in a vector store, it’s hard to find and remove.

    • Prompts and agent context
      Operational AI agents work with live customer data. Sensitive values should be tokenized before they reach the LLM and restored on output only for users who are authorized to see them. Reversible tokenization matters here, because an agent that updates an account or emails a customer needs the real values to finish the job.

    • AI evaluation data
      Teams replay production interactions in lower environments to find the root cause of a bad answer. Those snapshots carry the same PII as production, so they need the same protection.

    AI also changes the math on re-identification. An LLM can connect quasi-identifiers that look harmless one by one, and free text hides PII that field-level rules never see.  privacy.

    How is AI changing data de-identification?

    AI is making de-identification faster to set up and better at finding PII. That matters, because discovery is the hardest step. In the same survey, 82% of respondents named discovering and classifying PII as their top challenge in scaling data protection.

    AI now helps in three places:

    1. Discovery
      Language models find PII in free text, PDFs, and transcripts, where pattern matching misses names and addresses written in a sentence.

    2. Configuration
      AI suggests the right masking function for each field and builds rules from plain-language instructions.

    3. Implementation
      AI automates the setup work that once took months of scripting.

    AI classifiers still miss things. Sample the output and review it on a regular schedule, especially for unstructured data.

    How does K2view make data de-identification work across the enterprise?

    K2view Data Masking is a standalone solution that’s quick to implement, thanks to AI. Teams can deploy it on its own and add other K2view products later, if and when they need them.

    • Fast implementation with AI
      K2skills (prebuilt AI skills for common setup tasks) and K2assistant (an AI assistant built into the product) automate implementation. Time to value is short, training is minimal, and non-technical users can configure it.

    • Business entity approach
      K2view organizes data by business entity, such as a customer, account, or claim. Masked data keeps its referential integrity and semantic consistency across every system.

    • In-flight masking
      Data is masked as it moves, so PII is never exposed at rest.

    • Dozens of masking functions
      Customizable, out-of-the-box functions make setup fast and stay flexible as requirements change.

    • Integration with any source
      Coverage is complete across databases, mainframes, packaged apps, and files, so compliance holds as your stack changes.

    • A foundation for more
      The same platform supports test data management and synthetic data generation, so the first investment keeps paying off.

    For AI, K2view AI Runtime Governance is also a standalone solution. It tokenizes sensitive values before they reach the LLM and restores them only for authorized users.

    Conclusion  


    Data de-identification lets teams use real data for testing, analytics, sharing, and AI without exposing the people behind it. It works best when you find PII first, pick a technique for each field, and keep values consistent across systems. HIPAA, GDPR, and CCPA each set a different bar, and AI adds new exposure points in training data, RAG sources, prompts, and evaluation logs. A business entity approach keeps de-identified data accurate and complete, so it stays useful wherever it goes.

    Take the K2view Data Masking product tour
    to see entity-based de-identification in action.
     

     

    Achieve better business outcomeswith the K2view Data Product Platform

    Solution Overview
    Get Gartner Report
    Gartner data masking report

    Gartner® Market Guide
    for Data Masking

    Learn how to mask data for regulatory compliance.

    Get Gartner Report
    © COPYRIGHT 2026 K2VIEW Your Privacy Choices
    Manage cookies

    We use cookies to enhance your experience and to analyze site traffic as described in our Cookie Policy. By accepting, you consent to our use of cookies.

    Always active

    These cookies are essential for the site and services to function properly and cannot be disabled.

    These cookies help us understand and improve the use and performance of our services and how visitors interact with the various areas and features on our site.

    These cookies are used to deliver advertisements, to provide more personalized advertising to visitors, and to track the effectiveness of K2view’s advertising campaigns.

    These cookies enable our services to provide enhanced functionality and personalization. If not enabled, some parts of our site may not work as intended or offer the full user experience.

    K2view does not sell or share personal information. However, you still have the right to exercise your choice to opt out of the sale or sharing of your personal information at any time.

    By switching the toggle to the left and clicking “Save,” you indicate that you do not want us to sell your personal information or share it for online targeted advertising.

    You may update your preferences at any time using the toggle. Any change you make will override your previous selection.