AI Redaction Accuracy: How to Test PII Detection

Key Takeaways

• A single accuracy percentage hides the failures that matter. Measure precision, recall and F1 for every entity type.
• Recall is the privacy metric. Every false negative is a piece of personal data that reached the model or the reader.
• How you count a match changes the score. Decide between strict and partial span matching before you compare tools.
• Your test set is only as good as its labels. Use written guidelines, two annotators and an adjudication step.
• Scanned documents, other languages, tables and context-dependent identifiers break systems that look perfect on clean text.
• Accuracy is not permanent. Rerun the same test suite after every model, rule or threshold change.

Most teams find out how well their redaction works by accident, when someone spots a name that should have been masked. That is a poor way to learn. Read AI data masking puts it, detection is the real bottleneck: a masking layer can only protect what it finds.

The good news is that PII detection can be measured with the same discipline as any other classifier. The method below works whether you are evaluating a vendor, tuning an open-source model or checking your own pipeline before it touches production data.

Why a Single Accuracy Number Misleads

Vendors like to quote one figure, often described as accuracy. In a typical document, though, the vast majority of words are not personal data at all. A system that masked nothing would still label most tokens correctly, so a headline accuracy score can look excellent while names, account numbers, and addresses slip through untouched.

What you actually need to know is narrower. Of the identifiers that exist in the text, how many did the system catch? Of the things it masked or how many were really identifiers? And does the answer change for phone numbers versus free-text names, or for typed emails versus scanned forms? Those questions decide whether a workflow is safe, and they matter even more once you will understand the difference between redaction and anonymization, because each technique depends on detection being right first.

Precision, Recall and F1, Applied to PII

Three numbers do most of the work. Precision is the share of detected items that really were personal data. Recall is the share of real personal data that the system detected. F1 is the harmonic mean of the two, so it only rises when both rise. In redaction terms, low precision means over-masking and damaged documents, while low recall means leaks.

For privacy work, recall deserves the most attention, but precision is not optional. A system that masks every capitalised word will post high recall and produce text no model or analyst can use. That trade-off is sharper when masked values are replaced with placeholders the model must reason over, as in prompt anonymization. Report all three numbers together and never accept F1 on its own as proof that leaks are rare.

How You Count a Match Changes the Score

Before any scoring, decide what counts as a correct detection. Token-level scoring checks each word separately, so catching “Mary” but missing “Ellen” in “Mary Ellen Carter” earns partial credit. Entity-level scoring treats the whole name as one unit, which is closer to how a leak actually happens: a half-masked name is often still a name.

At entity level there is a second choice. The CoNLL-2003 shared task, a long-standing NER benchmark covering English and German news text, counted an entity as correct only when its boundaries and type exactly matched the gold label. The SemEval-2013 drug-recognition task popularised a finer breakdown with strict, exact-boundary, partial-overlap and type-only schemes, which the open-source nervaluate library implements. Partial matching is useful for diagnosis, because it shows whether a model is finding the right area but trimming the edges.

Our advice is to use strict entity-level matching as the headline figure and partial matching as a diagnostic. Also check for overlapping detections, where two models claim the same characters with different labels. Our own engineering write-up on building a privacy-first anonymizer explains why overlapping spans have to be merged before text is rewritten, and your test should confirm the merged output is what gets scored.

Building a Gold-Standard Test Set

A gold-standard set is a sample of documents where every identifier has been labelled by people following written rules. The rules matter as much as the labels. Decide in advance whether a job title is personal data, whether a hospital name counts, whether “next Tuesday” is a date to mask and how to tag a partial address. Without those decisions, two careful people will label the same page differently and your scores will measure their disagreement rather than the system.

Use two independent annotators on at least part of the set and measure how often they agree, either with Cohen’s kappa or by scoring one annotator against the other with F1. Then have a third person adjudicate the differences. The 2014 i2b2/UTHealth de-identification corpus, one of the best-known public benchmarks, was built this way: 1,304 longitudinal medical records describing 296 patients were double-annotated, arbitrated and proofread, and the average token-based F1 of annotators against the final gold standard was 0.927. If trained annotators working from detailed guidelines land there, expect your own first round to need refinement.

Real documents give the most honest picture, but they carry the very data you are trying to protect, so keep them inside a controlled environment and limit who sees them. Synthetic data is safer to share and lets you create rare cases on demand, yet generated text tends to be cleaner and more regular than real records, which flatters results. A blend works best: real samples for the headline score, and synthetic data for rare entity types and edge cases. Start from what actually flows through your systems; AI data discovery shows where that is.

Score Every Entity Type Separately

An overall F1 can hide a disastrous category. Email addresses and card numbers follow patterns and are usually caught; free-text names, street addresses and medical record numbers in unusual formats are not. Break every result down by entity type and set a target for each, with the strictest targets on the identifiers that would cause the most harm if exposed.

This is also where your own data classification earns its keep. The entity list should come from a classification scheme agreed with legal and security, not from whatever labels a model happens to output. Financial teams, for example, will care about account and routing numbers far more than a generic model does, which is why redaction for financial data needs its own test cases.

Edge Cases That Break Redaction

Clean, typed English text is the easy part. Scanned PDFs and photos go through optical character recognition first, and OCR errors turn “Smith” into “Srnith” or split an account number across two lines. Test the full pipeline, from the original file to the redacted output, not just the text model, and include faxes, handwriting, stamps and low-resolution images. Our look at AI PDF converters covers why file conversion itself is a common leak point.

Language and format matter too. A model trained on English news may miss names in Spanish, Hindi or German, or names written in other scripts, so add samples in every language your users actually write in. Tables, CSV exports and chat logs strip away the sentence context NER models depend on, and call transcripts add spelling-free phone numbers like “four one five”. Teams that process customer conversations at scale, such as BPOs using redaction, should weight these formats heavily.

The hardest cases are identifiers that depend on context. “Jordan” is a person in one sentence and a country in the next. A rare diagnosis, a job title and a small town can together point to one person even though none is a direct identifier. These quasi-identifiers are why anonymization takes more than masking names. Latanya Sweeney’s 2000 study of 1990 US Census data estimated that 87% of the population was likely unique on 5-digit ZIP code, gender and date of birth; a 2006 re-analysis by Philippe Golle put the figure at 63% using 2000 Census data. Either way, combinations deserve their own test cases.

False Negatives, False Positives and Threshold Tuning

The two kinds of error carry very different costs, so treat them differently. A false negative means personal data reached a model, a log or a person who should not have seen it. A false positive means something harmless was masked, which can break a summary, confuse an agent or frustrate staff into bypassing the tool. Neither is free, but only one is a privacy incident.

Most detection models output a confidence score, and the threshold you pick moves the balance. Lower it and recall rises while precision falls; raise it and the reverse happens. Tune thresholds per entity type on a separate validation set, never on the test set you report, or you will overfit and overstate results. For high-risk categories, accept more over-masking. The aim of a privacy firewall is that nothing sensitive gets through, and a little lost readability is a fair price.

Regression and Drift Testing

A redaction system that passed testing in March can fail in October without anyone touching it. Inputs change as new document templates, products, languages and customer types appear. Models change when vendors update them. Rules change when someone adds an exception to fix a complaint. Keep the gold-standard set under version control, run it automatically on every change, and block releases when recall for any high-risk type drops below its target. This belongs in the wider PII governance across your AI pipelines.

Add a small, rotating sample of fresh production documents, reviewed by people, so the test set does not freeze while real traffic moves on. Track per-type recall over time, not just a pass or fail result, because slow drift is easier to fix early. Many failures go unnoticed because nobody looks, a pattern covered in the AI incidents most businesses never detect.

What the Regulations Expect

No regulation sets a recall target for PII detection, but the legal standards explain why testing matters. Under GDPR, Article 4(5) defines pseudonymisation, and Recital 26 makes clear that pseudonymised data is still personal data, while truly anonymous information falls outside the regulation. Whether a dataset is anonymous depends on whether people could reasonably be identified, which a leaky detector undermines. Our guide to enterprise AI and GDPR covers the wider obligations.

In US healthcare, the HIPAA Privacy Rule offers two routes to de-identification. Safe Harbor requires removing 18 listed identifiers, from names and dates to device serial numbers and full-face photos, with no actual knowledge that the remainder could identify someone. Expert Determination requires a qualified expert to find the re-identification risk very small and document the analysis. HHS explains both in its de-identification guidance, including the point that the standard makes no distinction between structured fields and free text. Both routes assume the identifiers were found in the first place. Read it alongside our HIPAA guide to generative AI.

For broader practice, NIST SP 800-188, finalised in September 2023, advises agencies to adopt de-identification standards with measurable performance levels and to run re-identification studies, building on the earlier survey in NISTIR 8053. That is a useful framing for any organization: set measurable targets, test against them, and check what an attacker could still learn. This article is general information, not legal advice.

Where Questa AI Fits

Testing tells you how good detection is. The next question is where detection runs and what happens to the data while you find out. Questa AI is built around detecting and masking personal, financial, health and other sensitive entities before content reaches any model, using a dual-model pipeline that pairs general NER with a specialised PII model. Questa Blackbox can run that anonymization inside your own environment, and its governance dashboard tracks redaction activity, protected entities and audit trails, which gives a testing program real production evidence to compare against.

Whichever tool you choose, hold it to the method above: per-type recall on your own documents, strict matching and regression tests on every change. If you are comparing approaches, our guides on choosing an AI privacy solution and AI DLP versus an AI privacy firewall set out what to ask. The sooner you measure, the sooner you know what is already leaking.

Frequently Asked Questions

Conclusion

Redaction is only as trustworthy as its weakest entity type on its messiest document. Build a labelled test set from your real data, score each identifier type with strict matching, treat recall as the privacy measure and keep retesting as inputs and models change. Do that and you stop hoping your redaction works and start knowing how well it does.