PII Detection with Named Entity Recognition

By Edward Grundy, Bay Information Systems. Published 27 May 2026, updated 16 October 2026.

PII detection in text usually starts with named entity recognition (NER): finding the spans of text that refer to specific things (people, organisations, places, dates, sums of money) and labelling each with its type. “Sarah Johnson joined Pfizer’s London office in March 2024” contains four entities: a person, an organisation, a location and a date.

NER has many uses. It becomes a compliance control when the text holds information that must not leave the organisation, reach a third-party API or enter a model’s training data, and at that point it is the first stage of the pipeline that keeps personal data where it belongs. We can work through what counts as PII, the three operations NER supports, how each fails, how to measure them, and the tools available.

What counts as PII

Personally identifiable information is any data that can identify a specific person, directly or in combination with other data. The standard NER entity types (people, organisations, locations) cover part of it. The categories that matter most for compliance go further:

• Direct identifiers: name, national insurance number, passport number, date of birth, NHS number, driving licence number.
• Contact data: email address, phone number, home address, IP address.
• Financial identifiers: bank account number, sort code, card number.
• Health data: diagnoses, prescriptions, clinical notes, and anything that can be linked to a medical record.
• Identifiers in combination: a job title, an employer and an approximate location can single out one person with no name present.

NER models trained on news text handle names, organisations and locations reasonably well, and handle the compliance categories inconsistently. National insurance numbers and sort codes rarely appear in news, so they are scarce in the training data.

Three operations

NER supports three operations on sensitive text, each with its own purpose and its own cost of error.

Detection establishes whether a document contains PII. The output is a flag or a confidence score, used for routing (whether a document goes to a human reviewer before processing) and for audit logs.

Extraction returns each identified span with its type and position: the entity text, its type and its character offsets. Audit trails, anonymisation pipelines and data catalogues consume this record.

Redaction replaces identified spans with placeholders or type labels. “Sarah Johnson joined Pfizer’s London office” becomes “[PERSON] joined [ORG]’s [LOC] office”, and the redacted text can go to an LLM or another downstream process without the identifying details.

The three are often treated as one, and each needs its own evaluation. A model that detects PII with high recall can extract so many false positives that the redacted text becomes unusable, and a model that redacts accurately on news text can miss domain-specific identifiers entirely.

How PII detection fails

False negatives expose data. An entity the model misses passes through unredacted, and in a pipeline that sends text to an external API or stores it somewhere retrievable, a missed name or account number is a real exposure. Miss rates vary by entity type: names common in English-language news are detected well, and names from other scripts, unusual spellings and domain terms less well.

False positives remove meaning. Aggressive redaction can strip every date from a clinical document and leave it clinically useless, and a legal document with every organisation redacted can lose the context that makes it readable. A single F1 score hides both costs.

Domain shift causes most failures in practice. Models trained on news text meet legal documents, clinical records, financial reports and internal email, where the entity types they rarely saw in training are common. An NHS trust running a standard NER model over clinical notes will find it misses a significant share of patient identifiers that follow no pattern found in newswire text.

Extraction errors propagate. Text bound for a memory store or a structured pipeline usually passes through NER and other extraction steps first, so an extraction error becomes a wrong record. A drug name labelled as a person, or a summary that drops a negation, is then retrieved and acted on as fact. Nothing in the pipeline finds the error unless an evaluation layer checks extraction quality (How to Evaluate an AI Pipeline covers the approaches).

How to evaluate PII detection accuracy

An aggregate F1 score cannot say whether a model misses account numbers or over-redacts dates, and those are separate problems. We can measure at three levels instead.

Per entity type. A model can score well overall and badly on the categories compliance depends on (NHS numbers, IBANs, domain-specific identifiers), because those make up a small share of all entities. Report precision and recall separately for each entity type the pipeline relies on.

Against your own documents. CoNLL-2003 and similar corpora measure performance on newswire. Compliance pipelines process clinical notes, legal documents and internal correspondence. A small hand-labelled test set built from real or realistic anonymised samples of your own document types gives a figure that applies to your pipeline (How to Evaluate an AI Pipeline describes how to build one).

By operation. Detection, extraction and redaction fail at different costs, so each gets its own metric. For detection, recall matters more than precision, because a missed document skips review altogether. For redaction, the figure to track is the false negative rate: the share of identifiers that survive redaction. A missed identifier and an over-redacted one carry unequal risks, and F1 weighs them equally.

Metrics drift away from the task as documents and models change, so a periodic human audit of outputs belongs in the evaluation schedule.

Where PII removal stops

NER works on the surface form of sensitive information, and a document’s content can remain sensitive after every identifier is gone.

A pharmaceutical company processing internal documents about a drug in late-stage trials can remove every name, date and location, and the documents will still describe the drug’s mechanism, the trial design, the patient population, the early efficacy signals and the suppliers involved. None of those is a named entity, and all of them are sensitive.

The same holds in defence, legal proceedings and competitive intelligence, and anywhere the relationships between entities (who is working with whom, on what, at what stage) carry as much risk as the entities. A contract with every name redacted can still reveal the structure of a deal, and a research paper stripped of its authors can still identify the institution through its methods and references.

PII removal solves a specific, narrow problem: keeping directly identifying information out of plain text where it should not appear. Where the content itself is sensitive, the processing has to stay inside a boundary the organisation controls (Why Private Inference).

Tools for PII detection

The options run from small rule-augmented models to large general-purpose classifiers.

spaCy (en_core_web_sm through en_core_web_trf) is the standard production NER library for English. Its English pipelines are trained on OntoNotes and label 18 entity types, dates and money among them. The small CNN model runs efficiently on CPU with an F-score in the mid-0.8s on its own evaluation, which suits low-stakes detection; the transformer pipeline is more accurate at a higher memory cost. Both can be extended or fine-tuned on domain data.

dslim/bert-base-NER is a BERT model fine-tuned on CoNLL-2003, common as a reference point and as a component in larger pipelines. It labels the four CoNLL types (person, organisation, location, miscellaneous), performs well on news text, degrades on other domains, and covers no compliance-specific identifiers without fine-tuning.

Microsoft Presidio sits above the model layer. It combines NER models (spaCy or BERT-based) with rule-based recognisers and regular expressions for the identifiers neural NER misses: NHS numbers, card numbers, email addresses, IBANs. For PII pipelines in regulated sectors it is the usual starting point.

OpenAI Privacy Filter (openai/privacy-filter, Apache 2.0, on Hugging Face), released in April 2026, is a bidirectional token classifier derived from the gpt-oss architecture and post-trained for PII detection and masking. It has 1.5B parameters with about 50M active per token, so it runs on a laptop or in a browser, and its 128,000-token context handles long documents without chunking. It covers eight PII categories, lets precision and recall be tuned at runtime, and is designed for on-premises use in high-throughput sanitisation work.

For high-volume pipelines with well-defined entity types, fine-tuned BERT-class models remain competitive with much larger ones. For out-of-domain text and new entity types, LLM-based NER with few-shot prompting can do better, at a much higher cost per document and lower throughput. Volume, domain and the cost of a missed entity decide between them.

PII detection inside a pipeline

In most production systems NER runs as a preprocessing step: text enters, entities are found, and the redacted text moves on. The step has to run wherever text crosses a boundary: between components, into memory, and out as output.

Marigold (our own open-source inference platform) lets redaction be declared as a typed stage in a workflow. A pipeline that ingests documents, redacts PII, runs an extraction or summary and writes to a retrievable store declares the redaction step with its own inputs, outputs and error handling, so the same configuration applies to every document. (To discuss pipeline design for your own documents, get in touch.)

Where the documents’ content is the sensitive part, as in the pharmaceutical and defence cases above, the same pipeline runs on infrastructure the organisation controls, and the text stays inside that boundary throughout.