How to Discover PII in Unstructured Data

Start with the question you're actually trying to answer

Finding PII in unstructured data is not a scanning exercise, it's an inventory exercise that happens to require scanning. The goal is to be able to say, with evidence, what personal data lives in your file shares, tickets, chat logs and documents, where, and how sensitive it is — not just to generate a list of regex hits. Build discovery around that goal and the tooling choices get much easier.

Why unstructured data is where the real exposure lives

Structured databases get modelled, documented and access-reviewed because someone had to design the schema. Unstructured data — shared drives, support tickets, email archives, chat exports, scanned PDFs, meeting transcripts — accumulates personal data by accident: a customer pastes their passport number into a support ticket, a recruiter saves a CV with a home address to a folder that syncs to everyone, a call transcript captures a cardholder reading out their number. Nobody designed this, so nobody controls it, and it is usually far larger in volume than the structured systems the organisation actually governs.

Framework: inventory, sample, detect, tune, remediate

1. Inventory repositories before you inventory content. List every place unstructured data lives: file shares, SaaS document stores, ticketing systems, email, chat platforms, backups, and any AI tools already connected to them. You cannot scan what you have not listed, and this step alone usually surfaces forgotten repositories.
2. Sample before you scan everything. Running a full scan of a decade of file shares on day one produces an unmanageable pile of findings and burns credibility with the teams who have to act on them. Pull a representative, stratified sample (by repository, age, and owning team) first, to estimate the scale and pattern of what you're dealing with and to calibrate your detectors.
3. Choose detector types deliberately, and combine them.

• Pattern/regex detectors — fast and precise for structured-looking strings (credit card numbers, national ID formats, email addresses). High precision, but blind to identifiers with no fixed format and to context.
• Dictionary/lookup detectors — match against known lists (employee names, customer IDs, known-sensitive terms). Good for known entities, useless for anything not on the list.
• Context/proximity rules — flag content where a weak signal (a first name) appears near a strong one (a diagnosis term, an account number), improving recall for data that pattern matching alone misses.
• Classifier/ML-based detectors — trained models that recognise identifiers in free text without a fixed pattern (addresses, informal references to health or financial status). Better recall on messy real-world text, but need tuning and produce more false positives out of the box. A discovery program that relies on only one of these will either miss a lot (regex-only) or drown reviewers in noise (classifier-only, untuned).

1. Tune for precision and recall against your actual risk tolerance, not a default. Decide upfront what a false negative costs you (a missed health record in a public-facing folder) versus what a false positive costs you (reviewer time). For anything touching regulated categories of data, bias tuning toward recall and accept more review overhead; for low-risk internal categories, bias toward precision so people don't ignore the output.
2. Score and prioritise findings, don't just list them. Combine sensitivity of data type, repository exposure (public link, org-wide access, restricted), and volume into a simple risk score so remediation starts with the worst combination — sensitive data, broadly accessible — rather than working alphabetically through a spreadsheet.
3. Build a remediation workflow, not a report. Every finding needs a next action: restrict access, delete, redact, move to an approved repository, or accept and document the risk. Route findings to the repository owner with a deadline and a way to mark it resolved, and re-scan to confirm.
4. Repeat on a cadence. Unstructured data regenerates constantly. A one-time discovery project is a snapshot; a recurring scan (quarterly, at minimum for high-risk repositories) is a control.

Precision, recall, and the review capacity trap

Detector tuning is where discovery projects most often stall, and the reason is usually organisational rather than technical: whoever configures the tool sets it to maximum recall to "be safe," and it generates far more findings than the team assigned to review them can process. A backlog of unreviewed findings is functionally the same as not scanning at all, except it now looks like progress on a dashboard.

Set tuning targets against your actual review capacity. If one analyst can review 200 findings a week, do not configure a scan of a 2-million-file repository to flag at maximum sensitivity in one pass. Instead, phase it: run a high-precision pass first to find and remediate the clearest, highest-confidence exposures, then progressively loosen thresholds as review capacity allows or as automation (auto-remediation for high-confidence, low-ambiguity findings such as an exact credit-card-number pattern) takes some of the volume off human reviewers.

Repository-specific considerations

Different repository types carry different discovery challenges worth planning for separately:

• File shares and cloud drives — check both content and sharing settings; a properly redacted file that's still shared with "anyone with the link" is not fixed.
• Ticketing and support systems — attachments and free-text fields are the usual blind spot; ticket subject lines are often reviewed while ticket bodies and attached screenshots are not.
• Chat and messaging exports — high volume, very informal language, weakest fit for pattern detectors; context and classifier detectors do more of the work here.
• Scanned documents and images — require OCR before any text-based detector can see the content at all; confirm your pipeline actually runs OCR rather than silently skipping non-text files.
• Backups and archives — often forgotten because they are not "live" systems, but they hold the same personal data, sometimes older and less well governed than the live copy.

A realistic example

A 300-person logistics company ran discovery across its shared drives and found, in an initial sample of 5,000 files, that 6% contained what looked like personal data — mostly employee and customer records in HR and sales folders, as expected. But context-based rules also flagged a "Miscellaneous" folder with world-readable link sharing, containing scanned delivery signatures with handwritten names and addresses that OCR had made searchable. Pattern matching alone had missed it because the addresses did not follow a consistent format; a classifier detector combined with the "publicly shared" exposure signal is what surfaced it as the top-priority finding. Remediation was straightforward once found: restrict the link, move the folder into the document management system, and add a retention rule — but it would not have been found by regex alone.

Common failure modes

• Scanning everything at once with unfamiliar tooling, producing thousands of unreviewed findings and no working process to act on them.
• Using only pattern detectors and concluding "we don't have much PII in unstructured data" — you have not looked properly.
• No owner assigned to findings, so remediation stalls even after detection works.
• Treating discovery as a one-off audit rather than a repeatable control with a cadence.
• Ignoring exposure/access level in prioritisation, so a low-sensitivity but publicly-exposed file gets less attention than a high-sensitivity but properly-restricted one.

Do this next

• List every unstructured repository in your organisation, including ones nobody has scanned before.
• Run a sample-based pilot scan combining at least two detector types before committing to full-scale scanning.
• Build a scoring model that weighs sensitivity and exposure, not just data type.
• Assign every finding an owner and a deadline, and re-scan to confirm remediation.
• If Microsoft 365 is a major repository for your organisation, the Microsoft 365 sensitive data checklist is a good starting point for that specific environment, and the data inventory and RoPA starter kit helps you turn discovery findings into a maintained inventory rather than a one-time report.
• For a structured way to build this into an ongoing program rather than a project, the practical privacy program implementation course covers discovery alongside the rest of the operational program, and the assessment can help you gauge how mature your current discovery process is.