phi-leak-guard: A Practical Tutorial for PHI Leak Detection

Sending patient data to a third-party API to check if that same data is leaking is a logical paradox and a security risk. Most LLM-eval libraries like promptfoo or autoevals either ignore Protected Health Information (PHI) or require an external API call, which is a non-starter for clinical-grade AI workflows.

I needed a deterministic, local way to fail a build the second an LLM leaks a patient identifier, so I built phi-leak-guard. It's a zero-dependency TypeScript library that integrates directly into Vitest or Jest.

Integration and Usage

Setting this up as a regression gate in your CI/CD pipeline is straightforward. Since it runs locally, no data ever leaves your process.

import 'phi-leak-guard/vitest';

test('clinical summary never leaks PHI', () => {
  expect(summarize(patientNote)).toContainNoPHI();
});

When a leak is detected, the library specifies the identifier type and whether it was a pattern match or a validated hit (e.g., [nhs-number] "943 476 5919" (validated)).

Why Deterministic Validation Beats Naive Regex

The biggest pain point in PII/PHI detection is the false positive. A standard 10-digit regex will flag every single order number or timestamp as a sensitive ID. To solve this, the library uses validation logic rather than just pattern matching:

• NHS Numbers: Uses Modulus-11 checksums.
• Vehicle VINs: Implements ISO-3779 check digits.
• IPv4: Validates octet ranges.
• SSN/NINO: Applies structural and prefix rules.

If a number doesn't pass the checksum, it isn't flagged. This ensures the build only fails when there is a high probability of an actual leak.

Coverage and Limitations

This is a risk-reduction tool, not a full compliance certification. It focuses on the 18 HIPAA Safe Harbor identifiers. While it handles direct identifiers well, it cannot detect paraphrased re-identification or every possible name. For those needing deeper analysis, there is a pluggable seam to integrate a Named Entity Recognition (NER) model.

To see exactly what is being monitored, you can run coverageReport(), which explicitly lists what the library can and cannot detect (e.g., it won't pretend to detect biometrics in plain text).

Deployment

For anyone building a medical AI workflow, you can add this to your dev dependencies:

npm install --save-dev phi-leak-guard

The library is MIT licensed, supports ESM/CJS, and hits a 0.97 recall on synthetic benchmarks.

https://github.com/selvassn/phi-leak-guard