Securing AI Observability: PII vs. Searchability
In my experience rolling out our internal AI platform, we realized that traditional log management is broken for the AI era. Historically, if you had DB access, you could see the data. But now, non-engineers are using MCP (Model Context Protocol) to query logs. Suddenly, your observability stack is a backdoor to personal data for people who never had direct database permissions.
If you scrub all PII, your AI becomes useless. If you leave it in, you're one leak away from a disaster. We had to redesign the trust boundary entirely.
Here is how we structured our multi-layer PII defense to keep the data searchable for AI without exposing raw sensitive strings:
1. BigQuery Policy Tags: We implemented strict Column-Level Security (CLS). We use a three-tier taxonomy (pii_high, pii_medium, pii_low). If a reader doesn't have the specific permission for that column, the query simply fails with Access Denied. No masking, just hard access control.
2. ETL DLP (Data Loss Prevention): During our transformation pipelines, we use Cloud DLP to redact plain-text PII from derived tables. Instead of just deleting data, we use structural placeholders like [EMAIL_ADDRESS] or [PHONE_NUMBER]. This allows the AI to understand the context of a log entry without seeing the actual value.
3. Application-Side Log Hashing: To prevent PII from ever reaching Loki, we handle it at the source. We use an HMAC-SHA256 hash via a hashEmail function. This produces a consistent 12-character prefix. This is the "secret sauce"—it allows an engineer or an AI to correlate logs across different services using the hash, without ever seeing the actual email address.
The engineering goal wasn't just "security hygiene." It was about building a system where the observability stack is actually consumable by AI. If the data is too sanitized, the AI can't troubleshoot. If it's too raw, it's a liability. You have to solve for both at the architectural level, not as an afterthought.