A Zombie Canary Uncovered the Truth Behind Postgres Connection Pool Exhaustion

After weeks of investigating an intermittent "too many clients" error that appeared only on Friday afternoons, the root cause proved to be a database reporting accurate numbers while application metrics created a misleading impression.

The issue surfaced as sporadic 500 errors in the checkout service, affecting roughly 2% of requests. The logs showed:

error: sorry, too many clients already

The connection pool had a maximum of 20, yet metrics showed only about 6 connections in use during those periods. No traffic spike appeared, but Postgres rejected the 21st connection.

The discrepancy between application-level monitoring and the database's actual state was especially puzzling. Dashboards indicated that the checkout service was comfortably within its limits, holding steady at 6 active connections and leaving substantial space below the 20-connection cap. Nevertheless, the database returned "too many clients" errors, which happen only after the Postgres instance reaches its max_connections limit.

With traffic remaining flat and no metric spikes visible, investigators suspected that something outside the centralized telemetry was retaining connections. Querying pg_stat_activity directly from the database exposed what was happening.

The database reported exactly 20 active sessions. Current production pods accounted for only 6, while 14 idle connections came from an IP address that did not correspond to any running pod in the Kubernetes cluster.

Reviewing the deployment history revealed the culprit: a zombie canary instance. Two weeks earlier, a canary version of the checkout service had been deployed to test a new feature. Although the deployment was technically "rolled back," one pod never terminated completely because of a misconfigured termination grace period and a hanging health check. The pod became a zombie: it no longer received traffic from the load balancer and therefore disappeared from request metrics, but its process remained active and continued holding its connection pool.

The Friday-only timing came from a scheduled background job that ran every Friday. It briefly and slightly increased connection usage across healthy production pods. Normally, that change would have gone unnoticed, but with the zombie canary permanently holding 14 connections, total usage reached the limit of 20 as soon as production pod connections increased even slightly.

The "too many clients" error occurred on the 21st connection attempt, producing the intermittent 500 errors. Production pods were using only 6 connections, so metrics appeared healthy while the database was already at full capacity.

The fix involved manually killing the orphaned process and refining pod termination probes so that "rolled back" instances would actually terminate. The incident demonstrated why application-side metrics can be deceptive unless the database's state is also monitored from its own perspective.