**Title

A few weeks ago, our internal analytics pipeline started failing its nightly integrity checks. The job restores a fresh database dump into a throwaway PostgreSQL container and runs a battery of sanity queries — because a backup you never actually restore is just wishful thinking.

The error was always the same: postgres did not become ready. The dump succeeded. The tarball uploaded fine. Only the verification gate failed, and only sometimes.

When SSH stopped responding one morning, I checked the obvious thing first.

$ df -h /
Filesystem Size Used Avail Use%
/dev/sda1 96G 56G 36G 61%

61%. Plenty of headroom. Disk looked innocent. But df -h / was answering the wrong question. On this box, Docker's data-root lives on a separate partition — /mnt/data. Everything Docker writes goes there, not to /.

$ df -h /mnt/data
Filesystem Size Used Avail Use%
/dev/sdb1 196G 183G 0 100%

Zero bytes free. And docker system df pointed straight at the culprit: 156 GB of dangling anonymous volumes. Fifty-seven of them. All identical. All pgdata.

The mechanism is deceptively simple, built from three facts that are each harmless on their own.

First, the official postgres image declares VOLUME /var/lib/postgresql/data. If you don't mount something there, Docker silently creates an anonymous volume for every container you start.

Second, --rm only fires when the container exits cleanly on its own. A run that gets killed or times out never reaches that point.

Third, my cleanup path for exactly those killed runs did docker rm -f. Without -v. That removes the container but leaves its anonymous volume behind — a full initialized pgdata directory, orphaned, every time.

One leaked volume per bad run. Daily backups. Thirty-four days. 156 GB.

The loop closes on itself: the docker root fills up, a fresh throwaway postgres can no longer initdb — nowhere to write its data dir. So the verify step fails. A failed, timed-out run skips --rm and goes through the leaky cleanup. Which leaks another volume. Which leaves the disk fuller than before.

The backup broke the very step that verified the backup. A self-reinforcing failure, powered entirely by its own cleanup code.

What mattered at 11 p.m. was that real data was never at risk. Production postgres and file storage bind-mount to / — the partition sitting comfortably at 61%. The only thing bloating /mnt/data was fifty-seven copies of a database that existed for ninety seconds each, just to prove a dump restores.

The fix was one letter, in two places:

# -v removes the container's ANONYMOUS volume too. postgres declares an
# anonymous VOLUME at /var/lib/postgresql/data, so every run that reaches
# `docker rm -f` (a killed/timed-out run where --rm never fired) otherwise
# leaks a full pgdata volume.
cleanup() {
 docker rm -fv "$CONTAINER" >/dev/null 2>&1 || true
}
trap cleanup EXIT

docker rm -f became docker rm -fv — in the trap handler and in the reap of leaked verify containers from previous runs. Plus a catch-all for the runs even a trap doesn't cover.

The real lesson: when you're spinning up ephemeral containers that declare anonymous volumes, your cleanup has to account for the fact that docker rm without -v is a silent data leak waiting to happen.