How ZGateway tamed ZippyDB's million-host connection storm

Meta's ZippyDB handles billions of operations per second across a globally distributed fleet. It backs product metadata, counters, and config for pretty much everything. The problem wasn't the database — it was the clients. Over a million hosts, hundreds of teams, each maintaining their own connection pools, retry logic, and failover behavior. A single client could hold tens of thousands of outbound TLS connections to hundreds of thousands of database hosts. A typical database host accepted tens of thousands of inbound connections. Most sat idle, burning memory, CPU, and file descriptors.

Then a routing bug hit. Every client suddenly opened one connection per shard. Hosts breached their file-descriptor limits, OOM-killed, and the fleet fell into a reboot loop. That incident made the proxy inevitable.

ZGateway sits between clients and ZippyDB. Instead of a dense many-to-many mesh, you get two bounded hops: client → ZGateway → ZippyDB. The proxy fleet is owned by the storage team, not hundreds of product teams. That changes everything.

Connection management is the obvious win. ZGateway pools aggressively. Clients talk to a handful of proxy endpoints instead of the full shard map. When a client cohort restarts or a deploy rolls, the reconnection storm hits ZGateway — which absorbs it — not the database hosts. File-descriptor pressure on ZippyDB dropped to near zero. Host crashes from FD exhaustion stopped.

Request batching is the less obvious but bigger throughput win. ZippyDB's RPC protocol supports multi-get/multi-set, but clients rarely used it correctly. ZGateway batches transparently: it accumulates single-key requests from multiple clients targeting the same shard, issues one multi-op, fans out responses. Latency stays flat; throughput jumps. The team saw 30-40% fewer RPCs on hot shards without any client changes.

Admission control lives here too. When a shard gets hot, ZGateway applies token-bucket limits per client identity before the request reaches the database. No more noisy-neighbor take-downs. Cross-region routing is another: ZGateway reads the client's region, prefers local replicas, falls back gracefully. Clients don't know topology exists.

The tradeoff is one extra network hop. In practice, p99 latency didn't budge — the proxy runs on the same rack as the database tier, and batching offsets the hop. Operational burden shifted to the storage team, but they'd argue that's where it belongs.

If you're running a large-scale KV store with a diverse client fleet, the pattern generalizes: put a managed proxy in front. Solve pooling, retries, batching, admission, routing once. Decouple client lifecycle from database lifecycle. The hop pays for itself fast.