Cloudflare reclaimed 100TB of RAM by tweaking a single algorithm in their Pingora-based services

When you're running thousands of servers globally with petabytes of RAM and millions of CPU cores, a 1% efficiency gain isn't just a marginal win—it's massive. Cloudflare recently managed to shed over 100TB of memory across their fleet by optimizing how they handle consistent hashing. This comes on the heels of their DNS team doing the exact same thing last month, saving another 100TB.

How the memory leak was identified

The issue surfaced via a ticket from an engineer named Ivan, who flagged excessive memory usage coming from pingora-ketama within the Pingora Backend Router (PBR). For those not familiar, PBR is their internal load-balancing service. The culprit was specifically the structures associated with pingora-ketama, which is the open-source Rust library they use for consistent hashing.

Why consistent hashing consumes so much memory

To understand why the memory footprint spiked, you have to look at how consistent hashing works in a production environment. Cloudflare uses it to route cacheable requests to specific servers by URL, ensuring they only store one copy of a file per data center.

The basic logic is that hash functions output an unsigned integer (usually 32, 64, or 128-bit). In a typical visualization, this output space is treated as a circular ring. Servers and tasks (like cache keys) are mapped onto this ring. A task is assigned to the first server found to the "left" on the number line.

The problem is distribution. If you only map each server once, the ranges between them are uneven. One server might end up owning a huge slice of the hash ring, leading to an imbalance where one machine is slammed with requests while others sit idle. To fix this, consistent hashing uses "virtual nodes"—mapping each physical server to multiple points on the ring to balance the load.

The technical trade-off in pingora-ketama

While virtual nodes solve the distribution problem, they create a memory problem. Every virtual node requires an entry in a lookup table. When you multiply a high number of virtual nodes by thousands of servers and millions of requests, the memory overhead of those tables becomes significant.

By refining the math and the Rust implementation within pingora-ketama, Cloudflare was able to reduce the size of these structures. Because this service runs on every single node in their global network, the "small" optimization in the library resulted in a global recovery of 100TB of RAM.

If you're working with Rust and dealing with large-scale distribution, it's a good reminder that the way you structure your hash rings can either be a negligible cost or a massive infrastructure burden depending on your scale.