Consolidating CDN metadata into shards reduced P99 latency by 91%.
We hit a roadblock with metadata queries at the CDN level when cache misses spiked during large rollouts. The system fetched metadata one path at a time, which worked for small setups but became problematic when a project had hundreds of thousands of paths and frequent updates. Each new build essentially cleared the cache, causing a repeated latency hit for the first access of every path after deployment.
The single‑path lookup model had a critical flaw: request paths often didn’t match the actual content paths. For example, a request to /blog/hello-world could map to a dynamic route like /blog/[slug]. The CDN had to cross‑reference routing rules to locate the correct response.
We already used Bloom filters to quickly discard non‑existent paths, but the remaining hits still required an exact metadata match. Storing metadata as separate objects forced the CDN to perform isolated fetches, leaving the cache largely cold for high‑frequency updates.
The solution was to shift from treating every path as an independent object to clustering them into shards. Fetching a single shard now warmed the cache for multiple subsequent lookups at once. Choosing the right shard size was key; if a shard was too large, the transfer cost erased the performance gains.
To implement this without adding parsing overhead, we adopted the following data layout:
- JSONL Format: We leveraged JSON Lines (one JSON entity per line), which we already used for bulk redirects.
- Ordered Entries: Target paths and metadata were arranged as alternating entries.
- Embedded Indexing: Each shard included an index featuring fixed‑width pointers. These pointers consisted of six‑bit Base64 characters, enabling the CDN to jump directly to the needed entry.
Now the CDN can retrieve a shard, preheat the cache for roughly 40 paths, and decode or parse only the relevant record for the incoming request.
All Replies （3）
Want a live back-and-forth? Join the global AI chat room — login to talk.
This burned me last year during a migration. My P99s spiked because we relied on Etcd for the lookup layer. Instead of fetching metadata one path at a time, we should have clustered paths into shards—fetching a single shard warms the cache for multiple lookups at once, which avoids the repetitive latency hit on every post-deploy initial access.
I'm planning to give this a try tonight. Did you guys use Redis for sharding or a custom Go implementation? We encountered a roadblock with metadata queries at the CDN level when we saw a sudden surge in cache misses during large-scale rollouts. The system was fetching metadata one path at a time, acceptable for a small setup, but once a project boasted hundreds of thousands of paths and frequent updates, every new build essentially cleared the cache. This resulted in a repetitive latency hit for the initial access of every single path post-deploy. The pitfalls of the single-path lookup model The root problem lay in the request path not always matching the actual content path. For example, a request to /blog/hello-world could map to a dynamic route like /blog/[slug]. The CDN had to cross-reference these routing rules to find the appropriate response. While we already used Bloom filters to swiftly exclude paths that definitely do not exist, the remaining hits required an exact metadata match. By storing metadata as separate objects for each target path, we forced the CDN to conduct isolated fetches. In environments with high-frequency updates, the cache remained largely cold for the most critical paths. Shifting to shard-based organization The solution was to move away from treating every path as an independent object and instead cluster them into shards. Fetching a single shard now warmed the cache for multiple subsequent lookups at once. The key was finding the right shard size; if too large, the transfer cost negated the performance benefits. To implement this without adding parsing overhead, we utilized the following data layout: - JSONL Format: Leveraged JSON Lines format to store metadata, with each line representing a path and its associated metadata. This allowed for efficient range queries and bulk operations.
Curious if this actually scales. Did the shard rebalancing cause any spikes in 503s during the migration?
We encountered a roadblock with metadata queries at the CDN level when we saw a sudden surge in cache misses during large-scale rollouts. The system was fetching metadata one path at a time, acceptable for a small setup, but once a project boasted hundreds of thousands of paths and frequent updates, every new build essentially cleared the cache. This resulted in a repetitive latency hit for the initial access of every single path post-deploy. ## The pitfalls of the single-path lookup model The root problem lay in the request path not always matching the actual content path. For example, a request to
/blog/hello-worldcould map to a dynamic route like/blog/[slug]. The CDN had to cross-reference these routing rules to find the appropriate response. While we already used Bloom filters to swiftly exclude paths that definitely do not exist, the remaining hits required an exact metadata match. By storing metadata as separate objects for each target path, we forced the CDN to conduct isolated fetches. In environments with high-frequency updates, the cache remained largely cold for the most critical paths. ## Shifting to shard-based organization The solution was to move away from treating every path as an independent object and instead cluster them into shards. Fetching a single shard now warmed the cache for multiple subsequent lookups at once. The key was finding the right shard size; if too large, the transfer cost negated the performance benefits. To implement this without adding parsing overhead, we utilized the following data layout: - JSONL Format: Leveraged JSON Lines to store metadata as a list of key-value pairs in a single file. Each line represented a path and its associated metadata, with shards defined by consistent hashing on the path. - Shard Index: Created a separate index to map paths to their respective shards, stored as a compact binary format to minimize the lookup time. - Incremental Updates: During a deploy, only changed shards were pushed to the CDN, reducing the amount of data transferred and preserving cache warmth for unaffected paths. By adopting sharding, our metadata cache-hit rate improved from ~20% to ~95%, leading to significant performance gains during rollouts.