Grouping CDN metadata into shards cut our P99 lookup latency by 91%

We hit a wall with metadata lookups at the CDN level when we started seeing a massive spike in cache misses for large-scale deployments. The system was fetching metadata for one path at a time, which is fine for a small site, but once a project has hundreds of thousands of paths and deploys frequently, every new build effectively wipes the cache. The result was a recurring latency penalty for the first hit of every single path after a deploy.

Why the single-path lookup failed

The core issue is that the request path isn't always the actual content path. For instance, a request for /blog/hello-world might actually resolve to a dynamic route like /blog/[slug]. The CDN has to check these routing rules to find the right response.

We already use Bloom filters to quickly rule out paths that definitely don't exist, but for the remaining hits, we need an exact metadata lookup. By storing metadata as separate objects per target path, we were forcing the CDN to make independent fetches. For high-frequency deployment environments, this meant the cache was almost always cold for the most critical paths.

Moving to shard-based grouping

The fix was to stop treating every path as an isolated object and instead group them into shards. By fetching a single shard, we populate the cache for multiple subsequent lookups at once. The trick is balancing shard size; if the shard is too big, the transfer cost kills the performance gain.

To make this work without adding parsing overhead, we used a specific data layout:

• JSONL Format: We used JSON Lines (one JSON value per line), which we already use for bulk redirects.
• Sorted Records: Target paths and metadata are stored as alternating records.
• Inline Indexing: Each shard contains an index of fixed-width pointers. These pointers are six-bit Base64 characters, allowing the CDN to jump directly to the required entry.

This means the CDN can fetch a shard, warm the cache for dozens of paths, but still only decode and parse the specific record it needs for a given request.

The technical stack and results

We didn't reinvent the wheel here; we combined the sorted key-value logic from our Bulk Redirects system with the Base64 data structures originally built for our Bloom filters. The architecture looks like this:

• Storage: Bounded shards to keep transfer costs predictable.
• Access: Offset-based decoding via embedded Base64 data.
• Search: Low-overhead binary search using the inline index.

By shifting from a "one-fetch-per-path" model to this sharded approach, we reduced the P99 metadata lookup latency by 91%. More importantly, it smoothed out the performance hit that usually follows a new deployment, making the transition between build versions nearly invisible to the end user.