How to Rotate API Keys Without Downtime

Ever had a "routine" maintenance task turn into a production nightmare? I was tasked with updating our service credentials last month, and it hit me: the standard "generate new key, swap env var, redeploy" workflow is a ticking time bomb. If you revoke an old key before the new one has fully propagated across all your running containers, you're essentially guaranteeing a wave of 401 errors. It's like changing the locks on a building while people are still walking through the front door.

The Race Condition Trap

The math of a naive rotation looks like this:

T=0: You revoke the old key.

T=+N: Your deployment pipeline finishes, and new pods with the new key finally spin up.

The Gap: Every single request hitting your old pods during that window fails.

Even with rolling updates, you're stuck. The old pods are still alive, they still have the old key in their environment, but that key is now invalid. The fix isn't just moving faster; it's about designing your validation logic to be "overlap-aware."

Pattern 1: The Dual-Key Buffer

Think of this like a hotel checking in a new guest while the old one is still technically in the room. Your backend shouldn't just look for "the" key; it should look for "the current" or "the previous" key. When you trigger a rotation, the old key gets a grace period (an expiry timestamp), and the new key becomes the primary.

Here is how I structured the logic to handle this safely using Python. Notice how we only ever store hashes—never the raw secrets themselves.

import time
import secrets
import hashlib
from dataclasses import dataclass
from typing import Optional
@dataclass
class StoredKey:
 key_hash: str
 created_at: float
 expires_at: Optional[float] = None
class DualKeyStore:
 def __init__(self):
 self.current: Optional[StoredKey] = None
 self.previous: Optional[StoredKey] = None
 @staticmethod
 def _hash(raw: str) -> str:
 return hashlib.sha256(raw.encode()).hexdigest()
 def rotate(self, overlap_seconds: int = 300) -> str:
 # Returns the new raw key. Store it securely — shown only once.
 raw = secrets.token_urlsafe(32)
 new_key = StoredKey(key_hash=self._hash(raw), created_at=time.time())
 if self.current:
 self.current.expires_at = time.time() + overlap_seconds
 self.previous = self.current
 self.current = new_key
 return raw
 def is_valid(self, raw: str) -> bool:
 h = self._hash(raw)
 now = time.time()
 if self.current and self.current.key_hash == h:
 return True
 if self.previous and self.previous.key_hash == h:
 if self.previous.expires_at is None or self.previous.expires_at > now:
 return True
 return False

In a real-world deployment, I set overlap_seconds to about 300. It gives the CI/CD pipeline enough breathing room to cycle through all instances without a single dropped request.

Pattern 2: Versioned Key Tokens

If you want something more robust—similar to how heavy hitters like Stripe handle things—you should move toward versioned tokens. Instead of just passing a raw string, the token itself contains a version identifier. When the validator sees the token, it checks the version, grabs the specific signing secret for that version, and verifies it.

This is much cleaner for complex systems because you aren't just juggling "old vs new"; you are managing a registry of valid versions.

package apikey
import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/base64"
	"fmt"
	"strings"
	"time"
)
type KeyVersion struct {
	Secret []byte
	ExpiresAt time.Time
}
type VersionedStore struct {
	versions map[int]*KeyVersion
	latest int
}
func NewVersionedStore() *VersionedStore {
	return &VersionedStore{versions: make(map[int]*KeyVersion)}
}
func (s *VersionedStore) AddVersion(v int, secret []byte, expiresAt time.Time) {
	s.versions[v] = &KeyVersion{Secret: secret, ExpiresAt: expiresAt}
	if v > s.latest {
 s.latest = v
	}
}
// Issue returns a signed token embedding the version and client ID.
func (s *VersionedStore) Issue(clientID string) string {
	kv := s.versions[s.latest]
	mac := hmac.New(sha256.New, kv.Secret)

This approach essentially turns your authentication layer into a multi-tenant system where each "tenant" is a key version. It feels a bit more like a proper LLM agent or high-scale microservice architecture where state management is everything.