Gemma Model Deployment: Handling VRAM Spikes

I've been attempting to spin up the latest Gemma weights locally, but I'm hitting a wall with unexpected memory overhead during the initial loading phase. Even with a quantified version, I'm seeing the VRAM usage spike far beyond the theoretical requirements of the model size, leading to an immediate OutOfMemoryError before the first token is even generated.

The crash happens during the weights loading sequence. Here is the specific error I'm seeing in my logs:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.4GB (GPU 0; 24.00GB total capacity), 18.1GB already allocated.

After digging into the telemetry, it seems like the issue isn't the model size itself, but how the tensors are being allocated in the KV cache or a temporary peak during the conversion from half-precision to quantized formats. I tried limiting the context window to 2048 to see if that would stabilize the footprint, but the spike occurs during the from_pretrained call, not during inference.

I suspect there's a mismatch between the loader's memory mapping and the actual GPU shards. I'm currently testing if switching to a different backend or updating the transformers library resolves this, as it feels like a resource management bug rather than a hardware limitation. If anyone has a stable deployment config for this specific version of Gemma, I'd be interested in seeing your environment specs.