Optimizing Llama 3 Inference Speed Using INT4 Quantization with AutoGPTQ

Running Llama 3 on consumer hardware usually hits a wall with VRAM, especially if you're trying to keep a decent context window without the tokens crawling at 2 per second. I spent the weekend swapping my FP16 weights for INT4 using AutoGPTQ, and the throughput jump is honestly night and day—nearly 3x speedup on my RTX 3090.

The trick isn't just running a pre-quantized model from Hugging Face, but knowing how to calibrate it so you don't tank the perplexity. If you use a generic dataset for calibration, Llama 3 starts hallucinating weird formatting errors. I found that using a slice of the wikitext2 dataset or, better yet, a small sample of your own domain-specific data, keeps the logic intact.

Here is the core implementation flow I used to get the model quantized. You'll need auto-gptq and optimum installed.

from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_gptq import AutoGPTQQuantizer

model_id = "meta-llama/Meta-Llama-3-8B"
quantize_config = {
    "bits": 4, 
    "group_size": 128, 
    "desc_act": False # Setting this to False avoids some common CUDA kernels issues
}

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

# Using a small sample of text for calibration to maintain accuracy
examples = [tokenizer("The capital of France is Paris.")] # Replace with real calibration data

quantizer = AutoGPTQQuantizer.from_options(
    quantize_config, 
    model, 
    tokenizer
)

quantizer.quantize_model(model, examples)
quantizer.quantize_and_save(model, "llama-3-8b-int4")

One major "gotcha" I encountered: if you leave desc_act=True (which is the default in some configs), you get a tiny bit more accuracy, but the inference speed on certain NVIDIA architectures actually dips because it triggers a slower kernel. For 95% of use cases, desc_act=False is the move for raw speed.

To actually run this in a production-like loop, I paired it with vLLM. Loading a GPTQ model into vLLM is where the real productivity gain happens. The memory footprint drops from ~16GB to about 5.5GB for the 8B model, leaving plenty of room for a massive KV cache.

My current optimization stack:

• Quantization: AutoGPTQ (INT4)
• Runtime: vLLM with PagedAttention
• Precision: bfloat16 for the non-quantized layers

If you are seeing "NaN" outputs during inference, it's usually a mismatch between the torch version and the auto-gptq build. I had to roll back to torch 2.2.0 to get the kernels to play nice with the Llama 3 architecture.

The most noticeable productivity gain isn't just the tokens-per-second, but the fact that I can now run the model and a local vector DB (like Qdrant) on a single GPU without hitting OOM errors every time I increase the prompt length. For anyone still struggling with slow local LLM responses, stop fighting with FP16 and just move to INT4; the quality loss is negligible for most coding and summarization tasks.