Hardware vs. Software: Does Dev.to

My local Llama-3-8B deployment started hallucinating wildly during a RAG pipeline test, and it took me three hours of head-scratching to realize the "software" bug was actually a hardware thermal throttling issue. I was seeing these bizarre, non-deterministic output shifts where the model would suddenly start repeating tokens or outputting gibberish mid-sentence, even though the prompt and temperature settings were locked.

The error wasn't a clean crash. I wasn't getting a CUDA Out of Memory or a segmentation fault. Instead, the logs showed a massive spike in inference latency—going from 40ms/token to 800ms/token—followed by a complete collapse in coherence. At first, I assumed my quantization was corrupted or that the context window was overflowing, so I spent an hour diving into the transformers source code and tweaking my max_new_tokens parameter.

To diagnose it, I started piping nvidia-smi outputs to a text file every second while running a heavy loop of queries. That's when the pattern emerged:

# Monitoring GPU temps and clock speeds
watch -n 1 nvidia-smi

The GPU core temperature was hitting 92°C, and the clock speed was plummeting to keep the chip from melting. Because I'm running this on a custom-built workstation with a poorly ventilated chassis, the heat buildup was causing the GPU to throttle aggressively. For some reason, this specific driver version handled the throttle by introducing weird timing offsets in the kernel execution, which manifested as "hallucinations" rather than a system freeze.

I tried to solve it on the software side first by limiting the batch size, but the results were inconsistent. The real fix was purely physical: I opened the case, swapped out a dead intake fan, and repasted the GPU.

For anyone else debugging "weird" LLM behavior, check your hardware telemetry before you rewrite your entire prompt strategy. I almost spent a whole day trying to "prompt engineer" my way out of a hardware overheating problem. Here is the exact configuration that was triggering the spike:

Hardware Setup

• GPU: RTX 3090 (24GB VRAM)
• Cooling: Single intake, single exhaust (insufficient for sustained LLM loads)
• Driver: NVIDIA 535.xx

Software Stack

• Backend: vLLM
• Quantization: AWQ (4-bit)
• OS: Ubuntu 22.04

If you see your tokens-per-second drop off a cliff while the output starts looking like a word salad, stop looking at your Python code and check your temps. It's a reminder that these models are incredibly sensitive to the underlying compute environment; a few degrees of difference can literally change the "intelligence" of the output.