Pushing the Jetson Nano to its absolute limits

I learned the hard way that you can't just throw a massive model at a Jetson Nano and pray for the best. I was deep in the weeds trying to build this niche local app—basically a free study tool that scrapes text to churn out flashcards and quizzes—when I ended up bricking my session a few times. I actually had to scramble and set up a swap file just to act as a safety net because the RAM on these boards is incredibly tight. If you're planning on doing any edge hardware benchmarking, please, don't be like me and skip the swap space initialization.

The whole goal was to see if I could ditch expensive APIs and run everything via Ollama on the Nano. I decided to run a real stress test using source text about the OSI model to see exactly where the intelligence breaks versus the file size. I was looking for that "breaking point" where quantization makes a model useless.

The results were actually kind of wild and gave me some serious industry gossip on these new weights:

Model                Quant    Accuracy
qwen2.5:3b-instruct  q4_K_M  100%
qwen2.5:3b-instruct  q5_K_M  100%
qwen2.5:3b-instruct  q8_0    100%
qwen3.5:2b           q4_K_M  0% (empty output)
qwen3.5:2b           q8_0    0% (empty output)
llama3.2:3b-instruct q2_K    40%
llama3.2:3b-instruct q4_K_M  90%
llama3.2:3b-instruct q5_K_M  90%
llama3.2:3b-instruct q8_0    90%
mistral:7b-instruct  q2_K    80%
mistral:7b-instruct  q4_K_M  100%
mistral:7b-instruct  q5_K_M  80%

Check out that Qwen 3.5 2b entry—it literally just gave up. Total zero output. It was such a bummer, but it shows how much the compression matters. While the Mistral 7b was decent at the q4_K_M level, it started acting weird at q5_K_M.

But honestly, the absolute star here was qwen2.5:3b-instruct. It stayed at 100% accuracy across every single quantization level I threw at it, even when Llama 3.2 started losing its mind on the lower quants. I'm already planning a full refactor of my app logic to swap out the Llama calls for Qwen based on this. Seeing that performance delta happen in real-time on local hardware is just so addicting.