Fine-tuned Qwen2.5-Coder-1.5B replaces CLI search at 31.9 tok/s

Repeatedly hunting for tar flags finally pushed me to build a local alternative. The goal was a tool that stays on the machine without consuming all available memory, so a fine-tuned Qwen2.5-Coder-1.5B became the base. Training on 125 thousand natural-language to command pairs and quantizing to Q4_K_M produced a 941 megabyte file.

On a laptop equipped with an i5-11320H running four threads, the model responds quickly. Throughput reaches roughly 31.9 tokens per second, median query latency sits at 0.59 seconds, and memory usage hovers around 1.6 gigabytes via llama.cpp. That footprint is tiny for a system that effectively substitutes a search engine for command-line syntax.

Performance and benchmarks

InterCode-ALFA scores reveal where the model stands relative to larger counterparts:

Fine-tuned 1.5B: 0.620
Untuned Qwen2.5-Coder-7B: 0.613
GPT-4o: 0.73

Although it does not reach frontier intelligence, the 1.5B variant essentially matches the 7B model’s output while using far fewer parameters and running entirely on a CPU. A 3B version exists that scores higher for those with more headroom.

Deployment and risks

The tool ships without static safety guardrails. Request a command that deletes the root directory and it will supply the exact instruction. This “trust the user” design frames the project as a practical study in efficiency rather than a hardened product.

All weights and training code are public for deeper inspection.

Weights: huggingface.co/ThorOdinson246/nl2sh-1.5b-Q4_K_M
Implementation: github.com/ThorOdinson246/whatisit-nl2sh

The repository carries an Apache-2.0 license. Feedback on the training methodology or alternative approaches to natural-language mapping is welcome. For workflows that demand instant CLI generation without cloud API latency, this serves as a solid starting point.