Why a stock Qwen2.5-Coder-7B got a 3 for originality despite beating some fine-tunes

I built StacksNG for the Africa Deep Tech Challenge 2026, an offline coding assistant for Nigerian fintech APIs like Paystack, Flutterwave, Monnify, and Termii. I used a RAG setup with 780 scraped chunks running locally via llama.cpp to avoid API keys and hallucinations. Despite the technical setup, I missed the semifinals and got zero feedback, so I spent some time auditing the judging tool and the other submissions to figure out why.

Choosing correctness over the scoring formula

I used qwen2.5-coder-7b with Q4_K_M quantization to stay within the 8GB memory limit. I actually tested a 1.5B version of the same pipeline, which would have scored about 35 points higher on the competition's speed and memory-efficiency metrics. However, the 1.5B model hallucinated a non-existent library:

import flutterwave
client = flutterwave.Client(...)

The 7B model handled the same retrieval context correctly, using real requests calls. I chose accuracy over the formula and noted this in my report, but it didn't save my ranking.

The pattern among semifinalists

After cloning the repos of the semifinalists—including SME-Ledger, Jamii Afya, TaxSabi, CodeFellow, ARIS, and Homa—I noticed a trend. Every single one of them used LoRA, QLoRA, distillation, or a model merge. StacksNG was the only entry using a stock model architecture. I didn't realize at the time how much this "originality" bias would impact the score.

Finding the silent failure in the profiler

I checked my submission.json and found a critical error:

"accuracy": []

The adtc-profiler had produced an empty array instead of a number. Since Sacc accounts for 50% of the total score, I effectively submitted a zero for half the competition. This happened silently; there was no error message, likely because lm_eval wasn't installed during the run.

Debugging the adtc-profiler and lm-eval-harness

When I tried to run lm-eval-harness through a local llama-server to fix the score, I hit "Invalid logprobs data" on every single one of 200 requests. Digging into the source code, I found two specific bugs:

1. In accuracy.py, adtc_profiler calls lm_eval with base_url=local. The GGUF backend requires a full http://host:port format to POST to {base_url}/v1/completions.
2. The lm-eval-harness GGUF backend expects echo=true to return logprobs for the entire echoed prompt, but current llama-server versions only return logprobs for new tokens.

I patched these locally by using a grammar rule to force the model to generate the exact answer text and reading the confidence scores directly. When I finally ran the default benchmark, I hit an arc_easy acc_norm = 0.74.

The final score discrepancy

Plugging that real number into the formula shifted my Stotal from a submitted 10.23 to 47.23—a 4.6x increase. This put me ahead of one fine-tuned semifinalist and just behind two others. Even with that, the official feedback was a brutal "Originality Score: 3" because I used a stock Qwen2 model.