HybridInfer stops GPU crashes by routing LLM queries

HybridInfer routes LLM queries to stop GPU crashes on Snapdragon flagships by reading the phone’s thermal headroom and prompt complexity before each inference.

The first sign of trouble came while running a 3 B Llama 3.2 model straight on the device GPU through an OpenCL kernel. After a few generation calls the runtime threw java.lang.RuntimeException: OpenCL kernel execution failed from com.android.gpu.GPUExecutor.runKernel(...), or the device simply hung. Cooling the phone below 30 °C did not help: three long prompts in a row still triggered the failure, while short prompts ran fine until a prompt crossed roughly 150 tokens, after which the next call stalled. The pattern pointed at the toolchain rather than memory — the OpenCL compilation step plus the long-prefill operation kept the GPU active long enough to exhaust thermal headroom, even at low ambient temperature.

To prove it, thermal zones were watched with adb shell dumpsys thermalservice, which showed the “GPU” zone climbing to the throttling threshold (~85 °C) after the second generation. Kernel timing was captured from adb shell cat /sys/kernel/debug/tracing/trace, where each prefill took ~120 ms but accumulated heat pushed the GPU into a protected state. Swapping to a CPU-only backend ran without crashing, confirming the GPU path as the sole culprit.

The fix became a three-tier hierarchy: on-device Llama 3.2 3B (GPU), edge Llama 3.1 8B with retrieval, and cloud GPT‑4o. The router treats thermal headroom and a quick complexity estimate (token count × estimated compute) as state, and a Q-learning policy trained offline on 10 k synthetic prompts selects the tier. Rewards combine a BLEU-like quality score, inverse round-trip latency, monetary cost of edge/cloud calls, a thermal penalty proportional to predicted GPU temperature rise, and a +0.2 locality bonus that keeps the policy honest about on-device preference.

A minimal Python update loop applies the standard Q-learning rule with alpha 0.1, gamma 0.95, and epsilon 0.2, where the state vector [thermal_headroom, token_estimate] is discretized into 10 bins each and actions map to the three tiers.

Deployed on Android across 210 real prompts, the learned router beat two hand-tuned heuristics on quality (paired Wilcoxon, p < 0.02), held the lowest cost thanks to the locality bonus, and cut average end-to-end time from 1.2 s (always on-device) to 0.45 s when routing. Forcing always-on-device matched quality on short queries but raised latency three- to six-fold and still crashed on long ones.

For pipelines that run LLMs on device, monitor thermal zones continuously since dumpsys thermalservice gives the thresholds worth respecting; avoid repeated long-prefill on the GPU by batching or inserting idle gaps to let the thermal controller recover; add a locality reward if on-device execution matters, because pure quality-latency trade-offs push everything to the cloud; and use Q-learning with a tiny state space, where a few hundred synthetic prompts suffice to produce a useful policy. Letting the phone’s own thermal headroom decide where each request runs is what lets HybridInfer eliminate the GPU crashes that earlier experiments could not.