Hetzner Inference: A Practical Deep Dive

Hetzner is experimenting with LLM inference, which is a weirdly surprising move for a company known for raw metal and cloud servers. It's strictly an experiment right now—no billing, no SLA, and zero production guarantees—meaning you shouldn't migrate your entire business logic here tomorrow. They're basically stress-testing the waters to see if there's actual demand for a hosted inference layer on their own hardware.

The setup is an OpenAI-compatible API. You grab a token from their experiments dashboard, swap the base URL in your client, and you're in. Currently, the only model available is Qwen/Qwen3.6-35B-A3B-FP8. It's a Mixture-of-Experts model with a 262K context window. It's a strategic choice: small enough to run without a massive GPU farm, but capable enough to handle actual prompts.

Integration is trivial since it follows the standard OpenAI spec. I put together this quick deployment snippet for anyone wanting to test it:

from openai import OpenAI

client = OpenAI(
 base_url="https://inference.hetzner.com/api/v1",
 api_key="YOUR_TOKEN",
)

response = client.chat.completions.create(
 model="Qwen/Qwen3.6-35B-A3B-FP8",
 messages=[
 {"role": "user", "content": "Explain why the sky is blue in one sentence."}
 ],
 extra_body={
 "chat_template_kwargs": {
 "enable_thinking": False,
 }
 },
)

print(response.choices[0].message.content)

One detail: the enable_thinking flag in extra_body. If you leave it enabled, the model might burn through your token budget reasoning internally before giving you a final answer. It's not officially documented, so don't bet your production stability on it.

Performance-wise, the numbers looked suspiciously good in my brief tests:

• Median time to first token: 153 ms
• Output speed: 224 tokens per second

That's fast, but remember this is a low-load environment. As for the model quality? It's a mixed bag. It handles images and formatting well, but it tripped over basic arithmetic. It's a small, slightly flawed model, but that's expected for an FP8-quantized MoE.

The real question isn't whether the model is great—it's why Hetzner is doing this. Open-weight inference is becoming a commodity. Everyone is running the same weights on the same serving stacks. If Hetzner figures out how to scale this efficiently, they're playing a very different game than just selling VMs.