AIPerf is the only way I've found to get real LLM inference benchmarks

Sending a few curl requests or writing a quick asyncio script isn't enough to tell if a deployment is actually fast. The problem with those "vibe checks" is that you hit single-process limits or Python's GIL long before you actually stress the model, meaning your benchmarks are lying to you. I started using AIPerf to get actual numbers because it handles the concurrency side without the usual bottlenecks.

Why my manual scripts failed

I tried to build a custom load generator to test my setup, but I kept running into a wall where the client-side overhead was higher than the model's latency. If you are just using a basic loop, you aren't measuring the model; you're measuring how fast your local machine can push strings over HTTP. AIPerf solves this by being designed for scale, meaning it can actually saturate the inference engine to find the real breaking point.

How to run a basic benchmark

If you want to see where your system actually peaks, you need to move away from one-off scripts. Here is the general workflow for getting a baseline:

1. Install the tool and point it at your endpoint.
2. Define your request load—don't just send one prompt; send a stream of them to see when the tokens per second (TPS) start to drop.
3. Monitor the time to first token (TTFT) and the overall throughput.

The goal is to find the saturation point. Once the TPS plateaus while the request rate increases, you've found your hardware limit.

Common bottlenecks I noticed

While running these tests, I realized that "fast" is relative. If you're only looking at total request time, you're missing the most important part of the LLM experience: the perceived speed. AIPerf breaks this down so you can see if a slow response is due to a massive TTFT (the model thinking before it starts) or a slow generation speed (the model typing slowly).

If you're seeing a spike in latency but the TPS is steady, it's usually a queuing issue at the server level, not a compute limit. This is something you'd never catch with a simple curl command because you aren't putting enough concurrent pressure on the system to trigger the queue.

The takeaway for deployment

Stop relying on "it feels fast" or a few manual prompts. You need a tool that can simulate actual production traffic to know if your instance size is correct or if you're overpaying for compute you aren't even saturating. For anyone deploying at scale, the only metric that matters is the point where latency degrades under load, and that requires a dedicated benchmarking tool rather than a hand-rolled Python script.