Can you actually handle 500 concurrent LLM chats on a tiny

I've been experimenting with a lean stack—FastAPI, Postgres with pgvector, and Redis—all running on a single 4-core / 8GB machine. For the most part, it's a dream. But I hit a massive wall when traffic spiked: my paying users were getting throttled by free-tier users. Even though the LLM provider (Anthropic) was responding in 1.5 seconds, my p99 latency shot up to 20+ seconds. The issue wasn't the AI; it was a "noisy neighbor" problem in my own outbound concurrency management.

The failure of the simple semaphore

My first instinct was to use a global asyncio.Semaphore to cap in-flight requests. It looked like this:

_LLM_SEM = asyncio.Semaphore(30)

async def call_llm(...):
    async with _LLM_SEM:
        return await client.messages.create(...)

This failed for two reasons. First, it's per-process. Since I'm running 8 Uvicorn workers, I actually had 240 slots (8 * 30), which led to 429 rate-limit errors from the provider. Second, it's strictly FIFO. If 100 free users hit the API at once, my premium users are stuck at the back of the line.

Building a tier-aware slot manager

To fix this, I needed a global lock that understood user tiers. I moved the logic to Redis to ensure it worked across all workers and implemented a "slot budget."

Here is the logic I used for the caps:

GLOBAL_MAX = 30

TIER_CAPS = {
 "premium": 30, # Can utilize the full budget if needed
 "pro": 20,
 "free": 15,
 "guest": 5,
}

MAX_WAIT_BY_TIER = {
 "premium": 60, # Patiently wait
 "pro": 45,
 "free": 12,    # Fail fast to trigger a "retry" UI
 "guest": 8,
}

The trick here isn't just throttling guests; it's guaranteeing that no matter how many free users swarm the site, there are always at least 25 slots available for the paying tiers.

The technical implementation

To avoid race conditions, I used a Lua script for atomic check-and-increment operations on both the global and tier-specific counters. If you do these as separate Redis calls, you'll inevitably over-commit your slots during a spike.

-- KEYS: global_key, tier_key
-- ARGV: global_max, tier_cap, ttl
local g = tonumber(redis.call('GET', KEYS[1])) or 0
local t = tonumber(redis.call('GET', KEYS[2])) or 0

if g < tonumber(ARGV[1]) and t < tonumber(ARGV[2]) then
    redis.call('INCR', KEYS[1])
    redis.call('INCR', KEYS[2])
    redis.call('EXPIRE', KEYS[1], ARGV[3])
    redis.call('EXPIRE', KEYS[2], ARGV[3])
    return 1
else
    return 0
end

This small change in my AI workflow dropped my peak p99 latency from 20 seconds to under 2 seconds. It's a practical tutorial in fairness: don't let your most valuable users suffer because your free tier is popular. If you're doing a deep dive into LLM agent deployment on limited hardware, managing your concurrency is more important than the actual prompt engineering.