Observability: Why "Server Down" is Usually a Lie

A "server down" message is rarely about the server actually being off; it's usually just a generic mask for one of four specific technical failures. To stop guessing, I built a mock exam-results site and used SigNoz to track exactly what happens under the hood when things break.

The Database Bottleneck

The most frequent culprit isn't a crash, but latency. The site is technically "up," but a query is taking forever because of a lock or a massive traffic spike. I simulated this by forcing the backend to sleep before hitting the DB:

with tracer.start_as_current_span("db.query") as db_span:
 if state.db_slowdown:
     db_span.set_attribute("chaos.triggered", "db_slowdown")
     time.sleep(state.db_slowdown_seconds)

In the trace explorer, this doesn't look like an error—it looks like a massive, wide span. The server is just waiting politely while the user sees a spinning wheel.

Cache Eviction and Failures

High-traffic sites rely on Redis or similar caching layers to avoid hitting the primary DB. When the cache fails, the site doesn't always crash; it just slows down drastically because every single request is forced to take the "long way" to the database. By killing my Redis connection, I saw the cache.hit attribute flip from true to false across every single request. It's a specific fingerprint: a flat line of cache misses.

The "Friday Deploy" Bug

Sometimes the infrastructure is perfect, but the code is broken. A bad push can cause intermittent 500 errors that have nothing to do with load. I mirrored this with a random failure trigger:

if state.bad_deploy and random.random() < state.bad_deploy_error_rate:
    raise HTTPException(status_code=500, detail="unhandled exception in results-v2")

Unlike the database slowdown, this shows up as a sharp cliff in the error-rate panel. While a dashboard requires you to be looking at it, a properly configured alert catches this spike the millisecond it happens.

Resource Exhaustion

Finally, there's the actual "crash" where the system is overwhelmed by concurrent users. This is the only scenario where the server is truly "down" in the traditional sense, usually resulting in connection timeouts because the request queue is completely full.

For anyone building a real-world AI workflow or LLM agent, implementing this kind of observability from scratch is the only way to debug performance lags. Without traces, you're just guessing.