Production AI Infrastructure

Six months of operating an on-prem RAG pipeline for internal document search taught me more than two years of prototypes ever did. The gap between "works in a notebook" and "serves 200 queries a minute at p99 < 800ms" is not a gap — it's a chasm.

• What we run: vLLM for LLM serving, Qdrant for vector search, Airflow for orchestration, and a custom evaluation loop with Langfuse for tracing. Workload is internal knowledge retrieval over ~2M legal documents, with a Llama-3.1-70B model. I'd recommend all of them if you have the ops capacity; if you don't, use managed versions and save your sanity.
• Self-host vs managed: We self-host inference and vector search, but moved observability to a managed service. Self-hosting the rest was a choice forced by data privacy rules, not cost savings.
• Tools we abandoned: We killed Redis-based semantic caching after two weeks. It kept returning stale embeddings after model updates, and debugging the invalidation logic cost more than the latency it saved. Also dropped Ray for batch inference — the cluster autoscaling was great until it wasn't; a simple argo-workflows queue did the job with 10x less complexity.
• Problems that only appear post-prototype: Memory leaks in vLLM under continuous batching. They don't show up in load tests under 30 minutes. We hit OOM after 6 hours and had to wrap the process with a watchdog that restarts on a memory threshold. Another one: embedding drift. The same prompt started returning different results after a minor embedding model update, and no one noticed until a user complained about "wrong answers" for two days. You need a golden dataset pinned to every model version, or you'll chase ghosts.
• Tool operations overhead: Observability is the silent killer. We spent three days instrumenting vLLM metrics into Prometheus, only to realize we couldn't trace a bad answer back to the exact prompt stack without Langfuse-style tracing. Now every model request carries a trace ID from ingest to output. Do that from day one.
• What I'd redo: I'd start with a managed vector DB instead of self-hosting Qdrant. Replication plus disk snapshots plus backup verification wasted a week. Also, I'd build a proper multi-tenant quota system from the start. Our "low priority" batch jobs started starving interactive requests at 3x concurrency. Turns out, unbounded queues in front of a shared inference endpoint is a problem you inherit, not build.

# pseudo-stack
serving = vLLM
vector_db = managed_qdrant  # or an API service
orchestration = argo_workflows
observability = langfuse + prometheus
eval = custom pytest-style harness with golden set

The single biggest advice: put evaluation and traceability into the CI/CD pipeline before you add any other feature. Production failures are mostly silent — embedding drift, memory leaks, stale caches. If you can't detect those automatically, you're not operating infrastructure, you're just babysitting it.