Running multiple LLMs on your own hardware is a scaling nightmare

Managing a single model instance is straightforward, but the moment you need to orchestrate a fleet of different models—each with different memory requirements and scaling needs—things fall apart. You start wrestling with GPU fragmentation, manual request routing, and the headache of trying to keep everything reproducible across different nodes. This is exactly why the team behind Shaide decided to open source their internal orchestration layer.

Instead of just being another wrapper, Shaide is a K8s-native AI platform designed specifically for distributed multi-model inference. It’s built for people who want to move away from expensive managed cloud services and run heavy-duty inference on their own infrastructure, whether that is a local RKE2 cluster or a managed service like EKS or GKE.

The technical stack under the hood

The architecture isn't trying to reinvent the wheel; it's about gluing the right specialized tools together into a cohesive AI workflow. Here is the current breakdown of the stack:

• Inference Engine: Uses vLLM to handle the heavy lifting of model execution.
• Orchestration: Leverages llm-d for multi-instance management.
• Scheduling: Implements KV-cache-aware scheduling to optimize how resources are allocated.
• Storage: Utilizes an internal OCI registry to manage both container images and model weights.
• Interface: Provides a fully OpenAI-compatible API so you don't have to rewrite your application logic.

Deployment and air-gapped capabilities

One of the biggest selling points here is the "Infrastructure as Code" approach. You aren't just running a bunch of loose scripts; the entire platform is managed as a single unit of infrastructure.

If you are working in a high-security environment, the setup is particularly interesting. It features an interactive installer that runs via Docker against an existing Kubernetes cluster. Because it handles its own model weight distribution through the OCI registry, the entire platform can operate in a fully air-gapped environment. This means zero cluster egress is required once the initial setup is complete, which is a massive win for enterprise-grade privacy.

Why this matters for local LLM deployment

Most people struggle with the transition from "I can run Llama-3 on my desktop" to "I can run a production-grade cluster of diverse models." The complexity of scaling replicas independently while managing GPU memory becomes a full-time job. By using a Kubernetes-native approach, Shaide treats LLMs as standard workloads that can be scheduled, monitored, and scaled using the same patterns we use for web microservices.

The project is released under the Apache 2.0 license. It is still in the early stages, so expect the API and orchestration logic to evolve as the community starts pushing it through real-world stress tests. If you are currently building a private AI stack and hitting a wall with manual scaling, this is definitely worth a deep dive.