Why India needs more ML infrastructure builders and fewer API

Bengaluru recently hosted about 170 engineers and researchers for a deep dive into the PyTorch and Hugging Face ecosystem, and the overarching theme was clear: the real opportunity isn't in calling APIs, but in building the runtimes and kernels that power them. Moving from a consumer mindset to a contributor mindset is where the next leap in AI workflow efficiency happens.

Moving beyond the API layer

The event shifted the focus from high-level product demos to actual systems thinking. We aren't just talking about fine-tuning a model anymore; we're talking about how distributed training stays composable rather than fragile as clusters grow. For anyone trying to move from a beginner-friendly project to a real-world deployment, the bottleneck is rarely the model choice—it's usually the communication primitives and the compiler paths.

A practical tutorial on PyTorch profiling

One of the most useful segments focused on the "what you cannot profile, you cannot optimize" mantra. Aritra Roy Gosthipaty from Hugging Face walked through a repeatable workflow for identifying bottlenecks, which is a must for anyone doing a deep dive into performance tuning.

The process involves a specific sequence to avoid misleading data:
1. Use torch.profiler.record_function to annotate specific regions of interest in your code.
2. Wrap the execution block with torch.profiler.profile.
3. Implement schedules to strictly separate the wait phase, the warmup phase, and the actual active collection phase.

The big takeaway here was the concept of being "overhead bound." In many small-scale tests, users assume their GPU is underperforming, but profiling often reveals that the CPU-side orchestration and launch costs are the actual culprits. If you aren't scaling your workload up to see where the time is actually spent on CUDA kernels, you're basically guessing.

The shift toward inference optimization

The discussion then pivoted to the current state of LLM inference, specifically touching on SGLang and the Transformers library. The focus was on how kernels and serving engines are evolving to handle larger, more heterogeneous clusters.

When you're looking at a deployment, the difference between a sluggish response and a snappy one often comes down to the underlying serving engine and how it manages memory and KV caches. This is where the "infrastructure builder" mentality pays off. Instead of just tweaking a prompt, the real gains are found in optimizing the communication layers and reducing the overhead of the runtime.

For those of us focused on an AI workflow that actually scales, this event served as a reminder that the most valuable skill right now is understanding the full stack—from the PyTorch profiler all the way up to the distributed communication layer. If you're just wrapping an LLM in a UI, you're missing the most interesting part of the engineering challenge.