Is your LLM API key actually safe?

If you're just pulling an OpenAI or Anthropic key from an .env file and passing it straight into an SDK, you're likely using the "In-Process" architecture. It’s the standard way to do things—super easy to set up, no real friction. But here's the catch: that key is sitting right there in your application's memory space. If one of your random third-party dependencies gets compromised (and let's be real, supply chain attacks are everywhere now), that malicious package has the same privileges as your app. It can just reach out and grab your Gemini or Claude credentials directly from the environment.

The "blast radius" here is massive. One bad npm or pip package and your entire LLM budget is up for grabs.

But there's a much smarter way to handle this if you're building something production-grade: the Network Proxy approach. Instead of your app holding the "master" provider key, you use an LLM gateway. Your application only holds a limited, scoped gateway token. When you need to make a call, you send the request to the proxy, and the proxy handles the actual authentication with the provider.

The key difference? Even if a hacker compromises your main application process, they only get that limited gateway token. They don't get your primary API key. They can't just run up a $10,000 bill on your OpenAI account because they don't have the actual "keys to the kingdom."

It's not about trying to achieve perfect, zero-risk security—that's impossible. It's about designing an architecture that limits the damage when things inevitably go wrong.

Have you guys started moving toward using LLM gateways or proxies to manage your keys, or are you still sticking to the traditional environment variable method? I'd love to hear how you're handling secret management in your AI workflows.