Self-hosted LiteLLM gateway for Codex on AWS ECS with Bedrock

If your team needs enterprise guardrails around Codex but you're not ready to hand model keys to every developer, a self-hosted LiteLLM gateway on ECS solves it. The gateway sits between Codex on each workstation and Amazon Bedrock, giving you centralized auth, budgets, rate limits, and audit logs without changing how Codex runs its local tool loop.

What this actually gets you

Codex keeps doing what it does best — reading your repo, proposing edits, running tests inside its sandbox. The only difference: every model call hits /v1/responses on your LiteLLM endpoint instead of OpenAI directly. LiteLLM validates the request, checks the caller's virtual key against your policy, then assumes its ECS task role to invoke the Bedrock model (Claude 3.5 Sonnet, GPT-4o, whatever you've approved). The response streams back through the same path.

You get:

• Per-team or per-project virtual keys with hard spend caps
• Request/response logging to CloudWatch for compliance
• WAF rules in front of the ALB for IP allowlists or geo-blocking
• RDS PostgreSQL backing LiteLLM's usage and budget tables
• Secrets Manager + KMS for key storage — no plaintext creds in task definitions

Deploy the stack

The reference implementation lives in the guidance-codex repo under the feat/enterprise-gateway-readiness branch. Clone it and follow the LiteLLM quickstart — it provisions everything via CDK.

git clone -b feat/enterprise-gateway-readiness \
  https://github.com/openai-on-aws/guidance-codex.git
cd guidance-codex
# Bootstrap CDK if you haven't in this account/region
cdk bootstrap
# Deploy the gateway stack (ALB → Fargate → RDS → Bedrock)
cdk deploy LiteLLMGatewayStack

The stack outputs the gateway URL and a sample virtual key. Save both.

Configure Codex to use your gateway

On each developer machine, set two environment variables before launching Codex:

export OPENAI_BASE_URL=https://<gateway-alb-dns>/v1
export OPENAI_API_KEY=sk-litellm-<your-virtual-key>

Then run codex normally. The CLI will POST to your /v1/responses endpoint, and you'll see the traffic in CloudWatch Logs under the Fargate task's log group.

Validate the loop works end-to-end

1. Streaming — ask Codex to "write a quicksort in Python with comments." You should see tokens appear incrementally in the terminal.
2. Function calling — have it "list all .py files in this repo and summarize each." Codex will emit a list_files tool call, run it locally, then send the result back through the gateway for the next turn.
3. Budget enforcement — exceed the virtual key's daily limit; the gateway returns a 429 with a clear error body.

When this is overkill

• Small team, high trust — IAM Identity Center with direct Bedrock access is simpler. No gateway to maintain.
• Don't want to run infra — Portkey's managed gateway handles auth, budgets, and observability with a SaaS control plane. You trade operational burden for a monthly bill.
• Only need OpenAI models — LiteLLM adds latency (one extra hop). If you're not routing to Bedrock or mixing providers, a plain API key works fine.

Worth the effort?

For orgs that need audit trails, hard spend controls, and zero long-lived credentials on laptops — yes. The CDK stack is production-grade (multi-AZ Fargate, RDS with deletion protection, WAF managed rules). Expect ~$150–300/month for the gateway infra at modest traffic, plus Bedrock model costs. The repo's quickstart gets you to a working endpoint in under an hour.