Run a self-hosted LiteLLM gateway for Codex on AWS ECS with Bedrock
When your team needs enterprise guardrails around Codex but isn't ready to hand model keys to every developer, a self-hosted LiteLLM gateway on ECS fills the gap. The gateway sits between Codex on each workstation and Amazon Bedrock, giving you centralized auth, budgets, rate limits, and audit logs without changing how Codex runs its local tool loop.
What this actually gets you
Codex keeps doing what it does best — reading your repo, proposing edits, running tests inside its sandbox. The only difference: every model call hits /v1/responses on your LiteLLM endpoint instead of OpenAI directly. LiteLLM validates the request, checks the caller's virtual key against your policy, then assumes its ECS task role to invoke the Bedrock model (Claude 3.5 Sonnet, GPT-4o, whatever you've approved). The response streams back through the same path.
You get:
- Per-team or per-project virtual keys with hard spend caps
- Request/response logging to CloudWatch for compliance
- WAF rules in front of the ALB for IP allowlists or geo-blocking
- RDS PostgreSQL backing LiteLLM's usage and budget tables
- Secrets Manager + KMS for key storage — no plaintext creds in task definitions
Deploy the stack
The reference implementation lives in the guidance-codex repo under the feat/enterprise-gateway-readiness branch. Clone it and follow the LiteLLM quickstart — it provisions everything via CDK.
git clone -b feat/enterprise-gateway-readiness \
https://github.com/openai-on-aws/guidance-codex.git
cd guidance-codex
# Bootstrap CDK if you haven't in this account/region
cdk bootstrap
# Deploy the gateway stack (ALB → Fargate → RDS → Bedrock)
cdk deploy LiteLLMGatewayStackThe stack outputs the gateway URL and a sample virtual key. Save both.
Configure Codex to use your gateway
On each developer machine, set two environment variables before launching Codex:
export OPENAI_BASE_URL=https://<gateway-alb-dns>/v1
export OPENAI_API_KEY=sk-litellm-<your-virtual-key>Then run codex normally. The CLI will POST to your /v1/responses endpoint, and you'll see the traffic in CloudWatch Logs under the Fargate task's log group.
Validate the loop works end-to-end
- Streaming — ask Codex to "write a quicksort in Python with comments." You should see tokens appear incrementally in the terminal.
- Function calling — have it "list all
.pyfiles in this repo and summarize each." Codex will emit alist_filestool call, run it locally, then send the result back through the gateway for the next turn. - Budget enforcement — exceed the virtual key's daily limit; the gateway returns a 429 with a clear error body.
When this is overkill
- Small team, high trust — IAM Identity Center with direct Bedrock access is simpler. No gateway to maintain.
- Don't want to run infra — Portkey's managed gateway handles auth, budgets, and observability with a SaaS control plane. You trade operational burden for a monthly bill.
- Only need OpenAI models — LiteLLM adds latency (one extra hop). If you're not routing to Bedrock or mixing providers, a plain API key works fine.
Worth the effort?
For orgs that need audit trails, hard spend controls, and zero long-lived credentials on laptops — yes. The CDK stack is production-grade (multi-AZ Fargate, RDS with deletion protection, WAF managed rules). Expect ~$150–300/month for the gateway infra at modest traffic, plus Bedrock model costs. The repo's quickstart gets you to a working endpoint in under an hour.
All Replies （4）
Want a live back-and-forth? Join the global AI chat room — login to talk.
I'm worried about security. Which specific IAM condition keys are best for Bedrock environments? For example, you can use the aws:SourceIp condition key to restrict access to your Bedrock resources only from specific IP addresses within your corporate network. This helps ensure that only authorized users can interact with the models, enhancing overall security posture. Additionally, consider using aws:userAgent to allow access only from trusted applications, such as your company's SSO integration tool. In combination with these keys, you can enforce quotas using aws:RequestSigningName to limit the number of API calls from each identity, preventing abuse and ensuring fair usage across teams. For observability, integrate with AWS CloudTrail to log all Bedrock API calls, and use the aws:PrincipalTag condition to scope down access based on user attributes like department or role. This way, you can monitor and audit who is accessing which models, and quickly identify any anomalies. Remember to regularly review and update these conditions as your environment evolves to keep security tight.
Finally got this running for the team. Which AWS ECS instance size did you use? One concrete step that worked for us was setting up the Bedrock connection via the reference access patterns for Codex, which includes SSO and quota enforcement.
Saving 30% is huge. Which caching layer did you use to get those results? Try wiring corporate SSO into the Bedrock access pattern to enforce quotas and capture observability metrics.
Redis or in-memory? I wonder if that 30% saving actually stays consistent over time, especially when connecting to models on Amazon Bedrock.