bedrock-gateway-rust

An OpenAI-compatible HTTP gateway for Amazon Bedrock: Chat Completions, Responses, Completions and Embeddings from one Rust binary.

🌐 Website: firlab.app/bedrock-gateway · 📖 中文文档 / Chinese README

Point any OpenAI SDK, tool, or agent at this gateway and it routes requests to AWS Bedrock without a single line of client-side changes. The runtime is axum + tokio + aws-sdk-bedrockruntime — fully async, no GC pauses, no Python dependency. It replaces an earlier Python/FastAPI implementation while preserving wire-exact OpenAI API compatibility.

---

Highlights

• 100% Rust — single statically-linked binary, distroless Docker image (~12 MB compressed), no GC, low memory footprint, high concurrency under load.
• Wire-exact OpenAI compatibility — existing OpenAI SDKs, clients, and agents connect with zero code changes. No invented top-level fields; Bedrock-only features go through the standard extra_body mechanism.
• OpenAI Responses API — full support for POST /api/v1/responses, including streaming. Required by codex (wire_api = "responses"). Stateless; store and previous_response_id are accepted and silently ignored.
• Legacy text completions — POST /api/v1/completions for editors like Zed edit-prediction; reuses the Converse path and returns the text_completion shape (stream + non-stream).
• Automatic prompt caching — cache-point injection is on by default. The gateway places cache points across tools, system prompt, and messages automatically (up to max_cache_checkpoints per model). No client changes needed. Thresholds and limits are config-driven per model; a family fallback entry covers new Claude models automatically.
• Extended thinking / reasoning — Claude budget_tokens, adaptive_thinking, and DeepSeek string-form reasoning all supported. Map OpenAI's reasoning_effort levels (low / medium / high / xhigh / max) through extra_body; the gateway picks the right Bedrock wire format per model.
• Cross-region inference profiles — all seven geographic prefixes (us. / eu. / apac. / jp. / au. / ca. / global.) work transparently. Capability matching strips the prefix; Bedrock calls always use the original model ID.
• Zero-hardcoding — all model knowledge lives in config/models.toml. Adding a new model or tuning cache thresholds never requires a recompile.
• Four deployment targets — standalone binary, Docker, ECS/Fargate + ALB (CloudFormation one-click), and Lambda + Function URL (Lambda Web Adapter, no Lambda-specific Rust code).
• Dual auth, dual direction — client-to-gateway via Bearer token (SSM / Secrets Manager / env); gateway-to-Bedrock via Bedrock API Key bearer or SigV4 credential chain.
• Structured observability — per-request request_id, cached_tokens, cache_hit, ttfb_ms, duration_ms in structured JSON logs. No prompt text or secrets ever logged.

---

Supported Endpoints

All endpoints are prefixed by API_ROUTE_PREFIX (default /api/v1).

Method	Path	Description
POST	/api/v1/chat/completions	Chat completions — streaming (SSE) and non-streaming
POST	/api/v1/completions	Legacy text completions (text_completion) — for Zed edit-prediction; reuses the Converse path
POST	/api/v1/responses	OpenAI Responses API — stateless, streaming and non-streaming
POST	/api/v1/embeddings	Embeddings — Cohere, Titan, and Nova families
GET	/api/v1/models	Live model catalog from Bedrock control plane
GET	/api/v1/models/{id}	Single model lookup (supports inference profile IDs)
GET	/api/v1/health	Liveness probe — returns 200 OK

---

Client Compatibility

Client	Wire API	Endpoint	Status
opencode	OpenAI Chat Completions	POST /api/v1/chat/completions	✅ Supported
hermes	OpenAI Chat Completions	POST /api/v1/chat/completions	✅ Supported
Zed (edit prediction)	OpenAI Legacy Completions	POST /api/v1/completions	✅ Supported
codex	OpenAI Responses API	POST /api/v1/responses	✅ Supported
claude code	Anthropic Messages	POST /v1/messages	⏳ Roadmap

---

Quick Start

Prerequisites

• AWS credentials available (instance role, AWS_PROFILE, or access key pair)
• Bedrock model access enabled in your target region
• Docker, a release binary, or Rust (the version in rust-toolchain.toml) to build from source

30-second start with Docker

docker run \
  -e API_KEY=sk-my-secret-key \
  -e AWS_REGION=us-east-1 \
  -e AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-... \
  -p 8080:8080 \
  sunerpy/bedrock-gateway-rust

Using an EC2 instance or ECS task with an IAM role? Omit AWS_BEARER_TOKEN_BEDROCK and the access key pair — the SDK picks up instance credentials automatically.

Install the binary

The install script downloads the latest release for your platform, checks it against the release's SHA256SUMS and puts bedrock-gateway in ~/.local/bin:

curl -fsSL https://raw.githubusercontent.com/sunerpy/bedrock-gateway-rust/main/scripts/install.sh | sh

irm https://raw.githubusercontent.com/sunerpy/bedrock-gateway-rust/main/scripts/install.ps1 | iex

TOOL_VERSION=0.18.0 installs a given release (0.18.0 or later) and TOOL_INSTALL_DIR changes the directory. The archives are also on GitHub Releases, and cargo install bedrock-gateway-rust builds the binary from crates.io.

Build from source

cargo build --release
API_KEY=sk-my-secret-key AWS_REGION=us-east-1 ./target/release/bedrock-gateway

Local dev

API_KEY=testkey cargo run

Verify

curl http://localhost:8080/api/v1/health
# OK

curl http://localhost:8080/api/v1/chat/completions \
  -H "Authorization: Bearer sk-my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"anthropic.claude-3-5-sonnet-20241022-v2:0","messages":[{"role":"user","content":"Hello!"}]}'

OpenAI Python SDK — drop-in replacement

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/api/v1",
    api_key="sk-my-secret-key",
)

response = client.chat.completions.create(
    model="anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=[{"role": "user", "content": "Explain serverless in one sentence."}],
)
print(response.choices[0].message.content)

Streaming, tool calling, vision, and embeddings work the same way — no special configuration.

---

Deployment

Four targets share the same binary and the same environment-variable interface.

Target	Guide
Standalone binary	Run cargo build --release, ship target/release/bedrock-gateway + config/
Docker (distroless)	Pull sunerpy/bedrock-gateway-rust — Deploy → Docker
ECS / Fargate + ALB	CloudFormation one-click — Deploy → ECS / Fargate
Lambda + Function URL	Lambda Web Adapter, no Lambda code — Deploy → Lambda

ECS / Fargate quick deploy

aws cloudformation deploy \
  --template-file deployment/BedrockGatewayFargate.template \
  --stack-name bedrock-gateway \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    ApiKey=sk-my-secret-key \
    VpcId=vpc-... \
    SubnetIds=subnet-...,subnet-...

Lambda quick deploy

# Build the Lambda container image
docker build -f deployment/lambda/Dockerfile -t bedrock-gateway:lambda .
# Push to ECR, then:
aws cloudformation deploy \
  --template-file deployment/BedrockGatewayLambda.template \
  --stack-name bedrock-gateway-lambda \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    ImageUri=<ECR_URI>:latest \
    ApiKeySecretArn=arn:aws:secretsmanager:...

Lambda has a 10-minute maximum timeout. For long-running streaming sessions, prefer ECS/Fargate.

---

Configuration

All settings are environment variables. The gateway accepts both an APP_ prefix and bare names (e.g., API_KEY or APP_API_KEY); bare names take precedence.

The three required variables

Variable	Purpose
API_KEY	Bearer token your clients send to the gateway. Pick any string.
AWS_BEARER_TOKEN_BEDROCK	Bedrock API Key the gateway presents to AWS (recommended). Or omit and use SigV4.
AWS_REGION	AWS region for Bedrock calls. Defaults to us-west-2.

These two keys go in opposite directions — don't mix them up:

• API_KEY is what your clients send to the gateway.
• AWS_BEARER_TOKEN_BEDROCK is what the gateway sends to AWS Bedrock.

AWS credential options

The gateway auto-selects — no switch needed:

1. AWS_BEARER_TOKEN_BEDROCK (alias BEDROCK_API_KEY) — Bedrock API Key, sent as Authorization: Bearer. Bypasses SigV4 entirely. Recommended for new deployments.
2. SigV4 credential chain — resolves AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY, then AWS_PROFILE, then IMDS / ECS task role.

Production secret management

Prefer fetching API_KEY from a secrets store. Priority order:

1. API_KEY_PARAM_NAME — SSM Parameter Store
2. API_KEY_SECRET_ARN — Secrets Manager (secret must contain an api_key field)
3. API_KEY — plaintext env var

All optional variables

Show all optional variables

Model defaults

Variable	Default	Description
DEFAULT_MODEL	anthropic.claude-3-5-sonnet-20241022-v2:0	Model used when the client omits model
DEFAULT_EMBEDDING_MODEL	cohere.embed-multilingual-v3	Embedding model used when model is omitted

Feature flags

Variable	Default	Description
ENABLE_CROSS_REGION_INFERENCE	true	Transparently route cross-region inference profiles
ENABLE_APPLICATION_INFERENCE_PROFILES	true	Surface inference profile metadata in GET /models
ENABLE_PROMPT_CACHING	true	Auto-inject cache points for Claude and Nova models
CONFIG_DIR	config	External config directory path (overrides embedded defaults)

Server

Variable	Default	Description
PORT	8080	HTTP listen port
BIND_ADDR	0.0.0.0	Network interface to bind
API_ROUTE_PREFIX	/api/v1	Path prefix for all endpoints
MAX_BODY_SIZE_MB	20	Maximum accepted request body size in MB (raised from axum's 2 MB default so base64-encoded image payloads fit)

Logging

Variable	Default	Description
LOG_LEVEL	info	Verbosity: trace / debug / info / warn / error
DEBUG	false	Enable verbose error details in responses

Timeouts and retries

Variable	Default	Description
AWS_CONNECT_TIMEOUT_SECS	60	TCP connection timeout to AWS
AWS_READ_TIMEOUT_SECS	900	Response read timeout (15 min, accommodates long streams)
RESPONSES_STREAM_IDLE_TIMEOUT_SECS	180	Maximum silence between upstream Responses events before a terminal response.failed
MODEL_CATALOG_REFRESH_SECS	3600	Interval between background re-listings of the Bedrock model catalog, so new models appear in /models and pass the image check without a restart; 0 lists at boot only
AWS_MAX_RETRY_ATTEMPTS	8	Retries on transient throttling or 5xx failures

GPT-5.x / bedrock-mantle

Variable	Default	Description
MANTLE_BASE_URL_TEMPLATE	https://bedrock-mantle.{region}.api.aws/openai/v1	Upstream URL for GPT-5.x models (Responses surface). {region} is replaced with AWS_REGION at call time.
MANTLE_CHAT_BASE_URL_TEMPLATE	https://bedrock-mantle.{region}.api.aws/v1	Upstream URL for gpt-oss chat models (/chat/completions surface, no /openai prefix). {region} is replaced with AWS_REGION at call time.

Config files (no recompile needed)

Config files live in config/ alongside the binary and are read at startup. They are also embedded in the binary as a fallback — the gateway always starts, even if the files are absent.

File	Purpose
config/models.toml	Model capability registry: capability flags, reasoning paths, cache thresholds
config/regions.toml	Cross-region routing rules
config/embeddings.toml	Embedding model registry (Cohere / Titan / Nova families)
config/app.toml	Application defaults — lowest priority

Adding a new model requires only a config/models.toml entry:

[[model]]
match = "your-provider.your-model-id"
capabilities = []
[model.params]
max_tokens = 8192
context_window = 200000
# reasoning_path = "budget_tokens"  # uncomment if the model supports extended thinking

---

API Usage

All endpoints require Authorization: Bearer <API_KEY>.

Base URL: http://localhost:8080/api/v1

Streaming chat

curl http://localhost:8080/api/v1/chat/completions \
  -H "Authorization: Bearer sk-my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"anthropic.claude-3-5-sonnet-20241022-v2:0","messages":[{"role":"user","content":"Count to 5 slowly."}],"stream":true}'

Embeddings

curl http://localhost:8080/api/v1/embeddings \
  -H "Authorization: Bearer sk-my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"cohere.embed-multilingual-v3","input":["Hello world","Bedrock gateway"]}'

Extended thinking

response = client.chat.completions.create(
    model="anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=[{"role": "user", "content": "Solve step by step: ..."}],
    extra_body={"reasoning_effort": "high"},  # none|minimal|low|medium|high|xhigh|max
)
# Reasoning appears inline as <think>...</think> inside content

GPT-5.x (Responses and Chat Completions)

GPT-5.4, GPT-5.5, and GPT-5.6 Sol/Terra/Luna use the Mantle Responses upstream. They are also available on /api/v1/chat/completions through the built-in stateless Responses-to-Chat adapter. Reasoning summaries render as <think>, reasoning token usage is preserved, and function-tool continuations use authenticated rsc_v1 capsules. See the adapter design.

Non-streaming:

curl http://localhost:8080/api/v1/responses \
  -H "Authorization: Bearer sk-my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.5","input":"say hello","max_output_tokens":64,"stream":false}'

Streaming (raw SSE, terminates on response.completed — no [DONE]):

curl http://localhost:8080/api/v1/responses \
  -H "Authorization: Bearer sk-my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.4","input":"count to 5","stream":true}'

Chat Completions with reasoning:

curl http://localhost:8080/api/v1/chat/completions \
  -H "Authorization: Bearer sk-my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.6-sol","messages":[{"role":"user","content":"Solve 17 * 23"}],"reasoning_effort":"high","stream":true}'

codex configuration

Add to ~/.codex/config.toml:

[model_providers.bgw]
name = "Bedrock Gateway"
base_url = "http://localhost:8080/api/v1"
wire_api = "responses"
requires_openai_auth = false

[model_providers.bgw.env_key]
name = "BGW_API_KEY"
description = "Bearer token for the Bedrock gateway"

export BGW_API_KEY=sk-my-gateway-key
codex --provider bgw --model anthropic.claude-3-5-sonnet-20241022-v2:0

---

Architecture

src/
├── main.rs / lib.rs         # Entry point, crate root
├── error.rs                 # AppError, OpenAI error envelope
├── telemetry.rs             # tracing subscriber, dynamic log level
├── openai/                  # Wire types: ChatRequest, ChatResponse, ResponsesRequest, ...
├── domain/                  # Provider-agnostic traits: ChatProvider, ResponsesProvider, ...
├── config/                  # Settings, ModelCapabilityConfig, RegionRoutingConfig, ...
├── bedrock/                 # AWS Bedrock backend: translate, cache, reasoning, stream, ...
└── server/                  # axum router, auth middleware, AppState

The codebase is strictly layered — dependencies flow downward only. The Bedrock backend implements the domain traits; adding a non-Bedrock backend means implementing those traits without touching any existing code.

For the full architecture reference, contract rules, and contributor guidelines, see the Architecture & Contracts guide.

For caching behavior, reasoning budget paths, and cross-region inference profile details, see the Caching & Reasoning deep dive.

---

Supported Models

The authoritative list is config/models.toml and the live GET /api/v1/models endpoint. The registry currently covers:

• Claude — Sonnet 4.x / 5 / 5.5, Haiku 4.5 / 5.5, Opus 4.x / 5 / 5.5, Fable 5 / 5.1 (via cross-region inference profiles such as global.anthropic.claude-sonnet-5-5). response_format / Responses text.format is sent to Bedrock as structured output for every model; Bedrock answers 400 for a model without it (for example Fable 5)
• Amazon Nova — multimodal and text models
• DeepSeek — v3 (string-form reasoning path)
• xAI Grok 4, Moonshot Kimi K3 — temperature / top_p are dropped (the models reject them)
• GPT-5.x — gpt-5.4, gpt-5.5, and gpt-5.6-sol/terra/luna on /api/v1/responses plus adapted /api/v1/chat/completions, via the AWS Bedrock mantle Responses upstream. Region availability is declared in config/models.toml. Requires AWS_BEARER_TOKEN_BEDROCK; reasoning + tool continuation additionally requires the shared CHAT_REASONING_CAPSULE_* keyring.
• gpt-oss — gpt-oss-120b and gpt-oss-20b on /api/v1/chat/completions only, via the AWS Bedrock mantle upstream (chat_backend = "mantle"). Byte-level raw SSE passthrough; the gateway appends data: [DONE] at the stream tail. Not available on /completions. Listed in GET /models under the bare alias names gpt-oss-120b / gpt-oss-20b. Region-gated: us-east-1 / us-east-2 / us-west-2. Requires AWS_BEARER_TOKEN_BEDROCK to be set (otherwise those models are disabled with a WARN; the gateway still starts).
• GPT-6.x — gpt-6.1-sol, gpt-6-sol, gpt-6-luna, and gpt-6-astra on both /api/v1/chat/completions and /api/v1/responses through Bedrock Converse. The bare names alias to the global.openai.* cross-region inference profiles; the us. / global. profile ids also work directly. reasoning_effort maps to reasoning.effort, temperature / top_p are dropped (the models reject them), and response_format (chat) or text.format (Responses) uses the OpenAI text.format shape (strict schemas honored)
• Any Bedrock foundation model or inference profile accessible in your account — the catalog is listed from the control plane at startup and re-listed every MODEL_CATALOG_REFRESH_SECS

Adding a model requires only a config/models.toml entry and no recompile.

---

Build, Test, and Contributing

cargo build --release          # release binary → target/release/bedrock-gateway
cargo test --all-features      # unit + golden replay tests (no AWS credentials needed)
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --all

Gate before every push (make hooks runs it on git push):

cargo fmt --all -- --check && cargo clippy --all-targets --all-features -- -D warnings && cargo test --all-features

Golden fixtures in tests/golden/ are pinned offline and run in CI without AWS credentials. Live integration tests require real credentials:

BEDROCK_INTEGRATION=1 AWS_PROFILE=<your-profile> cargo test -- --ignored

See the Contributing Guide for contributor guidelines.

---

Links

Resource	Link
中文文档 / Chinese README	Read in Chinese
Contributing	Contributing Guide
Code of Conduct	Code of Conduct
Caching and reasoning	Caching & Reasoning deep dive
Docker deployment	Deploy → Docker
ECS/Fargate deployment	Deploy → ECS / Fargate
Lambda deployment	Deploy → Lambda
Architecture and contracts	Architecture & Contracts guide

---

License

MIT-0

---

Acknowledgements

This project is inspired by the Python gateway concept in aws-samples/bedrock-access-gateway. The Rust implementation is a full rewrite with wire-exact OpenAI compatibility, automatic prompt caching, Responses API support, and production-grade observability.