CrabTalkProvider translation
Send OpenAI format. CrabLLM translates to Anthropic, Gemini, Bedrock, and Azure automatically.
providers/overview →[ Open source · Rust ]
One API. Every model.
Route requests to OpenAI, Anthropic, Gemini, Azure, Bedrock, or Ollama through one OpenAI-compatible endpoint. Sub-millisecond overhead. Single binary. No runtime.
/* Start in 30 sec … */
$ cargo install crabllm crabctl - $P50_OVERHEAD
- 0.26 ms
- $PEAK_MEMORY
- 34.9 MB
- $RUNTIME
- single binary
[ What you get ]
Six capabilities, out of the box
Provider translation, routing, streaming, auth, caching, budgets — the gateway handles them so your client doesn’t.
Routing & fallback
Weighted random selection across providers. Exponential backoff retry. Automatic failover.
features/routing →Streaming first-class
SSE proxied without buffering. Per-chunk extension hooks. Keep-alive pings.
features/streaming →Virtual keys & auth
Per-key model access control. Rate limiting, usage tracking, and budget enforcement.
features/auth →Caching & rate limits
SHA-256 response cache. Per-key RPM and TPM limits. Sliding-window enforcement.
features/extensions →Budget enforcement
Per-key spend limits in USD. Automatic cost tracking from token usage and pricing config.
features/extensions →[ Performance ]
Not the bottleneck
Gateway overhead at 5,000 concurrent requests per second — identical limits (2 CPUs, 512 MB) against a mock backend with instant responses. Full results →
Latency
| Gateway | P50 | P99 |
|---|---|---|
| CrabLLM | 0.26ms | 0.54ms |
| Bifrost | 0.61ms | 1.26ms |
| LiteLLM | 159ms | 227ms |
Memory
| Gateway | Peak RSS |
|---|---|
| CrabLLM | 34.9 MB |
| Bifrost | 171.7 MB |
| LiteLLM | 541.8 MB |