Phase 17 · Lesson 19 of 29

AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost

A gateway sits between your apps and model providers. Core features are provider routing, fallback, retries, rate limiting, secret references, observability, guardrails. Market split in 2026: LiteLLM is MIT OSS with 100+ providers, OpenAI-compatible, but breaks down around ~2000 RPS (8 GB memory, cascading failures in published benchmarks); best for Python, <500 RPS, dev/prototyping. Portkey is control-plane-positioned (guardrails, PII redaction, jailbreak detection, audit trails), went Apache 2.0 open-source March 2026, 20-40 ms latency overhead, $49/mo production tier. Kong AI Gateway built on Kong Gateway — Kong's own benchmark on same 12 CPUs: 228% faster than Portkey, 859% faster than LiteLLM; $100/model/month pricing (max 5 on Plus tier); enterprise-fit if you're already on Kong. Bifrost (Maxim AI) — automatic retries with configurable backoff, fallback to Anthropic on OpenAI 429. Cloudflare / Vercel AI Gateways — managed, zero-ops, basic retry. Data residency drives the self-host decision; Portkey and Kong sit in the middle with OSS + optional managed.

Lesson 19 of 29Learn~60 minutesPython (stdlib, toy gateway-routing simulator)

Before this: Lesson 1 of Phase 17 (Managed LLM Platforms), lesson 16 of Phase 17 (Model Routing)

On this page

Learning Objectives

  • Enumerate the seven core gateway features (routing, fallback, retries, rate limits, secrets, observability, guardrails).
  • Map four 2026 gateways (LiteLLM, Portkey, Kong AI, Bifrost) to scale ceilings and use cases.
  • Cite the Kong benchmark (228% vs Portkey, 859% vs LiteLLM) and explain why it matters for >500 RPS.
  • Choose self-hosted vs managed given data residency and ops budget.

The Problem

Your product calls OpenAI, Anthropic, and a self-hosted Llama. Each provider has a different SDK, error model, rate limit, and auth scheme. You want failover (if OpenAI 429s, try Anthropic), a single credential store, unified observability, and rate limits per tenant.

Reinventing this at the app layer couples every service to every provider. A gateway layer consolidates it into one process with one API (typically OpenAI-compatible) that fans out to providers.

The Concept

Seven core features

  1. Provider routing — OpenAI, Anthropic, Gemini, self-hosted, etc. behind one API.
  2. Fallback — on 429, 5xx, or quality failure, retry elsewhere.
  3. Retries — exponential backoff, bounded attempts.
  4. Rate limits — per-tenant, per-key, per-model.
  5. Secret references — pull credentials from vault at runtime (never in app).
  6. Observability — OTel + GenAI attributes (Phase 17 · 13) + cost attribution.
  7. Guardrails — PII redaction, jailbreak detection, allowed-topics filters.

LiteLLM — MIT OSS, Python

  • 100+ providers, OpenAI-compatible, router config, fallback, basic observability.
  • Breaks down around 2000 RPS in Kong's benchmark; 8 GB memory footprint, cascading failures under sustained load.
  • Best fit: Python app, <500 RPS, dev/staging gateways, experimental routing.
  • Cost: $0 for OSS; cloud free tier exists.

Portkey — control plane positioning

  • Apache 2.0 OSS as of March 2026. Guardrails, PII redaction, jailbreak detection, audit trails.
  • 20-40 ms per-request latency overhead.
  • $49/mo for production tier with retention + SLA.
  • Best fit: regulated industries needing guardrails + observability bundled.

Kong AI Gateway — the scale play

  • Built on Kong Gateway (mature API gateway product, lua+OpenResty).
  • Kong's own benchmark on 12-CPU equivalent: 228% faster than Portkey, 859% faster than LiteLLM.
  • Pricing: $100/model/month, max 5 on Plus tier.
  • Best fit: already on Kong; >1000 RPS; willing to license.

Bifrost (Maxim AI)

  • Automatic retries with configurable backoff.
  • Fallback to Anthropic on OpenAI 429 is a canonical recipe.
  • Newer entrant; commercial.

Cloudflare AI Gateway / Vercel AI Gateway

  • Managed, zero-ops. Basic retry and observability.
  • Best fit: Edge-serving JavaScript apps on Cloudflare/Vercel.
  • Limited compared to Kong/Portkey on guardrails and rate limits.

Self-hosted vs managed

Data residency is the forcing function. Healthcare and finance default self-host (LiteLLM or Portkey OSS or Kong). Consumer products default managed (Cloudflare AI Gateway) or middle-tier (Portkey managed). Hybrid: self-hosted for regulated tenant, managed for others.

Latency budget

  • LiteLLM: 5-15 ms overhead typical.
  • Portkey: 20-40 ms overhead.
  • Kong: 3-8 ms overhead.
  • Cloudflare/Vercel: 1-3 ms overhead (edge advantage).

Gateway latency directly adds to TTFT. For TTFT P99 < 100 ms SLA, Kong or Cloudflare. For P99 < 500 ms, any.

Rate-limit semantics matter

Simple token-bucket works up to moderate scale. Multi-tenant requires sliding-window + burst allowance + per-tenant tiering. LiteLLM ships token-bucket; Kong ships sliding-window; Portkey ships tiered.

Gateway + observability + routing compose

Phase 17 · 13 (observability) + 16 (model routing) + 19 (gateways) are the same layer in production. Pick one tool that covers all three or wire them carefully: most 2026 deployments combine Helicone (observability) or Portkey (guardrails) with Kong (scale) for split roles.

Numbers you should remember

  • LiteLLM: breaks at ~2000 RPS, 8 GB memory.
  • Portkey: 20-40 ms overhead; Apache 2.0 since March 2026.
  • Kong: 228% faster than Portkey, 859% faster than LiteLLM.
  • Kong pricing: $100/model/month, 5 max on Plus tier.
  • Cloudflare/Vercel: 1-3 ms overhead at the edge.
mx-gateway-fallback

Use It

code/main.py simulates gateway routing with fallback across 3 providers under 429/5xx injection. Reports latency, retry rate, and fallback hit rate.

Ship It

This lesson produces outputs/skill-gateway-picker.md. Given scale, ops posture, compliance, latency budget, picks a gateway.

Exercises

  1. Run code/main.py. Configure fallback from OpenAI→Anthropic→self-hosted. What's the expected hit rate at 5% provider error rate?
  2. Your SLA is TTFT P99 < 320 ms and the model alone measures 300 ms at P99. Which gateways stay within budget?
  3. A healthcare customer requires self-hosted + PII redaction + audit. Pick Portkey OSS or Kong.
  4. Compare LiteLLM vs Kong: at what RPS ceiling should a team migrate?
  5. Design a rate-limit policy for a multi-tenant SaaS: free tier, trial tier, paid tier. Token-bucket or sliding-window?

Key Terms

Term What people say What it actually means
Gateway "API broker" Process sitting between apps and providers
LiteLLM "the MIT one" Python OSS, 100+ providers, breaks at 2K RPS
Portkey "guardrails gateway" Control plane + observability, Apache 2.0
Kong AI Gateway "the scale one" Built on Kong Gateway, benchmark leader
Bifrost "Maxim's gateway" Retries + Anthropic fallback recipe
Cloudflare AI Gateway "edge managed" Edge-deployed managed gateway, zero-ops
PII redaction "data scrub" Regex + NER mask before sending to model
Jailbreak detection "prompt injection guard" Classifier on user input
Audit trail "regulated log" Immutable record of every LLM call
Token-bucket "simple rate limit" Refill-based rate limiter
Sliding-window "precise rate limit" Time-windowed rate limiter; better fairness

Further Reading

Check yourself

Quick quiz

  1. What is the core role of an AI gateway in the lesson?

    1. A vector database for retrieval
    2. A model fine-tuning service
    3. A process sitting between apps and model providers that consolidates routing, fallback, retries, rate limits, secret references, observability, and guardrails behind one API
    4. A logging-only sidecar
    Show the answer

    C. A process sitting between apps and model providers that consolidates routing, fallback, retries, rate limits, secret references, observability, and guardrails behind one API

  2. What scale ceiling does Kong's benchmark report for LiteLLM?

    1. LiteLLM cannot be benchmarked
    2. It scales linearly past 10k RPS
    3. It breaks down around ~2000 RPS with 8 GB memory and cascading failures under sustained load
    4. It tops out at 50 RPS
    Show the answer

    C. It breaks down around ~2000 RPS with 8 GB memory and cascading failures under sustained load

  3. Per the Kong benchmark on equivalent CPU, how much faster is Kong AI Gateway than Portkey and LiteLLM?

    1. Identical
    2. About 10% and 20% faster
    3. 228% faster than Portkey and 859% faster than LiteLLM
    4. Slower than both
    Show the answer

    C. 228% faster than Portkey and 859% faster than LiteLLM

  4. Which gateway does the lesson position with 20-40 ms latency overhead and guardrails / PII redaction / jailbreak detection focus?

    1. Cloudflare AI Gateway
    2. Portkey
    3. Kong AI Gateway
    4. LiteLLM
    Show the answer

    B. Portkey

  5. What does the lesson say is the forcing function for self-hosted vs managed gateway decisions?

    1. Number of supported providers
    2. Cost per request
    3. Data residency requirements
    4. Whether the gateway is open source
    Show the answer

    C. Data residency requirements

  6. Which gateways stay within budget when the SLA is TTFT P99 < 100 ms?

    1. Only Portkey
    2. Any gateway
    3. Only LiteLLM
    4. Kong (~3-8 ms) or Cloudflare/Vercel edge gateways (~1-3 ms); Portkey at 20-40 ms is too heavy
    Show the answer

    D. Kong (~3-8 ms) or Cloudflare/Vercel edge gateways (~1-3 ms); Portkey at 20-40 ms is too heavy

Questions people ask

What is LiteLLM used for?

LiteLLM is an open-source tool that lets apps call many AI providers through one OpenAI-style interface. It can add fallbacks, track spend and manage keys. It is popular for prototypes and moderate traffic; very high traffic may need a heavier gateway.

Do I need an AI gateway?

A single small tool calling one model usually does not. Once you have several apps, several providers, or a need for central spending limits, logging and key control, a gateway saves work and lowers risk.

Run the code Read it on GitHub

Next lessonShadow Traffic, Canary Rollout, and Progressive Deployment for LLMsPrevious: Production Serving Stack — KV Offloading and Cache-Aware Routing

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes