~/divyam/stories  relay-ai-gateway

ai junction  ·  ai systems

Relay: An Enterprise AI Gateway

One OpenAI-compatible endpoint in front of 100+ LLM providers, with authentication, guardrails in both directions, a semantic cache, cost-aware routing and a live dashboard.

when
July 2026
role
AI Engineer
stack
PythonFastAPIlitellmPostgreSQLRedisChromaDBsentence-transformersReactTypeScriptDocker

Overview

Relay is a gateway that sits between applications and LLM providers. Instead of every service calling OpenAI, Gemini or Cerebras directly, each with its own key, error handling and cost tracking, they all call Relay.

It speaks the standard OpenAI chat-completions API, so an existing client switches over by changing one URL. Behind it, litellm reaches more than 100 providers.

Every request that passes through gets authentication, rate limits, safety checks in both directions, caching and cost tracking, without a line of client code.

Problem

Once more than one service talks to an LLM, the same plumbing gets rebuilt in each of them: provider keys passed around, retries on rate limits, ad-hoc prompt-injection checks, no shared cache, and no single place to see what the LLM bill or the safety posture looks like.

Every copy is a little different, and none of them gets tested properly.

Solution

Relay builds that layer once. Every request moves through the same pipeline: authenticate, rate-limit, scan the input, check the cache, route, call the model, scan the output, log.

Authentication accepts JWTs or API keys. Keys are stored only as SHA-256 hashes and shown once. Rate limits are set per tier and kept in Redis, with an in-memory fallback if Redis is down.

Input guardrails work in two layers. First, 31 regex patterns across six attack categories: instruction overrides, jailbreak personas, system-prompt extraction, injection delimiters, harmful content and credential exfiltration. Then a semantic check compares the request with known attacks, which catches paraphrases the patterns miss.

A semantic cache answers near-repeat questions without a new LLM call. Requests are embedded with all-MiniLM-L6-v2 and matched in ChromaDB at a cosine similarity of 0.95 or higher, with a 24-hour expiry.

Two optional features sit behind flags: smart routing, which sends simple prompts to a cheaper model and complex ones to a premium model, and LLMLingua-2 compression for long prompts.

On the way out, eight redaction rules strip personal data before a response leaves: social security numbers, card numbers, emails, phone numbers, API keys, bearer tokens, IP addresses and passwords.

Dashboard

A React and TypeScript dashboard shows everything live: each request with its cost, latency, cache hit and guardrail action, refreshed every three seconds, plus charts of routing savings and guardrail activity over time.

It also has a chat playground, a converter that turns PDF, DOCX and PPTX files into Markdown, and an evals page that replays curated cases one by one and lets you type your own input into the same functions, so nothing on it is hardcoded.

Highlights

  • One OpenAI-compatible endpoint for 100+ providers via litellm
  • 31 injection and jailbreak patterns plus a semantic paraphrase check, covering OWASP LLM Top 10 categories LLM01, LLM02 and LLM06
  • Eight PII redaction rules on every response
  • Semantic cache in ChromaDB with real sentence-transformer embeddings
  • Smart routing estimated 43.9% cheaper on a 32-prompt benchmark
  • 149 tests, including 48 red-team prompts and 61 golden regression cases, run in GitHub Actions on every push
  • One FastAPI process serves the API and the built dashboard; Docker runs PostgreSQL, Redis and ChromaDB

Evaluation

The tests call the real routing and guardrail functions directly, with no server, database or LLM involved. 48 red-team prompts cover every attack category, alongside clean prompts that must pass untouched, and a regression suite of 61 golden cases pins down the expected routing and guardrail decisions.

The cost benchmark is analytical. 32 synthetic prompts, 15 simple and 17 complex, are priced once at a premium model's rates and once with smart routing. Sending the simple ones to a cheaper model cut the estimated cost by 43.9%; complex prompts cost the same either way.

Repository

# Any OpenAI client works: point it at Relay instead of the provider
curl -X POST http://localhost:8001/v1/chat/completions \
  -H "Authorization: Bearer <jwt-or-api-key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "cerebras/gpt-oss-120b", "messages": [{"role": "user", "content": "Hello!"}]}'

# Relay adds two fields to the usual response
# { ..., "gateway_cached": false, "gateway_fallback": false }

Outcome

Relay is a complete, tested service rather than a demo: one process, one endpoint, and a dashboard that shows what every request cost and what the guardrails did to it.

Most of the work went into everything around the model call: deciding what counts as an attack, how close two questions must be to share an answer, and what a dashboard has to show before anyone can trust the layer in front of their LLMs.