Evolved from a local-first hybrid RAG prototype into a documented, ADR-governed inference gateway — balancing data privacy, inference cost, and operational resilience through a modular, vendor-neutral routing layer.
Enterprises want AI-powered insights without exposing sensitive data to cloud LLMs, absorbing unpredictable per-token costs, or losing track of why an architecture was built the way it was.
The Architecture
An Enterprise AI Gateway — a local-first hybrid inference platform routing queries through local models by default, falling back to cloud LLMs for complex reasoning, with semantic caching and a persistence layer to eliminate redundant calls and support audit.
Primary Outcome
Privacy-sensitive documents stay on-premises. Repeated queries are served from cache. Cloud LLM is a precision instrument, not the default firehose. Every major decision is recorded, not tribal knowledge.
Architectural Pattern
RAG + Hybrid Routing + Caching + ADR Governance. Each component independently replaceable — no vendor lock-in at any tier, no undocumented trade-offs.
Representative Scenario
Consider a mid-sized energy company that needs to analyse hundreds of vendor contracts and compliance documents — extracting obligations, financial terms, and risk flags — without sending sensitive legal data to a third-party cloud endpoint, and without losing a defensible record of how the routing and retention decisions were made.
Sensitive Data Constraint
Contracts contain confidential financial terms, counterparty obligations, and regulatory disclosures. Transmission to cloud APIs is not permissible under internal policy.
Cost at Query Volume
Business users frequently ask identical or near-identical questions across document sets. Per-token cloud costs compound rapidly without a caching strategy.
Response Consistency
Downstream workflows depend on deterministic, auditable outputs. Non-deterministic LLM responses for repeated queries undermine process reliability.
Architecture Diagram
A three-tier gateway: an orchestration layer (FastAPI), a retrieval, caching, and persistence layer (Qdrant + Redis + PostgreSQL), and a dual inference layer (local Ollama + cloud OpenRouter) connected through a priority-based routing policy. Requests arrive from Open WebUI or n8n-triggered workflows.
Request Flow
01
Query ingestion
A user submits a natural language query via Open WebUI, or an n8n workflow triggers a query programmatically. FastAPI receives the request and initiates the orchestration pipeline.
02
Semantic retrieval — Qdrant
The query is embedded and used to retrieve the top-k most semantically relevant document chunks from Qdrant. These chunks form the context window for the LLM prompt.
03
Cache lookup — Redis
Before invoking any LLM, the system checks Redis for an existing response to the same query-context pair.
Cache hit → return instantly, zero LLM cost
04
Inference routing decision
On a cache miss, the orchestrator routes to the appropriate inference tier based on query complexity and availability.
The response is stored in Redis before being returned to the user — ensuring future identical queries are served without model invocation.
06
Audit persistence — PostgreSQL
Query metadata — routing decision, provider used, cache outcome, and latency — is written to PostgreSQL, giving every response a traceable record independent of the ephemeral Redis cache.
Observability, Streaming & Cost Governance
A second build phase closed out three of the roadmap items below and added a governance layer the original design didn't have: the gateway now knows what it's doing, shows its work, and can't be run up in cost by a stranger with the demo link.
Request tracing & telemetry dashboard
Every request — classify, retrieve, cache lookup, route, infer, cache write — is traced end to end and stored in Redis with its own TTL. A live dashboard surfaces latency percentiles, cache hit rate, and provider/model/department distribution, recomputed on every poll rather than incrementally counted.
Streaming responses (SSE)
Answers stream token-by-token over Server-Sent Events instead of waiting on the full completion. Local-model failures still fall back to cloud transparently — but only before the first token has reached the client; once bytes are in flight, a failure surfaces as a terminal stream event instead of a silent retry.
Model tier governance
A single environment variable caps which model tiers are routable at all — not just de-prioritized. On the public demo, premium models are invisible to the classifier, the fallback chain, and the model picker alike, with one filter at the registry level rather than scattered checks.
Per-IP rate limiting
A hand-rolled Redis fixed-window limiter (5/min, 30/day by default) sits in front of the query endpoints only — health, models, and telemetry stay open. Atomic increments avoid a check-then-act race under concurrent requests from the same visitor.
Time-limited access codes
Replaces a permanent shared password with a code generated on demand and backed by Redis TTL — expiry is delegated to the store, not hand-rolled. Verification fails closed on any backend error, the deliberate opposite of the rate limiter's fail-open stance: an auth check must never stay permissive during an outage.
Dry-run load simulation
A stub inference server stands in for Ollama/OpenRouter so the full classifier → decision engine → cache → routing pipeline can be exercised under real concurrency — tens of simulated users, mixed streaming/non-streaming traffic — without spending a token.
Live Telemetry — Preview
A static snapshot of the /dashboard route for anyone reading this without going through sign-in. Real data, recreated here rather than screenshotted so it stays crisp and matches the page's own type system — the live version updates every five seconds.
Sensitive enterprise data is processed locally by default. Cloud models are never the primary path — only an explicitly triggered fallback for queries that exceed local model capability.
02 — RESILIENCE
Graceful degradation
If the local inference node is unavailable, the system automatically promotes the cloud path. No manual intervention required. The user experience is uninterrupted.
03 — ECONOMICS
Cost-aware caching
Redis intercepts repeated queries before they reach any LLM. For high-repetition enterprise query patterns, this can eliminate the majority of inference cost entirely.
04 — MODULARITY
Component independence
Each layer — LLM runtime, vector database, cache — is swappable without system-wide rearchitecting. Ollama can be replaced with vLLM; Qdrant with pgvector; Redis with a persistent store.
05 — GOVERNANCE
Decision traceability
Every major architectural trade-off is recorded as an Architecture Decision Record — context, decision, rationale, consequences, and alternatives considered — rather than left as undocumented tribal knowledge.
Architecture Decision Records
Major trade-offs are documented as ADRs rather than left implicit in code — the same discipline expected in a regulated enterprise architecture practice.
ADR
Decision
Status
0001
Adopt a local-first hybrid inference strategy
Accepted
0002
Introduce Redis as a response cache
Accepted
0003
Support a distributed Ollama inference node across a two-machine topology
Accepted
Measured Benefits
Capability
Architectural Mechanism
Data Privacy
Sensitive documents processed exclusively by Ollama on-premises. No document content exits the network perimeter.
Cost Control
Redis cache eliminates redundant cloud API calls. Per-token spend is bounded and predictable, not volume-linear.
Response Latency
Cache hits return in sub-10ms. Local inference (no network round-trip) is consistently faster than cloud for sub-threshold queries.
System Resilience
Automatic fallback to cloud on local failure. No single point of failure across the inference tier.
Vendor Neutrality
OpenRouter abstracts the cloud model provider. Switching from GPT-4 to Claude 3 is a config change, not a code change.
Auditability
PostgreSQL persists routing decisions and provider outcomes per query. Architectural rationale is captured separately in three ADRs, independent of the codebase.
Lessons Learned
Local models are production-viable for constrained domains. With careful prompt design and appropriate model selection (Gemma 2 9B), local inference handles the majority of enterprise document Q&A without quality degradation.
Caching is the highest-ROI optimisation in enterprise GenAI. Most teams over-index on model quality and under-invest in caching. In repetitive query environments, a well-designed cache can reduce cost by 60–80%.
Hybrid architectures require explicit routing logic. The decision of when to invoke local vs cloud cannot be left to chance. Building confidence scoring and complexity thresholds early prevents later architectural debt.
Simplicity compounds. Every component added to the pipeline introduces failure modes and operational overhead. Resist feature sprawl — prove each layer earns its place through measurable impact.
Documentation is a distinct deliverable, not an afterthought. Writing ADRs and a runbook alongside the code — rather than after the fact — forced clearer trade-off thinking and produced artefacts that carry into interviews and stakeholder conversations on their own merit.
Roadmap
Complexity-based query routing
Route by estimated token count and reasoning depth rather than a static primary/fallback rule — simple queries stay local, multi-hop reasoning escalates to cloud.
Users query only documents scoped to their role — enforced at retrieval, not just UI. The access-code system now shipped is a natural foundation for per-identity, not just per-visitor, gating.
$-denominated FinOps dashboard
Redis-backed budget tracking already downgrades routing when a spend cap is hit — the remaining gap is surfacing that spend visually per query and per time window, alongside the latency/cache metrics the telemetry dashboard already shows.
Durable trace storage
Traces currently live in Redis with a 24h TTL, sized for a live demo. Long-horizon analytics would need a persistence layer that survives past that window.
Portfolio Positioning Statement
“Designed and implemented an Enterprise AI Gateway integrating FastAPI, Next.js, Qdrant, Redis, PostgreSQL, Ollama, and OpenRouter — enabling privacy-aware enterprise document intelligence with semantic caching, streaming inference, resilient primary/fallback routing, request-level tracing, and cost-governed access control, all backed by ADR-documented architectural decisions.”