Evolved from a local-first hybrid RAG prototype into a documented, ADR-governed inference gateway — balancing data privacy, inference cost, and operational resilience through a modular, vendor-neutral routing layer.
Enterprises want AI-powered insights without exposing sensitive data to cloud LLMs, absorbing unpredictable per-token costs, or losing track of why an architecture was built the way it was.
The Architecture
An Enterprise AI Gateway — a local-first hybrid inference platform routing queries through local models by default, falling back to cloud LLMs for complex reasoning, with semantic caching and a persistence layer to eliminate redundant calls and support audit.
Primary Outcome
Privacy-sensitive documents stay on-premises. Repeated queries are served from cache. Cloud LLM is a precision instrument, not the default firehose. Every major decision is recorded, not tribal knowledge.
Architectural Pattern
RAG + Hybrid Routing + Caching + ADR Governance. Each component independently replaceable — no vendor lock-in at any tier, no undocumented trade-offs.
Representative Scenario
Consider a mid-sized energy company that needs to analyse hundreds of vendor contracts and compliance documents — extracting obligations, financial terms, and risk flags — without sending sensitive legal data to a third-party cloud endpoint, and without losing a defensible record of how the routing and retention decisions were made.
Sensitive Data Constraint
Contracts contain confidential financial terms, counterparty obligations, and regulatory disclosures. Transmission to cloud APIs is not permissible under internal policy.
Cost at Query Volume
Business users frequently ask identical or near-identical questions across document sets. Per-token cloud costs compound rapidly without a caching strategy.
Response Consistency
Downstream workflows depend on deterministic, auditable outputs. Non-deterministic LLM responses for repeated queries undermine process reliability.
Architecture Diagram
A three-tier gateway: an orchestration layer (FastAPI), a retrieval, caching, and persistence layer (Qdrant + Redis + PostgreSQL), and a dual inference layer (local Ollama + cloud OpenRouter) connected through a priority-based routing policy. Requests arrive from Open WebUI or n8n-triggered workflows.
Request Flow
01
Query ingestion
A user submits a natural language query via Open WebUI, or an n8n workflow triggers a query programmatically. FastAPI receives the request and initiates the orchestration pipeline.
02
Semantic retrieval — Qdrant
The query is embedded and used to retrieve the top-k most semantically relevant document chunks from Qdrant. These chunks form the context window for the LLM prompt.
03
Cache lookup — Redis
Before invoking any LLM, the system checks Redis for an existing response to the same query-context pair.
Cache hit → return instantly, zero LLM cost
04
Inference routing decision
On a cache miss, the orchestrator routes to the appropriate inference tier based on query complexity and availability.
The response is stored in Redis before being returned to the user — ensuring future identical queries are served without model invocation.
06
Audit persistence — PostgreSQL
Query metadata — routing decision, provider used, cache outcome, and latency — is written to PostgreSQL, giving every response a traceable record independent of the ephemeral Redis cache.
Sensitive enterprise data is processed locally by default. Cloud models are never the primary path — only an explicitly triggered fallback for queries that exceed local model capability.
02 — RESILIENCE
Graceful degradation
If the local inference node is unavailable, the system automatically promotes the cloud path. No manual intervention required. The user experience is uninterrupted.
03 — ECONOMICS
Cost-aware caching
Redis intercepts repeated queries before they reach any LLM. For high-repetition enterprise query patterns, this can eliminate the majority of inference cost entirely.
04 — MODULARITY
Component independence
Each layer — LLM runtime, vector database, cache — is swappable without system-wide rearchitecting. Ollama can be replaced with vLLM; Qdrant with pgvector; Redis with a persistent store.
05 — GOVERNANCE
Decision traceability
Every major architectural trade-off is recorded as an Architecture Decision Record — context, decision, rationale, consequences, and alternatives considered — rather than left as undocumented tribal knowledge.
Architecture Decision Records
Major trade-offs are documented as ADRs rather than left implicit in code — the same discipline expected in a regulated enterprise architecture practice.
ADR
Decision
Status
0001
Adopt a local-first hybrid inference strategy
Accepted
0002
Introduce Redis as a response cache
Accepted
0003
Support a distributed Ollama inference node across a two-machine topology
Accepted
Measured Benefits
Capability
Architectural Mechanism
Data Privacy
Sensitive documents processed exclusively by Ollama on-premises. No document content exits the network perimeter.
Cost Control
Redis cache eliminates redundant cloud API calls. Per-token spend is bounded and predictable, not volume-linear.
Response Latency
Cache hits return in sub-10ms. Local inference (no network round-trip) is consistently faster than cloud for sub-threshold queries.
System Resilience
Automatic fallback to cloud on local failure. No single point of failure across the inference tier.
Vendor Neutrality
OpenRouter abstracts the cloud model provider. Switching from GPT-4 to Claude 3 is a config change, not a code change.
Auditability
PostgreSQL persists routing decisions and provider outcomes per query. Architectural rationale is captured separately in three ADRs, independent of the codebase.
Lessons Learned
Local models are production-viable for constrained domains. With careful prompt design and appropriate model selection (Gemma 2 9B), local inference handles the majority of enterprise document Q&A without quality degradation.
Caching is the highest-ROI optimisation in enterprise GenAI. Most teams over-index on model quality and under-invest in caching. In repetitive query environments, a well-designed cache can reduce cost by 60–80%.
Hybrid architectures require explicit routing logic. The decision of when to invoke local vs cloud cannot be left to chance. Building confidence scoring and complexity thresholds early prevents later architectural debt.
Simplicity compounds. Every component added to the pipeline introduces failure modes and operational overhead. Resist feature sprawl — prove each layer earns its place through measurable impact.
Documentation is a distinct deliverable, not an afterthought. Writing ADRs and a runbook alongside the code — rather than after the fact — forced clearer trade-off thinking and produced artefacts that carry into interviews and stakeholder conversations on their own merit.
Roadmap
Complexity-based query routing
Route by estimated token count and reasoning depth rather than a static primary/fallback rule — simple queries stay local, multi-hop reasoning escalates to cloud.