There is a moment in the life of every AI product where a provider has a bad afternoon. It is rarely an outage with a status page and an apology. It is something quieter: elevated latency on one model, intermittent five-hundreds on one endpoint, or a rate limit that arrives at the exact hour your users are most active.
Whether that moment becomes an incident or a footnote is decided long before it happens. It depends on whether model calls are something your feature code does or something your system has a single path for.
In LILA it is a path. Every model request in the product goes through one gateway, and the interesting part of the gateway is how its pieces compose.
The router, and why feature code should not know a model name
The Python service exposes twenty-four named capabilities behind a registry rather than a set of ad hoc endpoints. Among them are chat, email triage, pattern detection, summarisation, extraction, reranking, embedding, transcription, speech synthesis, moderation and memory extraction. A capability declares what it needs and which model tier it wants. The router maps the capability to an actual model.
So feature code asks for summarize. It does not ask for a model name, a provider, a temperature or a context window.
This is the smallest decision in the design and the one everything else depends on. The day a model name appears in a feature module, swapping models becomes a search-and-replace across the codebase, cost attribution loses its chokepoint, and there is nowhere left to put a fallback. Reliability work is only cheap if there is exactly one place to put it. That is what the registry's indirection pays for: adding a capability does not mean touching reliability code, and improving reliability does not mean touching twenty-four features.
A chain and a breaker, which only work together
Two mechanisms sit under the router.
The fallback chain tries models in sequence. The provider is parsed from the model string, so a single chain can span OpenAI, Anthropic and Gemini instead of falling back only within one vendor's family. That matters because the failures worth surviving are usually provider-shaped. One company's endpoint is having a difficult hour, and the next model from the same company will be behind the same difficulty.
The circuit breaker is per provider. When it is open, the chain skips that provider entirely instead of paying its timeout on every request.
I would defend the composition harder than either piece. A fallback chain that keeps trying a provider already known to be failing is a delay multiplier. Each attempt pays a full timeout before the chain moves on, so the chain does nothing for availability. It multiplies the latency of every request that touches it and then serves the answer it would have served anyway. The breaker tells the chain what to skip, and without it the chain makes the bad afternoon worse. Retry logic with no memory of the last thirty seconds is the most common way a resilience feature becomes the outage.
The unglamorous layer below
Beneath the gateway are four primitives that have nothing to do with models.
Bounded queues, because an unbounded queue converts a slowdown into an outage. It does this politely. The system keeps accepting work, memory keeps growing, latency climbs past anything a user will wait for, and nothing has technically failed yet. Backpressure is how a slow dependency stays a slow dependency instead of becoming your incident.
Explicit timeouts on every call, with no unbounded waits anywhere. A hung request holds a connection, a queue slot and a user's attention.
Controlled retries, with the breaker deciding whether retrying is sane at all.
Defined degradation paths: reduced-capability behaviour that the system deliberately enters, rather than exceptions escaping to the user. This is the one I would put first if I could only have one. Degradation is a state the system knows it is in, which means it can be logged, alerted on and explained. Without it you still get degradation. You just get the kind nobody has a name for, where output gets worse and the first person to notice is a user who does not come back.
Around all of it sits per-request cost attribution, surfaced as a per-user cost dashboard, and structured logging throughout, so a failure can be diagnosed after the fact instead of living only in anecdote. There is also bring-your-own-key handling for users supplying their own provider credentials. That is a straightforward feature, and it becomes impossible to add later if cost tracking and routing are not already centralised.
Caching is a reliability feature wearing a cost disguise
Prompt caching is usually filed under cost optimisation. It belongs here too, because a cache hit is a request that cannot fail.
There is a cache adapter per provider behind a common registry, because OpenAI, Anthropic and Gemini differ materially in what is cacheable, how the prefix is identified and what invalidates it. One abstraction that pretends they are the same gets you something that half-works in three places.
It only functions at all because the layer above cooperates. The context assembler renders stable blocks with a cache-prefix hash so the prompt prefix is byte-identical between turns. Determinism there was a requirement rather than a nicety, and the reasons for that are worth their own essay. Non-deterministic context assembly does not give you a lower cache hit rate. It gives you approximately none, with no error anywhere to tell you.
What this design costs
A cross-provider fallback chain holds availability constant by letting something else vary, and the something else is behaviour. When a chain fails over from one vendor's model to another's, the request succeeds and the output is no longer the output you evaluated. Tone shifts. Instruction-following differs. Structured output that parsed reliably on one model may parse less reliably on the next.
So the honest statement about this architecture is that it converts a hard failure into a soft one, and soft failures are harder to see. I treat that as the cost of the design rather than as something I have solved. The mitigation I have is that degradation is an explicit state rather than an inference, and that model behaviour is measured in a harness rather than judged by feel: synthetic datasets scored against written rubrics, running in CI. Restraint in particular is scored as its own dimension, which is the part of the evaluation work I would defend hardest.
That harness has its own weakest link, which I would rather name than have found. Insight wording is graded by a separate model against a rubric. Judges drift, they are sensitive to rubric phrasing, and mine is not calibrated against human raters. I use those scores as a regression signal, a way of asking whether this got worse, and not as an absolute measure of quality. Calibrating against human agreement on a sample is the obvious next piece of work, and it is not done.
One more thing runs in CI that I did not expect to need: chaos scenarios, including clock-skew tests against time-dependent logic. Quiet hours, snooze windows, trust ramps and daily budgets are all functions of the clock. Time-dependent logic fails in ways ordinary tests do not reach, because ordinary tests run at whatever time the suite happens to run.
The version of this I built the first time
I did not arrive at "one path for everything" from reading about reliability. I arrived at it from having owned a delivery system where the same class of decision was made in more than one place.
At Cendra I owned the notification and real-time system, and the design decision I would still defend is that a single record drives every channel a notification can reach a person through: one outbox row rather than two independent producers. The reasoning is identical to the gateway's. When two code paths can independently decide the same thing, they will eventually disagree. That disagreement is a category of bug you fix repeatedly until you remove the second path.
A model gateway is that argument applied to inference. Provider selection, fallback, breaking, cost attribution and observability are decisions that must be made the same way every time, so they are made in one place, once.
LILA is mine alone, roughly nine and a half thousand commits since February 2026, self-counted in a private repository I own, so the gaps above are mine as well. I would rather list them than be asked.
Full architecture, including the capability registry and the delivery gate: LILA.