Every model call has a fixed token budget, and there is always more potentially relevant material than will fit inside it. That is the permanent state of any assistant with a memory, so something has to decide what gets left out on every turn. In most systems that something is an accident: the end of a prompt template, the tail of a slice, whatever the retriever's top-k happened to return that time.
I think that is the wrong home for the decision. What a system discards when it cannot hold everything is the most product-shaped choice in the pipeline. It decides whether the assistant reads as something that knows you or as something you have been introduced to again. Truncate the wrong end and it forgets the thing you said a minute ago. Retrieve too generously and the one relevant fact is buried under nine plausible ones, which looks like a model problem and is not one.
So in LILA the context assembler is a module in its own right, and the allocation is written down instead of emerging from whatever ran last.
Five layers competing for the same space
Context is assembled from five retrieval layers: the session, the conversation, memory, a knowledge graph over entities and relations, and RAG over indexed documents.
Naming them separately is half of the design, because each answers a different question. The session and the conversation are about what is happening now: the immediate state, and the turns that led to it. Memory is about what is durably true of this person. It is held episodically and promoted into long-term storage when it earns it. The graph is about how the entities in their life relate to each other, and RAG is about what is written down somewhere else.
Those are five different kinds of knowing, and they are not interchangeable. Once they collapse into a single blob called "the context", you lose the ability to say which kind was sacrificed on a turn that went badly. You can see that the answer was poor. You cannot see that it was poor because the graph layer took space the conversation needed.
Allocation is per use case, and the use case is the product
Each layer receives a ratio of the budget and a cap. A layer that cannot fill its allocation gives the remainder back, and the remainder is redistributed. A budget that leaves tokens unspent has become smaller than the one you designed without anyone deciding that.
The ratios differ per use case, and this is where the design stops being plumbing. A chat turn weights conversation and memory heavily, because the user is mid-thought and continuity is the whole experience. A morning briefing weights memory and RAG, because nobody is mid-thought when it arrives and the value is in synthesis across things they have not looked at yet. A speech-to-speech voice turn takes none of the five. In a spoken exchange latency matters more than depth, and a considered answer that arrives a beat late is a worse answer.
That last one is a claim about what the product is when someone speaks to it rather than typing, and a claim like that ought to sit in a table someone can argue with. If it lived as an if inside a prompt builder, nobody would argue with it, because nobody would find it.
The principle I would carry to any system is that allocation should be a declared policy per use case, not a default that each feature drifts away from. The moment two features assemble context slightly differently for reasons nobody wrote down, you have two products.
Determinism, which sounds pedantic and is load-bearing
Where two layers tie for a residual token, the tie is broken by position in a declared array, never by whatever order the map happened to iterate in. Same inputs, same context, every time.
That looks like fussiness over a single token. It is what the two most important properties of the module rest on, and neither of them is about tokens.
The first is that evaluation stops meaning anything without it. If identical inputs can produce two different contexts, then a score that moved between runs might be a regression or might be iteration order, and you cannot tell which. Every measurement built on top inherits that noise. The delivery gate and the pattern-detection rubric I use to score restraint, including the dimension that scores correctly saying nothing at all, are only trustworthy to the extent that the input to the model was reproducible. Determinism in the assembler is what makes the harness above it a measurement instead of an anecdote.
The second is caching, which is the part with a bill attached.
Rendering for a cache you do not control
The assembled context does not render as a string concatenation. It renders into stable blocks with a cache-prefix hash, so the prefix of the prompt stays byte-identical from one turn to the next.
Provider-side prompt caching is exact-prefix matching, and exact means exact. A reordered pair of blocks misses. An extra newline misses. A timestamp rendered near the top of the prompt misses on every turn, forever, and nothing in your telemetry says "you are paying full price because of a space". The cache either hits, or it silently does not, and you conclude that caching is overrated.
There are separate cache adapters for OpenAI, Anthropic and Gemini, because the three providers' caching contracts differ in what is cacheable, how the prefix is identified and what invalidates it. Pretending prompt caching is one feature with three implementations gets you a middle abstraction that works properly nowhere. That work sits in the model gateway, alongside the fallback and circuit-breaking machinery that keeps a provider's bad afternoon from becoming an outage.
This is unglamorous, and it changes both latency and cost, which are product properties. A user does not experience your cache hit rate. They experience a reply that arrives before they have started to wonder.
The part that is not finished
The ratios are hand-set defaults. I chose them from judgement about what each use case needs. The default table is frozen in code with a comment recording that it is a starting point and that retuning it from measurements is a separate scheduled piece of work. That work is not done. It is the largest piece of unfinished thinking in the system, and I would rather write that here than let someone find the comment and wonder what else is decorated.
What finishing it looks like is fairly clear, which is the frustrating part. The measurement I want is retrieval utility per layer per use case: of the tokens spent on the graph layer in this use case, how many appeared in the output or demonstrably shaped it. Layers that consistently return nothing the model uses should lose their share to layers that do. Doing that honestly means attributing influence rather than presence, which is hard, and it is why a plausible-looking number here would be worse than an admitted gap.
A hand-set table that admits it is hand-set can be improved by whoever reads it next. A hand-set table presented as principled cannot, because nobody knows it is a guess.
Why this is the module I would show first
It is where the product judgement and the engineering are the same act. Choosing that a voice turn carries no retrieved memory is a product decision. Making the tie-break deterministic so the choice can be measured is an engineering one. Rendering blocks so the prefix is stable is both, since it buys latency the user feels. A system where context is assembled ad hoc per feature has made all of those decisions without noticing it made any of them.
LILA is my own project, roughly nine and a half thousand commits since February 2026, self-counted in a private repository I own, with nobody else to verify the number. So every decision above is mine to defend, including the one I have described as unfinished.
More on the surrounding architecture, the memory promotion path and the delivery gate: LILA.