The Example Store Is Model State
Consider a classifier that retrieves labeled examples for an incoming query and gives them to an LLM to choose a label. This is a form of k-nearest-neighbor classification with an LLM decision rule: the store supplies the examples and the LLM decides how they imply a label. More elaborate systems may use lexical retrieval, reranking, filtering, or other machinery to select those examples.
The operational consequence is simple: the example store is model state. Adding, removing, or editing an example can change future classifications. A mislabeled row can pull decisions in the wrong direction; redundant rows can crowd better evidence out of a fixed context.
For the system to work, the input has to contain enough information to classify it, the taxonomy has to define distinctions that can be applied consistently, and the retrieval-and-generation pipeline has to surface and use evidence relevant to those distinctions. In a simple dense-retrieval system, much of the last requirement falls on the embedding space; reranking and the LLM can shift some of that burden downstream.
When the classifier gets a case wrong, a useful first question is: what evidence did it see?
Sometimes the neighborhood is effectively empty: retrieval returns examples, but none is useful. The store may not cover this part of the input distribution, or the case may fall outside the taxonomy. This is a coverage problem.
Sometimes the neighborhood is conflicted: plausible examples are retrieved, but they point toward different labels. Here the problem is separation.
Suppose a classifier distinguishes authentication_failure from authorization_failure. If the difficult cases involve users who authenticate successfully but cannot access a resource, adding more canonical examples of bad passwords and denied permissions may do little. Examples around the actual boundary are more useful.
A third case is harder to detect. The retrieved examples can be close, internally consistent, and wrong. A row may be mislabeled; the representation may collapse concepts the taxonomy distinguishes; or retrieval may be driven by features correlated with a label rather than relevant to it. Such a neighborhood can look healthy by ordinary retrieval diagnostics, so ground truth remains necessary.
Finally, the retrieved evidence can be good and the LLM can make the wrong decision. That is a decision-layer failure, not a reason to change the store.
Individual examples should therefore be evaluated by their effect on decisions rather than by whether they look good in isolation.
If accuracy is poor whenever a row is retrieved, the row is worth investigating, but the association is not causal; it may simply appear on difficult cases. A more informative test holds the evaluation cases fixed and changes the store.
This is very tractable for a retrieval-augmented classifier. Adding or removing an example requires no model retraining. The current store can be compared directly with candidate versions in which a row is removed, edited, or supplemented with another example.
The evaluation set should represent the decision problem. That requires representative labeled traffic to estimate overall performance, important slices where aggregate metrics could hide regressions, and challenge cases around difficult boundaries or high-cost errors.
Suppose an authentication_failure example is frequently retrieved for authorization errors. Candidate interventions include removing it, rewriting it, or adding an authorization example that competes with it. Evaluating those alternatives on the same cases reveals both the improvement being targeted and any collateral regressions.
Editing is especially relevant when a row has both useful and harmful effects. If removing it fixes one group of cases while breaking another, a narrower version may preserve its useful role. If the problem is missing competing evidence, an addition may be better than either editing or deletion.
A row has no context-free quality score. Its value depends on the other examples in the store, the retrieval policy, the decision model, and the input distribution. Two rows may substitute for one another; an example that matters today may become redundant after another is added.
Candidate changes should be judged against the decision function the classifier is intended to optimize. A change that improves quality without meaningful regressions is straightforward. When objectives conflict, improvement in the primary metric can be required subject to explicit bounds on protected slices, high-cost errors, latency, or cost. Similarity, novelty, and class balance may help generate hypotheses, but they are not the objective.
Counterfactual evaluation can also be cheaper than full end-to-end replay suggests. Run retrieval for the baseline and candidate stores first and identify the cases whose selected evidence changes. If removing a row alters the final context for only 2% of the evaluation set, only those cases generally require another expensive LLM call.
The changed cases also explain what an aggregate score can hide. If removing a row creates 47 wrong-to-right decisions and nine right-to-wrong decisions, those regressions reveal the useful role the row was serving and may motivate an edit or a competing addition rather than deletion.
Cases used to discover and tune a change should not be the only cases used to validate it. Candidate changes can be developed on one set and confirmed on held-out data; a later production window is particularly useful when the input distribution evolves.
The evaluator itself can be tested by perturbing a copy of a working store: flip an influential label, introduce redundant rows, remove examples from a covered region, or add an example at a known boundary. If the evaluation cannot detect effects introduced deliberately, it is unlikely to guide subtler edits reliably.
The store is therefore not a corpus that should simply accumulate plausible examples. It is editable model state.