Your AI Feedback Loop Is Only as Strong as Your Evaluation Infrastructure
How strong is your AI feedback loop? Measure segmentation coverage, segmentation density, and eval confidence to test whether evals reflect production reality.
Your AI system is three months into production. A new foundation model drops, benchmarks look better across the board, and the obvious question comes up: should you switch?
That decision depends on an evaluation set built before launch, when you were still working from anticipated cases instead of real production traffic. Running both models against the pre-launch set may show one outperforming the other, but it won’t tell you which will perform better on the cases the system handles now.
That mismatch is only the first problem. The harder question is whether the feedback loop surfaces the right failures, and whether it gives you evidence you can trust when evaluating a proposed change.
Measuring the Loop
Enterprise AI teams generally can’t retrain the foundation models they build on. When everyone is calling the same API, traditional software advantages erode. What matters instead is the system you build around the model: prompts, retrieval, evaluation sets, and a working understanding of failure modes. Production feedback is what makes that system specific to your traffic. A competitor can replicate your features, but they can’t replicate months of production data routing back into evaluation and iteration.
That advantage only compounds when evaluation keeps pace with production. Even with a feedback loop in place, your evaluation infrastructure can still fall behind: some production cases may fall outside the categories you use to evaluate performance, the categories themselves may miss the attributes that explain performance differences, and small samples can make results within those categories appear more reliable than they are.
Getting stakeholders to prioritize evaluation infrastructure can be hard. The work competes for roadmap space with features and launches, and the payoff isn’t always immediately visible.
Here’s what we’ve found works: establish a consistent measurement practice and help others understand why it matters. Yes, Goodhart’s Law is real, and any individual metric has limits. But disciplined measurement still gives teams a way to find the failures that aggregate performance numbers would otherwise hide.
We've found three measurements particularly useful for testing whether evaluation infrastructure still reflects production reality. All three depend on segmentation rules: any logic that classifies a production case into a meaningful category, ranging from simple filters (documents over 20 pages, pediatric patients) to classifiers that identify harder-to-detect attributes (handwriting, poor scan quality).
Each measurement captures a different dimension:
- Segmentation coverage: how much production traffic do your segmentation rules actually capture?
If your rules only classify half your traffic, your performance numbers only describe half your system. - Segmentation density: are your rules capturing the attributes that actually explain performance differences?
If your rules tag the obvious categories but miss the dimensions that drive failures, your segmentation can tell you where performance drops without telling you why. - Eval confidence: do you have enough eval cases in each segment to trust your numbers?
An 85% accuracy score based on 30 cases could mean the system’s actual performance is anywhere from 72% to 98%.
As production traffic grows, your picture of the system sharpens. Your segmentation rules capture more, your audits surface more performance-relevant attributes, and your eval cases accumulate. More coverage means you catch more failures. Catching more failures means you can prioritize fixes based on frequency and impact. The three measurements above tell you whether the feedback loop is getting stronger.
Segmentation Coverage
Aggregate performance numbers can mask significant variation across segments. Your system might be at 95% on straightforward cases and 60% on complex ones, but an overall accuracy score won’t tell you that.
Defining segmentation rules gives you a consistent way to categorize your production cases into groups where you’d expect your system to behave differently, so you can see where it’s actually working and where it’s failing. These categories don’t need to be unusual or extreme. Your system’s performance can vary across common everyday traffic, and those patterns matter just as much as the outliers.
We saw this firsthand in a prior authorization deployment. After launch, we reviewed the system’s production metrics: precision, or the percentage of the system’s auto-approvals that humans also approved, and recall, or the percentage of the human-approved cases that the system successfully auto-approved.
At first glance, production recall looked lower than it had in development, suggesting the model was missing approvals it should have caught. But when we sampled the apparent false negatives and inspected the underlying cases, a more specific pattern emerged. Many of the misses had the same defining attribute: the fax packet contained little or no clinical evidence, and in some cases only a cover sheet. Meanwhile, the clinicians approving those cases had access to additional clinical context in the electronic medical record (EMR). In other words, the AI system and the human reviewers were not making decisions from the same evidence.
The production metric counted these cases as missed approvals, which pulled down recall. That made the issue look like model performance, but the real problem was more specific: this subset of traffic sat outside the system’s supported input boundary. Until the system had access to the evidence required for automated review, fax-only cases with insufficient clinical evidence needed to be segmented separately and routed differently.
A pre-launch eval set rarely catches this kind of production mismatch. Finding it requires looking at production cases; identifying the attributes that explain performance differences; and turning those attributes into segmentation rules, routing logic, and eval set coverage.
To build effective segmentation rules, start with any attribute that could change how well your system handles a case. In a customer support system, that might mean intent, conversation length, the tone of the user’s message, number of turns, or even the user’s phone model.
Segmentation can also be based on what the system did, not just what it received. In medical document review, that could include which insurance criteria were evaluated or how much evidence the system extracted per criterion.
When building these rules, start with deterministic logic—conditions your system can evaluate with a clear yes or no. Is the document over 20 pages? Is the patient under 18? Rules like these are cheap to build, easy to maintain, and reliable.
Manual review will sometimes surface performance-relevant attributes that can’t be captured with a clear yes-or-no rule. In those cases, a small language model can act as a lightweight classifier, tagging the attribute at scale. For example, it can distinguish frustrated customer messages from neutral ones, giving you segmentation that simple filters can’t capture.
Once your segmentation rules are in place, check whether your eval set reflects the distribution of those segments in production. If 30% of production cases involve handwritten documents but none appear in your eval set, your accuracy score is blind to that segment. Use automated checks to track this overlap, and revisit it regularly as your production traffic evolves.
One caveat: that comparison only works for traffic your rules can already identify. If your production traffic is heterogeneous and your segmentation rules only classify 40% of it, you only have visibility into 40% of your system’s behavior. The other 60% is a blind spot, where your worst failures might be hiding.
Segmentation Density
High coverage doesn’t help if your segmentation is shallow. Say you’re segmenting medical cases by two dimensions: procedure type and patient age. But what if the number of documents in the case, or whether those documents contain conflicting evidence, is what’s actually driving your system’s failures? Density is about whether you’re slicing along the axes that actually explain performance, not just the ones that are easiest to see.
The most direct way to find missing dimensions is to look at your data. Randomly sample production cases and manually review them for attributes that might have affected performance, but that your rules don’t currently capture.
For example, you may think your segmentation is adequate because your rules tag procedure type and patient age. But manual review may surface other dimensions that explain performance differences: conflicting evidence across documents, low-quality scans, unusually long case packets, or cases where the system does not consistently extract the evidence needed for review. Low segmentation density means your rules cover the obvious categories but miss the attributes that explain the failures.
You can also use an LLM to scale this up. An agent can review each case and suggest new dimensions your segmentation rules might be missing, then compare those suggestions against the dimensions your existing rules capture. The gap between what the agent suggests and what your rules capture is a rough proxy for how much your segmentation is missing. You don’t need to trust the agent’s output blindly, but it’s a useful way to surface blind spots at scale without manually auditing everything.
The practical question is whether manual review keeps surfacing performance-relevant dimensions your segmentation rules still miss. If the answer is yes, your segmentation is too shallow. The goal isn’t a specific number; it’s getting to the point where manual review stops surfacing surprises.
Eval Confidence
Coverage and density only get you so far. If your sample sizes are too small, the numbers can look more precise than they are. An accuracy score based on 30 cases may look stable, but statistically it could still be off by double digits.
How do you know if you have enough evals for a segment? Start with the confidence interval for a proportion, which estimates a range for the system’s true accuracy within that segment. That range depends on the margin of error—the same measure commonly reported with election polls. For a binary outcome, meaning each eval is marked right or wrong, the margin of error at 95% confidence is:
Say your system is performing at about 85% accuracy on cases involving prior authorization for outpatient surgery. If you have 30 eval cases in that segment, your margin of error is about ±13%. That “85%” might feel precise, but the system’s actual accuracy could plausibly be anywhere from 72% to 98%. A range that wide can mean the difference between a system that’s struggling and one that’s performing well.
Bump that to 150 cases, and your margin of error drops to ±5.7%. Now you’re in a range where you can make real decisions: is this good enough to ship? Is the new model actually better than the old one? Did that prompt change help or hurt?
Improving eval confidence starts with knowing where you stand. Begin by computing the margin of error for each segment you already have. The segments with the widest margins are your first priority: that’s where your numbers are the least trustworthy, and where additional eval cases will have the most impact.
Closing the Loop
This measurement practice doesn’t come together overnight. Your production traffic evolves, new segments emerge, and your eval set has to keep pace. Staying current requires returning to production data regularly, because each review gives your evaluation and iteration process a clearer picture of what’s actually happening.
Once the measurement practice is in place, questions that used to rely on intuition become testable. A new model drops, and you can tell whether it performs better on your production traffic. A prompt changes, and you can see whether it improved your priority segments. The feedback loop runs, and you can trace how production data is improving the system.
To get there, start by sampling your production traffic. Pull a random set of cases, check them against your segmentation rules and your eval set, and compute the margin of error for your most important segments. If the numbers make you uncomfortable, you’ve found your first priority.
