The Validation Gap in Self-Improving AI

In most AI deployments, no matter how much teams test before launch, the real world will always find scenarios the test environment missed. Users ask questions differently than expected and edge cases naturally occur. Business context changes, policies evolve, and integrations fail in ways that were hard to simulate beforehand.

This means that AI systems need a reliable way to improve after launch. The goal is to turn real-world signals like support tickets, survey scores, evaluations, and logs into measurable performance improvements quickly enough that launching without complete information is an acceptable risk to take.

With modern observability systems, every production AI system keeps a complete, continuous record of its own shortcomings. A mishandled conversation appears in a transcript. A frustrated customer leaves a low survey score or a “thumbs down” reaction to a message. And the complete agent trace is available in a logging system.

Teams can usually see what’s broken. The evidence of what to fix isn’t scarce: if anything, it arrives faster than any team can review it. They have the expertise to investigate what went wrong. What slows them down is proving a change is safe before it ships.

Validation Is the Bottleneck

Trust gets expensive when a proposed improvement moves toward production. To see why, it helps to break self-improvement into its components:

The first two steps are getting easier. Models are increasingly capable of reading production traces, grouping related failures, and drafting changes that address recurring patterns. Similar-looking failures can still have different causes, but finding problems and proposing fixes no longer has to wait on the same manual process.

Validation is different. A proposed fix has to solve the problem it targets without introducing a regression somewhere else. In production, the stakes are high: a change meant to correct one behavior can degrade another part of the system before anyone catches it, exposing millions of users to the failure. Doing nothing often feels safer than shipping a fix the team can’t trust. That’s why every proposed fix has to pass through the same scarce expert review, which quickly becomes the bottleneck.

When Evidence Outpaces the Process

We saw this bottleneck while operating a customer support AI system for a Fortune 50 enterprise. The system handles millions of conversations a month across chat and voice, generating a flood of evidence about where the customer experience can improve. As the deployment scaled, that evidence accumulated faster than the team could manually investigate, validate, and ship changes.

Evidence included NPS surveys, conversation transcripts, and support tickets. Leadership could see patterns in the moments that frustrated customers, but visibility alone didn’t make it any faster to turn those patterns into deployed changes.

The team started where most teams would: with people reading the evidence. Analysts reviewed conversations, identified defects, and handed them to engineers to fix. That process resolved individual issues, but it didn’t scale: a single change could take over a week to move through investigation, implementation, and review. Each fix moved toward production as its own piece of work, without making the next issue easier to solve.

Additionally, the process also depended on individual analysts recognizing patterns across incidents that pointed to the same underlying failure mode. When a pattern went unnoticed, the same issue kept reappearing as new work. When every fix is a bespoke investigation that starts from a blank page, the effort you spent last week buys you nothing this week.

From Feedback to Evidence

Canary is the system we built to close that gap.

Rather than treat production feedback as something to monitor on a dashboard, Canary treats it as a body of evidence to learn from. It continuously reads production outcomes, clusters recurring failures into named patterns, and ranks those patterns by business impact. Then, it automatically turns the ones worth addressing into concrete changes to the deployed system: edits to the prompts, knowledge, and routing rules that determine how the AI behaves in production.

Determining which failures belong to the same pattern is not the same as grouping similar-looking interactions. A customer trying to cancel and a customer trapped in a broken self-service flow may sound nearly identical in a transcript, yet a change aimed at one would do nothing for the other. Encoding distinctions like these is what separates useful analysis from the mere accumulation of superficially similar signals. AI systems, especially coding agents, are excellent at this type of deep analysis, making it possible to scale out the judgment required to identify these patterns across production evidence.

Once a pattern is found, coding agents also excel at analyzing the context layer and suggesting targeted changes that should address the defect. But Canary concentrates its effort on the harder work: producing the evidence the team needs to decide whether that change can be trusted in production.

For each proposed change, Canary builds an evaluation set from the production conversations that surfaced the problem. It then runs the system twice: once in its current form, and once on a separate copy containing the proposed change. To make that comparison possible, we built a versioning system that could track every component of the AI system.

After running both configurations, Canary compares the results and reports what moved, giving the domain expert proof that the fix addresses the issue and doesn’t cause additional regressions. The expert no longer has to reason about the change in the abstract, because Canary gives them evidence tied directly to the production failures the change is meant to address.

Two aspects of the evaluation make that measurement trustworthy. First, the evaluation set includes additional cases beyond the examples used to develop the fix. That shows whether the improvement extends to other instances of the same pattern, or whether the system has overfitted to the examples that inspired the change. Second, the results are scored independently from the model that proposed the change, meaning the system doesn’t grade its own work.

In the end, the expert sees proof that the proposed change to the context layer addresses the problem, based on how it performs across the evaluation set. Their job becomes deciding whether the evidence passes their sniff test. A decision that used to take a week of manual review can now be made in minutes.

What Trusted Improvement Looks Like

The effect on this customer service system showed up quickly. Within weeks of introducing Canary, NPS improved—without a model migration, platform rebuild, or other major engineering initiative. The gains came from learning systematically from production and deploying only the changes that were proven to move the needle.

More importantly, the improvement held. The risk of losing that gain is what makes teams wary of moving quickly, because a new fix can improve the behavior it targets while degrading another part of the system that already worked. A metric may jump at first, only to sag a week later as a regression surfaces. With Canary, we were able to scale validation and gain confidence that changes would improve performance without undoing the progress already made. The work was now handled almost entirely by a combination of AI and software engineering.

As the loop continued, a less obvious effect emerged that we hadn’t fully anticipated. Each validated change altered how the system behaved, which shifted the mix of signals that showed up in production, which in turn gave Canary new evidence to learn from on the next pass. The loop didn’t just improve the system’s outputs—it also reshaped its own inputs. The system got better at getting better.

When Improvement Compounds

Before Canary, improvement in our customer support deployment followed the same pattern most organizations rely on: manual investigation, one fix at a time, on a human schedule. After Canary, validated changes could move through the system continuously, fundamentally changing the trajectory of improvement instead of accelerating the same manual process.

This mechanism isn’t specific to customer support, and today we use Canary across multiple verticals. Across projects, the same five elements remain: a signal to learn from, a way to identify recurring patterns, a set of levers that can actually be changed, a method for proving that a proposed change works, and a measurable outcome the change is meant to improve.

Every production AI system already supplies the first element: evidence of how it behaves in the real world. The question is whether that evidence becomes the foundation for a system that compounds, or simply accumulates as logs no one reads. Answering this takes more than a better manual process—it requires technology that allows trustworthy improvement to scale. By investing in technology, you bring down the cost of trust, so that improvement is no longer bottlenecked on team capacity. This is what unlocks self-improving systems.