Shipping AI agents and the importance of AI evaluation and observability

Your AI agent can pass every test but fail silently in production. Here’s the AI evaluation and observability loop that catches it. 

The problem is that nothing holds still 

Part 2 in our series When LLMs meet enterprise data discusses the gap between a working demo and something running in production. 

There's a moment in every enterprise AI project that feels like victory but it's actually the riskiest part of the whole thing. The agent works in a controlled pre-prod environment, stakeholders nod, a rollout date lands on the calendar. And then someone asks, "Okay, but how do we know it'll work next Tuesday, in production, in front of real customers?" 

That's the uncomfortable truth about AI agents that traditional software development doesn't prepare you for. When you write a traditional software function, you write a test, the test passes, and it keeps passing until someone changes the code. 

With LLM agents, nothing holds still. The model version changes underneath you. Someone edits a prompt. The underlying data shifts, because the agent runs on live customer data, and real customer records don't sit still. Correct stops being a fixed target. You can ship something that works perfectly today but not next week, and the hard part is you won't know. 

So before we made the agent smarter, we made it observable. 2 things mattered, and they build on each other – and together they quietly solve a 3rd problem we didn't see coming. 

Grading the agent on a loop 

The 1st was letting the model grade the model – using an LLM as a judge. Using AI to evaluate AI sounds like cheating, but think about the alternative. A human can review a few dozen agent responses before losing focus. We evaluate thousands of runs every time someone touches a prompt or changes a model version. That's not a job a person can do. 

The part that took us a while to internalize: This isn't a one-off test you run before launch. It's a loop that keeps running, in the same shape, before and after the agent goes live. 

Testing AI applications is a 5-step process. The key is that evaluation before launch (steps 1–3) and evaluation in production (step 4) use the exact same scoring mechanism, so the results are directly comparable, with step 5 closing the loop, as follows: 

1. Generate evaluation cases
   Build cases from real situations, not made-up ones – masked production data turned into realistic scenarios, each pinning down the input, the expected response, and the policies that must hold.

2. Provision the test data for each case 
   Load the full context each case requires – mixing masked production data with synthetic data to fill gaps – and keep it reproducible so you can rerun the same case after any change. 

3. Run and score the evaluations
   The judge rates each output across defined dimensions (correctness, relevance, policy compliance, tone) and rolls them into a weighted total, not a blunt pass/fail. Critical calls, like permissions, eligibility, and financial limits, bypass the judge and run through deterministic rules instead. 
4. Observe live behavior
   Once in production, the same dimensions, weights, and rules run against real traffic, so pre- and post-launch results are directly comparable. Failures flag themselves, so there’s no waiting on a support ticket. 
5. Feed issues back 
   Engineering gets the full evidence for each failure, fixes the cause, and adds the case to the regression suite so it's tested before the next release. 

 The obvious objection is: Who judges the judge? Before trusting it at scale, we calibrated it against a human-labeled sample and measured how often the 2 agreed. LLM judges like verbose answers and rate their own model's output more favorably – so you want to know that agreement rate before you lean on it.  

Was the judge perfect? No. But a judge you've measured is one you can trust to catch the big regressions – the ‘This prompt change broke 30% of answers’ kind – the moment they happen instead of 3 weeks later as a result of a support ticket. The moral of the story? For catching regressions at scale, a flawed-but-measured automated judge beats a perfect human who can only look at a fraction of the cases. 

Snapshotting the context and not just the answer 

The second piece is more important and gets talked about less. An agent's answer is only as good as the context it's exposed to. And in a live system, that context keeps changing. So when production evaluation flags a problematic response and you try to debug it the following day, the context has already changed. You're staring at logs, trying to reconstruct what the agent saw, and you can't, because the context moved on.  

The fix was being able to snapshot the exact data state and complete context an agent session ran against – for every agent session. Not just the answer, but the data and context underneath the answer. And context here is more than the data – it's everything that shaped the run: The model version, the tools available to the agent, the authenticated user, and the runtime policies in force. We capture the run's telemetry too – tokens consumed, latency, which tools actually fired – but that's a measurement of what happened, not an input that caused it. Freezing the inputs makes 2 things possible that you usually can't do. With K2view AI Context Optimizer™ you can: 

1. Check a past answer against the exact data and context it ran on – the frozen state – to judge whether it was actually correct, without re-running anything. 
2. Replay the session against that same frozen data and inputs, so you're debugging the exact conditions the agent saw, not an approximation of them. 

 That gets us most of the way to reproducing a bug instead of guessing at it. The LLM itself is still nondeterministic – the same prompt won't always return a byte-for-byte identical answer – but that's rarely where an ‘it only happens sometimes’ bug actually comes from. Far more often it's the context that shifted underneath you, and freezing the context is what turns the guessing game into something you can reproduce on demand. 

The model zoo and the politics of which one 

This is the problem we didn't see coming – the most operationally painful of all – and it's where those first 2 pieces paid off most. We don't get to pick one model and standardize on it. Our customers do that for us, and they all pick differently. Some are locked to Azure OpenAI, some to Bedrock, some have data-residency rules that rule out half the providers, and some run open-weight models in their own walls so nothing leaves the building. 

And then every one of these providers ships a new model version every few months, deprecates the old one on a timeline you don't control, and changes the pricing while they're at it. 

So, "Is the agent good?" is the wrong question. The real question a customer asks is, "Is the agent good on the model I'm allowed to use, at a cost I can live with?" 

And you have to be able to answer that question clearly, with data. Early on we couldn't. Someone would ask, "Should we move from this model to the cheaper one?" and there was no honest answer. That's not good enough when you're advising a customer on a decision that changes both their bill and their response quality. 

Here's what made this manageable: we'd already built the loop, and model comparison falls right out of it. The judge and the snapshots that power it hand you a clean benchmark practically for free. Pin a snapshot of real data, run the same set of questions through provider A, provider B, and the cheaper model from each, and score every run with the same judge. 

Now the comparison is fair: Same data, same questions, and same rubric – only the model changes. You get a table you can put in front of a customer: 

• This model is 4% more accurate but 3x the cost per answer. 
• That one is fast and cheap but struggles with multi-step questions.  

That turns "Which model should we use?" from a debate into a measurement. And because we're often the ones who deal with the support consequences of their choice, being able to measure the trade-offs (instead of defaulting to the most expensive model and praying for the best) is worth a lot. 

How we do it 

K2view AI Evaluation and Observability is responsible for token tracking, auditing of each step, and regression testing for agent outputs. That's the home for the LLM-as-judge scoring. 

Because each entity's data lives in its own self-contained Micro-Database (one database per entity), you can export a snapshot of that Micro-Database as a portable artifact – a customer's whole data state, captured as a file you can stash and reload. That's what makes ‘Replay this session against exactly the data it saw’ happen. The single-entity isolation that enables sandbox work (as described in Part 1 of this series) is the same thing that makes reproducible agent sessions work here. 

K2view supports flexible model management, switching the model (and provider) per invocation. So the agent flow stays the same and you swap only the model behind it, which is exactly what a clean benchmark needs. Snapshot for fixed data, judge for fixed scoring, per-invocation model switching to vary the 1 thing you're testing. 3 separate pieces that compose into 1 clean benchmark. 

The lesson 

For agents, evaluation and observability are not something you add in v2. They decide whether there will be a v2 at all. Build the ability to grade and replay before you build the clever features. You'll be tempted to skip it, but don't.

Coming soon: “How we stopped an AI agent from reading data it shouldn't have access to”