Table of contents
How we use LLM-as-a-judge and context replay to keep AI agents reliable in production and compare models fairly.
Key takeaways
- AI agent evaluation runs as a loop
The same scoring criteria run before and after the agent goes live, so pre- and post-launch results are directly comparable. - An LLM-as-a-judge makes evaluation scale
It scores thousands of runs after every prompt or model change. Measure how often it agrees with a human reviewer before you rely on it, and keep critical calls like permissions and financial limits on fixed rules. - Snapshot the context and not just the answer
An agent’s answer depends on the data, model version, tools, user, and policies it ran with. Saving that context lets you check a past answer and replay the session under the exact conditions that produced it. - Fair model comparisons need fixed data and fixed scoring
Pin a snapshot of real data and run the same questions through each model, scored by the same judge. Only the model changes, so cost and accuracy trade-offs become measurable. - K2view AI Evaluation and Observability covers the whole loop
It comes with the K2view AI Context Platform, doesn’t require any other K2view solution, and supports switching the model and provider for each invocation.
The problem with AI is that nothing holds still
Part 2 in our series, "When LLMs meet enterprise data", discusses the gap between a working demo and something running in production.
There's a moment in every enterprise AI project that feels like victory but it's actually the riskiest part of the whole thing. The AI agent works in a controlled pre-prod environment, stakeholders nod, a rollout date lands on the calendar. And then someone asks, "Okay, but how do we know it'll work next Tuesday, in production, in front of real customers?"
That's the uncomfortable truth about AI agents that traditional software development doesn't prepare you for. When you write a traditional software function, you write a test, the test passes, and it keeps passing until someone changes the code.
With LLM agents, nothing holds still. The model version changes underneath you. Someone edits a prompt. The underlying data shifts, because the agent runs on live customer data, and real customer records don't sit still. Correct stops being a fixed target. You can ship something that works perfectly today but not next week, and the hard part is you won't know.
So, before we made the agent smarter, we made it observable. Two things mattered, and they build on each other – and together they quietly solve a third problem we didn't see coming.
How do you use LLM-as-a-judge to evaluate AI agents?
The first was letting the model grade the model – using an LLM as a judge. Using AI to evaluate AI sounds like cheating, but think about the alternative. The average person can review a few dozen agent responses before losing focus. We evaluate thousands of runs every time someone touches a prompt or changes a model version. That’s not a job for people.
It took us a while to internalize that the judge doesn’t run once before launch. It runs in a loop that keeps the same shape before and after the agent goes live, so pre- and post-launch scores are directly comparable, and production failures flow back to dev as new regression cases, as described in our guide to AI evaluation and observability. Critical calls, like permissions, eligibility, and financial limits, bypass the judge and run through deterministic rules instead.
The obvious objection is, “Who judges the judge?” Before trusting the LLM at scale, we compared it to a human decision-maker and measured how often the two agreed. LLMs like verbose answers and rate their own model’s outputs more favorably, so you want to know that agreement rate before you lean on it.
Was the LLM judge perfect? No. But you could trust it to catch the big regressions – the “this prompt change broke 30% of answers” kind – the moment they happen, instead of three weeks later in response to a support ticket. For catching regressions at scale, a flawed but measurable judge beats a human expert who can only look at a fraction of the cases.
Why snapshot the context and not just the answer?
The second piece is more important and gets talked about less. An agent's answer is only as good as the context it's exposed to. And in a live system, that context keeps changing. So when production evaluation flags a problematic response and you try to debug it the following day, the context has already changed. You're staring at logs, trying to reconstruct what the agent saw, and you can't, because the context moved on.
The fix was being able to snapshot the exact data state and complete context an agent session ran against – for every agent session. Not just the answer, but the data and context underneath the answer. And context here is more than the data – it's everything that shaped the run: The model version, the tools available to the agent, the authenticated user, and the runtime policies in force. We capture the run's telemetry too – tokens consumed, latency, which tools actually fired – but that's a measurement of what happened, not an input that caused it. Freezing the inputs makes two things possible that you usually can't do. With K2view AI Context Optimizer™ you can:
- Check a past answer against the exact data and context it ran on – the frozen state – to judge whether it was actually correct, without re-running anything.
- Replay the session against that same frozen data and inputs, so you're debugging the exact conditions the agent saw, not an approximation of them.
That gets us most of the way to reproducing a bug instead of guessing at it. The LLM itself is still nondeterministic – meaning that the same prompt won't always return a byte-for-byte identical answer – but that's rarely where an "it only happens sometimes" bug actually comes from. Far more often it's the context that shifts underneath you, so freezing the context is what turns the guessing game into something you can reproduce on demand.
How do you compare LLMs fairly across providers?
This is the problem we didn't see coming – the most operationally painful of all – and it's where those first two pieces paid off most. We don't get to pick one model and standardize on it. Our customers do that for us, and they all pick differently. Some are locked to Azure OpenAI, some to Bedrock, some have data-residency rules that rule out half the providers, and some run open-weight models in their own walls so nothing leaves the premises.
And then every one of these providers ships a new model version every few months, deprecates the old one on a timeline you don't control, and changes the pricing while they're at it.
So, "Is the agent good?" is the wrong question. The real question a customer asks is, "Is the agent good on the model I'm allowed to use, at a cost I can live with?"
And you have to be able to answer that question clearly, with data. Early on we couldn't. Someone would ask, "Should we move from this model to the cheaper one?" and there was no honest answer. That's not good enough when you're advising a customer on a decision that changes both their bill and their response quality.
Here's what made this manageable: We'd already built an automated AI agent evaluation and observability loop, and model comparison is part of it. The judge and the snapshots that power it hand you a clean benchmark practically for free. Pin a snapshot of real data, run the same set of questions through provider A, provider B, and the cheaper model from each, and score every run with the same judge.
Now the comparison is fair: Same data, same questions, and same rubric – only the model changes. You get a table with answers you can put in front of your customer, like:
- This model is 4% more accurate but 3x the cost per answer.
- That model is fast and cheap but struggles with multi-step questions.
That turns "Which model should we use?" from a debate into a measurement. And because we're often the ones who deal with the support consequences of their choice, being able to measure the trade-offs (instead of defaulting to the most expensive model and praying for the best) is worth a lot.
How K2view evaluates AI agents in production
K2view AI Evaluation and Observability is responsible for tracking token use, tracing each step, and regression testing for dev fixes. That's the home for the LLM-as-judge scoring.
Because each entity's data lives in its own self-contained Micro-Database™ (one database per entity), you can export a snapshot of that entity as a portable artifact – for instance, a customer's whole data state, captured as a file you can stash and reload. That's what makes "Replay this session against exactly the data it saw" happen. The single-entity isolation that enables sandbox work is the same thing that makes reproducible agent sessions work here.
K2view supports flexible model management, switching the model (and provider) per invocation. K2view lets you switch the model and provider for each invocation. The agent flow stays the same and only the model behind it changes, which is what a clean benchmark needs. Snapshot for fixed data, judge for fixed scoring, per-invocation model switching to vary the one thing you're testing. In short, three separate pieces that compose into one clean benchmark.
The lesson
For AI agents, evaluation and observability are not something you add in v2. They decide whether there will be a v2 at all. Build the ability to grade and replay before you build the clever features. You'll be tempted to skip it, but don't.
Coming soon – AI data entitlement: Why a prompt is not secure
