AI applications must be evaluated and observed – against expected behavior, business context, governing controls, and production outcomes – continuously.
Traditional software remains deterministic, even when AI generates the code. AI applications are different because an LLM reasons, makes decisions, selects tools, and may take action at runtime.
The number of possible interactions is effectively infinite, and AI outputs are non-deterministic.
An LLM judge can evaluate responses across dimensions such as correctness, relevance, conciseness, politeness, and empathy.
Critical decisions must remain rule-based so they are deterministic and consistent.
The same evaluation framework used before release should run continuously in production.
Failed interactions are sent back to the AI engineering team to diagnose, fix, and retest – continually improving AI system behavior.
There are 2 major challenges for QA in AI-assisted development.
The first is speed. AI-assisted development enables teams to generate more code and test cases faster, increasing the demand for test-specific AI-composed validation data. The first 5 articles in this series explored the resulting AI test data bottleneck.
The second challenge is trust.
Traditional software – even when AI generates the code – is still deterministic. Its logic is fixed in the application, so the same inputs and conditions should produce the same results.
An AI application is different because an LLM is part of the application’s control flow. The LLM interprets the prompt, reasons over business context, makes decisions, selects tools, and may take action.
The number of possible combinations is effectively infinite.
Every interaction can vary according to the user, prompt, business context, conversation history, LLM version, available tools, policies, permissions, and live state of enterprise systems. Because the output is non-deterministic, the same request may also produce different results.
QA cannot define and test every possible interaction before release.
And when agentic systems operate at production speed and scale, organizations cannot rely on users or employees to notice and report incorrect behavior. An agent may make the wrong recommendation, expose the wrong data, invoke the wrong tool, or take an inappropriate action without producing a conventional software bug.
Testing AI-generated software therefore requires a different approach: Continuous evaluation and observability before and after release.
AI evaluation is often framed as measuring whether a response is accurate.
But AI outputs are rarely simply right or wrong.
Consider an AI application recommending the next best offer for a specific customer. Several offers may be reasonable. One may be highly relevant but poorly timed. A second may comply with policy but deliver less value. A third may be acceptable, but not optimal.
There are shades of correctness, relevance, completeness, appropriateness, and business value.
The risk also extends beyond the response itself. AI agents might:
Teams must therefore evaluate not only what the AI said, but what context it used, what it decided, which tools it invoked, what action it took, and what changed as a result.
Because AI outputs are not always binary, an LLM should be used as a judge.
Before evaluation begins, teams define the dimensions that matter for the use case. For a customer-facing application, these might include:
Correctness
The LLM judge scores each dimension separately, for example from 1 to 10, and explains the reason for each score.
The individual scores are then combined into a total score using custom weightings. Correctness and policy compliance may carry more weight than tone or conciseness.
This produces a more meaningful assessment than a simple pass or fail. It shows where the output was strong, where it fell short, and whether the total score met the required threshold.
But critical decisions should not be left to an LLM.
Permissions, eligibility, financial limits, prohibited actions, regulatory controls, and other high-risk requirements should be enforced through deterministic rules. That ensures the same conditions produce the same decision consistently.
Continuous AI evaluation and observability spans pre-production evaluation and live production evaluation in a closed feedback loop, as explained below:
Evaluation cases should begin with real business situations.
For example, if an AI agent handles billing concessions for VIP customers, teams can use masked VIP customer data from production to generate realistic conversations based on actual account conditions, payment history, previous concessions, open support issues, and eligibility rules.
Each case should define the conversation input, relevant business context, expected response or action, and the policies and rules that must apply.
This creates coverage grounded in the kinds of common, rare, negative, and policy-boundary situations the AI application will encounter in production.
Once the evaluation cases have been generated, the corresponding test data must be provisioned into the evaluation environment.
For the VIP billing-concession example, that means delivering the complete customer context needed to run each conversation – including account, billing, payment, support, eligibility, and policy data across the relevant systems.
The data may combine masked production-derived data with synthetic data. Production-derived data preserves realistic relationships and business patterns, while synthetic data fills gaps and creates conditions that may not exist in available datasets.
The data must also be reproducible so the same evaluation can be rerun after changes to the prompt, model, tool, policy, or agent configuration.
The AI application is tested against each evaluation case.
Before testing begins, teams define the dimensions the LLM judge will use to evaluate the output. For example:
The LLM judge scores each dimension separately – for example, from 1 to 10 – and explains the reason for each score.
The individual scores are then combined into a total score using custom weightings. Correctness and policy compliance, for example, may carry more weight than conciseness or tone.
This provides a more nuanced evaluation than a simple pass or fail. It shows where the output performed well, where it fell short, and whether its total score met the required threshold.
Critical decisions are evaluated separately through deterministic rules. Permissions, eligibility, financial limits, prohibited actions, and regulatory controls must produce consistent decisions under the same conditions.
Interactions that fall below the required score or violate a deterministic rule are flagged as failed. QA reviews these exceptions to determine whether the AI system’s behavior meets the required standard.
Once the application enters production, the same evaluation framework runs against live interactions.
To clarify, the same dimensions, scoring scale, weightings, thresholds, and deterministic rules used in steps 1-3 are now applied to step 4. This makes in-production test results directly comparable with pre-production test results.
The process automatically flags failed interactions and other exceptions for QA review, without depending on users or employees to recognize and report incorrect behavior.
This is the fundamental change: AI evaluation continues where the AI system is actually reasoning, deciding, and acting.
When production evaluation flags a failed interaction, the AI engineering team receives the evidence needed to verify and diagnose the problem.
That includes the conversation, business context, agent response or action, dimension scores, total score, violated rules, expected behavior, prompt, model, tools, and policies involved.
The team can then determine whether the cause was the prompt, context, model, tool selection, policy, rule, or application logic – and then fix it.
The failed interaction is also added to the regression suite so the corrected system can be tested against the same conditions before release.
Production evaluation flags the problem, the AI engineering team fixes it, and regression testing confirms that the fix works. This closed loop continually improves the AI system’s behavior.
K2view connects pre-production validation, production evaluation, and continuous improvement through the same entity-centric data foundation.
Before production, K2view uses masked production-derived data and synthetic data to generate and provision realistic evaluation scenarios around business entities such as customers, accounts, claims, orders, employees, and devices.
During production, K2view delivers live, governed business context scoped to the entity and task at hand.
After execution, K2view preserves lineage connecting the entity, context, policies, prompt, model, tools, actions, system changes, outcomes, evaluation scores, and cost.
This enables engineering teams to explain failures, reproduce them, fix the underlying problem, and verify the correction through regression testing. It also reduces the gap between the context used during evaluation and the live conditions the AI application encounters in production.
AI applications cannot be fully tested before release.
The possible interactions are effectively infinite, the outputs are non-deterministic, and the application may reason, decide, and act dynamically in production.
Enterprises therefore need to generate evaluation cases from real business context, provision the data needed to run them, score outputs using an LLM judge, enforce critical controls through deterministic rules, and apply the same evaluation continuously in production.
When failed interactions are fed back to the AI engineering team, fixed, and added to regression testing, evaluation becomes a continuous improvement loop.
That is why testing AI applications never stops.