August 11, 2026 · 10 min read
OpenMesh field notes
Beyond “looks pretty good”
A common AI development workflow still looks like this: build the prompt, try it on ten examples, adjust the prompt, show it to the team, decide that it looks pretty good, deploy.
That works for prototypes. It becomes dangerous when AI begins operating real workflows.
AI software is probabilistic. The same input does not always produce the same path or output. Once AI retrieves information, calls tools, and takes actions, there are many more ways for the system to fail.
Evals will become as fundamental to AI development as automated testing became to traditional software development.
Test the full path
Imagine an AI agent handling an insurance claim. It needs to understand what happened, retrieve the customer's policy, identify coverage, extract information from supporting documents, spot missing evidence, determine the next action, update the claims system, and communicate with the customer.
If the final recommendation is wrong, why? The model may have misunderstood the description. Retrieval may have returned the wrong policy. The correct policy may have been interpreted incorrectly. A tool may have failed. The wrong tool may have been called. The right decision may have been written into the wrong case state.
One final accuracy metric cannot tell you which part failed.
Evaluate the system in layers
A production evaluation framework should inspect the full path from input to outcome.
| Layer | What are we evaluating? |
|---|---|
| Understanding | Did the AI correctly understand the input? |
| Extraction | Did it identify the important information? |
| Retrieval | Did it retrieve the correct context? |
| Reasoning | Did it reach an appropriate conclusion? |
| Tool selection | Did it choose the correct tool? |
| Tool execution | Were the correct parameters used? |
| Workflow | Did the system move to the correct state? |
| Safety | Did it escalate when required? |
| Communication | Was the response clear and appropriate? |
| Outcome | Did the intended real-world result happen? |
Evals should start before production
Anthropic describes an evaluation as a test where an AI system receives an input and grading logic determines whether it succeeded. For complex agents, evaluations can inspect the final output and the full trajectory, including tool calls and intermediate steps.
Waiting for production failures is an expensive feedback mechanism. Without evals, teams enter a familiar cycle: a user reports a problem, an engineer finds an example, a prompt gets modified, the problem appears fixed, the modification quietly breaks another behaviour, a different user reports a problem. Customers end up doing the regression testing.
A proper eval suite changes the process. Before deploying a change, the team can run hundreds or thousands of previously identified scenarios and detect whether performance improved or regressed.
Build golden sets from reality
The best eval datasets often come from the workflow itself: common examples, difficult examples, known failures, ambiguous cases, edge cases, high-risk scenarios, and adversarial inputs. Then define what good performance looks like.
In healthcare triage, a golden set might contain historical or synthetic cases classified by clinical experts. For every case, you might know the expected risk classification, symptoms that should be extracted, the appropriate escalation path, information the model should not invent, and the expected workflow outcome.
Rerun that suite whenever the team changes the model, prompt, knowledge base, workflow, tool definitions, or retrieval system. Qualitative experimentation becomes measurable engineering.
Weight failures by consequence
Suppose an AI system achieves 95% overall accuracy. That number sounds impressive until the remaining 5% consists entirely of high-risk cases. The aggregate metric is almost meaningless.
Production evals need weighting. Slightly awkward wording can be low severity. A wrong customer category may be medium. An incorrect financial calculation is high. A missed urgent clinical escalation is critical.
Set different quality thresholds. Imperfect tone may be acceptable. Missed safety-critical escalations are not. Generic model benchmarks tell you relatively little about whether a system is ready for a specific workflow. The benchmark that matters represents your users, your workflow, and your failure modes.
Evals change how product teams work
One useful idea in Anthropic's guidance is eval-driven development. Instead of building first and deciding afterwards whether something is good enough, teams define expected capabilities and test cases earlier. That resembles test-driven development in traditional software.
A product manager may specify: when a customer requests cancellation and qualifies for retention offer A, the system should retrieve subscription status, identify eligibility, present the correct offer, and never cancel without explicit confirmation. That specification can become an evaluation. Product requirements become executable quality checks.
Evaluate real outcomes
The shift deepens when AI begins taking action. A customer service agent can generate a technically correct answer while the support ticket remains unresolved. An AI health coach can send an excellent recommendation that users consistently misunderstand.
As AI moves deeper into workflows, evaluation needs to move from output quality to task completion, then workflow correctness, then real-world outcomes.
A demo shows that the model can do something. An eval system proves that the product can be trusted to keep doing it.