Over the past few weeks, I’ve been reading quite a bit about AI evals. One thing that finally clicked for me is that evals aren’t just another AI buzzword—they’re the engineering layer that lets us build confidence in systems whose behaviour isn’t fully deterministic.
Coming from a traditional software engineering background, this was a bit of a mindset shift.
With conventional software, confidence comes from understanding the code. We write the logic, add unit tests and integration tests, and if everything passes, we have a pretty good idea of how the system will behave.
LLMs don’t work like that.
Even with the same prompt and the same input, the model can produce slightly different outputs. You can’t reason about every decision the way you would with normal code. Instead of trying to prove correctness, you measure behaviour.
That’s essentially what an eval is.
At a high level, an eval is simple. You give an agent a task, let it do its work, and then grade what happened. But after reading more about the topic, I realized there are actually two different things you’re grading.
The first is the obvious one: did the agent produce the correct outcome?
If you’re building an invoice-processing agent, did it classify the invoice correctly? Did it send it to the right workflow? Did it correctly identify a duplicate invoice? These are relatively easy to verify because you already know what the expected outcome should be.
The second part is what I found much more interesting: how did the agent reach that outcome?
Modern agents don’t just generate text. They call tools, retrieve information, update systems and sometimes perform actions on behalf of users. If you’ve instrumented the agent, you can capture this execution trace—the sequence of tool calls, arguments, state changes and actions it performed.
That trace can be evaluated as well.
Imagine an agent that ultimately classifies an invoice correctly, but before doing so it updates the supplier’s bank account or attempts to trigger a payment without approval. From an outcome perspective, the task succeeded. From a business perspective, that’s obviously unacceptable.
That’s why looking only at the final answer isn’t enough.
Some of these checks are straightforward. Did the agent call a tool it wasn’t supposed to? Did it modify fields that should have been read-only? Did it attempt an action before receiving approval? Those are deterministic rules and are easy to automate.
Other questions are harder. Was escalating to a human actually the right decision? Did the agent gather enough evidence before making a recommendation? Was its reasoning sensible? Those usually require judgment rather than exact rules, which is where techniques like LLM-as-a-Judge become useful.
One idea that really resonated with me is that outcome quality and behavioural correctness should be measured separately.
Suppose an agent classifies invoices correctly 95% of the time, but in 4% of runs it modifies data it should never touch. If you combine everything into one overall score, the system looks excellent. In production, however, that 4% could be the difference between a successful deployment and a serious incident.
The quality of the answer and the safety of the behaviour answer two different questions. They deserve separate metrics.
I also don’t think evals replace traditional testing. We still need unit tests, integration tests, contract tests and all the engineering practices we’ve relied on for years. Evals solve a different problem. They help us measure the parts of an AI system that are probabilistic rather than deterministic.
The more I learn about applied AI, the more I feel that writing prompts is actually the easy part. The harder—and far more valuable—engineering work is building systems that you can trust in production. Evals seem to be one of the key pieces that make that possible.
I’m still learning this space, but if you’re interested in going deeper, these resources helped me build a much clearer mental model:
- Anthropic’s resources on evaluating AI agents
- Lenny Rachitsky’s guide to evals
- Hamel Husain’s writings and FAQ on evals
If you’re coming from a software engineering background like I am, I’d probably start with Lenny’s and Hamel’s articles before diving into Anthropic’s more hands-on material.