Building an AI agent that looks good in a demo takes an afternoon. Building one that a team keeps using six months later is a different job. The difference isn't a cleverer prompt or a bigger model. It's whether you can answer one question every time you change something: did that make it better or worse?
If you can't answer that, every change is a guess. Eventually one of those guesses quietly makes the agent worse, someone gets a confidently wrong answer at a bad moment, and trust is gone. Internal tools rarely get a second chance.
So my rule is simple: evals before features.
Why spot checks lie
Everyone starts by trying a few examples by hand. Edit the prompt, run it on the three cases you remember, see that they look better, ship.
The problem is that LLM behavior is coupled in ways you can't see. A prompt change that fixes the case in front of you can break five others. A model upgrade that's better on average can be worse on exactly the inputs your users care about. A new tool can make the agent reach for it when it shouldn't. On three hand-picked examples, all of these look like wins.
Spot checks don't fail loudly. They fail by telling you what you hoped to hear.
What an eval actually is
An eval is just a test suite for behavior that isn't deterministic:
- A set of representative cases. Real inputs, including the ugly ones, with what a good outcome looks like.
- Scores that mean something. Not "looks good" — accuracy on the thing the agent exists to do, plus the costs of doing it.
- A habit of running it. On every prompt, model, or tool change, before release — not when someone remembers.
For the internal agents I build at work, that means measuring agent accuracy on representative cases and running the suite before every release. For a multi-agent incident system, it means labelled incident scenarios scored on root-cause accuracy, tool-call success rate, latency, and cost per run. For a RAG system, it means scoring faithfulness (does the answer stick to the sources?) and context recall (did retrieval find what was needed?).
Score the steps, not just the answer
The most useful thing I've learned about evals is to measure the intermediate steps, not only the final output.
An agent that gives a bad answer could have failed at retrieval, at reasoning, or at the tool call in between. If you only score the final answer, all you know is that something went wrong. Scoring each stage tells you where. In a RAG system, low context recall says fix retrieval; high recall with low faithfulness says fix generation. In a multi-agent system, scoring root-cause accuracy separately from fix quality catches the worst failure — a confident fix for a cause nobody established.
Cost and latency are part of quality
An agent that's 2% more accurate and three times slower might be worse for the person waiting on it. An agent that floods its context window with raw tool output gets slower, more expensive, and less accurate. That's why cost per run and latency belong in the eval next to accuracy. Trade-offs you can see are decisions; trade-offs you can't see are accidents.
Evals make you faster, not slower
The usual objection is time: building eval sets is work, and it doesn't ship features. But the alternative is slower. Without evals, every change needs careful manual checking, and people get nervous about touching the prompt at all. With evals, you can try a bold change, see the numbers in minutes, and keep it or throw it away.
Evals are what let you iterate on an agent with the same confidence a test suite gives you on normal code.
The checklist I use
Before an agent ships, and before every change after that:
- Do I have representative cases — including the failures I've already seen?
- Am I scoring the steps, not just the final answer?
- Are cost and latency tracked next to accuracy?
- Does every release run the suite automatically?
- Is a human approving anything irreversible until the numbers have earned more autonomy?
None of this is glamorous. It's the same discipline as testing, applied to systems that don't give the same answer twice. It's also what turns "a cool demo" into something a team actually relies on.