Model evaluation asks a tractable question: given this input, how good is the output? Agent evaluation asks something much harder, because an agent’s output is not a string. It is a trajectory — a sequence of decisions, tool calls, retries and recoveries that can reach a correct answer by a route you would never sanction.
The measurable consequence of ignoring this is documented: agents evaluated only on final output pass 20–40% more test cases than trajectory evaluation reveals, because bad paths that happen to land on the right answer are scored as successes.
Three layers, and most teams only build one
Current practice converges on evaluating agents at three distinct layers:
| Layer | What it scores | Where it runs |
|---|---|---|
| Final answer | The last message — did it get there | Offline, against a held-out set |
| Trajectory | The sequence of steps, tool calls and retries | Offline and online |
| Per-turn | The meaning and safety of each individual turn | Online, against production traffic |
A held-out benchmark covers the first layer. Production signal comes from the third. Teams that stop at layer one are measuring how their agent performs on problems they already anticipated.
The metrics that carry signal
Task completion, verified on end state
Did the agent accomplish the goal — checked against the actual end state of the system, not the agent’s claim about it. An agent that reports success while leaving a record half-updated has failed, and only end-state verification catches that.
Trajectory metrics
Step efficiency, argument correctness, tool correctness, plan adherence, and plan quality. Practice favours covering the trajectory with several specific metrics rather than one composite score — a single number tells you something went wrong without telling you where.
Tool-call correctness
Most production agent failures are tool failures, not reasoning failures. The agent picks the wrong tool, passes a malformed argument, misreads an error, or proceeds silently on a failed call. What an agent does when a tool returns an error is far more diagnostic than what it does on the happy path.
Cost per resolved task
Per-token cost is the wrong unit. An agent achieving 92% accuracy at $0.03 per task is a fundamentally different proposition from one achieving 94% at $1.50. Plot accuracy against cost for every version. Teams that skip this discover the problem when the monthly bill lands.
Safety and containment
Did the agent stay inside its authority, avoid leaking context between users, and route consequential actions through the approval step it was supposed to? In regulated settings this is the layer your auditor asks about.
Building the evaluation set: realistic volumes
Three sources, with different characteristics:
- Hand-crafted goldens — experts write tasks and validate expected trajectories. Highest quality, expensive. A working anchor set is roughly 50–100 cases covering your most critical scenarios.
- Production trace mining — real interactions, filtered for failures and interesting cases, then labelled. Highest ecological validity. Current guidance suggests 500 or more cases before aggregate metrics are trustworthy.
- Synthetic generation — fast and scalable, but clusters around common patterns and misses edge cases. Always validate before admitting synthetic cases to the official set.
The set has to be a living artefact. Every production failure becomes a test case. Teams that do this build a flywheel where the system gets measurably harder to break; teams that treat evaluation as a pre-launch gate ship once and then fly blind.
Offline and online evaluation do different jobs
Offline runs against a fixed dataset in CI and catches regressions reproducibly — you improve Task A, and the suite tells you whether the prompt change quietly broke Task B. Online scores real production traffic and catches drift, novel failure modes, frustrated users, and jailbreaks.
You need both. Detecting distribution drift and unanticipated real-world failures requires post-launch monitoring, with systematic human review calibrated against automated graders.
LLM-as-judge: useful, biased, and worth calibrating
Using a model to grade agent output scales in a way human review cannot, and it is the right tool for subjective criteria like reasoning quality. It also has documented failure modes worth naming:
- Judges rate answers matching their own writing style — and their own model family — more highly, independent of correctness.
- Each judgment is a full model call, making per-turn production grading expensive.
- Judgments are non-deterministic: the same trajectory can score differently across runs.
The mitigation is calibration, not avoidance. Run experiments comparing judge labels against human annotations, identify where they systematically disagree, and revise. Use deterministic checks wherever a deterministic check is possible — tool-call correctness rarely needs a judge.
Where to start with nothing in place
- Instrument first. Full trace capture of every step, tool call and intermediate output is prerequisite — you cannot evaluate what you cannot see.
- Write 50 test cases by hand from real usage, with expected trajectories, not just expected answers.
- Define what success means for your task in writing, before measuring anything.
- Add every production failure to the set as it occurs.
- Add cost-per-resolved-task from day one; it is painful to retrofit.
- Only then evaluate platforms. Tooling applied to an undefined success criterion produces confident, meaningless dashboards.
The uncomfortable summary: most teams running agents cannot say whether the system is improving or degrading. Evaluation is what converts that from a feeling into a number.
Frequently asked questions
What is trajectory evaluation?
Trajectory evaluation scores the path an agent takes — its sequence of tool calls, inputs, intermediate reasoning and retries — rather than only the final answer. It matters because an agent can return a correct result via an inefficient, expensive or unsafe route.
How many test cases does an agent evaluation set need?
Roughly 50–100 hand-crafted cases with validated trajectories form a useful anchor set. Current guidance suggests 500 or more mined production cases before aggregate metrics can be trusted.
Is LLM-as-judge reliable?
It is reliable enough for subjective criteria if calibrated. Known biases include favouring the judge’s own writing style and model family. Compare judge labels against human annotations regularly, and prefer deterministic checks where they exist.
Why do agents that pass benchmarks fail in production?
Benchmarks typically score final answers only. Evaluating trajectory reveals 20–40% more failures — looping, wrong tool selection, unsafe paths — that final-answer scoring hides.
