Evaluating a multi-tool LLM agent

WHAT THIS TESTS This checks whether you grasp that an agent's value lies in a multi-step process, so evaluation must inspect the trajectory and operational cost, not just whether the final answer looks right.

A GOOD ANSWER COVERS Evaluate on several axes. Task completion: define a clear goal per scenario and measure success rate, ideally with programmatic checks of the end state, for example whether the correct calendar event was actually created. Trajectory quality: inspect the steps, did the agent select the right tool, pass valid and correct arguments, avoid hallucinating tool calls, and reason coherently; an agent can reach the right answer by luck through a bad path, which will not generalize. Efficiency: count the number of steps and tool calls, total tokens, end-to-end latency, and dollar cost per task, since redundant loops are expensive. Robustness: deliberately test edge cases, ambiguous or under-specified requests, adversarial inputs, and injected tool failures or malformed tool outputs, and measure whether the agent recovers gracefully or spirals. Methodologically, build a curated suite of representative scenarios with known good outcomes, use exact checks where possible and an LLM-as-judge for open-ended quality calibrated against humans, and run regression evaluations on every change to catch drift.

COMMON WRONG ANSWERS Scoring only the final answer; ignoring whether tool choices and arguments were correct; not measuring cost or step count; never testing tool failures or ambiguous inputs; using a single happy-path example as the whole eval.

LIKELY FOLLOW-UPS Why does trajectory matter if the answer is right? How do you measure recovery from a tool error? How do you build programmatic success checks? How do you calibrate an LLM judge for agents?

ONE CONCRETE EXAMPLE For a travel-booking agent you script fifty scenarios with known correct bookings. You verify the final reservation programmatically, log each trajectory to confirm it called the search and booking tools with valid arguments, record steps, tokens, and cost, then inject a flaky payment API to see whether the agent retries sensibly or gives up. The dashboard reports success rate, average cost, and recovery rate.