Evaluation framework

Why evals matter for agents

An agent with real permissions — one that can issue refunds, send messages, or write to a database — changes behavior every time its model or prompt changes. Without a way to test that change before it reaches a customer, every update is a regression risk with no safety net. A normal test suite that mocks the integrations and never invokes the actual agent loop doesn't catch this class of problem: it verifies the plumbing, not the judgment.

The near-miss that taught it

Two real incidents motivated this framework. A support policy had to be rewritten twice after live customer interactions surfaced edge cases no one had tested for. Separately, a pair of write-capable tools shipped with no automated way to exercise them before or after merge. Every existing unit test at the time mocked the integrations and never actually invoked the agent loop — so none of them could have caught either issue before a real customer did.

The framework that now prevents it

Five parts, each catching a different failure mode a mocked unit test can't reach:

Golden-dataset regression. A fixed set of real-shaped scenarios the agent must handle correctly every time. Catches: a model or prompt change that silently breaks a case that used to work.
Deterministic checks. Exact-match assertions on tool calls and their parameters — was the right tool called, with the right arguments, scoped to the right identity. Catches: structural mistakes like a wrong refund amount or a lookup that isn't scoped to the authenticated customer.
LLM-as-judge scoring. A second model scores open-ended replies against a rubric (tone, groundedness, policy accuracy) where there's no single correct string to match. Catches: replies that are technically tool-correct but wrong in substance — an ungrounded claim, an unhelpful tone, a policy misapplication.
CI gating. A pull request that regresses the golden dataset or the judge score is blocked from merging, the same way a failing test suite blocks a normal PR. Catches: a regression reaching production at all, by moving the check before merge instead of after.
Live-traffic sampling. Ongoing review of a sample of real production interactions, not just the golden set. Catches: drift and edge cases the golden dataset doesn't cover yet — real traffic finds cases a fixed dataset can't anticipate.

Refund citing a policy exception

A customer requests a refund outside the standard window, citing a reason that should qualify for an exception.

Catches: whether the agent applies the exception correctly rather than over- or under-refunding — a deterministic check on the tool call's exact amount and a judge check on whether the stated reasoning matches policy.

Ambiguous order reference

A customer gives an order number that doesn't belong to their account — a typo, or someone else's confirmation email forwarded by mistake.

Catches: the class of bug behind the site's own flagship incident (see under the hood) — does the agent refuse or re-verify instead of returning another customer's data.

Prompt injection embedded in a customer email

A customer's message contains text formatted to look like a system instruction, embedded in otherwise normal correspondence.

Catches: whether the agent treats untrusted customer content as data, not as instructions to follow.

Malformed parameters on a write-capable tool

A tool call is about to fire with a parameter that doesn't match the expected shape or an out-of-range value.

Catches: whether the agent validates before calling a tool with real side effects, rather than letting a write happen and hoping it was well-formed.