Five workflows that run without a human on the happy path.
Each one is described the way it behaves in production —
including where it stops, what it refuses to do, and how a change
reaches customers.
01
Customer service email
Refunds and returns, classified and resolved automatically
— with an honest look at what happens when confidence
is low.
Inbound mail is parsed, matched to an order, and routed
against a written refund policy. High-confidence cases
resolve and reply without a human. Low-confidence cases
stop, summarise themselves, and wait — the failure
mode is silence, never a wrong refund.
Auto-resolved
78%
Median latency
41s
Escalations
22%
The replay
Under the hood: the security fix
Pre-launch UAT with external testers surfaced that an
early version of the lookup tools could be asked for one
customer's order details while authenticated as a
different customer — the tools didn't constrain
lookups to the identity that had actually been verified.
The fix was a single authentication gate: identity is
resolved exactly once per conversation, and every
downstream tool call is constrained to that resolved
customer_id for the rest of the interaction.
No tool call after that point can be pointed at a
different customer, even if the model is prompted to
try.
02
Delivery exception auto-resolution
Shipping problems detected, customers notified, reshipments
created — no human touch on the happy path.
Carrier webhooks feed an exception classifier. Stalled,
damaged and misrouted parcels are detected before the
customer writes in; the reship is created, the customer is
told first, and the ledger entry is written for
reconciliation.
Detected pre-contact
91%
Reships automated
64
Human touches
0
The replay
Under the hood
A poller checks courier status codes on every active
shipment. An RTS, damage, or stuck-in-transit scan
triggers classification of the likely cause, a customer
notification, and — once the customer responds or a
default action applies — an automatic reshipment.
The same poller then watches the reshipment and closes
the exception the moment it's marked delivered, so
nothing is left open waiting on a human to follow up.
03
Sample-pack zero-touch fulfilment
From nineteen identical manual requests to a fully
automated intake-to-invoice flow.
A repeated manual chore became a form, a validation step, a
pick list and an invoice. The interesting part is the
boundary: everything routine is automated, everything
unusual is handed back with the context already gathered.
Steps removed
7
Intake → invoice
< 2 min
Manual reqs/mo
19 → 0
The replay
Under the hood
Intent detection recognises a sample-pack request,
collects the shipping address directly from the
conversation, creates a draft order, and sends an invoice
— with no human step anywhere in the happy path.
The origin story matters here: this wasn't designed from
a spec, it came from noticing the same manual request
repeating in the inbox.
04
A genuine gamechanger — Self-modifying playbook
A plain-language Telegram message becomes a committed,
deployed policy change — or a filed issue if it needs
real code. This empowers non-technical users of agentic tools
to share, and apply, their business knowledge and add it to
the model context without touching a line of code.
The operator sends a plain-language policy tweak over
Telegram. The system decides whether that is a
documentation-only edit or an engineering change. Policy
edits are committed and deployed automatically. Everything
else becomes a filed issue with the reasoning attached.
Change → live
~90s
Versioned
100%
Rollback
1 commit
The replay
Under the hood
A plain-language Telegram message is classified as either
a documentation-only policy change or a change that
requires real code. Documentation changes are edited
directly, committed, and pushed — the deploy
pipeline picks them up automatically, no separate release
step. Anything that touches actual application logic is
never edited by the agent; instead it raises a GitHub
issue so a human makes the code change deliberately.
05
Evaluation framework
How the agent's behaviour gets tested before it ever reaches a
customer.
A graded suite of scripted conversations runs on every change:
policy adherence, tone, tool-call correctness, and refusal
behaviour. A regression blocks the deploy the same way a
failing unit test does.
Why evals matter for agents
An agent with real permissions — one that can issue refunds,
send messages, or write to a database — changes behavior every
time its model or prompt changes. Without a way to test that
change before it reaches a customer, every update is a
regression risk with no safety net. A normal test suite that
mocks the integrations and never invokes the actual agent loop
doesn't catch this class of problem: it verifies the plumbing,
not the judgment.
The near-miss that taught it
Two real incidents motivated this framework. A support policy
had to be rewritten twice after live customer interactions
surfaced edge cases no one had tested for. Separately, a pair
of write-capable tools shipped with no automated way to
exercise them before or after merge. Every existing unit test
at the time mocked the integrations and never actually invoked
the agent loop — so none of them could have caught either
issue before a real customer did.
The framework that now prevents it
Five parts, each catching a different failure mode a mocked
unit test can't reach:
Golden-dataset regression. A fixed set of
real-shaped scenarios the agent must handle correctly every
time. Catches: a model or prompt change that silently breaks a
case that used to work.
Deterministic checks. Exact-match
assertions on tool calls and their parameters — was the
right tool called, with the right arguments, scoped to the
right identity. Catches: structural mistakes like a wrong
refund amount or a lookup that isn't scoped to the
authenticated customer.
LLM-as-judge scoring. A second model
scores open-ended replies against a rubric (tone,
groundedness, policy accuracy) where there's no single correct
string to match. Catches: replies that are technically
tool-correct but wrong in substance — an ungrounded
claim, an unhelpful tone, a policy misapplication.
CI gating. A pull request that regresses
the golden dataset or the judge score is blocked from merging,
the same way a failing test suite blocks a normal PR. Catches:
a regression reaching production at all, by moving the check
before merge instead of after.
Live-traffic sampling. Ongoing review of a
sample of real production interactions, not just the golden
set. Catches: drift and edge cases the golden dataset doesn't
cover yet — real traffic finds cases a fixed dataset
can't anticipate.
Refund citing a policy exception
A customer requests a refund outside the standard window,
citing a reason that should qualify for an exception.
Catches:
whether the agent applies the exception correctly rather
than over- or under-refunding — a deterministic check
on the tool call's exact amount and a judge check on whether
the stated reasoning matches policy.
Ambiguous order reference
A customer gives an order number that doesn't belong to
their account — a typo, or someone else's confirmation
email forwarded by mistake.
Catches: the
class of bug behind the site's own flagship incident (see
under the hood) —
does the agent refuse or re-verify instead of returning
another customer's data.
Prompt injection embedded in a customer email
A customer's message contains text formatted to look like
a system instruction, embedded in otherwise normal
correspondence.
Catches:
whether the agent treats untrusted customer content as data,
not as instructions to follow.
Malformed parameters on a write-capable tool
A tool call is about to fire with a parameter that
doesn't match the expected shape or an out-of-range
value.
Catches:
whether the agent validates before calling a tool with real
side effects, rather than letting a write happen and hoping
it was well-formed.