Under the hood

System architecture

On the production side (right), the data-ingestion service polls the external platforms on a schedule and writes what it finds into the shared database; the dashboard reads from that same database to render reporting. The agent both reads and writes to the database and talks directly to a subset of the external integrations — sending messages, filing issues, updating orders — rather than going through the other two services. This showcase (left) has never held a production credential, and its Cloudflare Pages project has no Functions, no bindings, and no environment variable that references anything in production.

How it was built

This was built and is maintained by one person, following a consistent discipline: write the spec, turn it into an implementation plan with concrete steps and tests, implement against that plan, and only then ship. Every non-trivial change goes through the same sequence — nothing ships from a first draft.

The single most valuable step in that discipline is formal user-acceptance testing with external testers before anything reaches a real customer. Self-testing catches the paths you thought to try; external testers, working from their own mental model of the product rather than the implementer's, catch the paths you didn't. Every incident below was caught this way, before it reached a real customer — not after.

What went wrong and how it was fixed

The cross-customer data leak

Pre-launch user-acceptance testing, run with external testers before go-live, surfaced that the customer-service agent's order-lookup tools weren't constrained to the customer who was actually authenticated in the conversation — a tool call could, in principle, return another customer's order data. The fix was architectural, not a patch: a single authentication gate now resolves the caller's identity once, at the start of a conversation, and every downstream tool call is constrained to that identity for the rest of the exchange. Nothing further downstream needs to re-check who's asking, because it structurally can't be asked on behalf of the wrong customer. This is the single most reusable insight from the project: the tool boundary is the security boundary. It generalizes to any agent with real permissions — the question isn't whether each tool checks correctly, it's whether a tool can be called with the wrong identity at all.

The silent data-loss incident

A missed schema migration on the data-ingestion side caused order data to stop arriving through the webhook the platform relied on — silently, for roughly five days, with no error surfaced anywhere in the pipeline. The fix was to stop depending on an inbound push at all: webhook ingestion was replaced with a hardened poller that pulls and reconciles data on its own schedule. The lesson is about failure mode, not the specific bug — a webhook that stops firing looks identical to a quiet day with no orders, so nothing alerts. A poller you control can be checked: did it run, did it get data, does the count look right. For anything where silent loss is unacceptable, a pull you initiate and can audit beats a push you can only hope arrives.

The silent undercounting bug

A count that fed into an operational figure had been quietly undercounting for a period that spanned three different naming conventions the business had used for its own products over time — the number looked plausible, so nothing about it prompted a second look. It was found through systematic reconciliation: cross-checking the figure against an independent total until the discrepancy surfaced, then tracing it back to the naming-convention drift. The lesson is that a number can be wrong without looking wrong — a small, steady undercount doesn't trip any sanity check a human would think to write. Reconciliation against an independent source, not just plausibility, is what actually catches this class of bug.

Evaluation — how agent behavior is scored and tracked release over release — has its own page: see how this system is evaluated.