An AI agent with write access to production
An AI agent that performs a dangerous production data operation — replaying millions of historical customer records through Intuit's marketing pipeline — which previously required a human specialist to babysit for one to three days. It runs unattended for days, survives its own server being destroyed and replaced mid-run, coordinates three teams over Slack and JIRA, writes to production through an audited approval process, and independently verifies the data arrived correctly at every destination.
The central engineering problem was not making it work — it was making it fail safely.
Customer data flows from a source system through a pipeline to three destinations: a data warehouse, Adobe Experience Platform, and Marketo. When something lands wrong downstream, the fix is a replay — re-publish the historical records so they flow through again.
A replay is not read-only, and that is the entire problem. To run one you must temporarily lower the priority of live production data in the production database, wait hours or days, then raise it back. Forget that last step, or die partway through, and production stays misconfigured indefinitely while live customer data keeps flowing at the wrong priority. Nobody gets an alert, because nothing crashed.
By hand this was one to three days of intermittent attention: coordinate three teams over chat, hand-edit a production database, watch dashboards, spot-check two destinations, and remember to undo the change. The correctness of the whole thing rested on someone not forgetting.
The agent runs on Kubernetes, which destroys and recreates its server routinely — a deployment, a machine failure, a memory limit. A replay takes days. So the agent will be killed mid-run, repeatedly, as a matter of course.
It is a 22-state machine whose complete state is written to durable storage after every step. The correctness argument is stronger than "we save progress": all seventeen functions that decide what happens next are pure functions of saved state. None reads the clock, the network, a random source or a module global. Resuming is therefore not a reconstruction that might disagree with what the system was doing — it is re-running the same calculation on the same inputs.
The component that nudges the workflow forward deliberately holds no state at all, and re-reads everything from durable storage every cycle. During a Kubernetes rollout the old and new servers run simultaneously for a few seconds and both drive the same replay. In most designs that is a serious double-processing bug; here two drivers produce the same result as one, by construction.
One failure durable state did not solve on its own: the agent authenticates with a ticket that expires in ten minutes, while a replay runs for days. Durable state answers what to do next. It does not answer what credentials to do it with.
If a replay lowers production priority and dies before restoring it, production is misconfigured. If a second replay then starts on overlapping data, it reads the current — already downgraded — value as "the original", and on finishing restores to that wrong baseline. The true original values now exist nowhere. No error, no alert.
So a replay that ends while still owing a restore does not release the global lock, and every subsequent replay is blocked until a human resolves it. The decisive detail is what happens when the system cannot tell: an unreadable registry means "cannot tell", which is treated as "obligation outstanding", and the lock is held.
That is a deliberate choice of a loud, visible, annoying failure over a silent, permanent, invisible one. The same reasoning appears wherever a stuck lock could be auto-cleared: silently releasing it would trade a safe stuck lock for a quiet, unsafe "production stays downgraded and nobody knows".
The agent uses an LLM for several things. It deliberately does not use one to decide whether a data-verification report passed. That verdict already exists as structured numbers; asking a model to re-read prose saying "passed" would put a production replay one hallucination away. The decision is four plain boolean comparisons.
Where the model is used, it writes the human-readable wording of gate notifications — it never chooses an audience, and never chooses a control-flow outcome. If it fails entirely the system falls back to a deterministic template, so a gate can never be blocked by the model.
And that confinement still nearly caused an incident. An LLM rewrote a gate notification and silently deleted two facts from it: a scope-mismatch warning and a pull request URL — from the one message that would have stopped a bad merge. The paraphrase was fluent, plausible and non-empty, so it passed every naive quality check. It had simply removed the single fact the message existed to convey.
The lesson generalises past this system: confining a model to "just the wording" is not sufficient protection when the wording is the interface to a human decision. A paraphrase that drops the warning does not make a wrong decision itself — it removes the evidence a human needed to make the right one. The fix pins the load-bearing facts and falls back to the template if any is missing, and the pins are derived from the message rather than hardcoded, so a later edit to the wording cannot leave the guard pinned to a string that no longer appears. A guard that silently stops guarding is worse than no guard.
Three approval gates were automated, and no gate was converted away from being a human gate. All of them remain human-approvable; the reject button still works everywhere.
The obvious implementation — convert the phase to an automatic one — would have removed the ability to reject at that gate, because the rejection path checks that the phase is human-approvable. The agent would have gained the power to approve while simultaneously stripping humans of the power to refuse. The asymmetry matters: the "no" is the decision carrying real weight, because rejecting terminates the run through the cleanup path that restores production.
Instead the agent supplies its decision through the same channel a human would use. Rollback becomes a config flag rather than a code change; automated and manual paths share one implementation; the audit trail, timeout semantics and the operator's mental model all survive. Two of the three automations ship disabled by default, and deliberately so — two switches that look symmetric have wildly different blast radii. Releasing one gate leads onward to a production replay; a mistake on the other leaves a priority un-restored, which is recoverable and visible.
Verification compares values record by record between the source and each destination. Destinations lag twenty to thirty minutes, so a missing record might be lost or merely late; formats differ, so byte-exact comparison produces false alarms.
The subtle problem is scope. If a replay was only ever configured to send to one destination, checking both and finding nothing at the second reports "both destinations checked, all clean" — a sentence indistinguishable from genuine full success, and from total delivery failure. A human reviewer can usually infer which from context. An automated approver cannot. Scope awareness had to exist before auto-approval was safe, and an unknown scope is never widened to "all destinations".
The same principle recurs in at least six independent places under different names. A payload containing none of the checked fields is an extraction failure, not a match. An attribute in no destination's field map is an error. A rule that does not cover this attribute is excluded from the tally entirely. Even the test double was changed, because a fake that fabricates "match" for an out-of-scope destination would let a test assert exactly the silent partial pass the work removed — a safety property can leak out through a test double, and few engineers think about that.
The write-up this page draws on carries its own section on weaknesses, which is the part most worth knowing about. A verdict state meaning "could not be decided" is fully plumbed, counted and blocked on, but no code path currently produces it. A timeout gap is documented and owned rather than silently relied upon — but shipped, not fixed. A single global lock permits one replay at a time organisation-wide and is held through 24–48 hours of human gates: a conscious trade that fails in the safe direction, and a real throughput ceiling. And the staged rollout of two automations has telemetry but no exit criteria — no threshold, no owner, no date.
He is equally direct about what is ordinary: durable workflows with checkpointing and human-in-the-loop interrupts are what the underlying framework provides. This is competent use of a durable execution engine, not the invention of one.