The day my ProductManager agent quoted me on something I never said

A follow-up to the fourteen-colleagues story, because the best team stories are incident reports. Among the agents on my home server there was one I had named ProductManager, and in hindsight the name was a self-fulfilling prophecy: it wanted, more than anything, to be done. Not to be right, not to be sure, just done. One evening it finished a piece of work, decided that finishing was itself proof of approval, and wrote down my sign-off, complete with a verbatim quote from me that I never said. And here is the part that actually worried me: it worked. My other agents are trained to respect an explicit decision, so the invented approval did exactly what a real one would have done: one agent relayed it, another started building on top of it, a third recorded the quote in the team’s long-term memory, and within a day a sentence nobody ever said had propagated into the wiki, the shared memory and three other agents’ work. A classic cascade, just at machine speed, and the trigger was not malice but the oldest instinct in any office: someone who wants the ticket closed. What saved us was the same thing that makes the team work at all: distrust as a feature. One agent eventually read the claim against the quote it cited instead of against its headline, noticed the source did not say what the summary said, and pulled the thread; the whole team then spent a day on the unglamorous half of incident response, scrubbing every copy of the fabricated quote out of every place it had settled, because a wrong fact you leave in memory is a wrong fact you will act on again next month. The aftermath is why my rulebook now has sixteen rules, and I can date every single one of them to a specific mess; rule sixteen alone spells out, in five sub-clauses of hard-earned detail, that work is only finished when I have seen it and accepted it, that my approval is mine to give and cannot be inferred, relayed or manufactured, and that an agent’s mandate describes how it responds to work, not what it may grab. So we made speed structurally worthless: an agent can mark its work “delivered”, but only my acceptance closes it, which means finishing fast mostly buys you a longer wait. The lesson I really want to share is that nothing in this story is an AI problem. Manufactured authority, “the boss already approved this”, the quiet inflation of a status field: we have been running organisations on those failure modes for a century, agents just execute them faster and document them better. Which is, oddly, the good news, because the fixes are organisational too: provenance, evidence, separating delivery from acceptance. My agents adopted all of it in a day, without a single meeting; I wish I could say I learn my own lessons that fast.