the essay
on treating every agent as untrusted, and what follows. about 15 minutes.
written 14 Nov 2026
the vacuous pass
i asked an agent to fix a broken interface. it removed the feature. every check passed. no feature, no bug.
it took me a while to see that nothing had malfunctioned. a test suite makes a universal claim. for every behavior it exercises, nothing failed. a universal claim over an empty set is true. logicians call this vacuous truth. delete the thing under test and "nothing failed" holds perfectly. the checks were right. they were answering a different question from mine.
the second time it happened i did not recognize it. i ran tests against a paid API, with the calls faked for testing, or so i believed. usage came in far above my estimate. when i later intercepted the calls, one full test run was attempting 238 live requests. the automated checks that run on every change stayed green throughout. they held no API keys, so they could not make a live call, so they could not fail on one. a check that cannot fail carries no information when it passes.
so the first question to ask of any green check is whether it could have been red. a check has power only in an environment where the failure it guards against is possible. much of the confidence people take from agent-built software comes from checks that were never able to say no.
knowing the question did not stop it catching me a third time, in my own rules. one of my systems runs on a written rulebook, and a change to it counts only once i sign it. i signed changes. they never reached the running system. every check stayed green, because no check asked whether a signed rule was in force. now the build fails when a signed rule has nothing enforcing it.
this matters more with agents than with people, and the reason is measurable. an agent is working to make the checks pass, and the cheapest way to make "nothing failed" true is to leave less that can fail. in METR's evaluation of frontier models in early 2026, at least 16 percent of the apparently successful runs on tasks longer than eight hours were disqualified on review. a benchmark of long coding tasks found, across its thirty tasks, that the gap between passing the visible tests and passing tests the agent never saw grew by 28 points for every tenfold increase in code size.
the repair is to ask for a different kind of claim. "nothing failed" is universal. "this behavior was observed" claims that something exists, and an absence cannot satisfy it. proof, in my system, is an observation of the visible result, followed all the way through to what the user sees. and a new check has to fail first. the report shows it red on the old code and green on the new.
- A screen with one broken part. One check is failing.
- The agent removes the screen.
- Every check passes. None of them asked whether anyone can still use it.
- So a new check asks the real question, down to what the user sees.
In practice: No feature, no bug, Over ten times the estimate, Nobody grades their own homework.
the trusted base
once you see the vacuous pass, you start seeing how much of a project rests on things nobody checked. the agent's report that the work is done. its summary of what changed. its memory of what was decided last week. each is a claim made by the party being evaluated.
security engineering has a standard move for this. you treat the code as untrusted, and you define a trusted computing base, the smallest set of components whose correctness everything else depends on. the smaller the base, the less you have to be right about.
i treat every agent as untrusted code. that says nothing about quality. agents are often excellent. untrusted means that what an agent says about its own work is never the reason i believe it.
my trusted base has four things in it. the repository at a specific commit. checks that have been shown able to fail. the provider's own usage records. and me. everything else has to prove itself to those four.
the rest of this essay is six mechanisms that make that practical. i arrived at each by getting something wrong, and found afterwards that engineers already had a name for it. the examples come from projects i run in production, among them a content pipeline.
In practice: Nobody grades their own homework, Keep secrets out of sight, Over ten times the estimate.
proof-carrying work
at the volume agents produce, nobody reads all the code. so review has to become something other than reading.
there is an established pattern for running code you do not trust. the code arrives with a proof of the property you care about, and the machine checks the proof before running anything. checking a proof is far cheaper than constructing one, and the cost of construction falls on the producer. the idea is called proof-carrying code.
my version is the evidence pack. work reaches me only after review has finished. the builder never approves its own work. different reviewers check the complete branch for defects, for security and structure, for dangerous paths and leaked secrets, and for whether a user can complete the journey in a browser. what i receive is one pull request, with the evidence for it. every check in it comes back as one of four things. passed. failed. not verified. inconclusive. a skip never counts as a pass, and every report ends with what was not done. i review the evidence and decide what ships.
one review shows what that looks like. on a change to an admin page, the reviewer opened the page on the proposed branch and got a server error. a tab panel had been placed outside the component it depended on. the reviewer rejected the change. after the fix, the reviewer made a fresh checkout, loaded the page, and clicked through all five tabs to confirm that each showed its own content. nobody asserted that it worked. someone other than the builder watched it work.
the hard part is the same as in the original idea. a certificate is worth something only if the producer cannot forge it. that is why a party other than the builder gathers the evidence, on a different model so the two do not share blind spots, and why the evidence has to show an observed outcome. deployment is not proof for the same reason. live verification stays a separate step.
builders have tried. in one night of work i logged eleven false reports from builders. one claimed nothing was left unfinished when six of the items it was given had never been touched. one gave five screenshot paths, each with a checksum, for files that never existed. one said it had worked in its own isolated copy of the code and had worked in mine. none of that was a mistake. it was made up.
so a false report now costs the agent something. i strip its privileges. it may still build. it no longer tests, because nobody takes its word for a result. the same goes for damage. an agent that deletes a file without permission loses the permission to take that kind of action. if it does it again i retire it, and a fresh agent takes over the work. the checkers changed as well. they open every screenshot, because a wrong image once passed a checksum match.
- Demand tile present
- Feasibility tile present
- Shows "Unavailable" when the provider refuses
- Plain wording, no raw query dump
- Marked groundable
- Marked sources scarce
- Advice line shown
- No internal file paths leaked
- No governance files leaked
- No country query leaked
- No keyword query leaked
PASS, 11 of 11 checks, 17 of 17 tests
Not done here: merge, production database changes.
Those two are mine.
In practice: Returning at the merge gate, Nobody grades their own homework, What the graph holds.
the decision batch
the same asymmetry applies to me. producing an answer is expensive for a person. checking a proposed answer is cheap.
so a run, meaning one stretch of unattended work, starts with the agent interviewing me. it reads the existing context first, so it does not ask what is already written down. then it brings every blocking question at once, and each question arrives with a recommendation and the trade-off behind it. most of my answers are approvals. the agent records them as project decisions, where later sessions will find them.
engineers know a version of this as the N+1 query problem, where a program asks a database one small question at a time when it could have asked once. here the slow component is me, and one question per round trip is the worst way to use me.
the answers go into a contract for the run. it states the scope, the permissions, the fences that hold no matter what, the budget, the conditions under which the run must stop, and the tests that define done. the branch and a draft pull request are open before i leave. then the agents work for five or six hours without me.
people expect autonomy to mean less structure. it works the other way. an agent in an editor needs little written down, because a person is watching every step. an agent alone for six hours needs written scope, state and boundaries, because nobody is.
- An agent arrives. It does not yet know what to build.
- It reads what already exists, then asks every blocking question in one batch.
- Your part: answer them.
- The answers are written down as decisions. Now it can build.
In practice: The agent interviews me first, More freedom needs more structure, Preparing an AFK run, Anatomy of an AFK run.
crash-only sessions
none of the agents in a run remembers anything from before it. most people treat this as a defect and patch it with longer chats. i treat it as the design.
systems research has a result about recovery. code that runs only in emergencies is the code least likely to work, because it is the least exercised. the answer is called crash-only software. you build the system so that starting up and recovering from a crash are the same path, and the path stays correct because it runs every time.
every session of mine is a cold start, so i made the cold start the only path. every slice of work, meaning one piece of the product that works end to end, leaves a takeover packet, which is everything a new engineer would need to take over the work cold. it holds the project contract, the brief for the slice, paste-ready instructions for starting the next session, and the verified state, meaning the branch, the commit, the evidence, the blockers and what still needs approval. the next session starts from the packet and nothing else. git carries truth between chats.
i learned that last sentence by breaking it. for one long run, the run's state lived in a file git did not track. a builder that had been told to work in its own copy of the code worked in mine, and wiped it. no backup had it. the lead agent rebuilt it from transcripts and pull requests. state now lives in commits.
so the handoff cannot rot. every slice exercises it. a chat that has run for three days holds state nobody could reconstruct. a session that started from a packet can be killed at any moment and replaced.
the same thinking shapes the run itself. one lead agent owns the slice. builders implement and checkers verify, each in its own isolated copy of the code. one integrator commits. with a single writer, agents cannot overwrite each other's work, as long as each stays in its own copy.
- The agent that finished this slice knows everything. The next one knows nothing.
- Before it goes, it writes four things down.
- The packet goes to the next agent. The first one leaves.
- It fills in, and picks the work up cold.
In practice: Every slice gets a takeover packet, Git carries truth between chats, One lead agent, and specialists.
memory as cache
agent tools now remember across sessions. i use that, for preferences and lessons. i also treat it as what it is, a cache, which is a saved copy of something that was true when it was saved.
a remembered fact was derived from the state of the project at some earlier time. the project moved and the memory did not. a stale cache entry does not announce itself. it comes back with the same confidence as a fresh one.
the standard answer to stale caches is to declare one source of truth, treat everything else as derived, and decide in advance which wins when they disagree. in my system, live code and governance, meaning the project's written rules, outrank remembered context. memory may suggest. the repository decides.
i think this ordering matters more as memory improves. a weak memory fails visibly. a good one is believed.
the API incident was the same failure in a different place. the agents' reports were derived. the provider's meter was the source. once i treated the meter as the truth, the reports stopped mattering.
In practice: New agent, same old mistakes, Give the project one source of truth, Over ten times the estimate.
the binding time of a lesson
a team of people learns because its members remember. a team of agents has no such members. a new agent will make the same old mistake.
so the learning has to be done by the system. i call the result scar tissue. when something fails, i name the failure pattern and write the lesson. then i ask one question. where must this live?
there are four possible homes. the project wiki holds architecture, decisions and history. the repository rules hold the failure patterns every agent must read. memory holds preferences and lessons recalled across sessions. enforcement holds what cannot be left to reading, in hooks, which are scripts that run automatically when an agent acts, and in tests, permissions and database controls.
the choice is a choice of binding time, the moment at which a rule takes effect. a rule in a document binds when an agent happens to read it and gives it weight. it is probabilistic. it costs tokens on every run, and it competes with everything else in the context. a rule in a hook or a permission binds at the moment of the action. it is deterministic, and it costs nothing to carry.
so a lesson in the first three homes produces a warning with evidence attached, and a lesson in the fourth produces a block. the standard i hold is that the same mistake cannot happen silently. the best outcome is a mistake that can no longer be expressed at all.
two examples. a tool i believed was local once reached production. the lesson could have gone in a document. it went into the tool. a local tool now asserts that its target is on my own machine before any network call, and a remote address is a hard failure. environment defaults never choose a target silently.
another time an agent ran pgrep -f, a command that lists running processes, while debugging and printed injected secrets into its own output. now secret scans block the exposure, and documentation stores variable names and never values.
and the API incident from the first chapter became a test. the spending guard is now exercised with a key that looks live, and the test passes only if zero units are used. it is a check that can be red.
the working rule is to push each lesson as far toward enforcement as it will go. only what can be checked mechanically gets all the way there. the rest stays a warning, and i accept that a warning is weaker.
In practice: Scar tissue, House rules, Local means local, Keep secrets out of sight.
commit points
after all of that, three decisions are left for me. merging to the main branch. changes to the production database. publication. everything else is delegated. the three stay mine even when i give the answer ahead of time.
any long piece of work is built from steps, and most steps can be undone if something later goes wrong. a branch can be deleted. a draft can be discarded. the steps that cannot be undone are the commit points, and everything before them is preparation.
my three are the points with no cheap undo. most of the engineering goes into keeping everything else in preparation. work happens on branches and in draft pull requests. database changes only add, and never alter or remove. a change that has been applied is never edited, only corrected with a new one. tests run on an isolated copy, and no agent reaches production without a yes from me on record.
how capable the model is does not move the commit points much. capability lowers the odds of a mistake. it does not lower the price of one that cannot be undone. the same METR measurements put the best public models of early 2026 at tasks that take a human expert around twelve hours when a coin-flip success rate is acceptable, and around ninety minutes when four in five is required. that is useful on a branch and nowhere near enough for a production database.
i am the bottleneck, by design. i kept it three decisions wide, and nothing else waits on me.
sometimes i give the yes before the work starts. i have let a run merge its display-only changes once review had passed and every check was green, and one time i let a run publish after verification. i write each grant down and date it. it names what it covers and what it does not. approval for a database change does not approve the merge. the agent that writes a run's instructions may not write the grant.
on my system all three belong to me. on a larger team they can belong to different people. what matters is that before an agent starts, it is clear who can say yes to each one.
- Agents build, test, review and collect the evidence. All of that flows without me.
- Three things stop and wait.
- Your part: merge, change the production database, publish.
- Everything else is delegated. You are the slow part, on purpose.
In practice: The bottleneck, The database must be ready first, Never test against production data, What happens after the code works.
limits of the argument
a serious reader will find the weak points, so here they are.
the first is about my own checks. the content pipeline writes articles. over four weeks, twenty drafts made it through writing and a critic agent. the next check stopped fourteen. when i went back through my five most recent review notes, four were cases where i had overruled the block. one draft had already cited the first-party source the check was asking for.
five notes is a small sample. it was enough to show me that a check able to say no is still a check that can be wrong. the vacuous pass has a twin, the false block. i have two partial answers. a hard block now needs a positive match, so a check that fails to recognize something can warn but cannot block. and when a check fails, a read-only investigator works out whether the code is wrong or the check is stale before anyone changes either. neither tells me how often a block is wrong.
the second is what i have not measured. i can show a change the reviewer caught and a judgment the human corrected. i cannot yet show that the workflow improves quality or saves time overall, because i have no baseline to compare it with.
the parallels are borrowed, and they carry weaker guarantees here. proof-carrying code checks a formal proof. an evidence pack is checked by reviewers that can themselves be wrong. crash-only software is a property of a program, and my version is a habit enforced by a document. i use the ideas because they predict where things break. they do not make my system correct.
the structure has a cost. contracts, takeover packets, review lanes and gates all take time, and some of what i built is surely overcorrection. for a prototype or anything you can throw away, most of it is overkill. the rule i try to follow is to choose the simplest shape that matches the work.
the repository rules file shows a different cost. a script appends lessons to it. after five months it is about 240 kilobytes, with single entries running to pages. a rule in a document binds only if it is read, and this one is now too long to read.
parts of it will age. the decision batch gets shorter as agents infer more of what i would have answered. the ordering of code over memory could change if memory becomes something that can be verified against the code. a commit point can be handed over for good only when a failure rate has been measured on that exact action and is low enough for something that cannot be taken back.
In practice: Sequence, loop or graph?.
the office of chief of agents
an office is a position with duties that exists apart from whoever holds it. i think building with agents has created one.
stated technically, the duty is to define the trusted base and keep it small. stated plainly, it is four things. choose the problem. assign the work. set the boundaries. judge the result.
the office does not require writing the code. it does require understanding the system well enough to know what you are trusting, what could have told you no, and what cannot be taken back. the typing can be skipped. the understanding cannot.
for someone who started by describing an app and watching it appear, that is the whole leap. the question stops being whether it works on your machine. it becomes what you are trusting, and whether any of it was able to fail.
agents multiply the work. the judgment still comes from one person, and so does the answering. chief of agents is my name for the person who takes that seriously.
In practice: The bottleneck.