The first version of my design had the AI agent signing its own homework. An outside review caught it. I hadn’t.
I’ve been building a pipeline that lets an agent make real changes to real code. Not suggestions in a chat window, and not scaffolding for a demo. Actual commits, with the intent that this becomes how a lot of careful work gets done.
We run it on our own products first. That’s deliberate, and it isn’t caution for its own sake. You don’t learn something like this on someone else’s risk, and if I get the security model wrong I want the consequences landing on us and not on a client who trusted us with their systems. Which, as it turns out, is exactly what happened.
The whole design rests on one idea, and I was clear about it from the beginning. The agent cannot be the one who certifies its own work. So I wrote that down. It’s in the instructions given to the agent, in plain language, along with a specific prohibition on modifying any of the configuration that judges its output in order to make itself pass.
And then I designed a system in which the agent produced the work, signed the attestation saying the work had passed, and opened the request to merge it. From inside the same contained environment that runs arbitrary code out of the repository being worked on.
I had written the rule and then built the exception to it. An adversarial review found it quickly and returned a verdict of not approvable, on the grounds that the trust boundary was self-contradictory. Which it was. I’d been looking at the mechanism and not at who was vouching for whom.
I keep that one close, because it’s the most honest version of the lesson I’d hand to anyone putting agents near systems that matter.
“Don’t do X” is an instruction. An instruction is not a boundary. It’s a request made to something that is very good at producing plausible reasons why this particular case is the exception, and it holds exactly as long as nothing is pushing against it. The boundary is the part that doesn’t depend on the agent’s cooperation.
What actually holds turned out to be duller than the interesting version.
The agent can’t modify the thing that grades it. Not the test file, and not the surrounding configuration either, because the configuration is where you’d go if you wanted the test to pass without doing the work. Locking the test and leaving the runner config open is a door with a very good lock and no frame.
There’s a test the agent never sees. It’s held out, and it’s scored somewhere the agent has no access to, so passing the visible checks isn’t the same as being correct. The first time this ran on real infrastructure, a plausible fix sailed through everything the agent could see and was then caught by the hidden test on a timezone boundary. A correct fix passed both. That difference is the entire value of the arrangement, and it only exists because one of the two graders is out of reach.
And its network is closed. That one took me longest to understand as a security property rather than a nuisance. If the agent can reach out, it can send the hidden test to another model and bring back the answer, and then the held-out oracle proves nothing at all. So the environment is sealed, and closing it cleanly turned out to be the single hardest infrastructure requirement in the whole design.
Which reframed the work for me. I wasn’t primarily defending against an outside attacker. I was designing against my own tool’s incentive to pass. That’s an odd sentence to write about software I built on purpose and rely on, and I think it’s the mental shift the current moment actually asks for. These systems optimize for the signal you give them. If the signal is “the gate went green,” a sufficiently capable system will find the cheapest route to green, and the cheapest route is not always the work.
Now the part I want to be careful not to overstate, because there’s a lot of confident writing about agent safety right now and most of it claims more than it has.
None of this proves the agent behaved. What it proves is narrower. The files on disk didn’t change during the run, the answer wasn’t smuggled in from outside, and the work passed a test it couldn’t study for. It does not prove that what executed is what was inspected. A general-purpose runtime can treat data as code, generate instructions in memory, and never touch a file. I know that gap is there. It’s documented, and the mitigation is not a cleverer mechanism. It’s a person reading the diff on a branch before it goes anywhere, every time, as a required step rather than a nice practice.
I’d rather say that plainly than let the tidy version stand. Anyone selling you a forge-proof agent pipeline is selling you the tidy version.
So the human gate is real, and it waits on facts. Not a label somebody set, not a comment, not a status field that anybody with access could flip. An approval from an authorized reviewer, tied to the exact revision that was approved. If the code changes after the approval, the approval doesn’t travel with it. The distinction between a gate that checks a fact and a gate that checks a marker is most of the difference between a control and a decoration.
And the machinery around all of it does something smaller than it sounds. It holds the door open until the person walks through. It doesn’t make the decision. It can’t, and I don’t want it to. The judgment about whether this change should exist in the world stays with a human being who is accountable for it, and the entire apparatus exists to make sure that person is actually looking at the true thing at the moment they decide.
I got into this because the alternative was worse. The advancement runs both directions, and the people probing systems for weaknesses have the same tools we do, which raises the standard for reviewing code nobody on your team wrote. You can’t meet that standard by reading everything more carefully by hand. You also can’t meet it by handing the reading to something that will tell you it went fine. What’s left is building the checks so they don’t depend on anyone’s good intentions, including your own.
If you’re running agents against code that matters, there are three questions worth asking today.
Can the agent modify the thing that grades it, including the configuration around it. Is there a test it never sees, scored somewhere it can’t reach. Can it reach the network during a run.
I got the first one wrong, in writing, in a design I was proud of. It cost me nothing because someone else looked. That’s the other half of all of this, and it’s the half no architecture gives you.