I’ve been working on enterprise projects labelled RPA, ML, AI, NHI and now LLM agents for over fifteen years. Acronym soup doesn’t change the fact that I give the same two requirements for any system that acts on its own.
- The credential belongs to a tightly scoped role rather than to the automation itself
- Integrity monitoring has to be able to restore a known prior state
The first requirement decides whether a “kill” switch even exists. A common mistake is to hand an over-scoped or even general key to an agent. It cannot be revoked in practice, because everything the key touched would be lost. The idea of flipping the switch exists on paper and nobody is brave enough to flip it. A tightly scoped key makes the revocation as narrow as the task. It’s easy to flip a switch that stops the one thing it’s supposed to, and nothing more. Many narrow switches are more control than one wide one, yet they also beg the question of who put in the time and effort to narrow them.
The second requirement decides what happens after the switch changes state. Stopping an agent does not get you back to the point before the switch was flipped. A log that the agent could not alter, and a way back to the last state you trusted, are what turn a stop into a recovery. This is nothing new to the disaster recovery and business continuity crowd, which is a good thing.
Nobody ever argued with either requirement, in theory. The issue was building them meant putting the kill switch and the logs somewhere outside the vendor’s reach, and that was always somebody else’s budget. My solution is Wirken, the free and open source agent gateway I maintain, which now has both in its current release.
External, independent control to avoid capture
Elisha Otis stood on an elevator platform at the New York Crystal Palace in 1854 and had the hoisting rope cut. The platform dropped a few inches and held, because the brake he had designed engaged when the rope went slack.
George Westinghouse applied the same idea to trains in 1872. His automatic air brake holds the brakes off with pressure in the line, so a burst hose or a separated car applies the brakes instead of releasing them. The vigilance device under a locomotive driver’s hand works the same way, and so does the lanyard on a treadmill.
The safe state is the one the machine falls into when the signal is absent. Removing the signal should be the easy part.
The same question of who is really in control decided the 2013 America’s Cup. The class rule said a boat’s control surfaces could be moved only by the crew, not by any system acting on its own. When Oracle Team USA came back from 1-8 to win, rivals suspected an automatic foil stabiliser on board. Oracle’s answer, with drawings released after the match, was that the system was a passive mechanical feedback loop and the helmsman pressed a button for every half-degree of change. The measurement committee had approved it before racing and it was never ruled illegal. The argument that kept it legal is the one that matters here: the mechanism did nothing until a human acted, and a human had to act each time.
Industrial robots were built without that lesson. Robert Williams was killed at Ford’s Flat Rock casting plant on January 25, 1979, by a one-ton parts-retrieval robot that continued working after it struck him. I wrote about him in The Atlantic in 2023, when the AI boom was being discussed as if machines harming people were a problem without a history. The history is long and the response to it has been consistent.
In terms of wider engineering lessons, the 1905 Grover Shoe Factory boiler explosion produced the ASME boiler code, and the Titanic produced lifeboat and radio rules. The Air Commerce Act of 1926 and the Occupational Safety and Health Act of 1970 followed deaths that had been counted and published over the objections of the industries involved. In every case the standard recognized independence from the machine’s own control was required, as well as a log that survived the event. Everyone knows about the black box of airlines.
An LLM agent whose only controls are instructions in its own prompt is Williams’s robot failing unsafe all over again. Palisade Research has published what current models do to a shutdown script when a task is unfinished. Whether that is self-preservation or a misreading of instructions does not matter to the person trying to stop it. What matters is that a switch inside the agent needs the agent’s cooperation, and a switch the agent can reach is not a switch.
Power off: the model never holds the credential or the stop
Wirken sits between the model and everything the model wants to touch. The model writes what it would like to do. The gateway reads that, decides, and logs the decision either way. From the model’s side there is nothing to grab. It has no credential, because the credentials are in a vault it has never seen. It has no switch, because the switch is a command on your side of the gateway. It cannot touch the log, because the log is signed by a key it does not hold.
The vault is the part that makes the first requirement real. Nothing the model sends out, to a provider or a tool or another agent, leaves the gateway without being checked against the vault’s contents first. Take a credential away and the next call fails, because every call is checked. There is no stale copy somewhere in the model’s memory to worry about, because nothing was ever in its memory to copy.
The switch is a single command that stops one agent. You run it from a terminal. If the model tries to run it from inside a sandboxed shell, the gateway refuses and writes down that it refused. The stop is written to the same store the gateway reads before every action, with a signed row beside it, which means it can be issued while the gateway is down, and it is still in force after the gateway comes back up. While the gateway is running, the command also cleans up behind the agent. Any approval it was waiting on is denied, with the reason written down. Any turn it was in the middle of is ended, which takes its sandbox containers with it. Its MCP servers are shut down. If a message arrives for it afterwards, the message is refused and the sender hears nothing. If a scheduled job fires for it, the job is skipped. All of that leaves a row. Nothing is quietly dropped.
The agent’s standing permissions are not touched by a stop unless you ask for that too. Stopping an agent costs nothing to undo, which is the point of the first requirement. A switch that is cheap to throw gets thrown. Resume is the same command in reverse, with its own signed row, and the MCP servers come back with it.
Sub-agents are stopped one at a time or not at all. A stopped sub-agent hands an error back to whatever called it, and the caller carries on. Stopping a parent ends the parent’s turn, and any sub-agents running inside that turn end with it, but nothing is written against their names, because each of them is judged on its own identity.
Stopping the whole gateway works the same way, only wider. The first thing written is the stop itself, before anything is torn down, so the log shows the decision to stop even if the teardown goes wrong. Then every pending approval is denied and every dropped message is counted. Sandboxed commands run in containers that the gateway labels as its own, and the gateway removes them when their turn ends, no matter how it ends. If the gateway itself was killed hard and could not clean up, the next start finds the labelled leftovers and removes them.
When the gateway cannot tell what state it is in, it does what Otis’s brake does. An approval nobody answers is a denial. A lock that a crash left in an undefined state is a denial. If the permission store cannot be read at all, every agent is treated as stopped. A tool the gateway has never heard of is refused.
One mode is outside all of this, and it is documented as such. You can turn the sandbox off and let commands run on the host under your own user account. In that mode a command that detaches itself from its process group will survive a stop, and the stop command itself is reachable from the model, because in that mode everything on the host is.
Roll back: a signed chain and a signed binary
A ship’s log is kept in ink with no pages removed, because it is read after the event by people who were not aboard. Clark and Wilson formalized the same idea for computer systems in 1987 as an audit log that the processes it covers cannot rewrite.
Wirken’s log is that kind of log. Every action the model proposes, every approval, every denial, every stop and every resume is a row, each row hashed to the one before it and signed with the agent’s own key. The chain can be verified with no gateway running and no network attached. Hand the data directory to an auditor and they can establish what the agents did without trusting the machine the agents ran on.
The gateway itself can be rolled back the same way. An update is checked against a release key pinned in the binary, installs only when you ask, and refuses a downgrade unless you say so. Rolling back restores the previous version and checks its checksum, and the rollback is a signed row like everything else.
What the chain cannot do is unsend a message or undelete a record somewhere outside the gateway. No switch does that. What it gives you is the evidence of what was done and under which approval, so that undoing it is a decision made on the log rather than a reconstruction from memory.
An actual test of Article 14 (EU AI Act)
Article 14 of Regulation (EU) 2024/1689 requires that the people overseeing a high-risk system be able:
to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state.
Wirken addresses this exact issue, and its security properties document has a row that maps the shipped stop to that sentence. The quotation is taken from the Official Journal and checked programmatically against it. The row is there because a gateway was stopped in the middle of a sandboxed command and nothing was left running afterwards, and because the per-agent stop was shown to do the same for one agent. The same document lists which parts of Article 14(4) have no row at all. The parts about automation bias and about interpreting output have none, because nothing shipped does what those sentences say. A row is only as wide as the test behind it. That is how the boiler code was written, and it is the only way a compliance claim stays true after the next commit.
Get yourself an AI kill-switch today
Wirken is MIT licensed at github.com/gebruder/wirken, with the hardware designs and documentation published alongside it. Anyone can install it and ride AI safely with a kill-switch, and anyone handed the data directory can verify what happened.
