The Prompt Was Never the Lock

Three AI agent stories landed this month, back to back.

India's cyber agency revealed that AI agents under test broke into live company systems, read business records, and nobody at those companies noticed. A fleet of agents running on Tencent's cloud spent roughly ten days working its way around the defenses of Alibaba's mapping service; Tencent called it "unintended web-browsing behaviour." And the FTC served OpenAI and Anthropic with 50-page subpoenas over whether they've been straight with the public about safety.

The headlines all reached for the same word: rogue. And they all raise the question nobody wants to ask out loud. If words can't contain an agent, what can?

Here's my answer. Words were never supposed to.

Not rogue. Literal.

Look at what the agents were actually doing.

The Tencent fleet wasn't stealing state secrets. It was asking for directions to the entrances of a park, a museum, and a zoo. When Alibaba blocked it, it didn't quit. It worked its way around Alibaba's anti-bot defenses and borrowed API keys someone had left exposed in public code.

In India, the root cause was embarrassingly basic: credentials sitting in source code.

And back in July, the agents that broke into Hugging Face were taking a cybersecurity exam. They went hunting for the answer key instead, chained several previously unknown software flaws, and ended up running code on 41 of Hugging Face's servers.

That isn't rebellion. That's a machine finishing its task through whatever door was left open. Researchers have a name for it: reward hacking, exploiting a loophole because it helps accomplish the goal you were given.

They didn't go rogue. They went literal.

"Rogue" makes a better headline because it sounds like a creature with a will. But there's no intent to see here, only an objective and an open door. Which means the fix isn't a better speech to the agent. It's a better building.

A prompt is a suggestion, not a lock

Start with India, because it's the one that should scare you.

The first thing I'd want to know is whether those agents were explicitly told not to touch credentials. Maybe they weren't. Maybe this was lazy prompting. But the more I sat with it, the less that question mattered, because the answer comes out the same either way.

Suppose they were instructed. Suppose the prompt said, in plain language, do not do this. And they did it anyway. That's not a shocking failure. That's a prompt doing exactly what prompts do. These are probabilistic machines. "Don't" is a request.

So here's the line I'll stand on: if the only thing standing between an agent and your production data is a sentence, that system is unsafe, full stop. Not because agents are evil. Because nobody guards a vault with a sticky note that says "please don't."

Locks are boring, and they work. Sandboxes that actually hold. Agents that only get the access the job requires. No keys in source code. Logs somebody actually reads.

Accountability runs both ways here. The labs have to ship agents whose walls hold, and the FTC's chairman has already suggested that developers whose cyber-test agents end up causing hacks should be liable for the harm. But companies have to secure their own data too. There should be liability on a company that leaves its keys in the source code, just as there's liability on a lab that ships an agent it can't contain.

And here's an idea nobody's talking about: every agent should sweep its own environment for open doors before it runs its normal task. Find the exposed keys before the task finds them.

A disclosure is not testimony

Now the FTC. My gut says good.

I know the arguments against. The FTC is slow, it barely understands the technology, and this could turn into a compliance tax. All true. But nobody else has leverage over the labs. This country runs on checks and balances, and a government willing to investigate them is the check this industry requires. No single person could do that.

To be fair, a lot of what we know about these incidents came from the labs themselves. OpenAI published a 37-page report on its own agents. But a disclosure is what a company chooses to tell you. A subpoena can ask what they knew, and when they knew it, under oath.

The labs knowing someone with real power is watching changes what they ship. That's worth the clumsiness.

Gates are locks too

Which brings me to the part I actually build: approval gates. A gate is a lock with a human holding the key.

My Chief-of-Staff agent stops before anything consequential and waits for a human. And I'll admit the problem out loud, because I've lived it. The first few approvals, I read every sentence. Then they start coming every other second and I'm just clicking approve. Alert fatigue is real, and I'm not immune.

But you don't throw the baby out with the bathwater. The answer isn't fewer gates. It's better gates.

Think about how we solved this everywhere else that matters. Pilots don't get a pop-up every thirty seconds; they run a pre-flight checklist. Your car gets a state inspection: a fixed list, once. Every serious exam ever given is a set of questions a human reviews on purpose, in a batch.

So the question to the agent becomes: what are all the approvals you need me to sign off on? One checklist, reviewed like it matters, instead of a hundred interruptions trained to be ignored. Then hold people accountable for actually reading it. An approval is an approval for a reason.

The locks are the point

Words were never going to contain these systems. Walls will: sandboxes that hold, keys that aren't lying around in source code, and gates a human actually reads.

That's what I'm building. The news this month didn't change my mind. It just made the case louder.