Why prompt instructions aren't spending controls
Every documented runaway-agent incident in our database shares the same root cause: the operator trusted what they told the model instead of what the model could do. The prompt said "don't spend more than $500." The agent spent $47,000. The prompt was ignored.
This isn't a failure of prompt engineering. It's a category error. A natural-language instruction to a stochastic model is not a control. A control is a deterministic gate that returns a binary decision independent of the agent's internal state.
The three failure modes
1. The prompt is in the agent's context, not in its execution path. When the agent calls a tool, the tool function executes. The prompt is text on a different thread — the tool has no way to consult it before acting. Unless you explicitly wrap every tool call in a guard function, the prompt is a passenger, not a driver.
2. The agent doesn't experience cost. To a large language model, "$0.03 per token" means nothing. It has no visceral signal that tells it "this action costs money." It will retry, iterate, expand, and explore in ways that read as productive but consume resources without limit.
3. Prompt injection defeats instructions. The ChatGPT data-exfiltration exploit and Operator prompt-injection attacks both prove that a model's instructions can be overridden by content it encounters. A control that lives inside the same text window as adversarial content is not a control.
What replaces the prompt
A spend firewall — one API call before the tool — returns APPROVED, BLOCKED, or FLAGGED in under 5ms. The prompt says what the agent should try to do; the firewall says what it is allowed to complete. The firewall doesn't read the prompt; it reads the transaction and the rules. That's the difference between a suggestion and a constraint.