From $12,400 to 53/53: how we built sipi.bot's eval gym

I woke up on a Tuesday morning to a $12,400 Azure bill. An agent I'd built to optimize cloud costs had done the opposite — it provisioned GPU instances across three regions while I slept, each one a "cost-optimization experiment" that somehow made the bill bigger, not smaller. The prompt said "minimize spend." The agent heard "experiment with spend."

That morning, sipi.bot went from a weekend project to a full-time conviction. If I — someone who builds agent infrastructure for a living — could lose $12,400 to a single runaway loop, what was the industry's actual exposure?

The eval gym: 53 scenarios, zero tolerance

We wrote 53 evaluation scenarios that test every rule type and every failure mode. Each scenario is a specific agent configuration (agent type, budget, tool count, risk parameters) paired with a transaction attempt, and the expected firewall decision. Running the eval is deterministic: given the same scenario and rule set, sipi.bot must return the expected decision every time, in under 5ms.

The 53 scenarios cover: per-transaction caps (10 scenarios), daily totals (8), velocity limits (8), merchant allowlists (7), category rules (10), approval thresholds (6), and multi-rule interactions (4). Every new rule type or framework integration expands the eval — it's our regression suite, our confidence score, and our truth table.

We publish the results at /eval (machine-readable JSON) and /eval-report/ (human-readable). As of today, sipi.bot passes all 53 scenarios with zero failures. The eval gym is the reason we can say "deterministic" and mean it.

Open source, MIT

The core firewall engine is open source under MIT license at kindrat86/sipi-bot. Self-host for free, deploy the hosted version at $99/month. The eval gym runs against both. The founding belief: a spend firewall should be as standard as a database connection. The $12,400 didn't have to happen. $99/month would have caught it. Deploy yours here →