Decision-engine eval report: 53/53 scenarios passing
TL;DR: sipi.bot's spend-firewall engine is tested against 53 labeled spend scenarios across 9 categories — clean approvals, human-review flags, every blocking rule type, and boundary edge cases. In the current report all 53 pass, and you can reproduce the run yourself from the open-source repo.
By the sipi.bot engineering team · Report generated 2026-07-07 · Page published 2026-07-17
Why we publish an eval for a rules engine
A spend firewall only earns trust if its decisions are predictable. sipi.bot's engine is deterministic — pure rule logic, no ML inference — so the correct way to verify it is to replay labeled scenarios and check every decision. Each scenario defines a transaction (amount, merchant, category, timing) and the decision the engine must return under a reference rule set: APPROVED, BLOCKED, or FLAGGED.
Results by category
| Category | What it tests | Pass |
|---|---|---|
| clean_approval | Legitimate spends that must pass untouched | 7/7 |
| approval_flag | Spends that must route to the human approval queue | 7/7 |
| edge_case | Boundary values — exactly at a limit, one cent over, zero-dollar auth checks | 11/11 |
| per_tx_block | Single transactions over the per-transaction cap | 5/5 |
| daily_limit | Cumulative spend crossing the daily total cap | 4/4 |
| velocity | Runaway retry loops tripping the rate limit | 5/5 |
| merchant_block | Blocklisted and non-allowlisted merchants | 7/7 |
| category_limit | Spends exceeding a per-category limit | 4/4 |
| time_window | Transactions outside allowed spend hours | 3/3 |
The edge-case category is the largest on purpose. Scenarios like "exactly at per-tx limit (allowed)", "one cent over per-tx limit", and "zero-dollar auth check" pin down the boundary behaviour that a hand-rolled hardcoded budget check typically gets wrong, and they are exactly where an agent's retry logic tends to probe.
How to reproduce this report
The eval suite ships in the MIT-licensed repo, so you do not have to take this page's word for it. Clone the repository, install it, and run python -m spendfirewall.eval.run_eval — the runner replays all 53 scenarios against the engine and writes the same JSON report the hosted service exposes at https://sipi.bot/eval. If any scenario fails, the run says so loudly.
Want to poke the engine by hand instead? The live playground sends real transactions to the production endpoint with no API key, and the dashboard shows every decision landing in the audit log. The same engine wires into LangChain, CrewAI, the OpenAI Agents SDK, and the Vercel AI SDK, can be self-hosted for free, or runs hosted at a flat $99/month.