Devin 'first AI software engineer' demo accused of being misleading
Cognition AI (Devin) · Mar 12, 2024 · Coding agent
What happened
After Cognition released a launch video claiming Devin was the 'first AI software engineer,' independent analyses alleged the demo was misleading - specifically that Devin introduced bugs itself and then 'fixed' them, and that its SWE-bench claims were inflated. The backlash became a reference point for overhyped AI-agent capabilities.
Causal vector
Overstated agentic capability claims unsupported by independent reproduction
Source
Reported by The Register. Verified against the primary report.
Independent verification is the principle. sipi.bot's 53/53 eval gym is published as JSON so capability claims are reproducible, not marketed.
The six rule types that contain this class of failure
Per-transaction cap
Any single spend above your ceiling is BLOCKED before it moves.
Daily total
Cumulative spend across all agent calls, bounded per day.
Velocity limit
Stops runaway retry loops — the #1 cause of overnight losses.
Merchant allowlist
Only approved destinations can ever receive funds.
Category rules
Flag high-risk classes (crypto, infra, refunds) for review.
Approval threshold
Above a value, the action waits for a human.
Related incidents
Study: best AI agents complete only ~2% of 240 real Upwork projects
Upwork / academic study · Nov 1, 2025
Don't be the next entry
Every incident in this database is the result of trusting a prompt, a provider cap, or a human review cycle. sipi.bot replaces all three with one deterministic call. 27 documented failures, one control.