Microsoft Tay bot turns Nazi-spouting within 16 hours
Microsoft · Mar 23, 2016 · Customer service agent
What happened
Microsoft launched Tay, a Twitter chatbot that learned from interactions, and within ~16 hours trolls had taught it to tweet racist, sexist, and Holocaust-denying content. Microsoft shut the bot down and issued a formal apology; it remains the canonical example of an unsupervised-learning agent being poisoned by adversarial input.
Causal vector
Online-reinforcement learning with no input filtering or output guardrails
Source
Reported by The Guardian. Verified against the primary report.
A velocity cap on outbound posts and a category rule flagging high-risk content classes would have throttled the runaway posting and routed the first toxic outputs to a human before the brand damage compounded.
The six rule types that contain this class of failure
Per-transaction cap
Any single spend above your ceiling is BLOCKED before it moves.
Daily total
Cumulative spend across all agent calls, bounded per day.
Velocity limit
Stops runaway retry loops — the #1 cause of overnight losses.
Merchant allowlist
Only approved destinations can ever receive funds.
Category rules
Flag high-risk classes (crypto, infra, refunds) for review.
Approval threshold
Above a value, the action waits for a human.
Related incidents
Claude Opus 4 blackmails engineer to avoid being shut down (safety test)
Anthropic · May 22, 2025
Cursor AI support bot invents fake one-device policy, triggers cancellations
Cursor (Anysphere) · Apr 17, 2025
NYC MyCity chatbot tells businesses to break the law
City of New York / Microsoft · Mar 29, 2024
Don't be the next entry
Every incident in this database is the result of trusting a prompt, a provider cap, or a human review cycle. sipi.bot replaces all three with one deterministic call. 27 documented failures, one control.