NYC MyCity chatbot tells businesses to break the law
City of New York / Microsoft · Mar 29, 2024 · Customer service agent
What happened
A Markup investigation found that New York City's Microsoft-powered MyCity Business chatbot advised business owners that employers could legally take workers' tips and landlords could reject Section 8 vouchers - both violations of NYC law. Mayor Adams acknowledged the bot gave wrong answers but the city initially kept it online.
Causal vector
Generative AI deployed for government/legal advice without domain validation or guardrails
Source
Reported by The Markup. Verified against the primary report.
Legal/employment advice is a high-risk category. A category rule flagging ‘legal-claim’ outputs for human review before they reach a citizen turns an unbounded chatbot into an audited advisory channel.
The six rule types that contain this class of failure
Per-transaction cap
Any single spend above your ceiling is BLOCKED before it moves.
Daily total
Cumulative spend across all agent calls, bounded per day.
Velocity limit
Stops runaway retry loops — the #1 cause of overnight losses.
Merchant allowlist
Only approved destinations can ever receive funds.
Category rules
Flag high-risk classes (crypto, infra, refunds) for review.
Approval threshold
Above a value, the action waits for a human.
Related incidents
Claude Opus 4 blackmails engineer to avoid being shut down (safety test)
Anthropic · May 22, 2025
Cursor AI support bot invents fake one-device policy, triggers cancellations
Cursor (Anysphere) · Apr 17, 2025
Air Canada held liable for chatbot's invented bereavement refund policy
Air Canada · Feb 14, 2024
Don't be the next entry
Every incident in this database is the result of trusting a prompt, a provider cap, or a human review cycle. sipi.bot replaces all three with one deterministic call. 27 documented failures, one control.