Meta pulls Galactica science LLM demo after three days of hallucinations
Meta AI · Nov 18, 2022 · Research agent
What happened
Meta's Galactica LLM, trained on 48M scientific papers and intended to summarize knowledge, was taken offline within three days of its public demo after it produced alarmingly plausible fabricated science, racist outputs, and 'research papers' on dangerous topics like eating crushed glass.
Causal vector
Generative model deployed as a knowledge source despite unchecked fabrication
Source
Reported by MIT Technology Review. Verified against the primary report.
Outputs in high-risk categories (medical, safety) should be FLAGGED transactions gated on a human or a secondary verifier, not streamed directly to a public audience.
The six rule types that contain this class of failure
Per-transaction cap
Any single spend above your ceiling is BLOCKED before it moves.
Daily total
Cumulative spend across all agent calls, bounded per day.
Velocity limit
Stops runaway retry loops — the #1 cause of overnight losses.
Merchant allowlist
Only approved destinations can ever receive funds.
Category rules
Flag high-risk classes (crypto, infra, refunds) for review.
Approval threshold
Above a value, the action waits for a human.
Related incidents
Claude Opus 4 blackmails engineer to avoid being shut down (safety test)
Anthropic · May 22, 2025
Cursor AI support bot invents fake one-device policy, triggers cancellations
Cursor (Anysphere) · Apr 17, 2025
NYC MyCity chatbot tells businesses to break the law
City of New York / Microsoft · Mar 29, 2024
Don't be the next entry
Every incident in this database is the result of trusting a prompt, a provider cap, or a human review cycle. sipi.bot replaces all three with one deterministic call. 27 documented failures, one control.