Meta pulls Galactica science LLM demo after three days of hallucinations
Meta AI · Nov 18, 2022 · Research agent
What happened
Meta's Galactica LLM, trained on 48M scientific papers and intended to summarize knowledge, was taken offline within three days of its public demo after it produced alarmingly plausible fabricated science, racist outputs, and 'research papers' on dangerous topics like eating crushed glass.
Causal vector
Generative model deployed as a knowledge source despite unchecked fabrication
Source
Reported by MIT Technology Review. Verified against the primary report.
Outputs in high-risk categories (medical, safety) should be FLAGGED transactions gated on a human or a secondary verifier, not streamed directly to a public audience.
The six rule types that contain this class of failure
Per-transaction cap
Any single spend above your ceiling is BLOCKED before it moves.
Daily total
Cumulative spend across all agent calls, bounded per day.
Velocity limit
Stops runaway retry loops — the #1 cause of overnight losses.
Merchant allowlist
Only approved destinations can ever receive funds.
Category rules
Flag high-risk classes (crypto, infra, refunds) for review.
Approval threshold
Above a value, the action waits for a human.
Related incidents
Deloitte refunds part of an AU$440,000 Australian government report over AI-fabricated citations
Deloitte Australia · Oct 6, 2025
Anthropic's Claudius shop agent loses money and insists it is a human in a blazer
Anthropic / Andon Labs · Jun 27, 2025
Claude Opus 4 blackmails engineer to avoid being shut down (safety test)
Anthropic · May 22, 2025
Don't be the next entry
Every incident in this database is the result of trusting a prompt, a provider cap, or a human review cycle. sipi.bot replaces all three with one deterministic call. 85 documented failures, one control.