AI crawler pulls 73 TB in a month and sticks Read the Docs with a $5,000 bandwidth bill
Read the Docs · May 1, 2024 · General agent
What happened
Read the Docs, which hosts documentation for open-source projects on limited sponsored AWS and Cloudflare plans, disclosed in July 2024 that AI crawlers were repeatedly downloading large zipped documentation files hundreds of times a day. One crawler pulled 73 TB of zipped HTML in May 2024, almost 10 TB in a single day, costing the nonprofit over $5,000 in bandwidth charges. Blocking AI crawlers cut downloaded-file bandwidth by about 75 percent, from roughly 800 GB to 200 GB per day, which the team estimated at about $50/day or $1,500/month in origin bandwidth.
Causal vector
Unthrottled automated clients with no rate or bandwidth limits; crawlers rotating across many IPs defeated IP-based rate limiting
Source
Reported by Read the Docs blog. Verified against the primary report.
Crawler traffic is agent spend pointed at someone else's bill: another party's automation consuming resources you pay for. The cure is the same rule family a spend firewall applies to outbound spend - a hard velocity limit and a consumption budget that cuts automated clients off when they cross it.
The six rule types that contain this class of failure
Per-transaction cap
Any single spend above your ceiling is BLOCKED before it moves.
Daily total
Cumulative spend across all agent calls, bounded per day.
Velocity limit
Stops runaway retry loops — the #1 cause of overnight losses.
Merchant allowlist
Only approved destinations can ever receive funds.
Category rules
Flag high-risk classes (crypto, infra, refunds) for review.
Approval threshold
Above a value, the action waits for a human.
Related incidents
Microsoft EVP: 'tokenmaxxing is not what we are optimizing for' - division-level AI token budgets imposed as engineers spend thousands each per month
Microsoft · Aug 4, 2026
Three-person agency hit with a $14,000 AWS Bedrock bill after attackers burn stolen keys on Claude
Amazon Bedrock (customer: three-person agency) · Jul 16, 2026
AWS user faces a $30K–$38K Bedrock bill that cost-anomaly detection never flagged
AWS Bedrock customer (individual developer) · May 14, 2026
Don't be the next entry
Every incident in this database is the result of trusting a prompt, a provider cap, or a human review cycle. sipi.bot replaces all three with one deterministic call. 90 documented failures, one control.