How to Monitor AI Costs in Real Time
Most teams discover their AI bill on the monthly invoice — too late, too aggregated, and with no way to attribute spend to a specific agent or session. The fix is real-time cost monitoring: per-token attribution, per-agent dashboards, anomaly detection, and webhook alerts. Here is the 5-step playbook.
The problem with provider-side reporting
OpenAI, Anthropic, and other providers show you aggregate usage on a dashboard, updated hourly or daily. That is useful for capacity planning but useless for incident response. By the time a retry loop shows up on the provider dashboard, it has been running for hours. You need sub-second visibility at the call site.
The 5-step playbook
1 Instrument every LLM call
Tag every call with agent id, session id, model, input token count, output token count, and computed cost. This is the atomic unit of cost attribution. Without it, every downstream metric is an aggregate guess.
{
"agent_id": "research-bot-01",
"session_id": "sess_abc123",
"model": "gpt-4o",
"input_tokens": 1240,
"output_tokens": 380,
"cost_usd": 0.0069
}2 Build a per-agent dashboard
Aggregate the instrumented calls by agent, session, model, and category. The dashboard should answer: which agent is spending the most? Which model? Which session? Is spend trending up or flat? A good dashboard lets you drill from total spend down to the single call that caused it.
3 Set anomaly detection rules
Flag spend that deviates from baseline. The simplest rule: alert when hourly spend exceeds 3x the trailing 7-day median. More sophisticated rules use per-agent baselines — a research agent spending $20/hour is normal; a chat agent spending $20/hour is an anomaly.
{"rule": "spend_anomaly", "params": {"multiplier": 3, "baseline_window_hours": 168}}4 Configure webhook alerts
Send alerts to Slack, PagerDuty, or email when thresholds are crossed. Latency matters — an alert that arrives 6 hours after the anomaly is useless. Aim for sub-minute alert latency. Route critical alerts (circuit-breaker trips, daily ceiling crossed) to PagerDuty; route informational alerts (anomaly detected, threshold approached) to Slack.
5 Pair monitoring with enforcement
Monitoring shows what happened. A pre-spend firewall stops what should not. Run both: sipi.bot to block or flag before money moves, and an observability layer (Langfuse, Helicone, OpenMeter) to trace and understand. Monitoring without enforcement gives you a beautiful dashboard of your runaway spend.
What good monitoring looks like
- Sub-second latency — you see a call's cost within milliseconds of it completing.
- Full attribution — every dollar is tied to an agent, session, model, and category.
- Anomaly surfacing — the system tells you something is wrong before you look.
- Drill-down — from total spend to the single anomalous call in three clicks.
- Alert routing — critical alerts wake a human; informational alerts go to a channel.
Frequently asked questions
How do I monitor my AI costs?
Instrument every LLM call with agent id, session id, model, and token count; aggregate into a per-agent dashboard; set anomaly detection rules; configure webhook alerts. Aim for sub-second visibility, not monthly bill shock.
Is monitoring enough to control AI spend?
No. Monitoring is reactive. Pair it with a pre-spend firewall (sipi.bot) that blocks or flags transactions before money moves.
What is per-token cost attribution?
Tagging every LLM call with exact input/output token counts and multiplying by the model's per-token price, so you know the cost of each call — not just the monthly provider bill.