LLM Provider Evaluation Template
Choosing a provider is a decision with many axes. This template makes it a scored matrix instead of a vibe.
The matrix
1. Task fit: benchmark score on YOUR tasks.
2. Rate: real cost per task, not per token.
3. Latency: p95 for your workload shape.
4. Limits: rate limits, context window, concurrency.
5. Reliability: uptime and incident history.
6. Governance: native budgets, key scoping, logs.
7. Score each 1–5, weighted by your priorities.
FAQ
What's the most common mistake?
Picking on rate alone — per-task cost is the real number.
How often should I re-evaluate?
Quarterly, or when pricing changes.
Related
Stop the next $12,400 night.
One API call (or MCP tool) in front of every agent transaction — APPROVED, BLOCKED, or FLAGGED, deterministic, ~5 ms, fully logged.
See plans — from $99/mo Try a live check