Introducing Model Maverick
Applications should never hardcode model names. Today we're opening early access to the engine that makes that possible.
By Donn Felker
The problem we kept hitting
Somewhere in every LLM-powered codebase there's a constant:
model = "gpt-4o" or claude-sonnet-4-6, chosen
months ago, for reasons nobody wrote down. Since then, providers have
shipped new models, updated old ones in place, changed prices, and
deprecated endpoints. The constant hasn't moved.
The tools that exist don't quite solve this. Public benchmark scores are contaminated and averaged into meaninglessness. Gateways route traffic brilliantly but have no measured opinion about which model deserves it. Trained routers make the pick for you but can't show their work. Eval platforms measure your prompts — if you author, run, and maintain the evals yourself, forever.
What we built
Model Maverick answers one question well: which model should run this workload?
Every night, it benchmarks frontier models — Anthropic, OpenAI, Google, and xAI at launch — against a private task corpus: 200+ tasks across ten capabilities and five authored difficulty tiers. Golden answers never enter a prompt, so public-benchmark contamination can't inflate a score. Graded results fold into a capability matrix of quality, pass-rate, cost, and latency — per model, per capability, per difficulty tier — each smoothed as a weighted running average of recent nights, so a single odd run can't swing a ranking.
On top of that matrix sits a policy engine. You define pools of allowed
models (your compliance boundary — a request can never widen its own),
per-capability weights, and standing cost and latency constraints. Then
your application calls POST /v1/recommend — or your agent
calls the recommend_model MCP tool — and gets back a ranked
list that doubles as a fallback chain, a machine-readable check spec for
validating output, and a recommendation_id that resolves to
a complete decision trace.
Maverick runs no inference. You keep Bedrock, Vertex, OpenRouter, LiteLLM, or direct SDKs, and execute on the model the data supports.
Honesty as a feature
The design decision we're proudest of is the least glamorous one: the
system tells you when it doesn't know. New models are flagged
measured: false until dense benchmarking says otherwise. If
your policy caps you below a workload's difficulty ceiling, you get the
best allowed model and clears_difficulty: false.
Every well-formed request returns a model — flags, never refusals, never
silent substitutions.
And when someone asks "why did it pick Haiku over Opus?", the decision trace answers from stored data alone: every surviving model with its score, every excluded model with its reason. Prompts themselves are never stored — only per-tenant keyed HMAC fingerprints.
Early access, and dogfood first
Tenant #1 is us: our own products route model selection through Maverick before any external tenant does. Every gap we find that way is a gap a customer never sees. We're now onboarding a small group of design partners — teams with real workloads to route, real compliance boundaries, and opinions about what the matrix should measure next.