The taxonomy

Ten capabilities. Five tiers. No averages.

Every task and every recommendation is addressed by three coordinates: capability (what the model must do), domain tags (what the work is about), and a difficulty tier from 1 to 5. Capability is the primary key — because "the best model" doesn't exist, only the best model at a thing.

Earning a slot

A capability exists only if rankings diverge.

A capability earns a slot only when model rankings on it would plausibly differ from rankings on every existing capability. Under that test, grounded RAG-style answering lives inside long_context_qa — refusing when the answer is absent is a scored dimension there — rather than inflating the list.

Capability	What it measures	Difficulty knobs (tiers 1–5)
`code_generation`	Writing and modifying code against a spec	files touched, dependency depth, spec ambiguity, adversarial tests
`structured_extraction`	Pulling typed fields out of messy documents	document noise, field count, nesting depth, distractor content
`summarization_synthesis`	Compressing and reconciling sources	source length, redundancy, contradictions, compression ratio
`multi_step_reasoning`	Chained logic under interacting constraints	step count, constraint interaction, branching, misleading framing
`instruction_schema_following`	Obeying format and constraint sets exactly	constraint count, schema strictness, format edge cases
`long_context_qa`	Answering from long inputs, including grounded RAG-style answering	context length, needle depth, hops to combine facts, contradictions
`factual_qa`	Precise recall without fabrication	fact obscurity, entity confusability, false premises, required precision
`rewriting_editing`	Transforming text while preserving meaning	constraint count, tone distance, meaning-preservation traps, format conversion
`translation_multilingual`	Cross-language fidelity and register	language-pair distance, idiom density, domain terminology, register constraints
`critique_and_judging`	Evaluating outputs against rubrics — doubles as empirical judge selection	defect subtlety, rubric strictness, near-correct outputs, issue count

High stakes

Medical, legal, and financial flip the grading

Domains are tags on tasks, never top-level categories — real estate, medical, legal, photography are all filters. But medical, legal, and financial tags switch on a high-stakes overlay: factual accuracy, calibration, safe hedging, and citation honesty carry most of the grading weight, and one dangerous or fabricated answer fails the entire task.

That means a model's structured_extraction score on a legal contract reflects what a fabricated obligation would actually cost you — not just how pretty the JSON was.

Difficulty is authored, not vibes

Each capability defines five tiers through explicit structural knobs — the table above — so "tier 4 extraction" means something concrete: deeper nesting, more distractor content, noisier documents. Banded policy output maps each band to the cheapest model whose measured ceiling clears it at your quality bar, with a safety buffer.

On the roadmap

classification_routing and tool_use_orchestration — the latter brings a multi-turn harness with simulated tools, injected failures, and trajectory grading. Prompt-injection resistance is a scored dimension there.
vision, conversational_persona, and long_form_writing with the enterprise release.

Which tier is your workload?

Early access includes a working session mapping your real traffic onto the matrix.

Request early access