How do you test a model router change before it ships?
AutoRouter6 min read
Key facts
- A routing test that skips the person's rules can pass while a shipped change stops a rule from holding, so the test has to run the same path production runs.
- Anthropic's guide to building evaluations advises designing them to mirror your real-world task distribution, edge cases included, and grading them automatically where possible (Anthropic, read November 4, 2026).
- Before a routing change ships, the AutoRouter team checks it against reviewed routing cases, including picks people rated down, run through the same rule path production uses.
Test a routing change by running real, reviewed requests through the same code path production uses, with the person's rules loaded, and grading each pick against the models a reviewer accepted. AutoRouter, the Mac app for Codex and Claude Code, checks its routing changes this way before they ship, against reviewed cases that include picks people rated down. A rating you send can become a case that later routing changes are checked against. AutoRouter wrote this post, and it describes a manual check by the AutoRouter team: AutoRouter doesn't publish benchmarks yet, and automatic evaluations on real coding tasks are coming soon.
Why can a routing test pass while production fails?
A routing test can pass while production fails when it calls the model-picking step without the layers production runs around it. Routing has several: the person's rules, how hard the turn is, the quality bar, the plans and the price. A test that hands requests straight to the picking step never runs the rules, so a change that stops a rule from holding still passes. Run the same entry point production runs, with saved rules loaded, so every layer takes part. What is an LLM router? describes the kinds of routers and their layers.
What goes into a set of reviewed cases?
Each reviewed case records one real request and the answer a reviewer accepted for it:
- The real prompt, with the settings, rules and plan state it ran under.
- The model or models a reviewer accepted for it.
- One sentence on why the case matters, so a later reader knows what it protects.
- A stable ID, so a failure points at one case.
Two checks keep the set honest. Reject two approved cases whose inputs match but whose accepted answers don't overlap, since no policy can pass both. When a product change retires a setting a case depends on, send the case back for review instead of relabeling it to pass.
Anthropic's guide to building evaluations advises designing them to mirror your real-world task distribution, edge cases included, and grading them automatically where possible (Anthropic). Routing cases fit both: the requests come from real use, and a pick either matches an accepted model or it doesn't.
Where do the most useful test cases come from?
The most useful cases come from picks people rated down, because each one shows where the policy fell short on a real request. AutoRouter lets you rate a pick when it falls short, and the AutoRouter team reviews every rating people send. The reviewed cases AutoRouter checks routing changes against include picks people rated down, so a rating can do more than flag one turn: once reviewed, it can become a case that later changes are checked against.
Real prompts need a basis for keeping them. AutoRouter keeps the message that opens each turn, up to 4,000 characters, to review routing, along with records of the model chosen, tokens and cost, and it doesn't store transcripts (AutoRouter privacy policy).
How do you grade a routing decision?
Grade each pick against the models a reviewer accepted for that case, and scope any accepted exception to that one case and plan. An exception keyed by a category, such as every hard turn on one plan, can hide a wrong answer in another case of the same category. Report direct matches apart from accepted exceptions, and fail the run on any miss, so a reader sees how much of a pass rests on exceptions.
| Test design | What it can miss |
|---|---|
| Calls the model-picking step without the rules | A change that stops a rule from holding |
| Accepts exceptions by category | A wrong answer in another case of the same category |
| Uses only hand-written prompts | How people phrase real requests |
| Relabels a case when a setting is retired | A case that no longer tests anything |
| Covers only common requests | Rare ones, such as a turn no other model can run |
What should a test of a rule cover?
A test of a rule should cover every case where obeying the rule costs something: the excluded model is the cheapest one that clears the bar, no other model clears it, or no other model can run the turn at all. AutoRouter's avoid rules show why each branch needs its own case.
An avoid rule is always respected: AutoRouter never sends the turn to the avoided model, even to continue an earlier turn that ran on it. It picks the next model that clears the quality bar, even when that model costs more. If no other model clears the bar, the strongest remaining model runs the turn, and if no other model can run the turn at all, AutoRouter refuses it and names the rule. A team that writes a rule to keep a model out can count on that rule holding.
- The avoided model is the cheapest that clears the bar: a pricier model that clears it wins.
- No other model clears the bar: the strongest remaining model runs the turn.
- No other model can run the turn: AutoRouter refuses it and names the rule, even when the previous turn ran on the avoided model.
What does a passing test not show?
A passing test shows that the policy gave an accepted answer on every case in the set, and nothing more. It doesn't show how often picks are right on live traffic, what turns cost, or whether the set covers what people send next. Use a green run to decide a change is ready for review, and watch live picks and ratings after it ships.
Questions
Does a passing test prove a change is better?
No. It proves the change gives accepted answers on the cases in the set. Whether picks improve on live traffic takes a separate measurement, such as how ratings move after the change ships.
How many cases does a set need?
Enough that each rule and each common kind of request has at least one case that fails when it breaks. Anthropic's guide favors more cases with automated grading over fewer hand-graded ones (Anthropic), and routing picks can be graded automatically against accepted models.
Should a rule change ship with a test case?
Yes. Add a case that fails without the change, so a later edit that undoes it shows up in the next run.
Does AutoRouter publish its routing test results?
No, AutoRouter doesn't publish benchmarks yet. The AutoRouter team reviews real routing decisions and every rating people send, ratings and tests of each new model retune the router, and benchmarks and automatic evaluations on real coding tasks are coming soon. How AutoRouter picks a model for each turn explains the order it follows.
Sources
- Anthropic: Define success criteria and build evaluations, read November 4, 2026
- AutoRouter privacy policy
