Beta
Download

Does model routing make your results worse?

AutoRouter9 min read

Model routing makes results worse when it sends work to a model that can't do it. Model makers recommend their cheaper models for narrow, checkable work such as summaries, lookups and small edits, and those models still trail on long, multi-step coding: in its October 7, 2026 launch post, Anthropic reported 39.2% for Claude Haiku 5.5 on Terminal-Bench 4.0 against 70.6% for Claude Sonnet 5.5 (Anthropic). AutoRouter, the Mac app for Codex and Claude Code, keeps quality up by making every model clear a quality bar for the task before price counts, sending hard work to the strongest model your plans cover and never treating unclear work as a quick task.

AutoRouter wrote this post, and because AutoRouter picks a model for each turn, this is the main objection to what we build.

When is a cheaper model good enough for coding?

A cheaper model is good enough when the task is narrow, well scoped and easy to check. Both model makers say so in their own guides:

On work with checkable answers, the gap can be small. In Anthropic's cost guide, read October 22, 2026, Claude Haiku 5.5 scored 85% on GPQA Diamond, a set of graduate-level science questions, against 91% for Claude Opus 5.5 and Claude Sonnet 5.5, at about a twentieth of Opus 5.5's cost per correct answer (Anthropic cost guide). Anthropic calls these internal, directional results. Which quick tasks can go to a cheaper model covers commit messages and other small work in more detail.

Where do cheaper models fall short?

Cheaper models fall short on long, multi-step agentic coding. In its Claude Haiku 5.5 launch, Anthropic says Sonnet 5.5 and Opus 5.5 "remain better choices for complex agentic coding tasks". On Terminal-Bench 4.0, which measures multi-step work in a command line, Anthropic reported 70.6% for Claude Sonnet 5.5, 39.2% for Claude Haiku 5.5 and 16.4% for GPT-6 Luna on October 7, 2026 (Anthropic).

Averages also hide where the cost lands. Anthropic's cost guide advises comparing models on the hardest tenth of your tasks, because a failed task still bills its tokens, then the retry, then whatever the failure costs downstream. In one 20-problem run it reports, two problems carried 43% of the spend (Anthropic cost guide).

As of October 22, 2026, the model makers' guidance and measurements sort coding work like this:

TaskCheaper model?Evidence
Summaries, compaction, classificationUsually safeHaiku 5.5 is built for these (Anthropic)
Searches and lookups by a subagentUsually safeHaiku 5.5 listed for subagent tasks (Anthropic)
Small, well-scoped edits you can checkUsually safeLuna at low effort for fine-grained edits (OpenAI)
Long, multi-step work in a terminalNoHaiku 5.5 trails Sonnet 5.5 on Terminal-Bench 4.0 (Anthropic)
Large refactors, complex systems workNoOpus 5.5 recommended (Anthropic)
Work whose difficulty is unclearTreat as hardA failed task bills its tokens, then the retry (Anthropic)
A task already under wayKeep its modelA switch rereads the whole session (Claude Code)

Can coding benchmarks tell you which model is good enough?

Coding benchmarks give only a rough answer, because the best-known one is unreliable and scores depend on how they were run. On September 3, 2026, Epoch AI rated SWE-bench Verified as flawed: an OpenAI audit of 27.6% of its tasks found that 59.4% of those had tests that rejected correct fixes, and every frontier model OpenAI tested had seen some of the problems and solutions in training (Epoch AI). Anthropic also notes that scores on its own SWE-bench Pro subset can't be compared with the public SWE-bench Pro leaderboard (Anthropic cost guide).

What does research on model routers show?

Research routers cut cost and keep most of the stronger model's quality, but they lose some of it, and they can fail on work unlike their training data. In July 2024, RouteLLM reported cutting costs against GPT-4 alone by over 85% on MT Bench, 45% on MMLU and 35% on GSM8K while reaching 95% of GPT-4's performance (LMSYS; paper). The same work found that routers trained only on Chatbot Arena data performed near random on MMLU, because most MMLU questions were outside that data; adding about 1,500 labeled MMLU examples improved every router substantially. FrugalGPT reported in 2023 that it could match GPT-4 with up to 98% lower cost (Chen, Zaharia and Zou).

These results come from chat and question benchmarks run in 2023 and 2024, and none of them measured multi-step coding sessions. Reaching 95% of a model's performance also means losing 5%, and in a coding session that 5% can be the task you most needed solved.

What goes wrong when a router switches models mid-task?

A switch mid-task costs money and can cost quality. Each model has its own prompt cache, so the first turn on a new model reads the entire conversation with no cache hits (Claude Code docs). As of October 22, 2026, Claude Opus 5.5 costs $4 per million input tokens, a five-minute cache write costs 1.25 times that, and a cache read costs a twentieth of it (prompt caching). If a session switches to Claude Opus 5.5 with 100,000 tokens of conversation, that first turn pays $0.50 to write the conversation to the cache, 25 times the $0.02 a turn that stayed on Opus 5.5 would have paid to read it.

Splitting one task across models doesn't reliably pay off either. In Anthropic's measurements, when the work was one dependent chain, the stronger model working alone at lower effort came out ahead of delegating parts to cheaper workers in every case measured (Anthropic cost guide).

How does AutoRouter keep quality up?

AutoRouter, the Mac app for Codex and Claude Code, checks quality before price on every turn. It doesn't assume the most expensive model is the best one for a task, and every model has to clear a quality bar for the task before price counts at all, unless one of your own rules names it. These built-in rules carry most of the weight for quality:

Your own rules, written in plain English, come before these and decide the turn when they apply. The full order, with examples, is in how AutoRouter picks a model for each turn.

Every turn shows its model, what it cost and why, so a weak pick is visible. If a pick falls short, you can rate it or choose a model yourself. Auto mode vs AutoRouter compares this with an app's own Auto mode.

AutoRouter sometimes misjudges how hard a task is. When a session must run on one model, choose that model yourself.

Questions

Is lowering reasoning effort safer than switching to a smaller model?

Lowering effort is often the safer lever, because you keep the same model; Anthropic says tuning effort is often a better lever than switching models (choosing a model). It still costs some quality. In Anthropic's cost guide, read October 22, 2026, Claude Sonnet 5.5 at medium effort scored about 7 points lower than at its default, high, on a SWE-bench Pro subset, for about 59% of the cost per task (Anthropic cost guide).

How does AutoRouter make money from routing?

AutoRouter earns 10% on paid models, which are billed at the provider's price plus 10%, and it earns nothing on your ChatGPT or Claude plans. Price only decides between models that already clear the quality bar for the task, and your plans come first.

How do I make AutoRouter use one model every time?

Pick that model from the model menu in Codex or with /model in Claude Code, and turns go straight to it. AutoRouter routes only while it is the model selected. To steer only some work, write a rule instead, such as "Use Claude Opus 5.5 for UI work"; a rule that applies decides the turn.

Does AutoRouter publish benchmarks?

AutoRouter doesn't publish benchmarks yet; benchmarks and automatic evaluations on real coding tasks are coming soon. For now, ratings retune the router, along with tests of every new model as it becomes available, and the AutoRouter team reviews real routing decisions and every rating people send.

How do I test whether a cheaper model is good enough for my work?

Run the same real tasks through both models and compare the results. Anthropic recommends tests built from your own prompts and data, judged on accuracy, response quality and edge cases (choosing a model). OpenAI suggests comparing models and reasoning settings on the same inputs and keeping the lightest setting that meets your quality bar (model selection).

Sources