# Does a cheaper model write deep research reports as well?

A trial run of a public benchmark by AutoRouter, written before any counted
call: the main run on October 28, 2026 at about 11:30 PM ET, and the Fable
5.1 follow-up on October 29, 2026, before any counted Fable call. It is an
unofficial run with a substitute grader, so its scores compare within this
study only.

## Question

A frontier model does the web research. Can a cheaper model write the final
report from that research without a real drop in quality? How does that
compare with giving the cheaper model the whole job, and does a more
expensive writer do better?

## Tasks

20 of the 50 English tasks in
[DeepResearch Bench](https://github.com/Ayanami0730/deep_research_bench)
(Du et al., 2025), drawn with `random.Random("drb-cheap-writer-20").sample`
before any run: ids 52, 53, 61, 65, 66, 68, 73, 74, 79, 80, 81, 82, 84, 86,
90, 91, 92, 93, 95 and 99, covering 14 of the benchmark's 22 topics. Task
prompts are used verbatim.

## Setups

| Code | Research (web search and fetch)    | Writes the report            |
| ---- | ---------------------------------- | ---------------------------- |
| OO   | Claude Opus 5.5                    | Claude Opus 5.5 (baseline)   |
| OS   | Claude Opus 5.5 (same notes as OO) | Claude Sonnet 5.5            |
| OH   | Claude Opus 5.5 (same notes as OO) | Claude Haiku 5.5             |
| HH   | Claude Haiku 5.5                   | Claude Haiku 5.5             |
| OF   | Claude Opus 5.5 (same notes as OO) | Claude Fable 5.1 (follow-up) |

Opus 5.5 researches each task once; OO, OS, OH and OF write from the
identical notes file. Every call runs through Claude Code in print mode on a
Claude subscription, with Claude Code's built-in WebSearch and WebFetch and
no other tools. Costs are Claude Code's own list-price estimate for each
call; no money changed hands per call.

## Grading

1. **RACE**, the benchmark's report-quality grader: the official English
   scoring prompt, per-task criteria and weights, and reference report.
   Score = target ÷ (target + reference) × 100, plus its four dimensions
   (comprehensiveness, insight, instruction following, readability).
2. **Faithfulness to the notes,** in place of the benchmark's FACT check,
   which needs a separate scraping key: the grader lists claims in the report
   that the research notes don't support, and citation numbers that point at
   the wrong source, reported per 1,000 words.
3. **Grader noise:** 10 reports (8 from the main run, 2 from the follow-up,
   picked by seed) are RACE-scored a second time.
4. The grader sees only the task, the report and the reference, never which
   model wrote what.

## Pass rule

A writer **holds up** if the 95% bootstrap interval of its mean paired RACE
difference against OO (same task, writer minus Opus) has a lower bound above
−2.0 points. The same test is reported for HH. For the follow-up, Fable is
**better than Opus writing** if that lower bound is above 0.

## Deviations from the official benchmark

- **Grader model:** GPT-6.1 Sol. The official RACE grader model was not
  available to this account, so scores do not compare with the public
  leaderboard.
- **Citation cleaning:** the benchmark strips citations with a model before
  scoring; here a fixed pattern removes inline `[n]` marks and the Sources
  section, identically for every setup.
- **Size:** 20 tasks rather than 100, one run each. FACT is not run.
- **Timing:** the Fable reports were written and graded about two hours
  after the main run, with the same grader and prompts.

## Exclusions

A report is excluded only if its call fails after retries or Claude Code
reports that a different model served it. None were excluded.

## Data

[results.jsonl](results.jsonl) has one line per research run (40) and per
written report (100): its task, setup, models, finish time, tokens,
list-price cost, word count, RACE scores, unsupported claims and citation
errors.