# Codex vs Claude Code on the same real bug fixes Written on November 9, 2026, before any run was collected. ## Question Given the same real bug reports from an open-source project, how often does Codex fix each bug compared with Claude Code? How long does each take, how many tokens does each use, and what would each run cost at API list prices? ## Test bed - Repository: [pallets/click](https://github.com/pallets/click), a widely used Python command-line library (about 12,000 lines of source and 1,600 to 2,100 tests, depending on the commit). - Each task starts from a fresh copy of click at the commit just before the real fix was merged. Test dependencies come from click's own `uv.lock` (`uv sync --frozen --group tests`, pytest 9.0.2). - **Claude Code**, headless (`claude -p`), on its default model (Claude Opus 5.5) and default effort. Permissions are skipped inside the copy, there is no session persistence, and only project settings load (no personal CLAUDE.md, hooks, plugins or MCP servers). It is signed in with a Claude Max subscription. - **Codex**, headless (`codex exec --json`), on GPT-6.1 Sol, the model OpenAI recommends for complex coding, at its default reasoning effort. It runs in workspace-write mode inside the copy with no personal AGENTS.md, profiles or MCP servers. It is signed in with a ChatGPT subscription. - No router sits in between. The model each tool asks for is the model that answers. - Three runs of every task in each tool: 30 runs. ## Tasks and their checks Four tasks are real bugs that click's maintainers fixed between May and October 2026. The prompt is the original bug report. Where the merged fix's tests check behavior the report doesn't spell out, the prompt adds that behavior as a short acceptance note, worded the same for both tools. The check is the test files from the merged fix, which neither tool can see during the run. They are copied in afterwards. A run passes when those tests pass and the rest of click's test suite still passes. 1. **A parameter named `help`** ([#2819](https://github.com/pallets/click/issues/2819), fixed in [#3678](https://github.com/pallets/click/pull/3678), base `8c1a0a7`). An argument or option named `help` breaks parsing and the automatic `--help` option. The merged fix's tests also assert an internal name (`_click_default_help`) and two new warnings, none of which the report asks for. The check is therefore the maintainers' `test_param_named_help` and `test_option_reusing_help_flag`, copied into [`test_study_help.py`](test_study_help.py) with the internal-name assertion removed (3 of its 5 tests fail before the fix). 2. **`version_option` ignores `package_name`** ([#3125](https://github.com/pallets/click/issues/3125), fixed in [#3582](https://github.com/pallets/click/pull/3582), base `916883a`). Acceptance note: when the name is an import name that differs from its installed distribution name (as with `PIL` and Pillow), resolve it through `importlib.metadata.packages_distributions()`. When it maps to more than one distribution, raise an error that says "multiple installed distributions" and names each one. When it maps to none, keep the existing "not installed" error. Hidden tests: `tests/test_basic.py` (2 fail before the fix). 3. **`show_envvar` output for a list or an empty `envvar`** ([#3884](https://github.com/pallets/click/pull/3884), base `bb695aa`). The prompt is the pull request's before-and-after table for the error hint and the help screen. Acceptance note: when the context has an `auto_envvar_prefix`, an empty `envvar` falls back to the automatic name on the help screen (`[env var: WEATHER_UNITS]`). Hidden tests: `tests/test_options.py` (9 fail before the fix). 4. **Empty usage line from `HelpFormatter.write_usage`** ([#3360](https://github.com/pallets/click/issues/3360), fixed in [#3434](https://github.com/pallets/click/pull/3434), base `63274a7`). Acceptance note: with no arguments the line is the prefix and program name with no trailing space, including with a custom prefix, an indent or a styled prefix, and a command whose `options_metavar` is empty or `None` prints just `Usage: `. Hidden tests: `tests/test_formatting.py` (7 fail before the fix). One task is everyday work: 5. **Tests for one module.** From base `8c1a0a7`, where click has no `tests/test_textwrap.py`: "Write pytest tests for src/click/\_textwrap.py in tests/test_textwrap.py, and make sure they pass." Passes when the file exists, its tests pass, and they cover at least 90% of `_textwrap.py`. Every prompt ends with "Run the tests with `.venv/bin/pytest -q`." Each check was confirmed before any run: for tasks 1 to 4, the hidden tests fail on the base commit and pass on the merged fix, and the whole suite passes on the merged fix. ## What is recorded Per run: pass or fail with the check's details, wall time, input, output and cached tokens as each tool reports them, and the cost in dollars at each lab's list prices. Claude Code reports this as `total_cost_usd`. For Codex, the cost is computed from its reported token counts and OpenAI's published GPT-6.1 Sol prices on the day of the run. On a subscription no money changes hands, so both figures are API-equivalent prices. Each observation is one dated line of `results.jsonl`. ## Changes before the first counted run Two pilot runs of task 3, one in each tool, tested the runner. Claude Code passed. Codex failed only the two hidden tests for the `auto_envvar_prefix` fallback, which the prompt didn't mention. The acceptance note was added to task 3, every hidden test was compared with its prompt, and task 1's check was narrowed as described above. Both pilot runs are left out of the results. ## Rescoring note The first coverage check for task 5 parsed the wrong column: click's coverage settings measure branches, which adds two columns to the report, so every run read as 0% covered. Rescoring the six saved task 5 workspaces with `coverage report --include=src/click/_textwrap.py` changed all six from fail to pass (94% to 100% covered). No agent was rerun. In two Claude Code runs of task 5, Claude Code also changed one line of `_textwrap.py`, which the prompt didn't ask for. Its tests showed that the wrapper can return a line one character wider than the width asked for, which Python's `textwrap` does not. The check still counts those runs as passes, and `results.jsonl` marks them with `source_changed`. ## What it can't say Five tasks in one Python library. The bug fixes are recent, but a model may still have seen click's code or a similar fix in training. Three runs per cell show spread, not a tight average. Each tool runs its own default or recommended model, so this compares the tools as shipped, not the models in isolation, and a different model or effort setting in either tool would change the result.