When the model improves, the harness needs another evaluation
WIGTN Plugin made recurring product-development practices reusable in Codex, including requirements, verification and release decisions. As repository exploration, implementation planning, code changes and testing became native model behaviors, we needed to test whether the older workflow still added value.
Here, a harness means the task order, inspection steps and completion rules placed around the model. Rather than port the Claude Code workflow unchanged, we held the model and tasks constant and measured whether its additional machinery improved the outcome.
We changed the harness, not the model or the task
We held GPT-5.6 Sol constant so that changes in the model would not be confused with changes in the harness. The Bare condition used Codex without the plugin; the legacy-harness condition added the previous, heavier WIGTN implementation process.
Every run started from a clean workspace inside a task-specific Docker environment. We fixed the problem statement and base commit, then preflighted both the expected failure and the reference patch. A run counted as resolved only when the official tests covering the new and existing behavior passed.
Each trial’s evaluation packet recorded wall-clock seconds, output tokens and tool or command calls. The table reports the median across the four runs in each condition. The released packet does not expose finer-grained timer boundaries, and the ARM host used x86 Docker emulation, so wall time is treated as an environment-sensitive secondary metric rather than a portable latency claim.
The sample contains only two tasks and two trials per condition. It does not establish that every harness becomes less useful as model capability increases. It tests one narrower question: with a strong model held constant, did the legacy harness add value?
Codex explores the repository, makes the change and runs the relevant existing tests.
The same model and task receive the legacy WIGTN harness, including planning, verification and completion records.
Two fixed bug-report tasks from the Astropy and Pytest repositories.
Two trials per condition for each task, with execution order reversed between repetitions.
Both conditions solved every run, but the legacy harness cost more
Bare Codex and the legacy harness each resolved all four runs. Across the eight runs, no matched pair differed in resolution.
The additional cost came from separate planning, inspection and completion logging, even for small bug fixes. In this sample, those steps did not change the code outcome.
GPT-5.6 Sol paired comparison, overall median
| Metric | Bare | Legacy harness | Delta |
|---|---|---|---|
| Resolved | 4/4 | 4/4 | No difference |
| Wall time | 120.21 sec | 302.62 sec | +151.7% |
| Output tokens | 3,613 | 8,715.5 | +141.2% |
| Commands | 12.5 | 16.5 | +32.0% |
The clearest benefit was enforcing team rules, not improving code outcomes
The FeatureBench pilot produced no intact, repeatable coding-performance lift. In a separate GPT-5.5 PRD-format evaluation, the plugin improved compliance with WIGTN’s predefined structure from 0/3 to 3/3.
The PRD evaluation measured format compliance rather than overall document quality. It checked requirement IDs, unresolved decisions and acceptance criteria. The result supports a narrower benefit: applying team rules the model cannot infer on its own.
We invalidated the first successful Seaborn Plugin run in FeatureBench after discovering that it had read an installed copy of the same project outside the workspace. The incident made one principle explicit: passing tests is not enough to trust an evaluation. Access to reference implementations and hidden tests must be audited as part of the result.
Evidence by task type
| Evaluation | Observed result | What it supports |
|---|---|---|
| SWE-bench Verified | Bare 4/4, legacy harness 4/4 | No quality lift in the selected coding sample |
| FeatureBench pilot | 0 intact, repeatable lifts | No general coding-quality claim |
| PRD format (GPT-5.5) | Bare 0/3 → Plugin 3/3 | Reproducible team-defined document structure |
Keep routine implementation light. Add structure where failure is expensive.
After the evaluation, we shifted the plugin from directing implementation to selectively applying requirements, verification and release rules. Low-risk local changes no longer follow the same path as work that requires durable requirements and evidence.
Find the cause, make the smallest change, run focused tests and perform only the relevant checks. Small changes do not create a WorkGraph or requirements table.
For authorization, data structures, concurrency and migrations, connect requirements and risks to code locations and executed evidence.
Create one only when interruption recovery or a durable plan is explicitly required. When a source changes, dependent work and checks return to stale.
Keep verification state separate from authority to commit, push or open a PR. Recorded state never replaces the user’s current request.
We checked that the revised structure stayed out of routine coding
We kept the experiment that measured the legacy harness separate from validation of the current design. After the redesign, we independently tested routine-coding noninterference, WorkGraph state rules and fixed-input regression behavior.
The current evidence supports a narrower claim: in this development sample, the selective design did not turn routine coding into a full product workflow. We have not established that it produces better plans or better general coding outcomes than Bare Codex.
- 01Bare Codex and the current WIGTN configuration both passed 12/12 hidden tests across 12 synthetic Python, JavaScript and Ruby bug fixes.
- 02There were zero out-of-scope edits, unsolicited PRDs, WorkGraphs or release states, and zero losses of user-authored drafts.
- 03WorkGraphs in all 12 isolated repositories passed the final schema check; all 144 provenance, requirement, task, check, risk and protected-path fields passed as well.
- 04We ran 67 fixed-input regression tests covering WorkGraph creation, change propagation, format migration, completion rules and release authority.
Where the claim stops
- L01The primary comparison is a development sample limited to two SWE-bench Verified tasks and two trials per task.
- L02Because GPT-5.6 Sol was held constant, the comparison does not establish that every harness becomes less useful as model capability increases.
- L03We invalidated the FeatureBench run that accessed a reference implementation and observed no intact, repeatable lift in general coding quality.
- L04Some raw execution packets are not included in a durable public repository, so this is not a complete third-party reproduction package.