Report index

Running a harness on frontier models, part 1: Claude Code

Seven months of a Claude Code plugin, measured at each commit. The instruction surface peaked at 49,275 lines and is 19,912 today, but it did not fall in a straight line: it grows back between frontier releases and gets cut at each one. What never got cut was the enforcement a model cannot skip.

David Cho 조현우
AI Product Engineer, WIGTN
Claude Code
01Problem

Do not make every engineering decision in one context

We started from a recurring failure mode: one coding session mixes requirements, architecture, implementation and review until early assumptions become invisible. Later reviewers then inherit decisions they never independently checked.

WIGTN Plugin separates those responsibilities into named stages with explicit artifacts. Parallelism is useful only when it creates independent scrutiny and bounded ownership, not when it merely increases the number of agents on screen.

02Workflow

Turn every handoff into an artifact

DefineProduce a PRD

Clarify goals, actors, constraints, acceptance criteria and phased scope.

ChallengeReview in parallel

Completeness, feasibility, security and consistency reviewers inspect the same artifact independently.

DesignInspect before deciding

Architecture choices follow repository scans, contracts and existing conventions.

BuildAssign bounded ownership

Backend, frontend, AI and operations work is split by file and interface boundaries.

ReviewApply quality gates

Readability, performance, testability, best practices and security receive separate checks.

ReleaseStop or publish deliberately

Failed gates block release; successful runs prepare an intentional commit and pull request.

WIGTN Plugin six-stage delivery workflow
FIG.The released workflow moves from PRD definition through independent review, bounded implementation and an explicit release decision.
03Coordination

Three layers keep shared decisions visible

Workflow memory model

LayerLifetimeCarries
Repository memoryPersistentConventions, commands, architectural rules
Shared contextOne coordinated buildContracts, file ownership, cross-team decisions
Task stateOne work itemProgress, blockers, verification evidence
04Origin

We shipped a marketplace of stack plugins and then took it apart

The repository opened on 12 January 2026 and took 72 commits in its first month. The shape was wrong. Work was divided by technology, with separate plugins for frontend, backend, mobile and AI development, each carrying its own skills for the frameworks in that stack.

On 14 February those five became one. The division had been by technology, but the work does not divide that way. A React change and a FastAPI change need the same sequence of definition, independent review, bounded implementation and a release decision. They do not need different plugins. The merge removed 6,321 lines and added 3,011.

Three weeks later, two commits on consecutive days removed a further 30,670. The first cut 54 components to 32 and its message names the reason: twenty-two of the removed skills documented React, Tailwind and Jest, which the model already knew. Skills went from 29 to 7 in that pass and to 3 in the one that followed; they stand at 7 again today, after five months of regrowth. What survives at any count is project-specific information only.

Agent, command and skill counts in February 2026, March 2026 and today
FIG.Directory listings at each commit. The agent row barely moves across all three snapshots while the skill row collapses from 29 to 3 and stands at 7 again today, which is the shape of the decision: roles were kept, documentation was not. The middle column is the second March pass, three days after the cut described above.
05Model upgrades

Every upgrade is a cut, and every quiet month undoes part of it

Checking out each milestone commit and counting the instruction text gives a shape we did not expect to have to admit. The surface does fall, from 49,275 lines to 19,912, but not in a straight line. It comes down hard at a model release and climbs back through the months in between, because that is when features get added. The March cut took 58% out in three days; the twelve weeks that followed put 24% back.

The cuts themselves are specific. In June, adapting to Opus 4.8 meant removing what the commit calls expressive over-harness, with no change to behaviour or logic: eight forced xhigh effort settings, and the fabricated numbers we had written into our own prompts, including hardcoded durations, speedup claims and example scores. In August, the Opus 5 pass kept the contracts and cut the rest.

Part 2 of this series put a number on the cost of leaving that growth alone. On two SWE-bench Verified tasks, GPT-5.6 Sol resolved 4 of 4 runs with the legacy harness and 4 of 4 without it, while the harness raised median wall time by 151.7% and output tokens by 141.2%. We published that against our own product, and it is what set the rule below.

Instruction surface in lines from February to August 2026, falling in steps with regrowth between them
FIG.Every .md file under agents, commands and skills, counted with wc -l at each milestone commit. The series is not smoothed. The regrowth between March and June is design styles, screen-spec, the diagram skill and the presentation generator being added.

Instruction surface immediately before and after each cut

DateTriggerBeforeAfterChange
2026.03.07v3 streamline49,27527,918−43%
2026.03.09v3 streamline, second pass27,91820,361−27%
2026.06.26Opus 4.825,30823,795−6%
2026.07.09Prompt slim24,36022,584−7%
2026.08.04Opus 522,74819,754−13%
06Enforcement

The prompts shrank; the enforcement grew

Deleting instructions is only safe if the things that must not be skipped stop being instructions. A prompt is advice a model can quietly drop when the context fills. A hook is not. Between July and August the plugin moved each rule that mattered out of the prompt and into something mechanical.

One of those replacements fixed nondeterminism in our own tooling. The gate used to sum findings into a score out of 100, and the same diff could score 78 on one run and 85 on the next, which decided whether the commit was blocked. The score is gone. A rollup counts critical and major findings instead, so the same findings now always produce the same decision.

Two independent gates between a commit and the repository
FIG.Traced from hooks/gate.sh. The gates are independent: the objective checks run on every commit and never read the message, while the review record is demanded only from commits that claim to have passed a review. The emergency path removes the claim, not the checks.

What replaced prompt instructions

Was written as an instructionIs now
Run the quality gate before committingA pre-commit hook that blocks the commit when no gate record exists
Report the review honestlyA checks script the hook runs itself, so the exit code is not written by the model
Do not reuse an old reviewA 30-minute freshness window on the gate record
Weigh security findings heavilySecurity-critical blocks the commit regardless of any other finding
Keep the checks enabledDeleting the script regenerates it; opting out requires writing down a reason
07Evaluation plan

The benchmark exists as a protocol, not yet as a result

The local evaluation harness defines four areas: long-horizon coding, code review, ambiguous PRD interpretation and frontend design. It fixes prompts, tools and effort, repeats each run three times and uses blind graders.

Only one comparison row currently contains results. Publishing a multi-agent advantage now would turn package counts into a performance claim they cannot support.

  • 01Report median and range across three runs, not the best run.
  • 02Score hidden-test correctness before stylistic preference.
  • 03Normalize cost as dollars per completed task or detected defect.
  • 04Release prompts, run logs, grader rubrics and failure cases together.
08Limitations

Where the claim stops

  • L01Agent and skill counts describe the package surface, not developer productivity.
  • L02The surface figures count lines of Markdown under agents, commands and skills. They measure how much instruction text ships, not how much of it the model reads on a given turn, and a smaller file is not evidence that what remains is correct.
  • L03The reductions are decisions this team made about its own tool, not a controlled comparison of harness sizes. The one controlled result we have is in part 2, and it covers two tasks on one model.
  • L04Operational timing observations are not a controlled single-agent comparison.
  • L05The current benchmark matrix is incomplete and must not be summarized as a result.