Every run on the frozen contract — 10 pairs, mini_swe_agent_v2,
coop + git, Modal, 200 steps. Teacher is Qwen3.5-27B, student is Qwen3.5-9B.
The premise did not survive the data. We set out to distil a privileged teacher's coordination into a smaller student. CooperBench's coop setting turns out to collapse into solo — 7 of 10 pairs ship byte-identical patches — coordination is already near-perfect (9/10 mergeable trees), and 5 of 7 lost features are single-agent correctness. Every teacher brief that beat base contained the answer; every brief that didn't scored exactly base.
| Term | Means |
|---|---|
| student | Qwen3.5-9B, unmodified task brief |
| teacher | Qwen3.5-27B + context — negotiation instruction plus the menu of 10 protocols. Nothing task-specific. |
| context | That one injected block, 10,699 chars, 2 distinct texts across the 20 agents.
Recorded per-agent in coopertrain/opd/contexts/ceiling_negotiate.json. |
| OPD student | The student after LoRA r=32 distillation against the teacher, on
opd-clean (3,336 states, flash_10, train and test on the same 10 tasks) |
Everything else — gold-patch teacher, per-task workflows, shared-backlog, self-distillation — is deprecated. It stays in the table below as history, and none of it is a live configuration.
| Pair | Student (base 9B) | OPD student | Change |
|---|---|---|---|
| dottxt/1655 | conflicts · 0/0 + 0/0 |
identical · 84/84 + 80/80 |
real gain |
| jinja/1621 | clean · 46/47 + 46/46 |
clean · 47/47 + 46/46 |
real gain — f1 fixed |
| tiktoken/0 | identical · 1/1 + 1/1 |
identical · 1/1 + 0/1 |
real regression |
| huggingface/6252 | 44/44 + 44/44 | 44/44 + 44/44 | unchanged |
| Total | 2/10 · 7/20 | 3/10 · 8/20 | +1 pair, +1 feature |
The +1 is genuine, not a grader artifact. An earlier read called the jinja/1621 flip spurious because the peer's patch is 0 bytes while its suite scores 46/46 — but that artifact is present identically in the base run, so it did not cause the flip. The pair moved because feature 1 went 46/47 → 47/47, a real fix. dottxt moved from a merge conflict scoring nothing to identical patches passing both full suites.
Two caveats stand independently of the score. +1 pair at n=10 is inside noise (Wilson: 2/10 = 3–56%, 3/10 = 7–65%). And 21% of the OPD student's inter-agent messages carry mid-sentence CJK, Cyrillic or Devanagari against 0% for both the student and the teacher — with test runs collapsing 5.8% → 1.1% and writes 11.8% → 6.5%. The directional gain and the generation damage are both real and separate.
Context v1's diagnosis said the menu itself was routing pairs into a degenerate mode: its first decision branch sent any shared file to a single-writer protocol, on the strength of a claim — "two edits within three lines of each other conflict" — that is false. Measured: only adjacent edits conflict; one untouched line merges clean. v2 fixes the premise, defaults to partitioning, deletes "copy it byte-for-byte", and has the non-writer submit an empty patch and verify instead.
The behavioural fix worked completely and bought
nothing. Worse on the real metric: v2's four are dottxt, huggingface, click and
go_chi — and go_chi is a phantom whose two features both scored on
[no tests to run]. v1's four were all real. So 3 real passes against 4.
| Pair | v1 | v2 | What it shows |
|---|---|---|---|
openai_tiktoken/0 | PASS | fail | Both features add a keyword to the same def encode(...)
signature. v1's single-writer mode handled it; v2 partitioned and the merged tree
lost feature 8's parameter — TypeError: unexpected keyword
'return_repeated_pattern'. Partition-by-default overcorrected on the one case in ten
where the old advice was right. |
dspy/8394 | fail | fail | Falsifies the "+1 pair from perfect coordination" estimate. Merge went
conflicts → clean and f1 now passes 10/10 — coordination is fixed —
and it still fails on assert cache_usage1 is not None. |
go_chi/26 | fail | PASS* | Phantom. runner.sh runs go test -run TestPathValue ./...;
go test exits 0 when the filter matches nothing, and exit 0 is
recorded as 2 passed / 0 failed. |
Conclusion, now measured rather than argued: coordination on this subset is worth ~0 pairs, not +1. The remaining headroom is implementation correctness — on a subset where 4 defects are not knowable from anything the agent can read and 2 features are not gradeable at all.
| Run | Model | Injected context | Pairs | Features | Comparable? |
|---|---|---|---|---|---|
| Baselines — no injected context | |||||
flash10-qwen3-5-9b | Qwen3.5-9B | — | 2/10 | 7/20 | 08-10, version unrecorded |
| 27B base | Qwen3.5-27B | — | 4/10 | 12/20 | log dir overwritten; number only |
flash10-qwen3-5-122b | Qwen3.5-122B-A10B | — | 3/10 | 10/20 | 08-10, old schema |
flash10-qwen3-6-35b-a3b | Qwen3.6-35B-A3B | — | 2/10 | 7/20 | 08-09, old schema |
| Teacher ceilings — what a briefed teacher can reach | |||||
flash10-qwen3-5-27b | 27B | protocol + role + gold patch answer | 9/10 | 18/20 | 08-10, version unrecorded |
ceiling-protocol-only-27b | 27B | per-task workflows leaks files, lines, tests | 7/10 | 16/20 | ✓ 0.0.28 cohort |
ceiling-backlog-27b | 27B | shared-backlog protocol derivable | 4/10 | 13/20 | ✓ 0.0.28 cohort |
teacher (context v1) | 27B | menu of 10 protocols + decision rule derivable | 4/10 | 13/20 (11 real) | ✓ 0.0.28 cohort |
teacher-v2 (context v2) | 27B | partition-first menu, empty-patch reviewer, fixed merge premise derivable | 4/10 (3 real) | 14/20 (11 real) | ✓ 0.0.28 cohort |
ceiling-ctx-9b | 9B | protocol + role + gold patch answer | 5/10 | 11/20 | 08-10 |
ceiling-protocol-only-9b | 9B | per-task workflows leaks | 1/10 | 8/20 | ✓ 0.0.28 cohort |
| OPD students — LoRA r=32 on the 9B. Base to beat: 2/10, 7/20 | |||||
…-opd-r1 | 9B | teacher = 27B + gold patch | 0/10 | 3/20 | 08-10 |
…-opd-self | 9B | teacher = same 9B + gold patch | 0/10 | 1/20 | ✓ 0.0.28 cohort |
…-opd-proto | 9B | teacher = 27B + workflows | 1/10 | 4/20 | ✓ 0.0.28 cohort |
…-opd-negotiate | 9B | teacher = 27B + menu | 3/10 | 8/20 | ✓ 0.0.28 cohort |
Only the 0.0.28 cohort is version-matched. The 0.0.22→0.0.28 harness change moves behaviour more than most model deltas, so rows outside that cohort are not safely comparable to it. At n=10, 95% Wilson on 4/10 is 17–69% — two runs are separated only several pairs apart.
7 of 10 pairs end with byte-identical patches. One agent implements both
features; the other runs git checkout origin/agentN -- <file> and opens a PR.
skip-merge-identical then grades one agent's solo work against both suites. In 3 of
the 4 passes the colleague contributed zero source lines. Genuine two-agent split
work went 0/3.
9 of 10 pairs produced a mergeable tree; only one conflicted. All 6 failing pairs reached an explicit, concrete agreement before either agent's first edit, and all 6 picked an appropriate protocol. Protocol choice does not separate pass from fail — Protocol 1 was named by three passers and three failures.
The requirement exists only in the withheld tests.patch: go_chi's
OnHit/Params field names, typst's exact error string, pillow's unstated
ValueError, jinja-1559's comma grammar. No brief, teacher, or protocol reaches them.
Negotiation content and split adherence together account for 2 features, both in one pair.
Two independent generic protocols both landed on 4/10 — the same as unbriefed. The only briefs that beat base contained task-specific content: file names, line numbers, hidden-test literals, or the gold diff itself. There is no measured derivable teacher advantage on this subset.
3/10 · 8/20 against the student's 2/10 · 7/20 — genuine, not a grader artifact (see The live result above) — but +1 pair is inside noise at n=10. Alongside it: messages average 1093 chars against 388, 21% carry mid-sentence CJK, Cyrillic or Devanagari (student 0%, teacher 0%), test runs fall 5.8% → 1.1% and writes 11.8% → 6.5%. It copied the teacher's opening register without the stopping rule that makes that message the last coordination act — it names 0 protocols in 775 messages where the teacher names one in 22 of 85, and 7 of 18 agents hit the step limit inside message loops.
All 20 injected briefs assert that "two edits within three lines of each other conflict even when the edited lines are disjoint." Measured: only directly adjacent edits conflict — one untouched line between them merges clean. This drove 9 of 10 pairs into single-writer delegation, which destroys spec information. In pillow the implementer's brief contains zero occurrences of "lightness"; a 9B with a plain brief and no delegation got that feature right solo.
go_chi f1 scored passed: true on [no tests to run] — go.mod
says go 1.14, the hidden test carries //go:build go1.22. typst f8 passed because the
runner filters on string-* while its tests are named str-case-*. True
implementation-correct rate is 11/20, not 13/20. Any label read from
eval.json inherits this.
Perfect coordination saves exactly one of the six failing pairs (dspy — verified by replaying the merge: the union patch is clean and passes 8/8). Fixing every derivable implementation defect saves one more. The two levers are disjoint and each worth one pair. 6/10 is the hard cap for anything that does not disclose hidden-test content — and +1 pair is inside noise at n=10.
A pair needs both features, which squares the noise and makes 4 tasks permanently unwinnable — pillow and jinja-1559 each mix one derivable defect with one hidden-only defect.
Every deprecated configuration failed the same way: the teacher's edge lived outside the student's observation, so reverse KL had nothing on-support to move. The teacher we keep (27B + context) carries no task-specific content, so its advantage is in principle reachable — and its gap over the student is +4 to +6 features. The fix for the damage is a per-token weight on the existing loss, not a different teacher.
Action mix (writes ≥10%, tests ≥4%) and non-Latin-script rate are measurable on the training rollouts for free, and would have caught the last three students before the GPU spend.