Qwen3.5-9B student, Qwen3.5-27B teacher. Two agents build separate features of one
repository, must not collide, and are scored on the merged tree. Every teacher context that
beats base contains the answer; every context whose advantage the student could derive scores
exactly base. That is why on-policy distillation has nothing to carry here. Newest first —
read bottom-up for the chronology.
27B base
4 / 10
nothing injected
+ any generic protocol
4 / 10
no gain, twice
+ the answer
9 / 10
gold patch + workflow
9B base
2 / 10
the student
OPD runs beating base
0 / 3
0/10, 0/10, 1/10
2026-08-11
Related work
Pedagogical RL predicts this failure, and names the mechanism
a privileged teacher produces trajectories that satisfy the reward "while still being highly unlikely under the student and thus not teachable, πθ(τ|x) ≪ 1"
gold-patch teachers → 0/10 twice, across a 27B teacher and a same-weights teacher
teacher recovery degrades as it is forced to continue student-generated prefixes (their Analysis A)
we score the teacher at a median 16.4k-token prefix — far past the range they tested
average NLL hides one catastrophic token; their spike penalty separates 7.57 from 0.63 where mean NLL reads 1.03 vs 0.49
every "KL is falling" claim here is a mean over ~90 output tokens; max/mean on the current run is 4×
What does not transfer: the cost. Their method GRPO-trains a teacher against a
spike-aware pedagogy reward, using ~2K rollouts. A CooperBench rollout is a pair of agents in
Modal sandboxes for 20–40 minutes, so one GRPO step over 10 tasks × 8 samples is ~3.5 hours of
wall clock. Pedagogical RL as written is not affordable here.
What does transfer, and is nearly free. Their §2 lists rejection sampling as the first
approximation to nearest-success sampling, and we have already paid for it: 58 trajectories
from pairs the grader marked as passing, across five ceiling runs. Combined with their §3B
surprisal gate — wt = σ(κ(log πS(τt) − γ)), full
weight where the student finds a token plausible, ~0 where it is shocking — that is an
affordable experiment needing no teacher RL and no new sampling.
But it is not OPD any more. The defining property of on-policy distillation is
that the states come from the student's own policy. Training on teacher-generated trajectories
is off-policy distillation with a learnability gate — the paper lists "direct off-policy" as a
baseline and describes its own method as "lightly off-policy". So the paper is not a fix
for OPD; its thesis is that on-policy sampling is the bottleneck. Adopting it means abandoning
OPD, which is a decision about goals rather than methods.
Two caveats on the cheap version. The 58 trajectories come from 29 pairs
across 10 tasks — small for SFT, with real overfitting risk. And the best 18 were generated with
the gold patch in context, so they contain shortcut steps that only make sense with the answer
visible; the gate is designed to strip exactly those, which makes it a fair test of the gate but
not clean supervision on its own.
run log
0 for 3
OPD training runs
Split by data pipeline, not by hyperparameters — every v1 run trained on a
corpus collected outside the eval contract, so their numbers cannot be compared with anything
collected since.
v1 — broken collection
v2 — contract-matched
step limit
0 — agents ran to 480 steps
200
serve
H100:1
H100:4
concurrency
8
10
task spread in corpus
78×
2.4×
runs
rounds 0, 2, 3, 4
opd-self-ceiling
best result
0/10 (base is 2/10)
0/10, 1/20 features
Each run trains a fresh LoRA from base — no run resumes another's
adapter, in either era. What v1 shares is the corpus, not the weights.
Run
Teacher's extra context
Brief tok
States
Eval
3 — ceiling
protocol + role + the benchmark's gold patch
3,150
4,173
0/10
2 — masked
peer spec, masked at 0.05 (15.9% of positions)
~200
1,500
0/12*
0 — first
peer spec
~196
449
not run
base 9B
—
—
—
2/10
Student is Qwen3.5-9B + LoRA r=32 in every run, trained fresh from base — no
run resumes another's adapter. * round 2 lost 4 pairs, so 0/12 counts only what ran and is not
comparable with the 0/20 figures; its logs have been deleted and it cannot be re-verified.
Why OPD is not working yet
Three runs have been evaluated and all three landed at or below the 2/10 base. What the
evidence rules in, and what it rules out:
Finding
Evidence
1
A teacher conditioned on the gold patch collapses the student
Two independent configurations — 27B teacher (round 3) and same-weights teacher (opd-self) — differ in teacher size, corpus, collection settings and context length, and produce the same result. The shared factor is the answer being in the teacher's prompt.
2
The student learns the answer-holder's behaviour without the answer
Against base: messaging +154%, --wait +156%, git inspection +59%, but test runs −37% and file writes −16%. A teacher holding the solution never needs to search or verify, so its distribution favours coordinating and publishing over implementing.
3
Coordination is not the binding constraint for this student
opd-self has the best coordination of any run — 9 clean merges, 1 conflict, 54% of messages answered within five turns — and scores 0/10. The 5/10 teacher has the worst merge record (6 conflicts, 38% answered) and wins anyway, because it writes correct code.
4
Healthy loss telemetry does not imply a better policy
KL 0.093 → 0.013 and agreement 0.945 → 0.950 measure how well the student matched a distribution it cannot reach from its own observations.
5
Single-epoch training may be too few passes
Round 0 — 449 states, 3 epochs — is the only run that ever showed sustained learning (KL 0.084 → 0.046). Every run since sees each decisive state exactly once.
What this rules out as the fix. Up-weighting coordination tokens: the student
already over-coordinates relative to base and is worse for it. Masking to the decisive positions:
round 2 did that and the model unanchored (drift 0.428) and stopped submitting. The deficit is
code correctness, not collaboration — which is why the next teacher variant carries the
feature's tests rather than its patch. Tests give the teacher a concrete target to
iterate against, so its advantage appears as behaviour the student can imitate — run the suite,
read the failure, fix, publish when green — instead of text the student can never derive.
Teacher's privileged context
9B
27B
none (base)
2/10 · 7/20
4/10 · 12/20
shared-backlog — generic procedure, no task content
—
4/10 · 13/20
prescriptive protocol — task-specific, no answer text
1/10 · 8/20
7/10 · 16/20
protocol + gold patch
5/10 → distils to 0/10
9/10 → distils to 0/10
27B protocol-only measured over 9 of 10 pairs. The teacher is chosen from
this matrix rather than assumed: a variant is worth distilling only if it beats base and
its advantage is something the student could execute without privileged input.
An anchored loss was proposed and abandoned (not run)
Every run so far minimised KL(student ‖ teacher) alone. That applies an unbounded
per-state pull with nothing holding the student to behaviours it already performs competently,
which is what the overshoot above looks like. The objective now carries a second term:
loss = KL(student ‖ teacher) + 0.5 · KL(student ‖ its own base)
The anchor is the student's own starting policy — adapters disabled on the same
weights, evaluated on the same top-k support — not a hand-picked prior. It is the mirror of what
round 2 removed: that run masked away the anchor entirely and drifted to 0.428.
Config
Value
held identical to opd-proto
27B protocol-only teacher, same 3,336-state corpus, lr 5e-5, r=32, max_len 16,384, no masking, no comm weighting
changed
anchor_weight = 0.5, epochs 3 → 2 (budget)
Falsifiable prediction, stated before the data. The anchor works if the trained model's
action mix stays near base instead of collapsing:
Metric
base
opd-proto
anchor works if
write files
11.8%
6.2%
≥ ~10%
run tests
5.8%
0.8%
≥ ~4%
git inspect
18.5%
33.5%
≤ ~22%
median patch
3,570 B
1,326 B
≥ ~3,000 B
pairs
2/10
1/10
> 2/10
This was never run. The mechanism it tests — unbounded per-state pull — is an
inference from the action-mix numbers, not an established cause, and the coefficient 0.5 was a
guess rather than a tuned or cited value. It was dropped in favour of measuring teachers whose
advantage is actually derivable by the student, which turned out to be the more decisive
question.
The loss curve is the same whatever the teacher is worth
Run
Teacher
Teacher's ceiling
KL by third
Eval
opd-self
same 9B weights + gold patch
5/10
0.0461 → 0.0373 → 0.0325
0/10
opd-proto
27B + per-task workflows
7/10
0.0858 → 0.0587 → 0.0426
1/10
opd-negotiate
27B + 10-protocol menu, fully derivable
4/10
0.0878 → 0.0579 → 0.0426
3/10
Three teachers whose ceilings are 5/10, 7/10 and 4/10 produce near-identical training
curves. Every one descends cleanly with agreement rising, and the two that finished scored
at or below the 2/10 base. Mean KL measures how well the student matched a distribution; it
carries no information about whether that distribution was worth matching. The max in-window KL
on the negotiate run is 0.2354 against a mean of 0.058 — a 4× tail the average hides, which is
the concrete form of the spike problem described in
Pedagogical RL.
Epochs matter — single-epoch runs were never learning
Run
Epochs
KL by epoch
Agreement
round 3
1
0.100 → 0.084 → 0.081 → 0.072 (quarters)
0.902 → 0.919
opd-self
1
0.049 → 0.040 → 0.032 → 0.034 (quarters)
0.946 → 0.948
opd-proto
3
0.0858 → 0.0587 → 0.0426
0.924 → 0.934 → 0.951
The three-epoch run is the first since round 0 to show a clean monotone descent with
agreement rising every epoch. Single-epoch runs see each decisive state exactly once and their
curves are dominated by which states happen to be hard, not by learning. It did not rescue the
benchmark score — but it removes "too few passes" as an explanation for the other four.
Four runs, and the same signature every time: the student stops writing code
Run
Teacher's advantage
Epochs
Pairs
Features
Median patch
base 9B
—
—
2/10
7/20
3,570 B
round 3
27B + gold patch
1
0/10
3/20
—
opd-self
same weights + gold patch
1
0/10
1/20
1,736 B
opd-proto
27B + protocol, no answer
3
1/10
4/20
1,326 B
opd-proto had every known defect fixed — a teacher whose advantage is behaviour
rather than knowledge (7/10 on protocol text alone), three epochs with monotone KL descent
(0.086 → 0.059 → 0.043), a contract-matched corpus, a verified harness, 109,825 loss positions.
It still lands below base.
Four students, and the effect that reproduces every time
Run
Teacher
Teacher's ceiling
Pairs
Features
base 9B
—
—
2/10
7/20
opd-r1
27B + gold patch
9/10
0/10
3/20
opd-self
9B + gold patch
5/10
0/10
1/20
opd-proto
27B + per-task workflows
7/10
1/10
4/20
opd-negotiate
27B + protocol menu
4/10
3/10
8/20
3/10 is not evidence the method works. It is within the measured noise floor — two
re-runs of identical weights moved feature pass 30% → 17%, so an intervention must clear roughly
two pairs to be distinguishable, and this is a two-pair gap from opd-proto's 1/10.
What actually reproduces: writing and testing are suppressed
Share of executed commands
base 9B 2/10
opd-proto 1/10
opd-negotiate 3/10
coordinate (send_message / --wait)
2.8%
8.1%
26.2%
explore
60.8%
53.6%
45.5%
write files
11.8%
6.2%
6.5%
run tests
5.8%
0.8%
1.3%
git inspect
18.5%
33.5%
25.4%
git publish
3.0%
2.8%
2.9%
Distillation reliably teaches the coordination half of the teacher's behaviour and
reliably suppresses the implementation half. Every student writes files at ~6% against the
base model's 11.8% and tests at ~1% against 5.8%. The negotiate student spends a quarter of
every command it issues on messages. Two runs with near-identical mixes — write 6.2% vs
6.5%, test 0.8% vs 1.3% — scored 1/10 and 3/10, which is what a noise floor looks like.
On a benchmark scored by whether tests pass, a policy that writes half as much code and tests
a fifth as often cannot win, however well it coordinates.
The student adopts the teacher's action mix, then overshoots it
Share of executed commands
base 9B
opd-proto
27B teacher
explore (cat/ls/grep/find)
60.8%
53.6%
44.5%
toward teacher
coordinate (send_message/--wait)
2.8%
8.1%
9.5%
toward teacher
git inspect
18.5%
33.5%
21.3%
overshot past it
write files
11.8%
6.2%
9.2%
overshot past it
run tests
5.8%
0.8%
6.1%
collapsed — teacher tests MORE
git publish
3.0%
2.8%
5.4%
no movement
This rules out the intuitive explanation. The teacher is not a model that skips
exploration because it knows the answer — it runs tests at 6.1%, more than the student
did, and writes files at 9.2%. The student nonetheless learned to test 0.8% of the time and
inspect git a third of the time. It moved in the teacher's direction on four categories
and past the teacher on the three that decide whether working code gets produced.
So the defect is in the loss, not the brief. Reverse KL on the student's own trajectory
applies an unbounded per-state pull with nothing anchoring the student to behaviours it was
already performing competently, and three epochs of that compounds past the target. Round 2
failed for the mirror-image reason: masking removed the anchor entirely and the model drifted to
0.428 and stopped submitting. A model writing 6% of the time and testing 0.8% of the time cannot
produce working code, which is what 4/20 features and a 1,326-byte median patch look like.
A generic coordination procedure is worth nothing
The shared-backlog protocol asks the pair to decompose the work into atomic items over
send_message, claim them explicitly, keep one owner per contested file, and
re-broadcast the board on every change. It contains no task-specific content at all — two
distinct texts across all twenty agents. It scores exactly what the unbriefed model scores.
27B run
Sends
With CLAIM
Numbered boards
Score
shared-backlog
190
37
64
4/10
prescriptive protocol
112
0
31
7/10
gold patch
92
2
32
9/10
The agents complied and it did not help. They built boards and made explicit claims —
compliance was not the failure. The relationship is inverse: 190 sends → 4/10, 112 →
7/10, 92 → 9/10, same model, same tasks. That is the third independent confirmation that
coordination volume is not the lever.
Better coordination, measured properly, still does not predict the score
Message count is the wrong metric, so here is coordination quality: what the merges
actually did, and how often a message got an answer within five turns.
Run
Clean
Identical
Conflicts
Sends
Answered ≤5 turns
Score
base 9B
7
1
1
87
26%
2/10
9B + gold-patch ctx
4
0
6
55
38%
5/10
opd-self
9
0
1
221
54%
0/10
The best-coordinated run scores zero and the worst-coordinated of the three scores highest.opd-self has the most clean merges, the fewest conflicts and double base's answer
rate; the 5/10 model has six conflicts and wins anyway, on code correctness. Its merges are clean
partly because there is less code to collide — median patch 1,736 B against base's
3,570 B.
What the +3 actually buys is privileged knowledge of the collision — which file both
agents touch, which region each takes, which signature to converge on. An agent sees only its own
feature spec, so it cannot derive that, which makes it structurally the same problem as the gold
patch: a teacher advantage the student has no route to.
This kills self-distillation and names the teacher. A prescriptive protocol — naming
exact files, signatures and error strings — moves the 9B from 2/10 to 1/10. It cannot
execute a protocol at all, so its 5/10 with the gold patch came from the code, not the
coordination, and there is no behavioural gap in it worth teaching. The 27B reaches 7/10
of its 9/10 ceiling on protocol text alone, with no answer anywhere in the prompt — an advantage
that is behaviour rather than knowledge, which is exactly the property the two collapsed runs
lacked.
Both runs optimise cleanly and both collapse
Reverse KL, 15-step mean, x-axis normalised because the runs differ in
length (345 vs 521 steps). Both descend monotonically. Neither produced a model that beats base.
Quarter of run
opd-self KL
opd-self agree
round 3 KL
round 3 agree
first
0.0490
0.9462
0.1002
0.9017
second
0.0397
0.9434
0.0838
0.9125
third
0.0317
0.9526
0.0813
0.9196
fourth
0.0342
0.9478
0.0724
0.9192
benchmark
0/10, 1/20
0/10, 3/20
The loss curve carries no information about whether the run helped. Both descend
~30% with agreement flat-to-rising, and both produce a model worse than the one they started
from. opd-self starts at half round 3's KL — teacher and student share weights, so only the
prompt differs — and still ends in the same place.
v2 · self-distillation from the ceiling teacher 0/10
Config
Value
student
Qwen3.5-9B + LoRA r=32 α=64, q/k/v/o + gate/up/down, no brief
teacher
the same 9B weights, adapters disabled — the only difference is the prompt
teacher's extra
ceiling_v1.json: protocol, role, decisive step, endgame, and the benchmark's gold patch. Mean 3,219 tok, 20 distinct
why this teacher
it is a measured configuration — 9B+context scores 5/10 against the student's 2/10, and every pair in that gap is one these exact weights have passed
The teacher had the answer in its prompt, so it never needed to search or test. Its
token distribution favours coordinating, checking git state and publishing over exploring the
repository and running the suite. The student matched that distribution closely — KL 0.013,
agreement 0.950 — without having the answer. It now behaves like an agent that already
knows the solution while not knowing it: 2.5× the messaging, 59% more git inspection, 37% fewer
tests, 16% fewer edits.
Two independent configurations now agree. Round 3 (27B teacher) and this run (same-weights
teacher) differ in teacher size, corpus, collection settings and context length, and produce the
same collapse. The one thing they share is a teacher conditioned on the gold patch.
The first attempt at this eval scored 0/10 for a reason that had nothing to do
with the model, and it took a nine-agent forensic pass to find. Two cooperbench installs
existed: eval_e2e.py invoked ["cooperbench", …], PATH resolved that to
miniconda's 0.0.22, while every version check in the script imported .venv's
0.0.28. The two harnesses disagree on everything that decides a score — 0.0.22 names the
shared daemon team, leaves the real GitHub as origin, installs no
gh shim and grades cat patch.txt; 0.0.28 uses origin,
installs the shim and grades the agent's PR. All 19 agents hit gh: command not found,
pushed correct commits to team/agentN, and were graded as empty submissions. The
stale copy is uninstalled and the script now resolves its CLI next to sys.executable.
v1 · round 3 — the only v1 run with a full diagnosis
Config
Value
student / teacher
Qwen3.5-9B + LoRA r=32 α=64 / Qwen3.5-27B
teacher's extra
frozen ceiling block, 3,150 tok, 19 distinct — protocol, role, decisive step, endgame, and the gold feature.patch verbatim
data
8,998 states from the 9B's own flash_10 rollouts
trained on
4,173 states (max_len 16,384 keeps 46%) · 521 steps · 89,355 loss positions
loss
reverse KL, top-20, no masking · lr 5e-5 · 1 epoch · accum 8
curve
0.243 → 0.055, agreement 0.913, drift 0.142
base
OPD r3
pairs
2/10
0/10
features
7/20
3/20
merge conflicts
1
5
empty patches
4/20
2/20
send_message calls
80
88
Conflicts rose and features fell. Messaging is essentially unchanged — 80 against 88 executed
send_message commands.
v1 · the role ablation (recovered from the adapter volume)
Four arms, same states, same 14,037 loss positions — a controlled comparison of what the
role half of the brief is worth.
Role arm
KL first
KL last
KL mean
Agreement
context-only (no role at all)
0.1283
0.0897
0.1090
0.888 → 0.903
announce
0.1303
0.0921
0.1112
0.884 → 0.907
owner
0.1367
0.0933
0.1150
0.885 → 0.903
negotiate
0.1359
0.0951
0.1155
0.880 → 0.904
The four arms are indistinguishable. The spread across them is 0.0065 — smaller than
the ~0.04 drop each arm achieves within itself — and context-only, which specifies no
role whatsoever, sits at the bottom of the range. On this measurement the role protocol adds
nothing the peer's file context does not already supply.
Trained on the v1 corpus, so the levels are not comparable with anything
collected since; the between-arm comparison is internally valid because all four shared states
and token counts exactly. Recovered from opd-outputs — these numbers had never been
written down.
What is actually established, and what is not
Claim
Status
Three runs have all scored at or below base
measured
Reverse KL falls and top-1 agreement rises in every run
measured
The merge/push path is guarded (non-zero adapter, coherence probe, no unexpected or missing keys) and round 3 passed all four
verified in code
Why the trained policy is worse
not established
A defect large enough to invalidate both evaluated runs. The rollout collector diverged
from the frozen eval contract on three settings — step_limit 0 against 200 (agents
ran to 480 steps), concurrency 8 against 10, and serving on 1 GPU against 4, which changes
the numerics through tensor parallelism. Every state both runs trained on was drawn from a
differently-served model running far past the evaluated step budget. Now aligned; the corpus is
being recollected before anything else is concluded.
Earlier drafts of this page carried three explanations for the regression —
a 3.3× messaging increase, a corpus anti-correlated with headroom, and compaction dropping the
brief. All three came from ad-hoc extraction rather than the production code path and are false.
They have been removed rather than corrected in place.
2026-08-11
MeasuringDesign
Negotiate-your-own: teach the choice, not the answer
Every agent on every task gets the identical prompt — a menu of ten protocols
and a decision rule — and must agree one with its peer before editing. Two distinct texts across
all twenty agents, so the only thing a student can extract is how to choose.
Teacher
Answer-free?
Distinct texts
What a student could learn
27B
per-task workflows
❌ leaks code, line numbers, hidden-test content
20
this task's answer
7/10
shared-backlog
✅
2
one fixed procedure
4/10 — no gain
mixed assignment
✅
16
seven fixed behaviours
not run
negotiate-your-own
✅
2
how to select a protocol
measuring
Why uniformity is the point. A per-task assignment teaches seven fixed behaviours and
lets the student memorise which task got which. One prompt everywhere removes that: the split
still differs per task, but it is derived by the pair from what they tell each other,
so the only transferable content is the derivation.
The decision rule the agents are given
Ten protocols to choose from — lead/reviewer, handoff, spec owner/builder, claim registry,
partition by region, sequential shifts, implementer/integrator, two-authors-one-gate,
merge-forward, shared backlog — and the questions that pick between them:
Do you both need the same file, with edits landing near each other?
→ 1, 2, 6 or 7 — exactly one of you writes that file. A split will not merge.
Same file, far apart, untouched code between?
→ 4 or 5 — but agree the exact boundary in a message first.
Share a type or signature rather than a region?
→ 3 — publish it before either of you writes against it.
Is one of you going to finish much earlier?
→ 6 or 7 — let the later one integrate.
Both features touching several files each?
→ 10 to track it, 8 so someone owns the final go/no-go.
Cannot tell yet?
→ Ask "which files and functions does your feature touch?" and decide once you know.
Do not start editing to find out.
The last line targets a measured failure rather than a guess: agents begin editing before
they know what their peer is touching, and the collision is fixed by then.
Result: 4/10 pairs, 13/20 features — and 7 identical merges
The identical merges are the prompt working, and the prompt was wrong. Four of the ten
menu entries tell one agent to implement both features — "one of you owns every shared
file and implements BOTH features in it" — and the decision rule routes there by default:
"if your edits land near each other, a split will not merge: use 1, 2, 6 or 7." Nine of
ten tasks have overlapping hunks, so almost every pair was steered into duplication.
27B run
Identical
Clean
Conflict
Score
gold patch + workflow
2
7
1
9/10
workflow only
2
7
0
7/10
base
2
7
1
4/10
shared-backlog
0
10
0
4/10
negotiate menu
7
2
1
4/10
Every run above base has the same profile: 2 identical, 7 clean. The only run that
inverted it is the one steered toward duplication, and it sits at base. An identical patch means
both agents implemented both features — the second agent's entire 200-step budget produced
nothing new, which is the opposite of what two agents are for. Clean merges are what division of
labour looks like, and that is what the winning configurations do.
The error is traceable: an early finding that identical was "the
only reliable path to a pass" came from two genuinely single-file tasks (typst, tiktoken).
Generalising it into a decision rule pushed nine tasks into duplication. Note that
shared-backlog already partitions by default — 0 identical, 10 clean — and still
scores 4/10, so fixing this is necessary but not sufficient.
27B run
Identical
Clean
Conflicts
Score
gold patch
2
7
1
9/10
per-task workflows
2
7
0
7/10
shared-backlog
0
10
0
4/10
negotiate-your-own
7
2
1
4/10
The protocol worked and the score did not move. Seven identical merges out of ten —
more than triple any other run, including the 9/10 ceiling. The pairs negotiated and converged
on a single shared artifact, which is precisely the outcome this whole line of work was aiming
at. shared-backlog did the same thing by a different route: ten clean merges, zero
conflicts. Both solved coordination on their own terms; neither bought a single pair.
The two teachers that do score higher have worse merge profiles — 2 identical, and a
conflict apiece — and win on the content of what gets written.
This inverts the thesis this page was built on. The oracle ablation showed
that given correct code, coordination decides, and that still holds. What is now
measured is the converse: given perfect coordination, wrong code still fails — and for
these models the code side binds first. Convergence is achievable with a prompt; correctness is
not. That is why every teacher whose advantage is procedural scores exactly what the unbriefed
model scores, and why there has been nothing transferable for OPD to carry.
2026-08-11
DesignProtocols
Protocols: eleven working relationships, none of which name a file
A protocol says who holds which job. A workflow says what to type. Only the
first can generalise to a task the pair has never seen, because only the first is derivable
from an agent's own id.
#
Protocol
agent1
agent2
1
lead / reviewer
senior engineer — implements both features, owns every shared file
reviewer — writes no feature code, reviews the diff, runs the merge check, tests the merged tree, signs off
2
implementer / integrator
writes only its own feature, publishes early and often
owns integration — waits, merges, resolves, runs both suites, publishes the union
3
spec owner / builder
writes the contract first, publishes it before implementing
builds against the published contract, never redefines it
4
driver / navigator
drives — makes edits, names the file before touching it
navigates — never edits, watches the peer's branch, warns on collision course
5
two authors, one gate
writes its own feature
same, plus is the release gate — nobody publishes until it has run the merged-tree tests
6
sequential shifts
works first, publishes, stops touching shared files
starts after, builds on top, submits the union
7
test-first pair
writes tests for both features first and publishes them
implements against those tests
8
claim registry
announces every file claim before editing; first claim wins
same rules, checks the registry before touching anything
9
merge-forward
rebases onto the peer's latest published branch before every commit
same — collisions surface continuously, not at the end
10
negotiated split
no assigned role — the pair must agree a split in the first ~10 steps, state it explicitly, and hold to it
11
shared-backlog
decompose into atomic items over send_message, claim them explicitly, one owner per contested file, re-broadcast on every change
#10 and #11 are the ones that teach a procedure rather than an answer. Every other
entry still presumes someone knows the collision shape in advance. #10 hands the pair a method
for producing a protocol — inspect, propose, converge, commit — and #11 gives that
negotiation a concrete artifact to argue over. Those are the two that could transfer to an
unseen task.
The one measured so far produced zero gain. shared-backlog on the 27B:
4/10 pairs, 13/20 features — identical to the unbriefed model. The agents complied fully
(190 sends, 37 explicit CLAIMs, 64 numbered boards), so this is not a
compliance failure. Meanwhile every teacher that does beat base contains the answer:
per-task workflows 7/10, gold patch 9/10. No teacher whose advantage is purely procedural has
yet produced a gain over base, which is the central unsolved problem for distillation here —
there is nothing transferable to distil.
No harness support is assumed or needed. CooperBench's shared task list is
team-mode only (team_poller is absent for solo/coop) and the team
remote is read-only to agents until submit, so a board cannot live in a file. It lives in
messages, which is the only live channel the pair has.
2026-08-11
InfraFixed
Three silent faults that produced wrong numbers
Each ran to completion, reported success, and gave an answer that was not
about the model.
Fault
What it looked like
What it actually was
Two cooperbench installs
a clean 0/10 with every patch empty
eval_e2e.py ran ["cooperbench", …], PATH resolved it to miniconda's 0.0.22 while every version check imported .venv's 0.0.28. 0.0.22 names the daemon team, leaves real GitHub as origin, installs no gh shim and grades cat patch.txt; 0.0.28 grades the agent's PR. All 19 agents hit gh: command not found, pushed correct commits to team/agentN, and were scored empty.
Collection outside the contract
a 20-minute rollout taking three hours
collect.py ran step_limit=0 (agents to ~480 steps) against the eval's 200, on 1 serve GPU against 4, at concurrency 8 against 10. The corpus had a 78× spread across tasks; contract-matched collection gives 2.4×.
modal container stop -y
every cleanup reporting success
that subcommand takes no -y (unlike modal app stop, which requires it). With 2>&1 hiding the error, each "cleanup" stopped nothing — 23 sandboxes survived a session of them.
The common shape: the check and the thing being checked were different objects. The
version check imported a package while the run executed a binary; the cleanup verified an exit
code that belonged to an argument-parsing error. Both now resolve from sys.executable
or are verified by re-reading the state they claim to have changed.
Submission is a PR, not patch.txt. Since 0.0.28 the agent opens
its own PR and grading reads git ls-remote --tags team refs/tags/pr/<agent>.
The line no patch.txt to publish is a legacy log message, not a failure — an empty
patch means no PR was opened. Before believing any 0/10, grep the trajectories for
gh: command not found: one run had 58 of them and 0 PRs while its two comparison
runs had 0 and 16–18.
The eval also now serves on H100:1. A 9B is ~18GB in bf16, so one card
holds it with ~60GB left for KV cache; four cost 4× and add a ~14-minute first-time TP=4 compile
for no measured benefit. run_meta.json records gpu, n_gpu, concurrency, max_num_seqs,
both parsers and the imported harness version — git history cannot answer "what did this run
use?", since a baseline whose metadata says step_limit: 200 traces to a commit
containing STEP_LIMIT = 0.
2026-08-10
Key resultAblation
How much of the ceiling is reachable by the 9B?
Give the student the identical context that took the 27B to 9/10. What it can
do with that is the upper bound on what distillation could transfer.
27B + context
9/10 · 18/20
9B + context
5/10 · 11/20
9B base
2/10 · 7/20
Task
9B base
9B + ctx
27B + ctx
Reading
dottxt 1655
0/2
2/2
2/2
context is enough
click 2800
1/2
2/2
2/2
context is enough
jinja 1621
1/2
2/2
2/2
context is enough
typst 6554
0/2
1/2
2/2
partial
hf_datasets 6252
2/2
2/2
2/2
already solved
tiktoken 0
2/2
2/2
2/2
already solved
dspy 8394
0/2
0/2
2/2
capability, not coordination
jinja 1559
0/2
0/2
2/2
capability
pillow 290
0/2
0/2
2/2
capability
go_chi 26
1/2
0/2
0/2
the protocol was wrong
The context alone more than doubles the student: 2/10 → 5/10, 7 → 11 features. The 9B
can execute the protocol when it is told what the protocol is, which is what makes distillation
worth attempting at all — the target behaviour is inside its reach.
It also draws the hard line. The 27B's remaining 4 pairs — dspy, jinja 1559, pillow —
stay 0/2 for the 9B even with the reference diff in hand. That gap is capability, not
coordination, and no amount of on-policy distillation from a coordination brief will close it.
Split of the 27B's 9/10
Pairs
Reachable by OPD?
already solved by the base 9B
2
nothing to gain
unlocked by context alone
3
yes — the target
needs 27B capability
4
no
go_chi is the one self-inflicted loss, and the ablation proves it. Both models scored
worse with the context than without it (1/2 → 0/2), which rules out the model and
convicts the brief. Its partition-by-region protocol split mux.go at the
line above and below h.ServeHTTP — but git merges hunks with three lines of
context, not individual lines, so two edits straddling one call site conflict however
cleanly the lines are divided. Any partition tighter than three lines is unmergeable by
construction. Relabelled handoff: one agent owns the whole file, the other copies it.
2026-08-10
ResultCeiling
The teacher ceiling is 9/10
Give each agent a per-task coordination protocol and the reference diff for
its own feature, and the 27B goes from 4/10 pairs to 9/10.
27B + protocol
9/10 · 18/20
27B base
4/10 · 12/20
122B-A10B base
3/10 · 10/20
9B base (student)
2/10 · 7/20
35B-A3B base
2/10 · 7/20
Pairs passed on flash_10. The protocol run is the same model, same harness,
same 10 tasks — only the prompt changed.
Task
Features
Merge
Min
Msgs
Protocol given
dottxt_ai_outlines 1655
2/2
clean
1.6
6
partition-by-region
huggingface_datasets 6252
2/2
clean
1.9
4
partition-by-region
pallets_jinja 1621
2/2
clean
2.5
7
partition-by-region
pallets_click 2800
2/2
clean
3.6
5
partition-by-region
pallets_jinja 1559
2/2
clean
3.8
2
interface-first
openai_tiktoken 0
2/2
identical
4.7
7
handoff
typst 6554
2/2
identical
8.4
14
handoff
dspy 8394
2/2
clean
10.2
8
interface-first
pillow 290
2/2
clean
11.5
22
interface-first
go_chi 26
0/2
conflicts
15.2
19
partition-by-region
The two identical rows are handoff working as designed — one
agent publishes, the other copies. typst had been a permanent 0/2 at every model
size. dspy was the predicted miss and passed anyway: it survived the oracle
ablation (reference code in hand, still conflicted), but under interface-first merged
clean in 10 min. Being handed the answer was not enough; being told who owns the signature was.
The one failure is not the one that was designed for
The protocol fixed what it named — no _test.go in either patch, the commit that
killed 3 of 4 earlier runs gone. They collided anyway in mux.go at
h.ServeHTTP, which both features wrap. Partition assumed separable regions; a
middleware chain and a metrics hook on one call site are not. This task is handoff,
misfiled — 19 messages spent negotiating a split that does not exist.
The ceiling run is deliberately contaminated — agents were handed reference diffs —
so it is a measurement of coordination headroom, not a score. The 20 mutated
feature.md files were restored from .orig immediately after the run and
verified clean (0 residual protocol or reference blocks) before any training data was touched.
Also 4× faster — median 4.2 min against 16.5. Coordination is quick when the split is
known; the two slowest pairs are the two where it was contested.
2026-08-10
BugFixed
Every conditioning number this project reported was measured through a broken prompt
The teacher's brief was being spliced into the middle of the teacher's own
reasoning block. Fixing the insertion point roughly doubles the measured effect.
Prompts end at the generation header, so prompt + b_ids put the peer's feature
spec inside the assistant's own chain of thought, as if the model had begun thinking it.
before … <|im_end|> <|im_start|>assistant <think> [BRIEF] ← inside its own reasoning
after … [BRIEF] <|im_end|> <|im_start|>assistant <think> ← end of the last user turn
The fix inserts before the <|im_end|> closing the last user turn. Gate
re-run on 96 states / 14,674 positions, briefed vs unbriefed control:
Teacher briefed vs unbriefed
Broken splice
Fixed
Change
mean KL
0.0404
0.0957
2.4×
median KL
0.00001
0.00055
55×
p90
—
0.1230
—
p99
—
2.1437
—
argmax flip rate
4.48%
6.29%
+40%
share of divergence in top 1% of positions
43.7%
43.0%
—
Retracts: "the brief barely does anything (+6%)" — the most load-bearing negative
result on this page — was an artifact. The median moving 55× matters more than the mean: the
effect was invisible at the typical position and showed up only in the tail.
Does not retract: divergence is still concentrated, 43% in 1% of positions.
Coordination is a few decisive tokens in a long trajectory either way.
2026-08-09
Design
Per-task workflows — what the ceiling was actually built from
Read every pair's two reference diffs, ask what split would have worked, and
the answers collapse into three shapes. These are workflows, not protocols — each is
written for one specific task and names its files, its line numbers and its code.
Of the 20 "protocol-only" blocks
Count
contain literal code the agent must write
6
reference the grading tests (“the hidden tests assert…”)
4
give exact line numbers (sed -i '61a …')
4
So the 7/10 they produce is not an answer-free result. Stripping the
diff --git block does not make a brief answer-free when the prose still says
hash_sha256 = Regex(r"[a-fA-F0-9]{64}"), names the insertion line, and quotes what
the hidden tests must not match. Any claim on this page that the protocol-only ceiling shows a
behavioural advantage is wrong; it shows a knowledge advantage in a different format.
Protocol
When
First agent
Second agent
Tasks
partition-by-region
shared file, hunks do not overlap
owns its region only
owns its region only
5
handoff
shared file, hunks overlap; one artifact is inevitable
publishes first
builds on top, submits the union
2
interface-first
shared symbol, independent bodies
defines the signature, publishes it
implements against exactly it
3
Every brief also carries a fixed endgame clause — dry-run the merge, test the merged tree,
abort, publish — because an unpublished branch scores zero, the 9B's largest single loss.
Sides come from sorted agent id, which both agents already hold, so the student can derive
its own role at inference. That is the binding constraint: the teacher may see the peer's
spec, the student never can.
Results above: partition 4/5, handoff 2/2, interface-first 3/3. The single partition failure
is go_chi, which the taxonomy misfiled.
What the teacher was actually given
Median block is 12,381 characters. Below is the real typst 6554
injection — permanent 0/2 for every model, back to 2/2 via an identical merge. Both halves are
shown because the asymmetry is the mechanism.
Block section
Median chars
Derivable by the student?
protocol + role + decisive step + endgame
2,397
role yes (sorted agent id); file overlap no
reference implementation of its own feature
9,973
no
agent1 — owns the file
## Coordination protocol for this task (handoff)
Your colleague is editing the SAME file(s) at the same time. Both patches are merged with
`git merge` before either test suite runs, so a collision fails BOTH of you. Detection is not
enough — every pair that failed had already run the dry-run and seen the conflict.
**Your role (agent1):** Own the file. Within ~10 steps message agent2: 'I am taking crates/typst/src/foundations/str.rs and writing ONE signature carrying both params — do not edit it, copy my branch when I push.' Then write the union: `pub fn first(&self, #[named] pattern: Option<Str>, #[named] #[default] case: Option<Str>) -> StrResult<Str>` (same for last). Body: if pattern is Some -> `if self.0.is_empty() { return Err(string_is_empty()); }` then scan graphemes(true) [.rev() for last] on `g.chars().next()`; else the plain next()/next_back(); finally `apply_case(grapheme, case)`. Helpers as free fns over &str/String only. Because there is no cargo, the code must be blind-compilable: params typed `Option<Str>` (NOT a new enum), no `cast!` block, no new trait impls, nothing passed to `{}` that is not a &str/String. Error text must be literal: `no character matches pattern "{}"`, `unknown pattern "{}"`, `unknown case option "{}"`. Commit, push origin/agent1, message the SHA.
**Their role — do not do it for them:** Do NOT write an independent version — that is the only thing that can reintroduce a conflict. Fetch, read agent1's diff once as a reviewer specifically for blind-compile hazards (undefined types, `{}` on a non-Display value, unbalanced braces, `Str` vs `&str` mismatches) and reply with corrections BEFORE copying. Then `git fetch origin && git checkout origin/agent1 -- crates/typst/src/foundations/str.rs`, commit, and assert `git diff origin/agent1 -- crates/typst/src/foundations/str.rs` prints nothing. Submit that. Touch no other line of the file.
**The step that decides this task:** agent1 writes the union with plain `Option<Str>` params (no invented enum, nothing `{}`-formatted that lacks Display) and the literal message `no character matches pattern "<p>"` emitted after an is_empty -> `string is empty` guard, then agent2 copies that exact file byte-for-byte. Convergence is already solved in this pair; code shape is the entire remaining delta.
Before you open your PR: `git fetch origin && git merge --no-commit --no-ff origin/agent2`,
run the tests on the merged tree, `git merge --abort`. Then open the PR. Do not stop before
publishing — an unpublished branch scores zero.
agent2 — copies, does not write
**Your role (agent2):** Do NOT write an independent version — that is the only thing that can reintroduce a conflict. Fetch, read agent1's diff once as a reviewer specifically for blind-compile hazards (undefined types, `{}` on a non-Display value, unbalanced braces, `Str` vs `&str` mismatches) and reply with corrections BEFORE copying. Then `git fetch origin && git checkout origin/agent1 -- crates/typst/src/foundations/str.rs`, commit, and assert `git diff origin/agent1 -- crates/typst/src/foundations/str.rs` prints nothing. Submit that. Touch no other line of the file.
The second block does not ask agent2 to coordinate — it forbids it from implementing.
Its job is reduced to reviewing agent1's diff, then git checkout origin/agent1 and
asserting an empty diff. That is what produces an identical merge. Note also what
the protocol knows that no agent could — "there is no cargo, the code must be
blind-compilable", and the literal error strings from the reference tests. Hindsight,
injected as foresight.
What carries into training and what does not. The whole block goes to the
teacher, reference diff included — that is the point of distilling from a measured ceiling. The
student never sees any of it; it only sees the teacher's resulting token distribution over its
own trajectory. Where the teacher's edge is a behaviour (who owns the file, publish
before stopping) the student can imitate it. Where the edge is the reference code itself, there
may be nothing for reverse KL to transfer — which is the leading explanation for the flat KL
above.
2026-08-08
Harness5 bugs
Five harness bugs were scoring as agent failures
Found by reading trajectories rather than summaries. Shipped in cooperbench
0.0.25–0.0.28.
Symptom in the logs
Actual cause
Fix
cargo: command not found; agent writes Rust for a whole task and never
compiles
Docker ENV PATH is not inherited by sb.exec, so
/usr/local/cargo/bin is invisible
prepend the known toolchain dirs
episode ends mid-run, recorded as an agent error
Modal decodes stdout as strict
UTF-8; one cat of a binary file raises and kills the run
read bytes,
decode leniently
agent hangs until the sandbox dies
Modal's stdin pipe never reaches EOF, so a
bare grep pattern blocks forever
exec < /dev/null plus a
hard timeout
send_message silently dead for an entire run
unterminated heredoc,
and << 'MSG' with a space where the parser requires
<<'MSG'
parse guard that tells the agent what it wrote wrong
merge conflicts on files nobody's feature owns
test-file stripping was
Python-only, so metrics_test.go, *.spec.ts and
*Test.java reached the merge
language-agnostic path matcher
Two runs are only comparable if the harness version matches. The 0.0.22
coordination fixes alone moved conflicts 5→3, agent errors 7→3, minutes per pair 221→46 and pass
0/10→1/10 — larger than most model deltas at n=10. Every baseline on this page was re-run after
the fixes.
2026-08-08
Analysis
Conflicts are seen, not missed — the gap is finishing
Every pair that ended in a conflict had already run the dry-run and watched it
fail.
Run
Task
Dry-runs
Saw the conflict
Shim warned
9B
dottxt 1655
4
2
0
27B
dspy 8394
10
4
1
35B
dspy 8394
8
4
0
35B
typst 6554
7
2
1
35B
dottxt 1655
4
1
1
35B
tiktoken 0
11
7
2
35B
go_chi 26
6
2
1
35B
pillow 290
16
12
0
122B
typst 6554
5
6
1
122B
tiktoken 0
19
4
1
10/10 ran the dry-run, 10/10 saw CONFLICT (content), all ten submitted
anyway.pillow 290 saw it twelve times. Detection is at 100%, so better checking
buys nothing. The next inference — that the fix is the action after detection — is also
wrong: the 9B and 27B are indistinguishable there (p=1.00, p=0.70).
The separation is loop closure:
Endgame behaviour
9B
27B
p
created a PR at all
60%
100%
0.0035
merge-check in the last 15 steps
45%
89%
0.0064
post-conflict merge re-checks per agent
1.29
3.86
0.011
re-verified a clean merge after a conflict
3/7
7/7
—
empty patches submitted
4/20
0/20
—
The 27B always finishes; the 9B stops. Same shape in messaging: the 35B mentions its
colleague as often as the 27B but converts that to a commitment 2.3× less, then acts on it 45%
vs 78%. It writes "Let me first message my colleague…" and the next action is cat.
An intention that never becomes the next command.
2026-08-07
Ablation
Oracle ablation: is it really coordination?
Hand each agent a known-correct diff for its own feature, on the two tasks that
had gone 0/2 for every model.
Task
Baseline (all models)
Reference given
Merge
Msgs
typst 6554
0/2 ×4
2/2
identical
13
dspy 8394
0/2 ×3
0/2
conflicts
17
typst confirms it: a permanent 0/2 became 2/2 once code was free, and the two patches came
back byte-identical at 5,244 bytes. dspy is the useful failure — the answer in hand,
more messages than typst, still conflicted. Talking more is not the remedy; the
objective is agreeing on one artifact. Superseded 2026-08-10: dspy
reaches 2/2 once interface-first names who owns the signature.
The 27B wins by doing less — half the commands, 77 median steps against 173, and 60%
of features against 35%. The 9B is not under-trying, it is re-trying: one agent hit the same
compile error 15 times and reverted its own file 25 times in 70 minutes.
The 35B is a coordination failure, not a coding one. Same 35% features, fewest steps,
but 17 messages against 120, --wait used twice all run, 6 conflicts where everyone
else had 1. Git fluency is comparable. It simply does not talk.
Pairs grouped by message volume (all runs pooled)
n
Conflicts
Features
≤ 3 messages
11
36%
45%
≥ 6 messages
17
18%
44%
Talking halves collisions and does nothing for correctness — but within the 35B alone the
relation vanishes, so most of the signal is across-model at n=3, and the 35B ran on SGLang where
the others ran vLLM.
Active parameters predict quality; total parameters do not. 27B 60% → 122B (10B
active) 50% → 9B 35% → 35B (3B active) 35%. The 122B has 4.5× the total parameters and loses on
every axis. Both MoE models under-communicate, confounded with the serving stack.
Why features fail
Cause
9B
27B
35B
Pooled share
passed
7
12
7
—
merge conflict
2
2
10
41%
submitted nothing
4
0
2
18%
near miss (≤2 tests)
1
2
1
12%
wrong code / no tests ran
6
4
0
29%
59% of failures are coordination or process, 29% are wrong code. That is the premise
the distillation rests on, now measured.
2026-08-06
Structure
Every pair collides
Set
Pairs
Share a file
Shared file is the only file
flash_10 (eval)
10
10 / 10
8 / 10
cooperdata_train_fast (training)
67
67 / 67
—
Comparing reference hunk ranges inside the shared file: 9 of 11 genuinely overlap, 2 are
separable by region.
Strategy
Viable?
divide by file
never — there is one file
divide by region
2 of 11
one agent implements both, other submits nothing
yes — 9 of 33 passing pairs to date
second waits for the first to publish, builds on top
yes — the handoff protocol, 2/2
The COLLISION RISK flag fires on every state, so it carries no information and
was dropped. A clean merge is not a working merge — the two tasks nobody solved failed as
build failures after git succeeded (11 Rust type errors; 'Cache' object has no
attribute '_stats_lock'). Git resolves text, the compiler resolves meaning.
reference
Setup
Teacher conditioning, sharding, cost, noise floor
What the teacher is conditioned on
A teacher transfers only what it holds and the student lacks. The first attempt had neither
advantage — same 9B weights under a 92-token brief — and reverse KL weights by
p_student, so it could only re-rank mass the student already assigned.
Teacher condition
Mean KL
Top-1 agreement
States
identity (same weights, same prompt)
0.00012
1.000
2
void (temperature on both sides)
0.00031
0.997
56
same weights + 92-token brief
0.1254
0.904
56
27B, student's own prompt
0.0615
0.944
336
27B + peer spec, broken splice
0.0650
0.932
336
27B + peer spec + role, fixed splice
0.0957
—
96
KL(p‖p)=0 is the global minimum, so identity at 1.2e-4 checks state extraction,
output alignment, prompt boundaries and the loss mask at once.
Sampled tokens are byte-identical either way, so divergence is attributable to the insertion
alone. Superseded 2026-08-10: round 1 gives the teacher the full ceiling block, reference
diff included — see Round 1.
Contamination
Collection
Tasks
Pairs
Shared tasks with eval
Shared repos
first (benchmark subset)
10
10
10 / 10
9 / 9
second (benchmark, non-eval)
16
49
0
7 / 10
current (CooperData)
67
356
0
0 / 24
Two faults in order: the collector drew from the eval subset itself, then the replacement
still shared 7 of 10 repos. Both train normally and produce healthy curves, so nothing
downstream catches them. The collector now asserts the intersection before starting.
Sharding
Layout
Max context
Peak per device
States usable
single device
8,192
93 GB
13%
student over 2, teacher on 1
16,384
63 / 101 / 57 GB
48%
student over 3, teacher on 1
24,576
42 / 108 / 90 / 59 GB
82%
Median prompt 16,750 against a median reply of 90 — 79 sequence tokens through two models
per token the loss reads. ~2,200 tok/s; a full pass is ~10 GPU-hours.
Noise floor
Same weights, same script
Pairs
Features
Conflicts
Msgs/pair
pre-harness-fix
0/10
3/20
5
2.3
run A
1/10
6/20
3
3.8
run B
1/9
3/18
4
2.2
Pair pass rate is stable only coincidentally — the passing pair's identity changes while
feature pass moves 30% → 17%. An intervention must clear ~two pairs to beat a re-run of the
same weights. 95% Wilson on 5/10 is 24–76%.