Qwen3.5-9B — Thinking, Coordination & Repair

Results

Merge repair produced the highest pair pass rate: 26.85%, compared with 8.33% for the baseline. Worker thinking did not improve accuracy in these runs. Removing the coordinator gave a modest accuracy increase. In these non-thinking runs, the coordinator-on setting had 31% lower mean pair completion time (29.57 vs. 43.00 minutes) and 54% lower mean per-trial P50 (22.00 vs. 48.15 minutes), despite no accuracy gain. P99 remains near 62–64 minutes across settings.

All metrics are averaged across three trials. Timing columns describe individual pair completion times, not whole-trial durations. Each trial contains the same 36 pairs; a pair passes only if both features pass together.

What the metrics measure

Mean pair pass rate: for each trial, divide the number of pairs with both features passing the official evaluation by 36, then average the three rates. This is an average single-trial success rate, not the fraction solved at least once across three attempts.

Pair time — P50: take the median of the 36 pair execution times in each trial, then average the three medians. With 36 pairs, each median is the average of the 18th and 19th sorted durations.

Pair time — mean: average the 36 pair execution times in each trial, then average those three means. Because trial sizes are equal, this also equals the mean across all 108 pair attempts.

Pair time — P99: compute the 99th percentile of the 36 pair execution times in each trial, then average those three percentiles. We use linear interpolation between sorted durations, equivalent to NumPy’s default percentile method. With only 36 observations per trial, P99 is close to the slowest pair and should be interpreted as a descriptive tail metric.

Execution time is the harness’s recorded duration_seconds: environment setup, concurrent worker execution, patch collection and merge, optional repair, and cleanup. It excludes scheduler and pair-queue waiting, campaign image staging, and subsequent official evaluation. All 36 durations count, including unsuccessful pairs and workers that reach their limits. The one-hour worker limit is not a hard limit on the full pair: setup, merge, repair, and cleanup add time.

Configuration

The coordinator is non-thinking whenever enabled. “Worker thinking” enables Qwen’s chat-template thinking switch for the two workers only; no separate long-thinking setting was tested.

• Tasks and scoring: the same 36 training-set feature pairs across 17 qualified task images, pinned to CooperBench 63b9d44. Official tests and reference patches are unchanged. All 432 pair attempts have official scores.
• Model and sampling: Qwen3.5-9B served by SGLang with five replicas; temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5.
• Workers: two concurrent mini-SWE workers per pair, each in its own Apptainer environment, with up to 1,000 steps and 3,600 seconds. Cooperation tools, shared Git, and the completion gate are enabled in every setting. The no-coordinator setting therefore retains these harness features.
• Repair: only the repair setting adds --repair-integrator --repair-attempts 2. After merging worker patches, an unhealthy tree triggers a non-thinking repair agent for up to two sequential attempts of 25 steps each; it stops early when the health check passes. No separate repair time limit is set. Repair does not use official scores to choose between the original and repaired patches.
• Other controls: pre-submission merge, focused repair, and the behavioral gate are disabled in all settings.
• Execution: three separate trials per setting; up to 10 pairs (20 workers) per trial and one official evaluator at a time. Each trial requests 20 CPUs, 128 GiB RAM, and an eight-hour limit on the same Stanford NLP cluster node. Three overlapping trials target up to 60 workers, plus event-triggered coordinator or repair calls.

Interpretation and caveats