Skip to content

Official Codebase of 3DCodeVerse

Python

docs: DECISIONS D40–D43 + release notes for the bench/lint changes

ziyao79b96f2f8a73
diff --git a/harness/docs/DECISIONS.md b/harness/docs/DECISIONS.md
index 2284ae6c..76232dc5 100644
--- a/harness/docs/DECISIONS.md
+++ b/harness/docs/DECISIONS.md
@@ -198,6 +198,33 @@ written) that were accepted because the code works that way and the tests pin it
guard among them — never ran on a PR. Decision: `pytest tests -m "not live and
not blender and not node"` on both python floors; `timeout-minutes` on both jobs.
+* **D40 A harness cell that waited instead of iterating is flagged, not trusted
+ (2026-08-25).** Context: under a day-long gemini-3.7-flash 503 storm 37 of 40 harness
+ runs stopped on the 45-minute ceiling with 0–2 completed rounds while the 3 that met a
+ calm window finished 2–4 rounds and scored 0.92–0.95; the paired mean over the stormed
+ cells (0.516) measured the weather. Decision: `compare_backends.flag_degraded` marks a
+ harness cell that stopped for budget with money left, ≤ 1 completed round, after
+ ≥ 40 min; the report counts them (`degraded` / `cut` columns, † per prompt) and
+ `paired_compare` adds an `all −degraded` row. Scores stay — the artifact is real.
+* **D41 Failures of the arm are zeros; failures of the provider are dropped (2026-08-25).**
+ Context: a `oneshot+repair` cell whose repair call died in a 503 storm was scored 0 on
+ its pre-repair file; a harness run whose planner failed validation twice was recorded
+ `error` / score None and vanished from the mean (5 of 14 articulated prompts). Decision:
+ a one-shot arm whose LAST attempt was lost to the provider is `infra_failed` (redone);
+ a harness `PlanningError` is `no_code` / 0.0, like a one-shot answer in the wrong
+ format; "Error creating WebGL context" (a saturated shared GPU) is an infra marker.
+* **D42 `compare_backends` follows the battery's track (2026-08-25).** Context: the
+ fixed evaluator was static_object-only. Decision: `FixedEvaluator(track=, language=)`
+ picks the runtime and rubric from the battery (`RUBRIC_BY_TRACK`); articulated cells add
+ the joint-sweep gate and the pose sheet read from the built URDF (no plan, every arm
+ alike); one-shot arms get a language-aware minimal contract (the D18 recipe as rules,
+ no example) and must answer with both files in the `=== FILE ===` envelope.
+* **D43 Python lints never raise on unparseable source (2026-08-25).** Context: a
+ generated bpy file made CPython 3.11's `ast.parse` raise `SystemError` and the round died
+ in `build_once`. Decision: `_ast_lint.safe_parse` catches SyntaxError / RecursionError /
+ SystemError / MemoryError / ValueError; every python lint reports a lint ERROR with a
+ flatten-the-literal hint instead.
+
## Rejected / deferred
* Registering `single-shot` as a CodingAgent kind (rejected: it has no tools/session).
diff --git a/harness/docs/RELEASE_NOTES.md b/harness/docs/RELEASE_NOTES.md
index c87a34f3..6c2983da 100644
--- a/harness/docs/RELEASE_NOTES.md
+++ b/harness/docs/RELEASE_NOTES.md
@@ -12,6 +12,17 @@ score computed under an odd `n_samples` (branch `ziyao/eval-integrity`, D36–D3
the old rule `economy` (n=2) was strictly harsher than `balanced` (n=1) and
`quality` (n=3).
* CI runs the whole offline suite (was 7 of 24 test directories).
+* `bench/paired_compare.py`: paired Δ, 95 % t-interval, exact sign test, per tier, plus a
+ judge-free table (build rate, gate errors) per arm; `bench/prompts/compare_v4.yaml`
+ (40 static prompts); `--judge-samples` on `compare_backends`.
+* `compare_backends`: storm-degraded harness cells are flagged (D40); arm failures are
+ zeros and provider failures are dropped (D41); the articulated track is supported (D42).
+* Python lints survive unparseable source (D43).
+* Measured with these (2026-08-25/26, gemini-3.7-flash, judge gemini-3.1-pro-preview n=3):
+ static compare_v4, n=40 — harness 0.633 vs one-shot+repair 0.487, Δ +0.146
+ [+0.077, +0.215], sign p < 0.001; vs one-shot 0.232, Δ +0.401. Articulated
+ articulated_v2, n=14 — harness 0.309 vs one-shot+repair 0.316, Δ −0.007 (CI crosses 0);
+ 5/14 harness runs failed at planning.
---