Skip to content

Official Codebase of 3DCodeVerse

Python

planner: category-defining features must reach the plan; log the judge experiment

gaoypengee7214b205d7
diff --git a/harness/codeverse/prompts/tracks/plan_static.j2 b/harness/codeverse/prompts/tracks/plan_static.j2
index 027953bd..a85634e8 100644
--- a/harness/codeverse/prompts/tracks/plan_static.j2
+++ b/harness/codeverse/prompts/tracks/plan_static.j2
@@ -18,7 +18,10 @@ WHAT TO PRODUCE (JSON, fields: {{ schema_fields }}):
request states where that number came from). Every nameable feature in the request and in the
constraints must be either its own part or explicitly named in another part's `description` as
geometry that part carries — a feature named nowhere will not be built, and the judge's
- checklist will mark it missing.
+ checklist will mark it missing. The same holds for the CATEGORY's defining features even
+ when the request does not spell them out: a violin has f-holes, a clock has hands, a wrench
+ has jaws — a plan that omits one plans the wrong object (measured: a "violin" plan with no
+ FHoles part built cleanly, passed every gate, and lost the round on intent).
(Use `instances` for identical copies: 4 legs = ONE part, instances=4,
bbox of ONE copy; `symmetry` says how copies are laid out.) For EACH part:
* `name` PascalCase, unique, descriptive (`SeatCushion`, `FrontLeg`, `Backrest`).
diff --git a/harness/docs/EVAL.md b/harness/docs/EVAL.md
index 56704ebc..de886116 100644
--- a/harness/docs/EVAL.md
+++ b/harness/docs/EVAL.md
@@ -522,3 +522,17 @@ rate, build-fail rate, cost, minutes); pairwise table; number of degraded verdic
re-run; **cells dropped as `infra_failed` and `budget_exhausted`, per arm** (§7 — an
omitted drop count is an unreadable table); links to `record.json` / `report.html`
under `bench/out/`.
+
+## Judge experiments log
+
+**2026-08-29 — flash "named-feature sweep" prompt variant: REJECTED.** Re-judged 9 recorded
+runs spanning stored 0.02–0.98 with `gemini-3.7-flash` n=3, base prompt vs a variant that
+inserts an explicit present/absent sweep of brief-named features before scoring. The one
+confirmed leniency case (a violin missing its f-holes, judged ~0.43 by pro; renders eyeballed)
+did not move (0.898 → 0.888) and mean overall σ doubled (0.014 → 0.032); espresso gained an
+honest interpenetration defect, chair/penny unchanged. Root cause of the violin miss sits in
+the PLAN (no FHoles part — see the defining-features rule added to `plan_static.j2` the same
+day), not in judge prose. Flash stays a ranking/fallback judge; pro stays the verdict judge.
+Two calibration-tool defects found the same day (colliding `run` labels overwriting judgment
+files; old records whose stored overall contradicts their own criterion scores) are fixed in
+`judges/calibration.py` and flagged in its report.