Official Codebase of 3DCodeVerse
Python
planner: category-defining features must reach the plan; log the judge experiment
gaoypengee7214b205d7
diff --git a/harness/codeverse/prompts/tracks/plan_static.j2 b/harness/codeverse/prompts/tracks/plan_static.j2index 027953bd..a85634e8 100644--- a/harness/codeverse/prompts/tracks/plan_static.j2+++ b/harness/codeverse/prompts/tracks/plan_static.j2@@ -18,7 +18,10 @@ WHAT TO PRODUCE (JSON, fields: {{ schema_fields }}):request states where that number came from). Every nameable feature in the request and in theconstraints must be either its own part or explicitly named in another part's `description` asgeometry that part carries — a feature named nowhere will not be built, and the judge's- checklist will mark it missing.+ checklist will mark it missing. The same holds for the CATEGORY's defining features even+ when the request does not spell them out: a violin has f-holes, a clock has hands, a wrench+ has jaws — a plan that omits one plans the wrong object (measured: a "violin" plan with no+ FHoles part built cleanly, passed every gate, and lost the round on intent).(Use `instances` for identical copies: 4 legs = ONE part, instances=4,bbox of ONE copy; `symmetry` says how copies are laid out.) For EACH part:* `name` PascalCase, unique, descriptive (`SeatCushion`, `FrontLeg`, `Backrest`).diff --git a/harness/docs/EVAL.md b/harness/docs/EVAL.mdindex 56704ebc..de886116 100644--- a/harness/docs/EVAL.md+++ b/harness/docs/EVAL.md@@ -522,3 +522,17 @@ rate, build-fail rate, cost, minutes); pairwise table; number of degraded verdicre-run; **cells dropped as `infra_failed` and `budget_exhausted`, per arm** (§7 — anomitted drop count is an unreadable table); links to `record.json` / `report.html`under `bench/out/`.++## Judge experiments log++**2026-08-29 — flash "named-feature sweep" prompt variant: REJECTED.** Re-judged 9 recorded+runs spanning stored 0.02–0.98 with `gemini-3.7-flash` n=3, base prompt vs a variant that+inserts an explicit present/absent sweep of brief-named features before scoring. The one+confirmed leniency case (a violin missing its f-holes, judged ~0.43 by pro; renders eyeballed)+did not move (0.898 → 0.888) and mean overall σ doubled (0.014 → 0.032); espresso gained an+honest interpenetration defect, chair/penny unchanged. Root cause of the violin miss sits in+the PLAN (no FHoles part — see the defining-features rule added to `plan_static.j2` the same+day), not in judge prose. Flash stays a ranking/fallback judge; pro stays the verdict judge.+Two calibration-tool defects found the same day (colliding `run` labels overwriting judgment+files; old records whose stored overall contradicts their own criterion scores) are fixed in+`judges/calibration.py` and flagged in its report.