Skip to content

World's first open-source enterprise world model.

A misworded verdict says which way it is wrong (#749)

WaylandYangf7da78d9a739
diff --git a/docs/design/extraction.md b/docs/design/extraction.md
index 77bec2a3..32ed79ae 100644
--- a/docs/design/extraction.md
+++ b/docs/design/extraction.md
@@ -103,11 +103,16 @@ qualifiers, time words and quote span, and a nod writes an open statement [0015,
`scripts/bench/recall.mjs` (SEC filings, pharma, ai-timeline; Re-DocRED fetched by script and never
used as prompt examples) scores entity-pair recall; `judge_open.mjs` has a judge model read the
-chunk and reports stated, misworded and not stated, and separately whether the statement reads on
-its own without the document (`alone`, the uninformative-phrase class of [prior-work](prior-work.md)
+chunk and reports stated, misworded and not stated, then in a second pass marks each misworded
+statement extrapolated (it goes beyond what the document gives the pair) or contradicted (the
+document says otherwise: direction, phrase or the value of another cell) [prior-work item 8, #749],
+and separately whether the statement reads on its own without the document (`alone`, the uninformative-phrase class of [prior-work](prior-work.md)
item 1); `identity.mjs`, `govern.mjs`, `temporal.mjs`
-and the lease bench cover the other domains. Every cut reports at least three domains, two runs per
-configuration, with the judge's calibration stated; no F1 against Re-DocRED, whose gold omits true
+and the lease bench cover the other domains. The judge's own variance is measured: the same 909
+statements judged twice came out misworded 5.0% and 3.6% with 97% of verdicts identical, and
+reads-alone 4.8%, 8.6% and 10.7% across three passes, so a difference under two points of misworded
+between two runs is noise and reads-alone is reported but not compared between single runs [#749]. Every cut reports at least three domains, two runs per configuration,
+with the judge's calibration stated; no F1 against Re-DocRED, whose gold omits true
facts [0044]. Thresholds: not stated at most 2%, entity-pair recall no lower than before, prompt
tokens per document reported. Measured at cut 1: not stated at or under 2% in 16 of 18
corpus-rounds; NVDA full documents 44 to 46 of 52 (one standard deviation about 2.7) against 43 and
diff --git a/docs/design/prior-work.md b/docs/design/prior-work.md
index 83be539e..fda4d04f 100644
--- a/docs/design/prior-work.md
+++ b/docs/design/prior-work.md
@@ -163,7 +163,11 @@ built.
citation presence is not citation support [Gao 2023]; a three-way label (attributable,
extrapolated, contradictory) is more useful than a binary one [Yue 2023]; abstention must be a
first-class output [Menick 2022]. `quote_not_in_chunk` is a drop reason and the judge is a
- separate pass. *Open:* the judge's "misworded" splits into extrapolated and contradicted.
+ separate pass; since #749 a second pass marks each "misworded" verdict extrapolated or
+ contradicted. The verdict pass itself is untouched, because the judge is sensitive to its own
+ wording: the same 909 statements came out 0.8% misworded when the three verdicts were reworded as
+ four and 2.1% when the kind was asked in the same call, against 5.0% and 3.6% from two passes of
+ the unchanged judge.
9. **The contract's shape has a cost.** Format restrictions degrade reasoning [Tam 2024]; guideline
prompts are not reliably obeyed without training [Sainz 2024]; quoting before stating improves
grounding [Huang 2024]; forcing citations finer than a sentence hurts [Wang 2026]. The compact
diff --git a/scripts/bench/judge_open.mjs b/scripts/bench/judge_open.mjs
index 37e4b4b6..997fc11d 100644
--- a/scripts/bench/judge_open.mjs
+++ b/scripts/bench/judge_open.mjs
@@ -10,6 +10,12 @@
// 三档:stated(原文说了,陈述说对了)/ misworded(原文说了两者有关系,陈述的说法不对)/
// not_stated(原文没说)。0044 的门槛:not_stated 不超过 2%。
//
+// 判成 misworded 的再走第二遍,标一个 kind:extrapolated(说过头了:限定、时间或细节是原文给
+// 别的东西的或没给的)或 contradicted(说反了:方向反了、短语说的是原文否认的、值是别的行或
+// 列的)——prior-work 第 8 条,Yue 2023 的三分。判决那一遍一字不改,kind 单独一遍:把三档改成
+// 四档,同一批 909 条的 misworded 从 5.0% 掉到 0.8%;在同一次调用里追问 kind,掉到 2.1%;
+// 裁判自己的抖动是同一批判两遍 5.0% / 3.6%(逐条一致 97%)。
+//
// 用法:
// node scripts/bench/judge_open.mjs --kb <id> [--sample 200] [--seed 7] [--out judged.json]
// node scripts/bench/judge_open.mjs --calibrate scripts/bench/truth/open-statements.json
@@ -109,6 +115,34 @@ function chunkText(chunkId) {
const render = (r) => `${r.subj} —${r.phrase}→ ${r.obj}${r.quals ? ` (${r.quals})` : ""}${r.times ? ` [${r.times}]` : ""}`;
+// 第二遍:只问判成 misworded 的,说过头了还是说反了
+const KIND = `You check statements extracted from a document. For each numbered statement the document does state a relationship between the subject and the object or value, but the statement describes it wrongly. Say which way:
+- "extrapolated": the statement goes beyond what the document gives this pair: a qualifier, a time or a detail the document gives for something else or does not give, or a phrase that claims more than the document's words;
+- "contradicted": the document says otherwise: the direction is reversed, the phrase says what the document denies, or the value is another row's or column's.
+First copy the words of the document that bear on it ("evidence"). Judge only from the document. Output one JSON object: {"results":[{"i":0,"evidence":"...","kind":"extrapolated|contradicted"}]}`;
+
+async function kindPass(kb, judged) {
+ const ep = judgeEndpoint(kb);
+ const byChunk = new Map();
+ for (const it of judged) if (it.verdict === "misworded") (byChunk.get(it.chunk) || byChunk.set(it.chunk, []).get(it.chunk)).push(it);
+ for (const [chunk, list] of byChunk) {
+ const text = chunkText(chunk);
+ const body = `Document (${list[0].file}):\n${text}\n\nStatements:\n${list.map((x, j) => `${j}. ${x.statement}`).join("\n")}`;
+ let res;
+ try {
+ res = await chat(ep, [{ role: "system", content: KIND }, { role: "user", content: body }]);
+ } catch (e) {
+ console.error(`${stamp()} kind 调用失败:${e.message}`);
+ continue;
+ }
+ for (const r of res.results || []) {
+ const it = list[r.i];
+ if (it && ["extrapolated", "contradicted"].includes(r.kind)) it.kind = r.kind;
+ }
+ }
+ return judged;
+}
+
async function judgeAll(kb, items) {
const ep = judgeEndpoint(kb);
const byChunk = new Map();
@@ -148,7 +182,8 @@ function report(judged, total) {
const n = judged.length;
const count = (v) => judged.filter((j) => j.verdict === v).length;
const pct = (a) => `${a}/${n}(${n ? ((100 * a) / n).toFixed(1) : 0}%)`;
- console.log(`\n开放陈述 ${total} 条,判了 ${n} 条:stated ${pct(count("stated"))},misworded ${pct(count("misworded"))},not_stated ${pct(count("not_stated"))},未判 ${count("unjudged")}`);
+ const kind = (k) => judged.filter((j) => j.verdict === "misworded" && j.kind === k).length;
+ console.log(`\n开放陈述 ${total} 条,判了 ${n} 条:stated ${pct(count("stated"))},misworded ${pct(count("misworded"))}(extrapolated ${kind("extrapolated")},contradicted ${kind("contradicted")}),not_stated ${pct(count("not_stated"))},未判 ${count("unjudged")}`);
console.log(`0044 的门槛:not_stated ≤ 2%${count("not_stated") / (n || 1) <= 0.02 ? ",过" : ",没过"}`);
const fragments = judged.filter((j) => j.alone === false);
console.log(`单看读不通(alone = false):${pct(fragments.length)}`);
@@ -187,6 +222,6 @@ if (args.calibrate) {
const { total, picked } = loadStatements(KB, args.sample, args.seed);
console.error(`${stamp()} 库里 ${total} 条开放陈述,抽 ${picked.length} 条`);
-const judged = await judgeAll(KB, picked);
+const judged = await kindPass(KB, await judgeAll(KB, picked));
report(judged, total);
if (args.out) fs.writeFileSync(args.out, JSON.stringify(judged, null, 1));