World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
The recall bench waits for alignment and governance to settle before it scores a round (#1005)
Jeremy Cappse639cadeee3b
diff --git a/scripts/bench/README.md b/scripts/bench/README.mdindex 00485487..6c3446de 100644--- a/scripts/bench/README.md+++ b/scripts/bench/README.md@@ -86,7 +86,9 @@ node scripts/bench/recall.mjs --table runs/588`resolution_reviews` 的疑似同一对,#877;提示词里没有 known 时同一个东西换个写法再列一遍,落在这里而不是静默合并。不分召回通道:一对裁过之后 reason 被裁决改写,`name_vector|` 留不下来。全库总数另按待裁 / 合并 / 分开和现在的reason 前缀记;只算这一轮排队之后排的对,`--score` 没有排队时刻,算的是库里现有的——每轮开头删实体时它们随之删掉)、丢弃账(`extraction_drops`-按原因)和排队到最后一块抽完的秒数。`--table` 给每组的均值 ± 标准差,再给事实集合(去重的主/谓/宾)+按原因)和排队到最后一块抽完的秒数。抽完之后还要等对齐、治理、时间消解、裁决这些任务收尾(它们还在补事实、裁对子),+等到没有该跑的任务才打分,不然每轮记的是抽完那一刻的快照;等了多久记成 `settleSeconds`,卡住十五分钟没收尾的轮次+`--table` 会单独点名。`--table` 给每组的均值 ± 标准差,再给事实集合(去重的主/谓/宾)的重合度:组内两两比是运行间的抖动,跨组两两比是 known 的影响——跨组落在组内的范围里,空 known 就没把输出挪出方差。diff --git a/scripts/bench/recall.mjs b/scripts/bench/recall.mjsindex bcc12f00..7ffff850 100644--- a/scripts/bench/recall.mjs+++ b/scripts/bench/recall.mjs@@ -329,6 +329,7 @@ function table(dir) {const row = (name, f) => `| ${name} | ${groups.map(([, rs]) => ms(rs.map(f))).join(" | ")} |`;const lines = [`| | ${cols.join(" | ")} |`, `| --- |${cols.map(() => " --- |").join("")}`];lines.push(row("总分(/52)", (r) => r.score.pass));+ lines.push(row("抽完后等收尾(秒)", (r) => r.settleSeconds));lines.push(row("排给裁决的对(全库)", (r) => r.pairs?.total));lines.push(row("其中合并的", (r) => r.pairs?.merged));for (const doc of docs) {@@ -341,6 +342,9 @@ function table(dir) {lines.push(row(`${doc} 秒`, (r) => d(r).seconds));}console.log(lines.join("\n"));+ // 没等到收尾就记下的轮次(卡住十五分钟才停):数是快照,单独点名,别混进均值里看不出来+ const unsettled = runs.filter((r) => r.settled === false);+ if (unsettled.length) console.log(`\n没等到收尾的轮次:${unsettled.map((r) => `${r.at}(${r.known})`).join("、")}`);// 事实集合的 Jaccard:每篇文档各算,再平均const jac = (a, b) => {@@ -431,4 +435,36 @@ for (;;) {await sleep(30000);}-record(score(), stats(queuedAt), { endpoint, pairs: pairs(queuedAt) });+// **抽完不等于这一轮完了**:对齐、治理、时间消解、裁决还在后头跑,一边补事实一边裁对子+// (同一篇财报,抽完那一刻 552 条事实,一小时后 565)。不等它们收尾,每轮记下的就是+// 「抽完那一刻」各不相同的快照,两组之间多出一份与 known 无关的抖动。+// 等到没有该跑的任务:running 的,和 run_at 一分钟之内到点的 queued(对齐排队时带几秒防抖,+// 前一个没完的会带延迟重排;更远的定时任务不算这一轮的)。进展看跑完的任务数,规矩同上:慢不算超时,卡住才算。+// 秒数那一栏读的是 extracted_at,不受这里等多久影响+const pendingJobs = `SELECT count(*) FROM jobs WHERE status = 'running'+ OR (status = 'queued' AND run_at <= now() + interval '1 minute')`;+const settleFrom = Date.now();+let settled = false;+last = -1;+stall = 0;+for (;;) {+ const left = num(pendingJobs);+ const finished = num(`SELECT count(*) FROM jobs WHERE status IN ('done','failed')`);+ if (left === 0) {+ settled = true;+ break;+ }+ const kinds = psql(`SELECT string_agg(kind || '×' || n, ' ' ORDER BY kind) FROM+ (SELECT kind, count(*) AS n FROM jobs WHERE status = 'running'+ OR (status = 'queued' AND run_at <= now() + interval '1 minute') GROUP BY kind) k`);+ console.log(`${stamp()} 等收尾:还有 ${left} 个任务(${kinds})`);+ if (finished === last) {+ if (++stall > 30) { console.log("十五分钟没有任务跑完,不等了:这一轮记下的不是收尾之后的数"); break; }+ } else stall = 0;+ last = finished;+ await sleep(30000);+}+const settleSeconds = Math.round((Date.now() - settleFrom) / 1000);+if (settled) console.log(`${stamp()} 收尾完了(抽完之后又等了 ${settleSeconds} 秒)`);++record(score(), stats(queuedAt), { endpoint, pairs: pairs(queuedAt), settled, settleSeconds });