Skip to content

World's first open-source enterprise world model.

The typed-graph bench measures a warm base on deepseek-flash: 42k tokens per document against 48k on a fresh one (#1065)

WaylandYangb5d6d5bbceb0
diff --git a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md
index c6f8a279..ce70d5f9 100644
--- a/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md
+++ b/docs/decisions/0044-the-ontology-is-a-view-over-what-documents-say.md
@@ -1,6 +1,6 @@
# 0044 · The ontology is a view over what documents say
-- **Status**: In progress · accepted 2026-09-17 · cut 1 built (#731), cut 2 mostly built (#751, #876, #878, #882), cut 5 is 0061, cut 6 built (#885) · cut 2 and cut 6 measured twice on 100 Re-DocRED documents, two groups each (bench README, the fourth run on gemini-3.5-flash 2026-09-25 and the ninth on deepseek-flash 2026-09-30): judged precision passes on both (80–89%), same-sentence recall passes on gemini (17.8% / 17.1%) and fails on deepseek-flash (12.3% / 13.2%, the extractor finds a third fewer entity pairs), tokens per document fail on both (31k–48k against the 6k threshold; alignment is charged per signature, and a fresh 100-document base is its dearest shape); the errata agent adds without retracting (0–2 retractions, none wrong) · open: the tokens-per-document threshold (restate it per signature, or accept the miss), cut 4
+- **Status**: In progress · accepted 2026-09-17 · cut 1 built (#731), cut 2 mostly built (#751, #876, #878, #882), cut 5 is 0061, cut 6 built (#885) · cut 2 and cut 6 measured twice on 100 Re-DocRED documents, two groups each (bench README, the fourth run on gemini-3.5-flash 2026-09-25 and the ninth on deepseek-flash 2026-09-30): judged precision passes on both (80–89%), same-sentence recall passes on gemini (17.8% / 17.1%) and fails on deepseek-flash (12.3% / 13.2%, the extractor finds a third fewer entity pairs), tokens per document fail on both (31k–48k against the 6k threshold on a fresh base, 25k and 42k on a warm one: a second 100 documents bring as many new signatures as the first, and extraction with time interpretation alone is at the threshold); the errata agent adds without retracting (0–2 retractions, none wrong) · open: the tokens-per-document threshold (accept the miss, or cut alignment's cost; restating it per signature does not change the reading), cut 4
- **Written**: 2026-09-16 (conventions in the [README](README.md))
- **Related**: [0022](0022-an-unknown-date-is-not-an-open-one.md) put a document's date in `attested_at` beside the world and record axes; [0025](0025-governance-reads-the-ledger-before-it-decides.md) and [0027](0027-an-automatic-merge-is-gated-by-what-it-can-undo.md) put agent decisions through a gate that weighs what they can undo; [0041](0041-a-name-is-a-claim-about-an-entity.md) made names facts and identity a matter of evidence; [0043](0043-every-review-queue-is-governed.md) sent every review queue through the governor; #714 found upload time used as the document date in extraction.
diff --git a/scripts/bench/README.md b/scripts/bench/README.md
index 093f1db9..db8546e8 100644
--- a/scripts/bench/README.md
+++ b/scripts/bench/README.md
@@ -1028,3 +1028,63 @@ bound / none / undecided。0044 的门槛:裁判精度不低于完整原型的
费用:deepseek-flash 两组加 20 篇试跑共 ¥28.7(输入 $0.15 / 百万、输出 $0.60 / 百万,思考 token 按输出计);
裁判约 600 次调用、每次 $0.0045(gemini 没关思考,每次三四百个思考 token),合计 $3 上下。
+
+### 温库第二批,deepseek-flash:token 门槛再量一次(2026-10-03,同一二进制 dev @ 798fb24,第九次组 2 的库再灌 100 篇)
+
+第九次真跑留下的口子是 token 门槛:四次 100 篇真跑没有一次接近,而新库 100 篇是对齐最贵的形态。这一次量
+另一头:`--into <第九次组 2 的库> --corpus redocred-100b --judge 200 --errata`,同一个模型、同一份设置
+(抽取 `none`,对齐与提规则固定 `low`),和 2026-09-25 gemini 的那次温库同一个做法。
+
+| 第二批 | 值 |
+|---|---|
+| 开放陈述 / 全库实体 | 1488 / 4092 |
+| 全库签名 bound / none / undecided(第一批跑完时 283 / 677 / 54) | 478 / 1396 / 92 |
+| 类型化 勘误前 → 后 | 367 → 603 |
+| gold recall 勘误前 → 后(同句) | 3.7%(4.8%)→ 6.9%(9.6%) |
+| entity-pair recall(开放层) | 9.2% |
+| 裁判精度 勘误前 / 后 | 89.2%(判了 186,14 条请求失败)/ 87.5%(200 条判满) |
+| 勘误 | 看了全库 200 篇 822 条:撤 0 / 改 1 / 加 521 / 留人 96 / 拒 14 |
+| 每篇 token | **4.2 万**(新库 4.8 万) |
+
+按调用,和第九次组 2 的新库并排(每篇,调用次数在括号里):
+
+| 调用 | 新库 | 温库第二批 |
+|---|---|---|
+| 短语对齐 | 1.7 万(142) | 1.35 万(111) |
+| 提规则 | 0.6 万(38) | 0.60 万(34) |
+| 类别词对齐 | 0.9 万(76) | 0.58 万(54) |
+| 实体裁决 | 0.3 万(102) | 0.58 万(208) |
+| 抽取 | 0.4 万(126) | 0.48 万(137) |
+| 勘误(复查 + 核对) | 0.2 万(110) | 0.25 万(123,全库 200 篇) |
+| 本体代理 | 0.4 万(26) | 0.21 万(12) |
+| 时间解释 | 0.1 万(85) | 0.14 万(84) |
+| 合计 | 4.8 万 | 4.2 万 |
+
+读数:
+
+- **温库只省了 13%**(gemini 那次 18%)。第二批 100 篇带来 952 条新签名(绑定 +195,判无 +719,
+ 未定 +38),第一批是 1014 条:签名跟着文档线性长,不是跟着库收敛。短语对齐每条新签名约 1.4 千 token
+ (新库 1.7 千),和 gemini 量到的一样。
+- **光是抽取加时间解释就到门槛了**:0.48 万 + 0.14 万 = 0.62 万,门槛是约 0.6 万。对齐一个 token 不花,
+ 这条门槛在这套管线上也过不了——它是照紧凑原型(一次抽取调用、类型映射走缓存)定的,而这里每篇
+ 约 1.4 次抽取调用(100 篇 137 次),每次带着整份抽取契约。
+- **对齐三项 2.5 万,占六成,其中输出占八成**(短语对齐输入 32 万、输出 103 万;类别词 8 万、49 万;
+ 提规则 16 万、44 万)。那是 deepseek 在固定的 `low` 下的思考,gemini 同一步每次只有几百。
+- **实体裁决随库变大**:208 次对 102 次。库里的实体从两千到四千,同名候选多了;这一项在温库上是涨的。
+- 第二批召回低于第一批(同句 9.6% 对 13.2%),原因同 gemini 那次:库里判过"无"的签名不再重问。
+
+**对着原文看**:这份语料每篇约 200 个词、270 个 token。温库每篇 4.2 万是原文的约 160 倍,新库 4.8 万约
+180 倍;光抽取 0.48 万就是 18 倍,门槛 0.6 万是 22 倍。文档越短这个倍数越难看——每次调用的提示词
+(契约、本体、候选)是固定开销,不随正文变短。
+
+所以「按签名重定」救不了这条门槛:签名数跟文档数成正比,换个分母,每篇的数还是那么多。能动的是两处,
+都要另外量:对齐在 deepseek 上的思考(`low` 换成关掉,gemini 上关掉时错判从 12% 到 31%,deepseek 没量过),
+和进模型之前先筛掉一部分签名。否则就是记下「不过」接受它。这是 0044 的决定,台子只给数。
+
+**这一轮的台子出过一次岔子,读数时要知道。** 评测脚本 03:00 起了一次,上传完之后进程没了(`runs.log`
+里没有它的退出行),04:06 又起了一次,原因没查出来;服务端一直在跑,文档只传了一遍,第二次接上时抽取
+已经做完。脚本自己的 token 表只算它自己那一段(04:06 之后,每篇 1.6 万,没有抽取),不能用;上面的数
+是服务端日志从 03:00:49 到 04:38:24 整段按 `call=` 归的。短语对齐因此分两段跑(04:06 之前 65 次,之后
+46 次),判定数前后是接着涨的(2664 → 3236),没有看到重问。
+
+费用:180 万输入、239 万输出,约 $1.70(余额少了 ¥10.6);裁判约 400 次调用,$2 上下。用时 98 分。