Skip to content

World's first open-source enterprise world model.

The benchmark grows: corrections, merges, a third lead, and a known gap (#346)

WaylandYang6436db52b302
diff --git a/scripts/bench/README.md b/scripts/bench/README.md
index 1325d3b1..c162ac0f 100644
--- a/scripts/bench/README.md
+++ b/scripts/bench/README.md
@@ -185,35 +185,83 @@ node scripts/bench/run.mjs --corpus pharma --ontology /tmp/schemaorg.ttl --label
——世界轴那一半由脚本自己按 `valid_from` / `valid_to` 过滤。图谱两个接口两根轴都在
服务端。这个不对称照实记在这里,不藏进分数里。
-## 第一轮的数字(2026-09-05,dev @ 1dd917b,DeepSeek-V3 + bge-m3)
+## 题目覆盖什么
-| 配置 | 账本 | 世界轴 | 记录轴 |
-|---|---|---|---|
-| 声明了唯一性公理 | **25 / 25** | 17 / 17 | 8 / 8 |
-| 本体自己长(`--no-declare`) | **15 / 25** | 8 / 17 | 7 / 8 |
+36 道题,一个虚构公司,三波文档。每一波灌完(含这一波的删除、更正、合并)记一个时刻
+`wave-N`,记录轴上的题目以它们为界。
-两组的差**全部落在世界轴上**,而且原因只有一个:时态引擎的自动闭合按
-`functional` / `inverse_functional` 跑,而这两位**永不自动推断**。没有声明的库里,
+| 场景 | 它量的是什么 |
+|---|---|
+| 三任接力 A→B→C | 唯一性闭合,在同一条链上发生两次 |
+| 薪资更正 28000 → 32000 | 属性事实的闭合,世界轴与记录轴各答各的 |
+| 入职早于说出它的文档 | 世界轴靠文档写明的起点,不靠它到达的时间 |
+| 删除一篇文档 | 撤回:记录轴倒回去看得见,今天看不见(#268 的墓碑) |
+| **人改一条事实的起点**(#311) | 改之后世界轴翻转;记录轴倒回改之前看得见旧区间 |
+| **人合并两个拼法** "L. Si" → "Li Si"(#337) | 合并之前的世界:被并掉的实体带着自己的事实重新长回来 |
+| 只知道「2024 年」入职 | 年精度:年内成立,年外不成立 |
+| 「不再担任,日期不详」 | `valid_to_precision = unknown`——已知缺口,见 #345 |
+| 三道答案是**空**的题 | 项目开始前、入职前、撤回后:把开区间读成「自古如此」是最容易犯的错 |
+
+`known_gap: true` 的题目**不进主分**,单独一栏。它们钉住的是已知还没做对的行为,
+修好那天自己翻绿;让主分一直背着一个我们已经知道的数,只会让人不再看它。
+
+## 数字(2026-09-05,dev @ d55a97e,DeepSeek-V3 + bge-m3)
+
+| 配置 | 主分 | 世界轴 | 记录轴 | 已知缺口 |
+|---|---|---|---|---|
+| 声明了唯一性公理 | **36 / 36** | 22 / 22 | 14 / 14 | 0 / 1 |
+| 本体自己长(`--no-declare`) | **29 / 36** | 16 / 22 | 13 / 14 | 0 / 1 |
+
+两组的差落在**需要自动闭合**的那几题上:三任接力(谁在管 Aurora 的四个时刻)和
+薪资更正(两个时刻)。原因只有一个:自动闭合按 `functional` / `inverse_functional`
+跑,而这两位**永不自动推断**(bootstrap_ontology.rs 写了理由)。没有声明的库里,
接任不会闭合前任——账本变成
```
-Zhang San | leads | Project Aurora | from 2023-01-10 | (无终点) | live
+Zhang San | leads | Project Aurora | from 2023-02-01 | (无终点) | live
Li Si | leads | Project Aurora | from 2024-07-05 | (无终点) | live
+Zhou Qi | leads | Project Aurora | from 2025-09-01 | (无终点) | live
```
-两个人同时在管一个项目。这时候问「2023 年六月谁在管」,答案里两个人都在,
-而这正是这个产品最想答对的那类问题。薪资也一样:28000 与 32000 并排开着。
+三个人同时在管一个项目。问「2025 年六月谁在管」,三个都在。**新用户开箱后的库
+就是这个样子**,见 #341。
+
+记录轴几乎不受影响(13/14 vs 14/14):它不依赖任何声明,只看写入时就记下的两列。
+声明那一组里,**合并倒回**与**更正倒回**都是满分——#337 与 #311 两刀在真实摄入
+路径上是对的,不只是在连库测试里对。
+
+第一版(25 题、没有更正/合并/三任接力)的数字是 25/25 对 15/25,留在 git 历史里。
+
+## 对话那一路:24 / 37
+
+同一个库,同一批题,改成在 Chat 里问(`--chat`)。账本那一路量的是账本,这一路量的
+是 **agent 带着工具能不能问到**。差的 13 题不是散的,三堆,每一堆指向一个具体的东西:
+
+| 堆 | 题数 | 是什么 |
+|---|---|---|
+| 记录轴 | 7 | **工具面上没有 `as_of`**。`entity_facts` 只有 `at`,`search_chunks` 连时间都没有;#317/#338 把 `as_of` 铺到了 store 和 HTTP,agent 那一层留了 `None`。模型只能去翻 `changes`,然后在轮数用完前一直在"计划"。见 #347 |
+| 属性 | 6 | **`fact_line` 不渲染字面值**。薪资、职位是 `object_value`,没有 `other_name`,模型看到的是 `salary → ? (2023-06-01 → 2024-02-20)`——区间和置信度都到了,唯独值没到。见 #348 |
+| 否定句 | 1 | 判分按子串:模型说「没有证据表明王五曾经…」,`not` 里的名字出现了,记成错。这是自然语言判分的已知弱点,不改 |
+| agent 没带 `at` | 1 | 周七的入职是 2024 年,模型答成 2023 年六月也在。这一题就是对话那一路该量的东西 |
+
+前两堆各是一处代码,修完对话应该到 35/37 上下——那个数才是产品的数。
-记录轴几乎不受影响(7/8 vs 8/8):它不依赖任何声明,只看 `recorded_at` /
-`invalidated_at`——那两列是写入时就记下的。
+**8/37 那个数作废过一次**:第一版探针读错了 SSE 字段(`.delta` 而不是 `.text`),
+每题拿到的都是空串,而空串不含 `not` 里的名字,八道 `expect: null` 的题就这样被记成
+通过。现在空回复一律记 `error` 不计分,接口失败也单独记。写在这里,因为它说明
+**对话那一路的判分要先验证"真的拿到了回复"**,否则一个解析错误会长得像一个分数。
-**这个差值本身就是第一轮最值得报的结果**:产品的招牌能力在一个「什么都没声明、
-让本体自己长」的库上是关掉的,而那恰恰是新用户开箱后的样子。
-跑出来的另外两件事,都记在这里而不是藏进分数:
+跑出来的其他东西,都记在这里而不是藏进分数:
- **`at` 不给 = 不过滤世界轴**,不是「现在」。`as_of` 不给才是「现在」。两根轴的
缺省口径不对称,`aurora-lead-unfiltered` 这道题把它钉住了。
-- 一条 `salary` 值被抽取丢掉一次(`extraction_drops` 里的 `malformed_item`):
+- **「不再担任,日期不详」被读成仍然成立**(#345):账本专门留了 `valid_to_precision =
+ unknown` 这个状态,而每一处世界轴过滤都按 `valid_to IS NULL` 判活。`known_gap`
+ 那一栏 0/1 就是它。
+- 判分脚本自己犯过一次错:一跳邻域里含邻居之间的边,第一版把它们两端的名字都算成
+ "看见的",于是「李四 2023 年在管 Aurora」被误判出来。现在只看碰到主语的边。
+ 这类错**会把产品的正确行为报成失败**,比漏报更坏,所以写在这里。
+- 一条 `salary` 值被抽取丢掉过一次(`extraction_drops` 里的 `malformed_item`):
声明了 `number` 之后,模型写成 "28000 CNY" 的那一次没能通过校验。宁缺勿脏是
写死的取舍,这里只记它发生过。
diff --git a/scripts/bench/temporal.mjs b/scripts/bench/temporal.mjs
index 8b63f5e9..5334b47e 100644
--- a/scripts/bench/temporal.mjs
+++ b/scripts/bench/temporal.mjs
@@ -18,7 +18,9 @@
// 用法:
// node scripts/bench/temporal.mjs --label rc5
// node scripts/bench/temporal.mjs --label rc5 --chat # 顺带量对话(要一个会话模型)
-// node scripts/bench/temporal.mjs --kb <id> --only-score # 复用已灌好的库,只重打分
+// node scripts/bench/temporal.mjs --kb <id> --stamps '{"wave-1":"…","wave-2":"…"}'
+// # 复用已灌好的库,只重打分
+// node scripts/bench/temporal.mjs --corpus zh # 换一套语料与题目(corpus.zh.json)
//
// 环境变量:BENCH_BASE / BENCH_EMAIL / BENCH_PASSWORD / BENCH_PSQL(同 run.mjs)。
@@ -101,8 +103,13 @@ function valueOf(fact) {
}
async function main() {
- const corpus = JSON.parse(fs.readFileSync(path.join(HERE, "temporal", "corpus.json"), "utf8"));
- const sheet = JSON.parse(fs.readFileSync(path.join(HERE, "temporal", "questions.json"), "utf8"));
+ const suffix = args.corpus ? `.${args.corpus}` : "";
+ const corpus = JSON.parse(
+ fs.readFileSync(path.join(HERE, "temporal", `corpus${suffix}.json`), "utf8"),
+ );
+ const sheet = JSON.parse(
+ fs.readFileSync(path.join(HERE, "temporal", `questions${suffix}.json`), "utf8"),
+ );
try {
await api("POST", "/api/v1/auth/register", {
@@ -118,7 +125,8 @@ async function main() {
await api("POST", "/api/v1/auth/login", { email: EMAIL, password: PASSWORD });
let kb = args.kb;
- let waveOne = args["wave-1"];
+ // 每一波灌完记一个时刻:wave-1、wave-2……记录轴上的题目以它们为界
+ const stamps = args.stamps ? JSON.parse(args.stamps) : {};
if (!kb) {
const ws = (await api("GET", "/api/v1/workspaces"))[0].id;
@@ -200,33 +208,61 @@ async function main() {
);
if (id) await api("DELETE", `/api/v1/documents/${id}`);
}
- if (wave.label === "wave-1") {
- // **这一刻就是记录轴上的界**。等一拍再取,免得同一秒里第二波的写入
- // 也落在这个时刻之内
- await sleep(2000);
- waveOne = new Date().toISOString();
- process.stderr.write(` wave-1 时刻:${waveOne}\n`);
- await sleep(2000);
+ // 人改一条事实的起点(#311):不原地改,作废旧行、写修正行——记录轴倒回去
+ // 能看见改之前的区间,这正是它要量的
+ for (const c of wave.correct || []) {
+ const subject = await entityByName(kb, c.subject);
+ if (!subject) throw new Error(`更正找不到实体:${c.subject}`);
+ const detail = await api("GET", `/api/v1/kbs/${kb}/entities/${subject}`);
+ const fact = (detail.facts || []).find(
+ (f) =>
+ f.other_name === c.object &&
+ (!c.valid_from_was || (f.valid_from || "").startsWith(c.valid_from_was)),
+ );
+ if (!fact) throw new Error(`更正找不到事实:${c.subject} → ${c.object}`);
+ await api("PATCH", `/api/v1/kbs/${kb}/facts/${fact.id}`, {
+ valid_from: `${c.valid_from}T00:00:00Z`,
+ valid_from_precision: c.precision || "day",
+ valid_to: fact.valid_to,
+ valid_to_precision: fact.valid_to_precision,
+ note: c.note,
+ });
+ process.stderr.write(` 更正:${c.subject} → ${c.object} 起点改为 ${c.valid_from}\n`);
}
+ // 人合并两个拼法(#337 那一刀量的就是合并之前的世界)
+ for (const m of wave.merge || []) {
+ const source = await entityByName(kb, m.source);
+ const target = await entityByName(kb, m.target);
+ if (!source || !target) throw new Error(`合并找不到实体:${m.source} / ${m.target}`);
+ await api("POST", `/api/v1/kbs/${kb}/entities/merge`, { source, target });
+ process.stderr.write(` 合并:${m.source} → ${m.target}\n`);
+ }
+ // **这一刻就是记录轴上的界**。前后各等一拍,免得同一秒里下一波的写入
+ // 也落在这个时刻之内
+ await sleep(2000);
+ stamps[wave.label] = new Date().toISOString();
+ process.stderr.write(` ${wave.label} 时刻:${stamps[wave.label]}\n`);
+ await sleep(2000);
}
}
- if (!waveOne) throw new Error("复用库时要给 --wave-1 <ISO 时刻>");
+ if (!Object.keys(stamps).length) throw new Error("复用库时要给 --stamps");
- // 实体按名字找一次,后面所有题目复用
+ // 实体按名字找一次,后面所有题目复用。**合并掉的实体在这里找不到**——
+ // 它已经不是一个节点了;问它的题目从留下的那一方或它挂过的项目那边问
const ids = new Map();
async function entityId(name) {
if (ids.has(name)) return ids.get(name);
- const found = await api("GET", `/api/v1/kbs/${kb}/entities?q=${encodeURIComponent(name)}`);
- const hit = (found.entities || []).find(
- (e) => e.name.toLowerCase() === name.toLowerCase(),
- );
- ids.set(name, hit?.id ?? null);
- return hit?.id ?? null;
+ const id = await entityByName(kb, name);
+ ids.set(name, id);
+ return id;
}
const results = [];
for (const q of sheet.questions) {
- const asOf = q.as_of === "wave-1" ? waveOne : q.as_of || null;
+ const asOf = q.as_of && q.as_of.startsWith("wave-") ? stamps[q.as_of] : q.as_of || null;
+ if (q.as_of && q.as_of.startsWith("wave-") && !asOf) {
+ throw new Error(`题目 ${q.id} 引用了没有记下的时刻 ${q.as_of}`);
+ }
const subject = await entityId(q.subject);
if (!subject) {
// 抽取没抽出这个实体:不是时态答错,单独一档(与 run.mjs 的 absent 同理)
@@ -237,10 +273,13 @@ async function main() {
if (q.at) qs.set("at", q.at);
if (asOf) qs.set("as_of", asOf);
const graph = await api("GET", `/api/v1/kbs/${kb}/graph/neighborhood?${qs}`);
+ // **只看碰到主语的边。** 一跳邻域里还有邻居之间的边(李四 works_for 公司),
+ // 它们在 `at` 那一刻可能成立,但说的不是主语——把它们两端的名字都收进来,
+ // 第一版就是这么把「李四 2023 年在管 Aurora」误判出来的
const names = new Set(
graph.edges
- .flatMap((e) => [e.source, e.target])
- .filter((id) => id !== subject)
+ .filter((e) => e.source === subject || e.target === subject)
+ .map((e) => (e.source === subject ? e.target : e.source))
.map((id) => graph.nodes.find((n) => n.id === id)?.name)
.filter(Boolean),
);
@@ -274,11 +313,21 @@ async function main() {
if (args.chat) {
for (const r of results) {
if (r.outcome === "absent") continue;
- const reply = await askChat(kb, r.ask);
+ let reply;
+ try {
+ reply = await askChat(kb, r.ask);
+ } catch (e) {
+ // 接口本身失败不算答错,单独记 error——否则一个 500 会被记成
+ // 「对话答不出来」,而它答都没答
+ r.chat = { outcome: "error", reply: String(e).slice(0, 200) };
+ continue;
+ }
+ // **空回复永远不算通过**:空字符串当然不含 not 里的名字
r.chat = {
- outcome:
- (r.expect === null || reply.includes(r.expect)) &&
- !(r.not || []).some((bad) => reply.includes(bad))
+ outcome: !reply.trim()
+ ? "error"
+ : (r.expect === null || reply.includes(r.expect)) &&
+ !(r.not || []).some((bad) => reply.includes(bad))
? "pass"
: "fail",
reply: reply.slice(0, 400),
@@ -286,24 +335,41 @@ async function main() {
}
}
- const tally = (pick) => {
- const rows = results.filter((r) => (pick ? r.axis === pick : true) && r.outcome !== "absent");
+ // known_gap 的题目不进主分:它们钉住的是已知还没做对的行为,单独列出来,
+ // 修好那天自己翻绿——而不是让主分一直背着一个我们已经知道的数
+ const tally = (pick, gaps = false) => {
+ const rows = results.filter(
+ (r) =>
+ (pick ? r.axis === pick : true) && r.outcome !== "absent" && !!r.known_gap === gaps,
+ );
const pass = rows.filter((r) => r.outcome === "pass").length;
return { asked: rows.length, pass, rate: rows.length ? +(pass / rows.length).toFixed(3) : null };
};
const report = {
label,
+ corpus: corpus.name,
declared: !("no-declare" in args),
kb,
- wave_one: waveOne,
+ stamps,
at: new Date().toISOString(),
absent: results.filter((r) => r.outcome === "absent").map((r) => r.subject),
- ledger: { all: tally(null), world: tally("world"), record: tally("record") },
+ ledger: {
+ all: tally(null),
+ world: tally("world"),
+ record: tally("record"),
+ known_gaps: tally(null, true),
+ },
chat: args.chat
? (() => {
- const rows = results.filter((r) => r.chat);
+ const rows = results.filter((r) => r.chat && r.chat.outcome !== "error");
const pass = rows.filter((r) => r.chat.outcome === "pass").length;
- return { asked: rows.length, pass, rate: rows.length ? +(pass / rows.length).toFixed(3) : null };
+ const errors = results.filter((r) => r.chat?.outcome === "error").length;
+ return {
+ asked: rows.length,
+ pass,
+ errors,
+ rate: rows.length ? +(pass / rows.length).toFixed(3) : null,
+ };
})()
: null,
questions: results,
@@ -313,32 +379,54 @@ async function main() {
`\n账本:${report.ledger.all.pass}/${report.ledger.all.asked}` +
`(世界轴 ${report.ledger.world.pass}/${report.ledger.world.asked},` +
`记录轴 ${report.ledger.record.pass}/${report.ledger.record.asked})` +
+ (report.ledger.known_gaps.asked
+ ? ` 已知缺口 ${report.ledger.known_gaps.pass}/${report.ledger.known_gaps.asked} 通过`
+ : "") +
(report.chat ? ` 对话:${report.chat.pass}/${report.chat.asked}` : "") +
`\n`,
);
}
+/// 按名字找一个**现在还是节点**的实体。
+async function entityByName(kb, name) {
+ const found = await api("GET", `/api/v1/kbs/${kb}/entities?q=${encodeURIComponent(name)}`);
+ const hit = (found.entities || []).find((e) => e.name.toLowerCase() === name.toLowerCase());
+ return hit?.id ?? null;
+}
+
/// 一次对话问答。SSE 流里只取正文。
async function askChat(kb, question) {
+ // 帧的形状与 `web/src/api.ts` 里客户端解析的一致:`event:` 一行定类型,
+ // `data:` 一行是 JSON;正文在 `delta` 帧的 `{ text }` 里。第一版只读 `data:`
+ // 又去找 `.delta`,于是每一题都得到空串——空串不含 `not` 里的名字,
+ // 八道 expect=null 的题就这样被记成了通过
const res = await fetch(`${BASE}/api/v1/kbs/${kb}/chat`, {
method: "POST",
headers: { "content-type": "application/json", cookie },
body: JSON.stringify({ message: question }),
});
const text = await res.text();
+ if (!res.ok) throw new Error(`chat → ${res.status} ${text.slice(0, 200)}`);
let answer = "";
- for (const line of text.split("\n")) {
- if (!line.startsWith("data:")) continue;
- const payload = line.slice(5).trim();
- if (!payload || payload === "[DONE]") continue;
- try {
- const frame = JSON.parse(payload);
- if (typeof frame === "string") answer += frame;
- else if (frame.delta) answer += frame.delta;
- } catch {
- answer += payload;
+ let error = null;
+ for (const frame of text.split("\n\n")) {
+ let event = "message";
+ let data = "";
+ for (const line of frame.split("\n")) {
+ if (line.startsWith("event:")) event = line.slice(6).trim();
+ else if (line.startsWith("data:")) data += line.slice(5).trim();
+ }
+ if (event === "delta" && data) {
+ try {
+ answer += JSON.parse(data).text ?? "";
+ } catch {
+ /* 半截帧,跳过 */
+ }
+ } else if (event === "error") {
+ error = data;
}
}
+ if (error && !answer) throw new Error(`chat error frame: ${error.slice(0, 200)}`);
return answer;
}
diff --git a/scripts/bench/temporal/corpus.json b/scripts/bench/temporal/corpus.json
index 589c8aff..c843c3d5 100644
--- a/scripts/bench/temporal/corpus.json
+++ b/scripts/bench/temporal/corpus.json
@@ -1,66 +1,11 @@
{
"name": "meridian",
"note": "虚构公司,刻意不用真实实体:模型读过的东西量出来的是记忆,不是这条流水线。句子写得钝——一句一个事实、日期写全、名字写全——因为这个台子要量的是时间那根轴,不是抽取的运气。",
- "waves": [
- {
- "label": "wave-1",
- "note": "第一波灌完之后记一个时刻。记录轴上的所有题目都以它为界:那时我们还不知道后来的交接与更正。",
- "documents": [
- {
- "filename": "aurora-charter.md",
- "doc_time": "2023-01-10",
- "text": "# Project Aurora charter\n\nProject Aurora started on 2023-01-10 at Meridian Systems.\n\nZhang San leads Project Aurora from 2023-01-10.\n\nProject Aurora is part of the Platform Group.\n\nThe Platform Group is part of Meridian Systems.\n"
- },
- {
- "filename": "lin-zhao-offer.md",
- "doc_time": "2023-06-01",
- "text": "# Offer record\n\nLin Zhao joined Meridian Systems on 2023-06-01.\n\nLin Zhao works for Meridian Systems from 2023-06-01.\n\nLin Zhao has the job title Staff Engineer from 2023-06-01.\n\nLin Zhao has a salary of 28000 CNY from 2023-06-01.\n"
- },
- {
- "filename": "aurora-advisor-note.md",
- "doc_time": "2023-03-02",
- "text": "# Assistant note\n\nWang Wu advises Project Aurora from 2023-03-02.\n\nWang Wu works for Meridian Systems from 2020-09-01.\n"
- }
- ],
- "delete": []
- },
- {
- "label": "wave-2",
- "note": "第二波在第一个时刻之后才灌:世界轴上它们讲的是 2024 年的事,记录轴上它们是「后来才知道的」。删除那一条制造撤回。",
- "documents": [
- {
- "filename": "aurora-handover.md",
- "doc_time": "2024-07-05",
- "text": "# Handover memo\n\nLi Si leads Project Aurora from 2024-07-05.\n\nLi Si works for Meridian Systems from 2021-04-01.\n"
- },
- {
- "filename": "lin-zhao-correction.md",
- "doc_time": "2024-02-20",
- "text": "# Payroll correction\n\nLin Zhao has a salary of 32000 CNY from 2024-02-20.\n\nThe raise was approved by the Platform Group.\n"
- }
- ],
- "delete": [
- "aurora-advisor-note.md"
- ]
- }
- ],
"axioms_note": "时态引擎的自动闭合**只按本体的声明跑**:`functional` / `inverse_functional` 永不自动推断(理由见 bootstrap_ontology.rs——它驱动的是自动改写账本)。所以这两条声明是被测配置的一部分,跑分前先建出来;`--no-declare` 跑的是「本体自己长出来」的那种库,两组的差就是这条声明值多少。",
"classes": [
- {
- "key": "person",
- "label": "Person",
- "description": "A human being."
- },
- {
- "key": "organization",
- "label": "Organization",
- "description": "A company or a group inside one."
- },
- {
- "key": "project",
- "label": "Project",
- "description": "A named piece of work with people responsible for it."
- }
+ { "key": "person", "label": "Person", "description": "A human being." },
+ { "key": "organization", "label": "Organization", "description": "A company or a group inside one." },
+ { "key": "project", "label": "Project", "description": "A named piece of work with people responsible for it." }
],
"axioms": [
{
@@ -86,6 +31,13 @@
"is_transitive": true,
"description": "A group or project inside a larger organization."
},
+ {
+ "key": "advises",
+ "label": "advises",
+ "kind": "relation",
+ "temporal": "state",
+ "description": "A person who advises a project without being responsible for it. Several advisers are possible."
+ },
{
"key": "salary",
"label": "salary",
@@ -104,13 +56,79 @@
"domain": "person",
"functional": true,
"description": "The person's job title. One title holds at a time."
+ }
+ ],
+ "waves_note": "每一波灌完(含这一波的删除、更正、合并)记一个时刻 `wave-N`。记录轴上的题目以它们为界。一波之内的顺序固定:先灌文档并等抽取完,再删除,再更正区间,再合并。",
+ "waves": [
+ {
+ "label": "wave-1",
+ "note": "起点。这时我们只知道张三在管 Aurora、林昭六月入职、王五在给 Aurora 当顾问。",
+ "documents": [
+ {
+ "filename": "aurora-charter.md",
+ "doc_time": "2023-01-10",
+ "text": "# Project Aurora charter\n\nProject Aurora started on 2023-01-10 at Meridian Systems.\n\nZhang San leads Project Aurora from 2023-01-10.\n\nProject Aurora is part of the Platform Group.\n\nThe Platform Group is part of Meridian Systems.\n"
+ },
+ {
+ "filename": "lin-zhao-offer.md",
+ "doc_time": "2023-06-01",
+ "text": "# Offer record\n\nLin Zhao joined Meridian Systems on 2023-06-01.\n\nLin Zhao works for Meridian Systems from 2023-06-01.\n\nLin Zhao has the job title Staff Engineer from 2023-06-01.\n\nLin Zhao has a salary of 28000 CNY from 2023-06-01.\n"
+ },
+ {
+ "filename": "aurora-advisor-note.md",
+ "doc_time": "2023-03-02",
+ "text": "# Assistant note\n\nWang Wu advises Project Aurora from 2023-03-02.\n\nWang Wu works for Meridian Systems from 2020-09-01.\n"
+ }
+ ]
},
{
- "key": "advises",
- "label": "advises",
- "kind": "relation",
- "temporal": "state",
- "description": "A person who advises a project without being responsible for it. Several advisers are possible."
+ "label": "wave-2",
+ "note": "交接、薪资更正、一次删除,外加一个**拼写不同的李四**(\"L. Si\")——消解认不出它们是同一个人,于是库里多出一个实体,等下一波由人合并。",
+ "documents": [
+ {
+ "filename": "aurora-handover.md",
+ "doc_time": "2024-07-05",
+ "text": "# Handover memo\n\nLi Si leads Project Aurora from 2024-07-05.\n\nLi Si works for Meridian Systems from 2021-04-01.\n"
+ },
+ {
+ "filename": "lin-zhao-correction.md",
+ "doc_time": "2024-02-20",
+ "text": "# Payroll correction\n\nLin Zhao has a salary of 32000 CNY from 2024-02-20.\n\nThe raise was approved by the Platform Group.\n"
+ },
+ {
+ "filename": "helios-kickoff.md",
+ "doc_time": "2024-08-12",
+ "text": "# Project Helios kickoff\n\nProject Helios started on 2024-08-12 at Meridian Systems.\n\nL. Si leads Project Helios from 2024-08-12.\n\nProject Helios is part of the Platform Group.\n"
+ }
+ ],
+ "delete": ["aurora-advisor-note.md"]
+ },
+ {
+ "label": "wave-3",
+ "note": "第三任接手(三任接力)、一个只知道年份的入职、一条不带日期的「不再担任」,以及两件**人做的事**:改一条事实的起点,合并两个拼法。",
+ "documents": [
+ {
+ "filename": "aurora-third-lead.md",
+ "doc_time": "2025-09-01",
+ "text": "# Leadership update\n\nZhou Qi leads Project Aurora from 2025-09-01.\n\nZhou Qi joined Meridian Systems in 2024.\n\nZhou Qi works for Meridian Systems from 2024.\n"
+ },
+ {
+ "filename": "lin-zhao-role-change.md",
+ "doc_time": "2025-10-15",
+ "text": "# Role note\n\nLin Zhao no longer holds the job title Staff Engineer.\n\nThe date of the change was not recorded.\n"
+ }
+ ],
+ "correct": [
+ {
+ "subject": "Zhang San",
+ "object": "Project Aurora",
+ "valid_from_was": "2023-01-10",
+ "valid_from": "2023-02-01",
+ "precision": "day",
+ "note": "The charter date was the approval date; Zhang San took over on 2023-02-01."
+ }
+ ],
+ "merge": [{ "source": "L. Si", "target": "Li Si" }]
}
]
}
diff --git a/scripts/bench/temporal/questions.json b/scripts/bench/temporal/questions.json
index 54f9b02f..f8c2d5f4 100644
--- a/scripts/bench/temporal/questions.json
+++ b/scripts/bench/temporal/questions.json
@@ -1,5 +1,5 @@
{
- "note": "每道题问的是**某一个时刻**的图。`axis` 说的是哪根轴在动:world = 世界轴(那时世界是什么样,参数 at),record = 记录轴(那时我们以为世界是什么样,参数 as_of)。`as_of: \"wave-1\"` 由跑分脚本换成第一波灌完那一刻的时间戳。\n\n判分不认谓词的名字:本体在不同库里长得不一样(works_for / employed_by),而这个台子量的是时间,不是词汇。判据是「这个实体在那一刻挂着的事实里,有没有出现 expect,有没有出现 not 里的任何一个」。",
+ "note": "每道题问的是**某一个时刻**的图。`axis` 说的是哪根轴在动:world = 世界轴(那时世界是什么样,参数 at),record = 记录轴(那时我们以为世界是什么样,参数 as_of)。`as_of: \"wave-N\"` 由跑分脚本换成第 N 波灌完那一刻的时间戳。\n\n判分不认谓词的名字:本体在不同库里长得不一样(works_for / employed_by),而这个台子量的是时间,不是词汇。判据是「这个实体在那一刻挂着的事实里,有没有出现 expect,有没有出现 not 里的任何一个」。\n\n`known_gap: true` 的题目单独计数:它们钉住的是一个已知还没做对的行为,写在这里是为了让它被修的那天看得见,不是为了把分数拉低。",
"questions": [
{
"id": "aurora-lead-2023-02",
@@ -7,9 +7,7 @@
"at": "2023-02-01",
"subject": "Project Aurora",
"expect": "Zhang San",
- "not": [
- "Li Si"
- ],
+ "not": ["Li Si", "Zhou Qi"],
"ask": "Who led Project Aurora in February 2023?"
},
{
@@ -18,9 +16,7 @@
"at": "2023-06-15",
"subject": "Project Aurora",
"expect": "Zhang San",
- "not": [
- "Li Si"
- ],
+ "not": ["Li Si", "Zhou Qi"],
"ask": "Who led Project Aurora in June 2023?"
},
{
@@ -29,9 +25,7 @@
"at": "2024-06-30",
"subject": "Project Aurora",
"expect": "Zhang San",
- "not": [
- "Li Si"
- ],
+ "not": ["Li Si", "Zhou Qi"],
"ask": "Who led Project Aurora at the end of June 2024, days before the handover?"
},
{
@@ -40,9 +34,7 @@
"at": "2024-08-01",
"subject": "Project Aurora",
"expect": "Li Si",
- "not": [
- "Zhang San"
- ],
+ "not": ["Zhang San", "Zhou Qi"],
"ask": "Who led Project Aurora in August 2024?"
},
{
@@ -51,22 +43,38 @@
"at": "2025-06-01",
"subject": "Project Aurora",
"expect": "Li Si",
- "not": [
- "Zhang San"
- ],
+ "not": ["Zhang San", "Zhou Qi"],
"ask": "Who led Project Aurora in June 2025?"
},
+ {
+ "id": "aurora-lead-2025-10",
+ "axis": "world",
+ "at": "2025-10-01",
+ "subject": "Project Aurora",
+ "expect": "Zhou Qi",
+ "not": ["Zhang San", "Li Si"],
+ "ask": "Who led Project Aurora in October 2025?",
+ "why": "三任接力:第二次闭合要发生在第一次闭合过的同一条链上。"
+ },
{
"id": "aurora-lead-now",
"axis": "world",
"at": "2026-01-01",
"subject": "Project Aurora",
- "expect": "Li Si",
- "not": [
- "Zhang San"
- ],
+ "expect": "Zhou Qi",
+ "not": ["Zhang San", "Li Si"],
"ask": "Who leads Project Aurora in January 2026?"
},
+ {
+ "id": "aurora-lead-unfiltered",
+ "axis": "world",
+ "at": null,
+ "subject": "Project Aurora",
+ "expect": "Li Si",
+ "not": [],
+ "ask": "Who has led Project Aurora at any point?",
+ "why": "`at` 不给时服务端**不过滤世界轴**——问的是「我们持有的全部」,不是「现在」。这一题把那个口径钉住:三任都该在,而不是只剩现任。"
+ },
{
"id": "aurora-lead-record-wave1-at-2024-08",
"axis": "record",
@@ -74,9 +82,7 @@
"at": "2024-08-01",
"subject": "Project Aurora",
"expect": "Zhang San",
- "not": [
- "Li Si"
- ],
+ "not": ["Li Si"],
"ask": "Before the handover memo arrived, who did we think would be leading Project Aurora in August 2024?",
"why": "两根轴的分界就在这一题:世界轴问的是 2024 年八月,而记录轴停在我们还没读到交接备忘的那一刻——那时的答案只能是张三。答成李四,就是拿今天的认知去答当时。"
},
@@ -87,21 +93,60 @@
"at": null,
"subject": "Project Aurora",
"expect": "Zhang San",
- "not": [
- "Li Si"
- ],
+ "not": ["Li Si"],
"ask": "As of the first ingest, who was recorded as leading Project Aurora?"
},
+ {
+ "id": "aurora-lead-record-wave2-at-2025-10",
+ "axis": "record",
+ "as_of": "wave-2",
+ "at": "2025-10-01",
+ "subject": "Project Aurora",
+ "expect": "Li Si",
+ "not": ["Zhou Qi"],
+ "ask": "Before the third lead was announced, who did we expect to be leading Project Aurora in October 2025?",
+ "why": "同一根链上再往前推一格:第二波结束时我们知道李四接了张三,还不知道周七。"
+ },
{
"id": "aurora-lead-record-now",
"axis": "record",
"as_of": null,
"at": null,
"subject": "Project Aurora",
- "expect": "Li Si",
+ "expect": "Zhou Qi",
"not": [],
"ask": "As we understand it today, who leads Project Aurora?"
},
+ {
+ "id": "aurora-lead-corrected-start-2023-01-20",
+ "axis": "world",
+ "at": "2023-01-20",
+ "subject": "Project Aurora",
+ "expect": null,
+ "not": ["Zhang San"],
+ "ask": "Who led Project Aurora on 20 January 2023?",
+ "why": "第三波里有人把张三的起点从 01-10 改成了 02-01(#311 那条路:不原地改,作废旧行、写修正行)。改完之后一月二十日该是空的。"
+ },
+ {
+ "id": "aurora-lead-corrected-start-2023-02-05",
+ "axis": "world",
+ "at": "2023-02-05",
+ "subject": "Project Aurora",
+ "expect": "Zhang San",
+ "not": [],
+ "ask": "Who led Project Aurora on 5 February 2023?"
+ },
+ {
+ "id": "aurora-lead-record-wave2-at-2023-01-20",
+ "axis": "record",
+ "as_of": "wave-2",
+ "at": "2023-01-20",
+ "subject": "Project Aurora",
+ "expect": "Zhang San",
+ "not": [],
+ "ask": "Before the start date was corrected, who did we have leading Project Aurora on 20 January 2023?",
+ "why": "更正之前的认知:那时账本上张三从 01-10 起就在管。记录轴倒回去要看得见改之前的区间——那正是 0019 说的「修正是记录轴唯一值得回放的内容」。"
+ },
{
"id": "aurora-parent",
"axis": "world",
@@ -135,9 +180,7 @@
"at": "2023-01-01",
"subject": "Lin Zhao",
"expect": null,
- "not": [
- "Meridian Systems"
- ],
+ "not": ["Meridian Systems"],
"ask": "Who did Lin Zhao work for in January 2023?",
"why": "入职是六月的事。一月的答案应当是空,而不是把开区间当成「一直如此」——`valid_from` 之前不成立,正是双时态存在的理由。"
},
@@ -147,9 +190,7 @@
"at": "2023-09-01",
"subject": "Lin Zhao",
"expect": "28000",
- "not": [
- "32000"
- ],
+ "not": ["32000"],
"ask": "What was Lin Zhao's salary in September 2023?"
},
{
@@ -158,9 +199,7 @@
"at": "2024-06-01",
"subject": "Lin Zhao",
"expect": "32000",
- "not": [
- "28000"
- ],
+ "not": ["28000"],
"ask": "What was Lin Zhao's salary in June 2024?"
},
{
@@ -169,9 +208,7 @@
"at": "2026-01-01",
"subject": "Lin Zhao",
"expect": "32000",
- "not": [
- "28000"
- ],
+ "not": ["28000"],
"ask": "What is Lin Zhao's salary in January 2026?"
},
{
@@ -181,9 +218,7 @@
"at": null,
"subject": "Lin Zhao",
"expect": "28000",
- "not": [
- "32000"
- ],
+ "not": ["32000"],
"ask": "Before the payroll correction arrived, what salary did we have on record for Lin Zhao?"
},
{
@@ -193,13 +228,11 @@
"at": "2024-06-01",
"subject": "Lin Zhao",
"expect": "28000",
- "not": [
- "32000"
- ],
+ "not": ["32000"],
"ask": "Before the correction, what did we think Lin Zhao's June 2024 salary was?"
},
{
- "id": "lin-zhao-title",
+ "id": "lin-zhao-title-2023-09",
"axis": "world",
"at": "2023-09-01",
"subject": "Lin Zhao",
@@ -207,6 +240,17 @@
"not": [],
"ask": "What was Lin Zhao's job title in September 2023?"
},
+ {
+ "id": "lin-zhao-title-ended-unknown",
+ "axis": "world",
+ "at": "2026-01-01",
+ "subject": "Lin Zhao",
+ "expect": null,
+ "not": ["Staff Engineer"],
+ "known_gap": true,
+ "ask": "What is Lin Zhao's job title in January 2026?",
+ "why": "「不再担任,日期不详」是账本专门留的一个状态(valid_to 为空、精度为 unknown)。世界轴按 `valid_to IS NULL` 判活时会把它当成仍然成立——这一题钉住那个口径,修好的那天它自己会翻绿。"
+ },
{
"id": "adviser-record-wave1",
"axis": "record",
@@ -216,7 +260,7 @@
"expect": "Wang Wu",
"not": [],
"ask": "As of the first ingest, was Wang Wu recorded as advising Project Aurora?",
- "why": "那条未经证实的笔记当时确实在库里。记录轴回到那一刻就该看得见它——账本记的是「我们曾经这么以为」,抹掉它等于替过去修饰。"
+ "why": "那条笔记当时确实在库里。记录轴回到那一刻就该看得见它——账本记的是「我们曾经这么以为」,抹掉它等于替过去修饰。"
},
{
"id": "adviser-now",
@@ -225,9 +269,7 @@
"at": "2023-06-01",
"subject": "Project Aurora",
"expect": null,
- "not": [
- "Wang Wu"
- ],
+ "not": ["Wang Wu"],
"ask": "Do we still hold that Wang Wu advised Project Aurora in June 2023?",
"why": "那篇文档已经删了,这条事实的出处没了。世界轴仍问 2023 年六月,而今天的认知里它不该再成立。"
},
@@ -248,34 +290,82 @@
"at": "2022-06-01",
"subject": "Li Si",
"expect": null,
- "not": [
- "Meridian Systems"
- ],
+ "not": ["Meridian Systems"],
"ask": "As of the first ingest, what did we know about where Li Si worked in 2022?",
"why": "上一题的反面:那时这个人还没进过我们的账本。世界轴上成立,记录轴上还不知道——两根轴各答各的。"
},
{
- "id": "aurora-lead-before-start",
+ "id": "helios-lead-record-wave2",
+ "axis": "record",
+ "as_of": "wave-2",
+ "at": null,
+ "subject": "Project Helios",
+ "expect": "L. Si",
+ "not": ["Li Si"],
+ "ask": "As of the second ingest, who was recorded as leading Project Helios?",
+ "why": "合并之前的世界(#337 那一刀):两种拼法是两个实体,Helios 挂在 \"L. Si\" 身上。记录轴倒回合并之前,那个被并掉的实体要重新长回来、带着它自己的事实。"
+ },
+ {
+ "id": "helios-lead-now",
+ "axis": "record",
+ "as_of": null,
+ "at": null,
+ "subject": "Project Helios",
+ "expect": "Li Si",
+ "not": ["L. Si"],
+ "ask": "As we understand it today, who leads Project Helios?",
+ "why": "合并之后:Helios 的事实搬到了留下的那个实体身上。"
+ },
+ {
+ "id": "li-si-helios-record-wave2",
+ "axis": "record",
+ "as_of": "wave-2",
+ "at": null,
+ "subject": "Li Si",
+ "expect": null,
+ "not": ["Project Helios"],
+ "ask": "As of the second ingest, did we have Li Si connected to Project Helios?",
+ "why": "合并之前,\"Li Si\" 这个实体跟 Helios 没有任何关系——那条事实当时在另一个实体身上。"
+ },
+ {
+ "id": "li-si-helios-now",
+ "axis": "record",
+ "as_of": null,
+ "at": null,
+ "subject": "Li Si",
+ "expect": "Project Helios",
+ "not": [],
+ "ask": "Today, is Li Si connected to Project Helios?"
+ },
+ {
+ "id": "zhou-qi-employer-2024-06",
"axis": "world",
- "at": "2022-12-01",
- "subject": "Project Aurora",
+ "at": "2024-06-15",
+ "subject": "Zhou Qi",
+ "expect": "Meridian Systems",
+ "not": [],
+ "ask": "Who did Zhou Qi work for in June 2024?",
+ "why": "只知道「2024 年」入职。年精度的起点落在年初,六月在年内,成立。"
+ },
+ {
+ "id": "zhou-qi-employer-2023-06",
+ "axis": "world",
+ "at": "2023-06-15",
+ "subject": "Zhou Qi",
"expect": null,
- "not": [
- "Zhang San",
- "Li Si"
- ],
- "ask": "Who led Project Aurora in December 2022?",
- "why": "项目 2023 年一月才开始。这一题防的是把开区间读成「自古如此」。"
+ "not": ["Meridian Systems"],
+ "ask": "Who did Zhou Qi work for in June 2023?",
+ "why": "年精度的另一边:2023 年在「2024 年」之前,不成立。"
},
{
- "id": "aurora-lead-unfiltered",
+ "id": "aurora-lead-before-start",
"axis": "world",
- "at": null,
+ "at": "2022-12-01",
"subject": "Project Aurora",
- "expect": "Li Si",
- "not": [],
- "ask": "Who has led Project Aurora at any point?",
- "why": "`at` 不给时服务端**不过滤世界轴**——问的是「我们持有的全部」,不是「现在」。这一题把那个口径钉住:两任都该在,而不是只剩现任。"
+ "expect": null,
+ "not": ["Zhang San", "Li Si", "Zhou Qi"],
+ "ask": "Who led Project Aurora in December 2022?",
+ "why": "项目 2023 年才开始。这一题防的是把开区间读成「自古如此」。"
}
]
}