Skip to content

World's first open-source enterprise world model.

A benchmark that asks the question this project is built around (#340)

WaylandYangd55a97e00be8
diff --git a/scripts/bench/README.md b/scripts/bench/README.md
index aae03618..1325d3b1 100644
--- a/scripts/bench/README.md
+++ b/scripts/bench/README.md
@@ -134,3 +134,86 @@ node scripts/bench/run.mjs --corpus pharma --ontology /tmp/schemaorg.ttl --label
不是替代,是一条候选预扫的线:把分块检索出的几十个候选类交给它打一遍,再把(片段,类型)
作为提示喂给抽取,有机会把抽取时的类型精度抬到消解后的水平、省掉一轮消解调用;它找到而
我们没找到的,直接进 drop signal 做漏抽信号。这个实验还没做。
+
+---
+
+# 时态问答的测量台(#306)
+
+`node scripts/bench/temporal.mjs --label rc5`
+
+上面那个台子量的是**类型消解**。这一个量的是这个产品真正围绕着造的那件事:
+**按某个日期回答问题,且在事实变过之后仍答对**。
+
+在它之前,仓库能引用的数字只有 Ontology2SQL 在 BIRD Mini-Dev 上的成绩。别人家的
+记忆基准(LongMemEval、DMR)测的是对话回忆,没有一个会问「2023 年三月这个项目是谁
+在管」,而语料里那个答案还变过两次。在别人的基准上拿个中游名次,说明的东西还不如
+把那个缺失的基准公布出来。
+
+## 两根轴各问一遍
+
+这是这份题目里别处没有的一维:
+
+| 轴 | 参数 | 问的是 |
+|---|---|---|
+| world | `at` | 那时**世界**是什么样 |
+| record | `as_of` | 那时**我们以为**世界是什么样(0019) |
+
+同一个实体、同一个日期,两根轴的正确答案可以不同——`aurora-lead-record-wave1-at-2024-08`
+就是这么一道:世界轴问 2024 年八月(那时李四已接手),记录轴停在我们还没读到交接
+备忘的那一刻(那时只知道张三)。答成李四,就是拿今天的认知去答当时。
+
+## 记录轴不是改时钟改出来的
+
+语料分**两波**灌,第一波灌完记下一个时刻(`wave-1`),第二波才带来交接、薪资更正与
+一次文档删除。所有 `as_of: "wave-1"` 的题目问的就是那一刻。这样量到的是产品真实
+走过的路——一次真的摄入、一次真的抽取、一次真的删除——而不是 `UPDATE` 出来的场面。
+
+## 判分
+
+- 判据是「这个实体在那一刻挂着的东西里,有没有 `expect`,有没有 `not` 里的任何一个」,
+ **不认谓词的名字**:本体在不同库里长得不一样(works_for / employed_by),而这个台子
+ 量的是时间,不是词汇。
+- `expect: null` 表示**那时不该有答案**(项目还没开始、人还没入职、事实已被撤回)。
+ 把开区间读成「自古如此」是双时态最容易犯的错,所以专门留了三道。
+- `absent`(抽取压根没抽出这个实体)单独一档,不算时态答错——与上面那个台子同理。
+- `--chat` 再跑一遍**对话**:模型带着工具自己去问。账本那一路量的是账本,对话那一路
+ 量的是 agent 那一层,两者的差就是工具用得好不好。
+
+## 已知的口径缺口
+
+属性事实(薪资、职位)不是边,得从实体面板取,而那个接口今天**只有 `as_of`,没有 `at`**
+——世界轴那一半由脚本自己按 `valid_from` / `valid_to` 过滤。图谱两个接口两根轴都在
+服务端。这个不对称照实记在这里,不藏进分数里。
+
+## 第一轮的数字(2026-09-05,dev @ 1dd917b,DeepSeek-V3 + bge-m3)
+
+| 配置 | 账本 | 世界轴 | 记录轴 |
+|---|---|---|---|
+| 声明了唯一性公理 | **25 / 25** | 17 / 17 | 8 / 8 |
+| 本体自己长(`--no-declare`) | **15 / 25** | 8 / 17 | 7 / 8 |
+
+两组的差**全部落在世界轴上**,而且原因只有一个:时态引擎的自动闭合按
+`functional` / `inverse_functional` 跑,而这两位**永不自动推断**。没有声明的库里,
+接任不会闭合前任——账本变成
+
+```
+Zhang San | leads | Project Aurora | from 2023-01-10 | (无终点) | live
+Li Si | leads | Project Aurora | from 2024-07-05 | (无终点) | live
+```
+
+两个人同时在管一个项目。这时候问「2023 年六月谁在管」,答案里两个人都在,
+而这正是这个产品最想答对的那类问题。薪资也一样:28000 与 32000 并排开着。
+
+记录轴几乎不受影响(7/8 vs 8/8):它不依赖任何声明,只看 `recorded_at` /
+`invalidated_at`——那两列是写入时就记下的。
+
+**这个差值本身就是第一轮最值得报的结果**:产品的招牌能力在一个「什么都没声明、
+让本体自己长」的库上是关掉的,而那恰恰是新用户开箱后的样子。
+
+跑出来的另外两件事,都记在这里而不是藏进分数:
+
+- **`at` 不给 = 不过滤世界轴**,不是「现在」。`as_of` 不给才是「现在」。两根轴的
+ 缺省口径不对称,`aurora-lead-unfiltered` 这道题把它钉住了。
+- 一条 `salary` 值被抽取丢掉一次(`extraction_drops` 里的 `malformed_item`):
+ 声明了 `number` 之后,模型写成 "28000 CNY" 的那一次没能通过校验。宁缺勿脏是
+ 写死的取舍,这里只记它发生过。
diff --git a/scripts/bench/temporal.mjs b/scripts/bench/temporal.mjs
new file mode 100644
index 00000000..8b63f5e9
--- /dev/null
+++ b/scripts/bench/temporal.mjs
@@ -0,0 +1,348 @@
+#!/usr/bin/env node
+// 时态问答的测量台(#306)。
+//
+// 这个仓库唯一能引用的数字是 Ontology2SQL 在 BIRD Mini-Dev 上的成绩,而它围绕着
+// 造的那件事——**按某个日期回答问题,且在事实变过之后仍答对**——没有任何东西在量。
+// 别人家的记忆基准测的是对话回忆,没有一个会问「2023 年三月这个项目是谁在管」,
+// 而语料里那个答案还变过两次。在别人的基准上拿个中游名次,说明的东西还不如
+// 把那个缺失的基准公布出来。
+//
+// **两根轴各问一遍**,这是这份题目里别处没有的一维:
+// world —— 那时世界是什么样(参数 at)
+// record —— 那时我们以为世界是什么样(参数 as_of,0019)
+//
+// 记录轴不靠改时钟造出来:语料分两波灌,**第一波灌完记下一个时刻**,第二波带来
+// 交接、更正与一次删除。`as_of: "wave-1"` 的题目问的就是那一刻——那时账本里还
+// 没有后来的事。这样量到的是产品真实走过的路,不是 UPDATE 出来的场面。
+//
+// 用法:
+// node scripts/bench/temporal.mjs --label rc5
+// node scripts/bench/temporal.mjs --label rc5 --chat # 顺带量对话(要一个会话模型)
+// node scripts/bench/temporal.mjs --kb <id> --only-score # 复用已灌好的库,只重打分
+//
+// 环境变量:BENCH_BASE / BENCH_EMAIL / BENCH_PASSWORD / BENCH_PSQL(同 run.mjs)。
+
+import fs from "node:fs";
+import path from "node:path";
+import { execSync } from "node:child_process";
+import { fileURLToPath } from "node:url";
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const BASE = process.env.BENCH_BASE || "http://localhost:18080";
+const EMAIL = process.env.BENCH_EMAIL || "[email protected]";
+const PASSWORD = process.env.BENCH_PASSWORD || "benchbench123";
+
+const args = Object.fromEntries(
+ process.argv.slice(2).map((a, i, all) => {
+ const m = a.match(/^--([^=]+)(?:=(.*))?$/);
+ return m ? [m[1], m[2] ?? (all[i + 1]?.startsWith("--") ? true : all[i + 1] ?? true)] : [a, true];
+ }),
+);
+const label = args.label || "temporal";
+
+let cookie = "";
+async function api(method, url, body) {
+ const res = await fetch(BASE + url, {
+ method,
+ headers: {
+ ...(body ? { "content-type": "application/json" } : {}),
+ ...(cookie ? { cookie } : {}),
+ },
+ body: body ? JSON.stringify(body) : undefined,
+ });
+ // getSetCookie(复数):多个 Set-Cookie 只取第一个会丢掉会话那条
+ for (const c of res.headers.getSetCookie?.() ?? []) cookie = c.split(";")[0];
+ const text = await res.text();
+ if (!res.ok) throw new Error(`${method} ${url} → ${res.status} ${text.slice(0, 300)}`);
+ return text ? JSON.parse(text) : null;
+}
+
+function psql(sql) {
+ const cmd =
+ process.env.BENCH_PSQL ||
+ "docker exec -e PGPASSWORD=utopia landscapebi-db-1 psql -U utopia -d utopia -tAc";
+ return execSync(`${cmd} "${sql.replace(/"/g, '\\"')}"`, { encoding: "utf8" }).trim();
+}
+const num = (sql) => Number(psql(sql) || 0);
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+
+/// 卡住才算超时,慢不算(与 run.mjs 同一条理由:抽取在服务端排队,驱动脚本
+/// 死掉不影响它们,而按总时长封顶会把跑成功的组报成失败)。
+async function until(fn, everyMs = 8000, stallMs = 8 * 60 * 1000) {
+ let last = null;
+ let deadline = Date.now() + stallMs;
+ for (;;) {
+ const r = await fn();
+ if (r === true) return;
+ if (r !== last) {
+ last = r;
+ deadline = Date.now() + stallMs;
+ }
+ if (Date.now() > deadline) throw new Error("等超时:八分钟没有任何进展");
+ await sleep(everyMs);
+ }
+}
+
+/// 一条事实在世界轴上是否覆盖 `at`。起点未知按**保守包含**处理,与服务端
+/// `edges_among` 的口径一致(0019 之前就是这么定的)。
+function coversWorldTime(fact, at) {
+ if (!at) return !fact.valid_to || new Date(fact.valid_to) > new Date();
+ const t = new Date(at);
+ if (fact.valid_from && new Date(fact.valid_from) > t) return false;
+ if (fact.valid_to && new Date(fact.valid_to) <= t) return false;
+ return true;
+}
+
+function valueOf(fact) {
+ const v = fact.object_value;
+ if (v === null || v === undefined) return null;
+ if (typeof v === "object") return v.value ?? v.summary ?? null;
+ return v;
+}
+
+async function main() {
+ const corpus = JSON.parse(fs.readFileSync(path.join(HERE, "temporal", "corpus.json"), "utf8"));
+ const sheet = JSON.parse(fs.readFileSync(path.join(HERE, "temporal", "questions.json"), "utf8"));
+
+ try {
+ await api("POST", "/api/v1/auth/register", {
+ email: EMAIL,
+ display_name: "bench",
+ password: PASSWORD,
+ });
+ } catch {
+ /* 已注册,走登录 */
+ }
+ await api("POST", "/api/v1/auth/login", { email: EMAIL, password: PASSWORD });
+ psql(`UPDATE users SET is_admin=TRUE WHERE email='${EMAIL}'`);
+ await api("POST", "/api/v1/auth/login", { email: EMAIL, password: PASSWORD });
+
+ let kb = args.kb;
+ let waveOne = args["wave-1"];
+
+ if (!kb) {
+ const ws = (await api("GET", "/api/v1/workspaces"))[0].id;
+ const stamp = new Date().toISOString().replace(/[:.]/g, "-");
+ // **每次一个新库**:带着上一轮结果的库量出来的数字不可比(run.mjs 的头
+ // 那段坑就是这么来的)
+ kb = (
+ await api("POST", `/api/v1/workspaces/${ws}/kbs`, {
+ name: `temporal ${label} ${stamp}`,
+ ontology_packs: [],
+ })
+ ).id;
+ await sleep(4000);
+
+ // **声明是被测配置的一部分。** 时态引擎的自动闭合只按本体走,而
+ // `functional` / `inverse_functional` 永不自动推断(bootstrap_ontology.rs 写了
+ // 理由:它驱动的是自动改写账本)。所以缺省先把语料声明的那几条建出来;
+ // `--no-declare` 量的是「本体自己长出来」的那种库,两组的差就是这条声明值多少
+ const declare = !("no-declare" in args);
+ if (declare) {
+ const classIds = new Map();
+ for (const c of corpus.classes || []) {
+ const made = await api("POST", `/api/v1/kbs/${kb}/ontology/entity-types`, {
+ key: c.key,
+ label: c.label,
+ description: c.description || "",
+ });
+ classIds.set(c.key, made.id);
+ }
+ for (const a of corpus.axioms || []) {
+ const body = {
+ key: a.key,
+ label: a.label,
+ kind: a.kind,
+ temporal: a.temporal || "state",
+ functional: !!a.functional,
+ inverse_functional: !!a.inverse_functional,
+ is_transitive: !!a.is_transitive,
+ description: a.description || "",
+ };
+ if (a.kind === "attribute") {
+ body.datatype = a.datatype;
+ body.unit = a.unit;
+ body.domains = [classIds.get(a.domain)].filter(Boolean);
+ }
+ await api("POST", `/api/v1/kbs/${kb}/ontology/relation-types`, body);
+ }
+ process.stderr.write(
+ ` 声明:${(corpus.classes || []).length} 个类、${(corpus.axioms || []).length} 条关系/属性\n`,
+ );
+ } else {
+ process.stderr.write(" 不声明:本体交给自动扩展长\n");
+ }
+
+ for (const wave of corpus.waves) {
+ process.stderr.write(`\n== ${wave.label} ==\n`);
+ for (const doc of wave.documents) {
+ await api("POST", `/api/v1/kbs/${kb}/ingest`, {
+ filename: doc.filename,
+ content: doc.text,
+ // 语料里写的是日期(读起来清楚),接口要的是时刻
+ doc_time: `${doc.doc_time}T00:00:00Z`,
+ });
+ }
+ const want = num(
+ `SELECT count(*) FROM documents WHERE kb_id='${kb}' AND deleted_at IS NULL`,
+ );
+ await until(async () => {
+ const done = num(
+ `SELECT count(*) FROM documents WHERE kb_id='${kb}' AND graph_status='done' AND deleted_at IS NULL`,
+ );
+ process.stderr.write(` 抽取 ${done}/${want} 篇\n`);
+ return done >= want ? true : done;
+ });
+ // 删除也是记录轴上的事件(#268):墓碑留着,这一波之前的时刻仍看得见它
+ for (const filename of wave.delete || []) {
+ const id = psql(
+ `SELECT id FROM documents WHERE kb_id='${kb}' AND filename='${filename}' LIMIT 1`,
+ );
+ if (id) await api("DELETE", `/api/v1/documents/${id}`);
+ }
+ if (wave.label === "wave-1") {
+ // **这一刻就是记录轴上的界**。等一拍再取,免得同一秒里第二波的写入
+ // 也落在这个时刻之内
+ await sleep(2000);
+ waveOne = new Date().toISOString();
+ process.stderr.write(` wave-1 时刻:${waveOne}\n`);
+ await sleep(2000);
+ }
+ }
+ }
+ if (!waveOne) throw new Error("复用库时要给 --wave-1 <ISO 时刻>");
+
+ // 实体按名字找一次,后面所有题目复用
+ const ids = new Map();
+ async function entityId(name) {
+ if (ids.has(name)) return ids.get(name);
+ const found = await api("GET", `/api/v1/kbs/${kb}/entities?q=${encodeURIComponent(name)}`);
+ const hit = (found.entities || []).find(
+ (e) => e.name.toLowerCase() === name.toLowerCase(),
+ );
+ ids.set(name, hit?.id ?? null);
+ return hit?.id ?? null;
+ }
+
+ const results = [];
+ for (const q of sheet.questions) {
+ const asOf = q.as_of === "wave-1" ? waveOne : q.as_of || null;
+ const subject = await entityId(q.subject);
+ if (!subject) {
+ // 抽取没抽出这个实体:不是时态答错,单独一档(与 run.mjs 的 absent 同理)
+ results.push({ ...q, outcome: "absent", saw: [] });
+ continue;
+ }
+ const qs = new URLSearchParams({ entity: subject, hops: "1" });
+ if (q.at) qs.set("at", q.at);
+ if (asOf) qs.set("as_of", asOf);
+ const graph = await api("GET", `/api/v1/kbs/${kb}/graph/neighborhood?${qs}`);
+ const names = new Set(
+ graph.edges
+ .flatMap((e) => [e.source, e.target])
+ .filter((id) => id !== subject)
+ .map((id) => graph.nodes.find((n) => n.id === id)?.name)
+ .filter(Boolean),
+ );
+
+ // 属性事实(薪资、职位)不是边,从实体面板取。**世界轴在这里由脚本过滤**:
+ // 那个接口今天只有 as_of,没有 at——量出来的缺口照实写在报告里
+ const detail = await api(
+ "GET",
+ `/api/v1/kbs/${kb}/entities/${subject}${asOf ? `?as_of=${encodeURIComponent(asOf)}` : ""}`,
+ );
+ const values = (detail.facts || [])
+ .filter((f) => coversWorldTime(f, q.at))
+ .map(valueOf)
+ .filter((v) => v !== null)
+ .map(String);
+
+ const saw = [...names, ...values];
+ const hasExpect = q.expect === null || saw.some((s) => s.includes(q.expect));
+ const hasWrong = (q.not || []).filter((bad) => saw.some((s) => s.includes(bad)));
+ results.push({
+ ...q,
+ as_of_resolved: asOf,
+ outcome: hasExpect && hasWrong.length === 0 ? "pass" : "fail",
+ wrong_seen: hasWrong,
+ saw,
+ });
+ }
+
+ // 对话那一路可选:它量的是**产品的答案**(模型带着工具自己去问),
+ // 上面那一路量的是账本本身。两者差在哪,正是 agent 那一层的水平
+ if (args.chat) {
+ for (const r of results) {
+ if (r.outcome === "absent") continue;
+ const reply = await askChat(kb, r.ask);
+ r.chat = {
+ outcome:
+ (r.expect === null || reply.includes(r.expect)) &&
+ !(r.not || []).some((bad) => reply.includes(bad))
+ ? "pass"
+ : "fail",
+ reply: reply.slice(0, 400),
+ };
+ }
+ }
+
+ const tally = (pick) => {
+ const rows = results.filter((r) => (pick ? r.axis === pick : true) && r.outcome !== "absent");
+ const pass = rows.filter((r) => r.outcome === "pass").length;
+ return { asked: rows.length, pass, rate: rows.length ? +(pass / rows.length).toFixed(3) : null };
+ };
+ const report = {
+ label,
+ declared: !("no-declare" in args),
+ kb,
+ wave_one: waveOne,
+ at: new Date().toISOString(),
+ absent: results.filter((r) => r.outcome === "absent").map((r) => r.subject),
+ ledger: { all: tally(null), world: tally("world"), record: tally("record") },
+ chat: args.chat
+ ? (() => {
+ const rows = results.filter((r) => r.chat);
+ const pass = rows.filter((r) => r.chat.outcome === "pass").length;
+ return { asked: rows.length, pass, rate: rows.length ? +(pass / rows.length).toFixed(3) : null };
+ })()
+ : null,
+ questions: results,
+ };
+ console.log(JSON.stringify(report, null, 2));
+ process.stderr.write(
+ `\n账本:${report.ledger.all.pass}/${report.ledger.all.asked}` +
+ `(世界轴 ${report.ledger.world.pass}/${report.ledger.world.asked},` +
+ `记录轴 ${report.ledger.record.pass}/${report.ledger.record.asked})` +
+ (report.chat ? ` 对话:${report.chat.pass}/${report.chat.asked}` : "") +
+ `\n`,
+ );
+}
+
+/// 一次对话问答。SSE 流里只取正文。
+async function askChat(kb, question) {
+ const res = await fetch(`${BASE}/api/v1/kbs/${kb}/chat`, {
+ method: "POST",
+ headers: { "content-type": "application/json", cookie },
+ body: JSON.stringify({ message: question }),
+ });
+ const text = await res.text();
+ let answer = "";
+ for (const line of text.split("\n")) {
+ if (!line.startsWith("data:")) continue;
+ const payload = line.slice(5).trim();
+ if (!payload || payload === "[DONE]") continue;
+ try {
+ const frame = JSON.parse(payload);
+ if (typeof frame === "string") answer += frame;
+ else if (frame.delta) answer += frame.delta;
+ } catch {
+ answer += payload;
+ }
+ }
+ return answer;
+}
+
+main().catch((e) => {
+ console.error(e);
+ process.exit(1);
+});
diff --git a/scripts/bench/temporal/corpus.json b/scripts/bench/temporal/corpus.json
new file mode 100644
index 00000000..589c8aff
--- /dev/null
+++ b/scripts/bench/temporal/corpus.json
@@ -0,0 +1,116 @@
+{
+ "name": "meridian",
+ "note": "虚构公司,刻意不用真实实体:模型读过的东西量出来的是记忆,不是这条流水线。句子写得钝——一句一个事实、日期写全、名字写全——因为这个台子要量的是时间那根轴,不是抽取的运气。",
+ "waves": [
+ {
+ "label": "wave-1",
+ "note": "第一波灌完之后记一个时刻。记录轴上的所有题目都以它为界:那时我们还不知道后来的交接与更正。",
+ "documents": [
+ {
+ "filename": "aurora-charter.md",
+ "doc_time": "2023-01-10",
+ "text": "# Project Aurora charter\n\nProject Aurora started on 2023-01-10 at Meridian Systems.\n\nZhang San leads Project Aurora from 2023-01-10.\n\nProject Aurora is part of the Platform Group.\n\nThe Platform Group is part of Meridian Systems.\n"
+ },
+ {
+ "filename": "lin-zhao-offer.md",
+ "doc_time": "2023-06-01",
+ "text": "# Offer record\n\nLin Zhao joined Meridian Systems on 2023-06-01.\n\nLin Zhao works for Meridian Systems from 2023-06-01.\n\nLin Zhao has the job title Staff Engineer from 2023-06-01.\n\nLin Zhao has a salary of 28000 CNY from 2023-06-01.\n"
+ },
+ {
+ "filename": "aurora-advisor-note.md",
+ "doc_time": "2023-03-02",
+ "text": "# Assistant note\n\nWang Wu advises Project Aurora from 2023-03-02.\n\nWang Wu works for Meridian Systems from 2020-09-01.\n"
+ }
+ ],
+ "delete": []
+ },
+ {
+ "label": "wave-2",
+ "note": "第二波在第一个时刻之后才灌:世界轴上它们讲的是 2024 年的事,记录轴上它们是「后来才知道的」。删除那一条制造撤回。",
+ "documents": [
+ {
+ "filename": "aurora-handover.md",
+ "doc_time": "2024-07-05",
+ "text": "# Handover memo\n\nLi Si leads Project Aurora from 2024-07-05.\n\nLi Si works for Meridian Systems from 2021-04-01.\n"
+ },
+ {
+ "filename": "lin-zhao-correction.md",
+ "doc_time": "2024-02-20",
+ "text": "# Payroll correction\n\nLin Zhao has a salary of 32000 CNY from 2024-02-20.\n\nThe raise was approved by the Platform Group.\n"
+ }
+ ],
+ "delete": [
+ "aurora-advisor-note.md"
+ ]
+ }
+ ],
+ "axioms_note": "时态引擎的自动闭合**只按本体的声明跑**:`functional` / `inverse_functional` 永不自动推断(理由见 bootstrap_ontology.rs——它驱动的是自动改写账本)。所以这两条声明是被测配置的一部分,跑分前先建出来;`--no-declare` 跑的是「本体自己长出来」的那种库,两组的差就是这条声明值多少。",
+ "classes": [
+ {
+ "key": "person",
+ "label": "Person",
+ "description": "A human being."
+ },
+ {
+ "key": "organization",
+ "label": "Organization",
+ "description": "A company or a group inside one."
+ },
+ {
+ "key": "project",
+ "label": "Project",
+ "description": "A named piece of work with people responsible for it."
+ }
+ ],
+ "axioms": [
+ {
+ "key": "leads",
+ "label": "leads",
+ "kind": "relation",
+ "temporal": "state",
+ "inverse_functional": true,
+ "description": "The person currently responsible for a project. One project has one lead at a time."
+ },
+ {
+ "key": "works_for",
+ "label": "works for",
+ "kind": "relation",
+ "temporal": "state",
+ "description": "The organization a person is employed by."
+ },
+ {
+ "key": "part_of",
+ "label": "part of",
+ "kind": "relation",
+ "temporal": "state",
+ "is_transitive": true,
+ "description": "A group or project inside a larger organization."
+ },
+ {
+ "key": "salary",
+ "label": "salary",
+ "kind": "attribute",
+ "datatype": "number",
+ "unit": "CNY",
+ "domain": "person",
+ "functional": true,
+ "description": "The person's monthly salary. One value holds at a time."
+ },
+ {
+ "key": "job_title",
+ "label": "job title",
+ "kind": "attribute",
+ "datatype": "text",
+ "domain": "person",
+ "functional": true,
+ "description": "The person's job title. One title holds at a time."
+ },
+ {
+ "key": "advises",
+ "label": "advises",
+ "kind": "relation",
+ "temporal": "state",
+ "description": "A person who advises a project without being responsible for it. Several advisers are possible."
+ }
+ ]
+}
diff --git a/scripts/bench/temporal/questions.json b/scripts/bench/temporal/questions.json
new file mode 100644
index 00000000..54f9b02f
--- /dev/null
+++ b/scripts/bench/temporal/questions.json
@@ -0,0 +1,281 @@
+{
+ "note": "每道题问的是**某一个时刻**的图。`axis` 说的是哪根轴在动:world = 世界轴(那时世界是什么样,参数 at),record = 记录轴(那时我们以为世界是什么样,参数 as_of)。`as_of: \"wave-1\"` 由跑分脚本换成第一波灌完那一刻的时间戳。\n\n判分不认谓词的名字:本体在不同库里长得不一样(works_for / employed_by),而这个台子量的是时间,不是词汇。判据是「这个实体在那一刻挂着的事实里,有没有出现 expect,有没有出现 not 里的任何一个」。",
+ "questions": [
+ {
+ "id": "aurora-lead-2023-02",
+ "axis": "world",
+ "at": "2023-02-01",
+ "subject": "Project Aurora",
+ "expect": "Zhang San",
+ "not": [
+ "Li Si"
+ ],
+ "ask": "Who led Project Aurora in February 2023?"
+ },
+ {
+ "id": "aurora-lead-2023-06",
+ "axis": "world",
+ "at": "2023-06-15",
+ "subject": "Project Aurora",
+ "expect": "Zhang San",
+ "not": [
+ "Li Si"
+ ],
+ "ask": "Who led Project Aurora in June 2023?"
+ },
+ {
+ "id": "aurora-lead-2024-06",
+ "axis": "world",
+ "at": "2024-06-30",
+ "subject": "Project Aurora",
+ "expect": "Zhang San",
+ "not": [
+ "Li Si"
+ ],
+ "ask": "Who led Project Aurora at the end of June 2024, days before the handover?"
+ },
+ {
+ "id": "aurora-lead-2024-08",
+ "axis": "world",
+ "at": "2024-08-01",
+ "subject": "Project Aurora",
+ "expect": "Li Si",
+ "not": [
+ "Zhang San"
+ ],
+ "ask": "Who led Project Aurora in August 2024?"
+ },
+ {
+ "id": "aurora-lead-2025-06",
+ "axis": "world",
+ "at": "2025-06-01",
+ "subject": "Project Aurora",
+ "expect": "Li Si",
+ "not": [
+ "Zhang San"
+ ],
+ "ask": "Who led Project Aurora in June 2025?"
+ },
+ {
+ "id": "aurora-lead-now",
+ "axis": "world",
+ "at": "2026-01-01",
+ "subject": "Project Aurora",
+ "expect": "Li Si",
+ "not": [
+ "Zhang San"
+ ],
+ "ask": "Who leads Project Aurora in January 2026?"
+ },
+ {
+ "id": "aurora-lead-record-wave1-at-2024-08",
+ "axis": "record",
+ "as_of": "wave-1",
+ "at": "2024-08-01",
+ "subject": "Project Aurora",
+ "expect": "Zhang San",
+ "not": [
+ "Li Si"
+ ],
+ "ask": "Before the handover memo arrived, who did we think would be leading Project Aurora in August 2024?",
+ "why": "两根轴的分界就在这一题:世界轴问的是 2024 年八月,而记录轴停在我们还没读到交接备忘的那一刻——那时的答案只能是张三。答成李四,就是拿今天的认知去答当时。"
+ },
+ {
+ "id": "aurora-lead-record-wave1-now",
+ "axis": "record",
+ "as_of": "wave-1",
+ "at": null,
+ "subject": "Project Aurora",
+ "expect": "Zhang San",
+ "not": [
+ "Li Si"
+ ],
+ "ask": "As of the first ingest, who was recorded as leading Project Aurora?"
+ },
+ {
+ "id": "aurora-lead-record-now",
+ "axis": "record",
+ "as_of": null,
+ "at": null,
+ "subject": "Project Aurora",
+ "expect": "Li Si",
+ "not": [],
+ "ask": "As we understand it today, who leads Project Aurora?"
+ },
+ {
+ "id": "aurora-parent",
+ "axis": "world",
+ "at": "2023-06-01",
+ "subject": "Project Aurora",
+ "expect": "Platform Group",
+ "not": [],
+ "ask": "Which group did Project Aurora belong to in June 2023?"
+ },
+ {
+ "id": "platform-parent",
+ "axis": "world",
+ "at": "2023-06-01",
+ "subject": "Platform Group",
+ "expect": "Meridian Systems",
+ "not": [],
+ "ask": "Which company was the Platform Group part of in June 2023?"
+ },
+ {
+ "id": "lin-zhao-employer-2023-07",
+ "axis": "world",
+ "at": "2023-07-01",
+ "subject": "Lin Zhao",
+ "expect": "Meridian Systems",
+ "not": [],
+ "ask": "Who did Lin Zhao work for in July 2023?"
+ },
+ {
+ "id": "lin-zhao-employer-2023-01",
+ "axis": "world",
+ "at": "2023-01-01",
+ "subject": "Lin Zhao",
+ "expect": null,
+ "not": [
+ "Meridian Systems"
+ ],
+ "ask": "Who did Lin Zhao work for in January 2023?",
+ "why": "入职是六月的事。一月的答案应当是空,而不是把开区间当成「一直如此」——`valid_from` 之前不成立,正是双时态存在的理由。"
+ },
+ {
+ "id": "lin-zhao-salary-2023-09",
+ "axis": "world",
+ "at": "2023-09-01",
+ "subject": "Lin Zhao",
+ "expect": "28000",
+ "not": [
+ "32000"
+ ],
+ "ask": "What was Lin Zhao's salary in September 2023?"
+ },
+ {
+ "id": "lin-zhao-salary-2024-06",
+ "axis": "world",
+ "at": "2024-06-01",
+ "subject": "Lin Zhao",
+ "expect": "32000",
+ "not": [
+ "28000"
+ ],
+ "ask": "What was Lin Zhao's salary in June 2024?"
+ },
+ {
+ "id": "lin-zhao-salary-now",
+ "axis": "world",
+ "at": "2026-01-01",
+ "subject": "Lin Zhao",
+ "expect": "32000",
+ "not": [
+ "28000"
+ ],
+ "ask": "What is Lin Zhao's salary in January 2026?"
+ },
+ {
+ "id": "lin-zhao-salary-record-wave1",
+ "axis": "record",
+ "as_of": "wave-1",
+ "at": null,
+ "subject": "Lin Zhao",
+ "expect": "28000",
+ "not": [
+ "32000"
+ ],
+ "ask": "Before the payroll correction arrived, what salary did we have on record for Lin Zhao?"
+ },
+ {
+ "id": "lin-zhao-salary-record-wave1-at-2024-06",
+ "axis": "record",
+ "as_of": "wave-1",
+ "at": "2024-06-01",
+ "subject": "Lin Zhao",
+ "expect": "28000",
+ "not": [
+ "32000"
+ ],
+ "ask": "Before the correction, what did we think Lin Zhao's June 2024 salary was?"
+ },
+ {
+ "id": "lin-zhao-title",
+ "axis": "world",
+ "at": "2023-09-01",
+ "subject": "Lin Zhao",
+ "expect": "Staff Engineer",
+ "not": [],
+ "ask": "What was Lin Zhao's job title in September 2023?"
+ },
+ {
+ "id": "adviser-record-wave1",
+ "axis": "record",
+ "as_of": "wave-1",
+ "at": "2023-06-01",
+ "subject": "Project Aurora",
+ "expect": "Wang Wu",
+ "not": [],
+ "ask": "As of the first ingest, was Wang Wu recorded as advising Project Aurora?",
+ "why": "那条未经证实的笔记当时确实在库里。记录轴回到那一刻就该看得见它——账本记的是「我们曾经这么以为」,抹掉它等于替过去修饰。"
+ },
+ {
+ "id": "adviser-now",
+ "axis": "record",
+ "as_of": null,
+ "at": "2023-06-01",
+ "subject": "Project Aurora",
+ "expect": null,
+ "not": [
+ "Wang Wu"
+ ],
+ "ask": "Do we still hold that Wang Wu advised Project Aurora in June 2023?",
+ "why": "那篇文档已经删了,这条事实的出处没了。世界轴仍问 2023 年六月,而今天的认知里它不该再成立。"
+ },
+ {
+ "id": "li-si-employer-2022",
+ "axis": "world",
+ "at": "2022-06-01",
+ "subject": "Li Si",
+ "expect": "Meridian Systems",
+ "not": [],
+ "ask": "Who did Li Si work for in June 2022?",
+ "why": "李四 2021 年就入职了,这件事却是 2024 年的交接备忘才说出口。世界轴问 2022 年答得出来,靠的是文档写明的起点,不是它到达的时间。"
+ },
+ {
+ "id": "li-si-employer-record-wave1",
+ "axis": "record",
+ "as_of": "wave-1",
+ "at": "2022-06-01",
+ "subject": "Li Si",
+ "expect": null,
+ "not": [
+ "Meridian Systems"
+ ],
+ "ask": "As of the first ingest, what did we know about where Li Si worked in 2022?",
+ "why": "上一题的反面:那时这个人还没进过我们的账本。世界轴上成立,记录轴上还不知道——两根轴各答各的。"
+ },
+ {
+ "id": "aurora-lead-before-start",
+ "axis": "world",
+ "at": "2022-12-01",
+ "subject": "Project Aurora",
+ "expect": null,
+ "not": [
+ "Zhang San",
+ "Li Si"
+ ],
+ "ask": "Who led Project Aurora in December 2022?",
+ "why": "项目 2023 年一月才开始。这一题防的是把开区间读成「自古如此」。"
+ },
+ {
+ "id": "aurora-lead-unfiltered",
+ "axis": "world",
+ "at": null,
+ "subject": "Project Aurora",
+ "expect": "Li Si",
+ "not": [],
+ "ask": "Who has led Project Aurora at any point?",
+ "why": "`at` 不给时服务端**不过滤世界轴**——问的是「我们持有的全部」,不是「现在」。这一题把那个口径钉住:两任都该在,而不是只剩现任。"
+ }
+ ]
+}