World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
A benchmark that asks the question this project is built around (#340)
WaylandYangd55a97e00be8
diff --git a/scripts/bench/README.md b/scripts/bench/README.mdindex aae03618..1325d3b1 100644--- a/scripts/bench/README.md+++ b/scripts/bench/README.md@@ -134,3 +134,86 @@ node scripts/bench/run.mjs --corpus pharma --ontology /tmp/schemaorg.ttl --label不是替代,是一条候选预扫的线:把分块检索出的几十个候选类交给它打一遍,再把(片段,类型)作为提示喂给抽取,有机会把抽取时的类型精度抬到消解后的水平、省掉一轮消解调用;它找到而我们没找到的,直接进 drop signal 做漏抽信号。这个实验还没做。++---++# 时态问答的测量台(#306)++`node scripts/bench/temporal.mjs --label rc5`++上面那个台子量的是**类型消解**。这一个量的是这个产品真正围绕着造的那件事:+**按某个日期回答问题,且在事实变过之后仍答对**。++在它之前,仓库能引用的数字只有 Ontology2SQL 在 BIRD Mini-Dev 上的成绩。别人家的+记忆基准(LongMemEval、DMR)测的是对话回忆,没有一个会问「2023 年三月这个项目是谁+在管」,而语料里那个答案还变过两次。在别人的基准上拿个中游名次,说明的东西还不如+把那个缺失的基准公布出来。++## 两根轴各问一遍++这是这份题目里别处没有的一维:++| 轴 | 参数 | 问的是 |+|---|---|---|+| world | `at` | 那时**世界**是什么样 |+| record | `as_of` | 那时**我们以为**世界是什么样(0019) |++同一个实体、同一个日期,两根轴的正确答案可以不同——`aurora-lead-record-wave1-at-2024-08`+就是这么一道:世界轴问 2024 年八月(那时李四已接手),记录轴停在我们还没读到交接+备忘的那一刻(那时只知道张三)。答成李四,就是拿今天的认知去答当时。++## 记录轴不是改时钟改出来的++语料分**两波**灌,第一波灌完记下一个时刻(`wave-1`),第二波才带来交接、薪资更正与+一次文档删除。所有 `as_of: "wave-1"` 的题目问的就是那一刻。这样量到的是产品真实+走过的路——一次真的摄入、一次真的抽取、一次真的删除——而不是 `UPDATE` 出来的场面。++## 判分++- 判据是「这个实体在那一刻挂着的东西里,有没有 `expect`,有没有 `not` 里的任何一个」,+ **不认谓词的名字**:本体在不同库里长得不一样(works_for / employed_by),而这个台子+ 量的是时间,不是词汇。+- `expect: null` 表示**那时不该有答案**(项目还没开始、人还没入职、事实已被撤回)。+ 把开区间读成「自古如此」是双时态最容易犯的错,所以专门留了三道。+- `absent`(抽取压根没抽出这个实体)单独一档,不算时态答错——与上面那个台子同理。+- `--chat` 再跑一遍**对话**:模型带着工具自己去问。账本那一路量的是账本,对话那一路+ 量的是 agent 那一层,两者的差就是工具用得好不好。++## 已知的口径缺口++属性事实(薪资、职位)不是边,得从实体面板取,而那个接口今天**只有 `as_of`,没有 `at`**+——世界轴那一半由脚本自己按 `valid_from` / `valid_to` 过滤。图谱两个接口两根轴都在+服务端。这个不对称照实记在这里,不藏进分数里。++## 第一轮的数字(2026-09-05,dev @ 1dd917b,DeepSeek-V3 + bge-m3)++| 配置 | 账本 | 世界轴 | 记录轴 |+|---|---|---|---|+| 声明了唯一性公理 | **25 / 25** | 17 / 17 | 8 / 8 |+| 本体自己长(`--no-declare`) | **15 / 25** | 8 / 17 | 7 / 8 |++两组的差**全部落在世界轴上**,而且原因只有一个:时态引擎的自动闭合按+`functional` / `inverse_functional` 跑,而这两位**永不自动推断**。没有声明的库里,+接任不会闭合前任——账本变成++```+Zhang San | leads | Project Aurora | from 2023-01-10 | (无终点) | live+Li Si | leads | Project Aurora | from 2024-07-05 | (无终点) | live+```++两个人同时在管一个项目。这时候问「2023 年六月谁在管」,答案里两个人都在,+而这正是这个产品最想答对的那类问题。薪资也一样:28000 与 32000 并排开着。++记录轴几乎不受影响(7/8 vs 8/8):它不依赖任何声明,只看 `recorded_at` /+`invalidated_at`——那两列是写入时就记下的。++**这个差值本身就是第一轮最值得报的结果**:产品的招牌能力在一个「什么都没声明、+让本体自己长」的库上是关掉的,而那恰恰是新用户开箱后的样子。++跑出来的另外两件事,都记在这里而不是藏进分数:++- **`at` 不给 = 不过滤世界轴**,不是「现在」。`as_of` 不给才是「现在」。两根轴的+ 缺省口径不对称,`aurora-lead-unfiltered` 这道题把它钉住了。+- 一条 `salary` 值被抽取丢掉一次(`extraction_drops` 里的 `malformed_item`):+ 声明了 `number` 之后,模型写成 "28000 CNY" 的那一次没能通过校验。宁缺勿脏是+ 写死的取舍,这里只记它发生过。diff --git a/scripts/bench/temporal.mjs b/scripts/bench/temporal.mjsnew file mode 100644index 00000000..8b63f5e9--- /dev/null+++ b/scripts/bench/temporal.mjs@@ -0,0 +1,348 @@+#!/usr/bin/env node+// 时态问答的测量台(#306)。+//+// 这个仓库唯一能引用的数字是 Ontology2SQL 在 BIRD Mini-Dev 上的成绩,而它围绕着+// 造的那件事——**按某个日期回答问题,且在事实变过之后仍答对**——没有任何东西在量。+// 别人家的记忆基准测的是对话回忆,没有一个会问「2023 年三月这个项目是谁在管」,+// 而语料里那个答案还变过两次。在别人的基准上拿个中游名次,说明的东西还不如+// 把那个缺失的基准公布出来。+//+// **两根轴各问一遍**,这是这份题目里别处没有的一维:+// world —— 那时世界是什么样(参数 at)+// record —— 那时我们以为世界是什么样(参数 as_of,0019)+//+// 记录轴不靠改时钟造出来:语料分两波灌,**第一波灌完记下一个时刻**,第二波带来+// 交接、更正与一次删除。`as_of: "wave-1"` 的题目问的就是那一刻——那时账本里还+// 没有后来的事。这样量到的是产品真实走过的路,不是 UPDATE 出来的场面。+//+// 用法:+// node scripts/bench/temporal.mjs --label rc5+// node scripts/bench/temporal.mjs --label rc5 --chat # 顺带量对话(要一个会话模型)+// node scripts/bench/temporal.mjs --kb <id> --only-score # 复用已灌好的库,只重打分+//+// 环境变量:BENCH_BASE / BENCH_EMAIL / BENCH_PASSWORD / BENCH_PSQL(同 run.mjs)。++import fs from "node:fs";+import path from "node:path";+import { execSync } from "node:child_process";+import { fileURLToPath } from "node:url";++const HERE = path.dirname(fileURLToPath(import.meta.url));+const BASE = process.env.BENCH_BASE || "http://localhost:18080";+const EMAIL = process.env.BENCH_EMAIL || "[email protected]";+const PASSWORD = process.env.BENCH_PASSWORD || "benchbench123";++const args = Object.fromEntries(+ process.argv.slice(2).map((a, i, all) => {+ const m = a.match(/^--([^=]+)(?:=(.*))?$/);+ return m ? [m[1], m[2] ?? (all[i + 1]?.startsWith("--") ? true : all[i + 1] ?? true)] : [a, true];+ }),+);+const label = args.label || "temporal";++let cookie = "";+async function api(method, url, body) {+ const res = await fetch(BASE + url, {+ method,+ headers: {+ ...(body ? { "content-type": "application/json" } : {}),+ ...(cookie ? { cookie } : {}),+ },+ body: body ? JSON.stringify(body) : undefined,+ });+ // getSetCookie(复数):多个 Set-Cookie 只取第一个会丢掉会话那条+ for (const c of res.headers.getSetCookie?.() ?? []) cookie = c.split(";")[0];+ const text = await res.text();+ if (!res.ok) throw new Error(`${method} ${url} → ${res.status} ${text.slice(0, 300)}`);+ return text ? JSON.parse(text) : null;+}++function psql(sql) {+ const cmd =+ process.env.BENCH_PSQL ||+ "docker exec -e PGPASSWORD=utopia landscapebi-db-1 psql -U utopia -d utopia -tAc";+ return execSync(`${cmd} "${sql.replace(/"/g, '\\"')}"`, { encoding: "utf8" }).trim();+}+const num = (sql) => Number(psql(sql) || 0);+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++/// 卡住才算超时,慢不算(与 run.mjs 同一条理由:抽取在服务端排队,驱动脚本+/// 死掉不影响它们,而按总时长封顶会把跑成功的组报成失败)。+async function until(fn, everyMs = 8000, stallMs = 8 * 60 * 1000) {+ let last = null;+ let deadline = Date.now() + stallMs;+ for (;;) {+ const r = await fn();+ if (r === true) return;+ if (r !== last) {+ last = r;+ deadline = Date.now() + stallMs;+ }+ if (Date.now() > deadline) throw new Error("等超时:八分钟没有任何进展");+ await sleep(everyMs);+ }+}++/// 一条事实在世界轴上是否覆盖 `at`。起点未知按**保守包含**处理,与服务端+/// `edges_among` 的口径一致(0019 之前就是这么定的)。+function coversWorldTime(fact, at) {+ if (!at) return !fact.valid_to || new Date(fact.valid_to) > new Date();+ const t = new Date(at);+ if (fact.valid_from && new Date(fact.valid_from) > t) return false;+ if (fact.valid_to && new Date(fact.valid_to) <= t) return false;+ return true;+}++function valueOf(fact) {+ const v = fact.object_value;+ if (v === null || v === undefined) return null;+ if (typeof v === "object") return v.value ?? v.summary ?? null;+ return v;+}++async function main() {+ const corpus = JSON.parse(fs.readFileSync(path.join(HERE, "temporal", "corpus.json"), "utf8"));+ const sheet = JSON.parse(fs.readFileSync(path.join(HERE, "temporal", "questions.json"), "utf8"));++ try {+ await api("POST", "/api/v1/auth/register", {+ email: EMAIL,+ display_name: "bench",+ password: PASSWORD,+ });+ } catch {+ /* 已注册,走登录 */+ }+ await api("POST", "/api/v1/auth/login", { email: EMAIL, password: PASSWORD });+ psql(`UPDATE users SET is_admin=TRUE WHERE email='${EMAIL}'`);+ await api("POST", "/api/v1/auth/login", { email: EMAIL, password: PASSWORD });++ let kb = args.kb;+ let waveOne = args["wave-1"];++ if (!kb) {+ const ws = (await api("GET", "/api/v1/workspaces"))[0].id;+ const stamp = new Date().toISOString().replace(/[:.]/g, "-");+ // **每次一个新库**:带着上一轮结果的库量出来的数字不可比(run.mjs 的头+ // 那段坑就是这么来的)+ kb = (+ await api("POST", `/api/v1/workspaces/${ws}/kbs`, {+ name: `temporal ${label} ${stamp}`,+ ontology_packs: [],+ })+ ).id;+ await sleep(4000);++ // **声明是被测配置的一部分。** 时态引擎的自动闭合只按本体走,而+ // `functional` / `inverse_functional` 永不自动推断(bootstrap_ontology.rs 写了+ // 理由:它驱动的是自动改写账本)。所以缺省先把语料声明的那几条建出来;+ // `--no-declare` 量的是「本体自己长出来」的那种库,两组的差就是这条声明值多少+ const declare = !("no-declare" in args);+ if (declare) {+ const classIds = new Map();+ for (const c of corpus.classes || []) {+ const made = await api("POST", `/api/v1/kbs/${kb}/ontology/entity-types`, {+ key: c.key,+ label: c.label,+ description: c.description || "",+ });+ classIds.set(c.key, made.id);+ }+ for (const a of corpus.axioms || []) {+ const body = {+ key: a.key,+ label: a.label,+ kind: a.kind,+ temporal: a.temporal || "state",+ functional: !!a.functional,+ inverse_functional: !!a.inverse_functional,+ is_transitive: !!a.is_transitive,+ description: a.description || "",+ };+ if (a.kind === "attribute") {+ body.datatype = a.datatype;+ body.unit = a.unit;+ body.domains = [classIds.get(a.domain)].filter(Boolean);+ }+ await api("POST", `/api/v1/kbs/${kb}/ontology/relation-types`, body);+ }+ process.stderr.write(+ ` 声明:${(corpus.classes || []).length} 个类、${(corpus.axioms || []).length} 条关系/属性\n`,+ );+ } else {+ process.stderr.write(" 不声明:本体交给自动扩展长\n");+ }++ for (const wave of corpus.waves) {+ process.stderr.write(`\n== ${wave.label} ==\n`);+ for (const doc of wave.documents) {+ await api("POST", `/api/v1/kbs/${kb}/ingest`, {+ filename: doc.filename,+ content: doc.text,+ // 语料里写的是日期(读起来清楚),接口要的是时刻+ doc_time: `${doc.doc_time}T00:00:00Z`,+ });+ }+ const want = num(+ `SELECT count(*) FROM documents WHERE kb_id='${kb}' AND deleted_at IS NULL`,+ );+ await until(async () => {+ const done = num(+ `SELECT count(*) FROM documents WHERE kb_id='${kb}' AND graph_status='done' AND deleted_at IS NULL`,+ );+ process.stderr.write(` 抽取 ${done}/${want} 篇\n`);+ return done >= want ? true : done;+ });+ // 删除也是记录轴上的事件(#268):墓碑留着,这一波之前的时刻仍看得见它+ for (const filename of wave.delete || []) {+ const id = psql(+ `SELECT id FROM documents WHERE kb_id='${kb}' AND filename='${filename}' LIMIT 1`,+ );+ if (id) await api("DELETE", `/api/v1/documents/${id}`);+ }+ if (wave.label === "wave-1") {+ // **这一刻就是记录轴上的界**。等一拍再取,免得同一秒里第二波的写入+ // 也落在这个时刻之内+ await sleep(2000);+ waveOne = new Date().toISOString();+ process.stderr.write(` wave-1 时刻:${waveOne}\n`);+ await sleep(2000);+ }+ }+ }+ if (!waveOne) throw new Error("复用库时要给 --wave-1 <ISO 时刻>");++ // 实体按名字找一次,后面所有题目复用+ const ids = new Map();+ async function entityId(name) {+ if (ids.has(name)) return ids.get(name);+ const found = await api("GET", `/api/v1/kbs/${kb}/entities?q=${encodeURIComponent(name)}`);+ const hit = (found.entities || []).find(+ (e) => e.name.toLowerCase() === name.toLowerCase(),+ );+ ids.set(name, hit?.id ?? null);+ return hit?.id ?? null;+ }++ const results = [];+ for (const q of sheet.questions) {+ const asOf = q.as_of === "wave-1" ? waveOne : q.as_of || null;+ const subject = await entityId(q.subject);+ if (!subject) {+ // 抽取没抽出这个实体:不是时态答错,单独一档(与 run.mjs 的 absent 同理)+ results.push({ ...q, outcome: "absent", saw: [] });+ continue;+ }+ const qs = new URLSearchParams({ entity: subject, hops: "1" });+ if (q.at) qs.set("at", q.at);+ if (asOf) qs.set("as_of", asOf);+ const graph = await api("GET", `/api/v1/kbs/${kb}/graph/neighborhood?${qs}`);+ const names = new Set(+ graph.edges+ .flatMap((e) => [e.source, e.target])+ .filter((id) => id !== subject)+ .map((id) => graph.nodes.find((n) => n.id === id)?.name)+ .filter(Boolean),+ );++ // 属性事实(薪资、职位)不是边,从实体面板取。**世界轴在这里由脚本过滤**:+ // 那个接口今天只有 as_of,没有 at——量出来的缺口照实写在报告里+ const detail = await api(+ "GET",+ `/api/v1/kbs/${kb}/entities/${subject}${asOf ? `?as_of=${encodeURIComponent(asOf)}` : ""}`,+ );+ const values = (detail.facts || [])+ .filter((f) => coversWorldTime(f, q.at))+ .map(valueOf)+ .filter((v) => v !== null)+ .map(String);++ const saw = [...names, ...values];+ const hasExpect = q.expect === null || saw.some((s) => s.includes(q.expect));+ const hasWrong = (q.not || []).filter((bad) => saw.some((s) => s.includes(bad)));+ results.push({+ ...q,+ as_of_resolved: asOf,+ outcome: hasExpect && hasWrong.length === 0 ? "pass" : "fail",+ wrong_seen: hasWrong,+ saw,+ });+ }++ // 对话那一路可选:它量的是**产品的答案**(模型带着工具自己去问),+ // 上面那一路量的是账本本身。两者差在哪,正是 agent 那一层的水平+ if (args.chat) {+ for (const r of results) {+ if (r.outcome === "absent") continue;+ const reply = await askChat(kb, r.ask);+ r.chat = {+ outcome:+ (r.expect === null || reply.includes(r.expect)) &&+ !(r.not || []).some((bad) => reply.includes(bad))+ ? "pass"+ : "fail",+ reply: reply.slice(0, 400),+ };+ }+ }++ const tally = (pick) => {+ const rows = results.filter((r) => (pick ? r.axis === pick : true) && r.outcome !== "absent");+ const pass = rows.filter((r) => r.outcome === "pass").length;+ return { asked: rows.length, pass, rate: rows.length ? +(pass / rows.length).toFixed(3) : null };+ };+ const report = {+ label,+ declared: !("no-declare" in args),+ kb,+ wave_one: waveOne,+ at: new Date().toISOString(),+ absent: results.filter((r) => r.outcome === "absent").map((r) => r.subject),+ ledger: { all: tally(null), world: tally("world"), record: tally("record") },+ chat: args.chat+ ? (() => {+ const rows = results.filter((r) => r.chat);+ const pass = rows.filter((r) => r.chat.outcome === "pass").length;+ return { asked: rows.length, pass, rate: rows.length ? +(pass / rows.length).toFixed(3) : null };+ })()+ : null,+ questions: results,+ };+ console.log(JSON.stringify(report, null, 2));+ process.stderr.write(+ `\n账本:${report.ledger.all.pass}/${report.ledger.all.asked}` ++ `(世界轴 ${report.ledger.world.pass}/${report.ledger.world.asked},` ++ `记录轴 ${report.ledger.record.pass}/${report.ledger.record.asked})` ++ (report.chat ? ` 对话:${report.chat.pass}/${report.chat.asked}` : "") ++ `\n`,+ );+}++/// 一次对话问答。SSE 流里只取正文。+async function askChat(kb, question) {+ const res = await fetch(`${BASE}/api/v1/kbs/${kb}/chat`, {+ method: "POST",+ headers: { "content-type": "application/json", cookie },+ body: JSON.stringify({ message: question }),+ });+ const text = await res.text();+ let answer = "";+ for (const line of text.split("\n")) {+ if (!line.startsWith("data:")) continue;+ const payload = line.slice(5).trim();+ if (!payload || payload === "[DONE]") continue;+ try {+ const frame = JSON.parse(payload);+ if (typeof frame === "string") answer += frame;+ else if (frame.delta) answer += frame.delta;+ } catch {+ answer += payload;+ }+ }+ return answer;+}++main().catch((e) => {+ console.error(e);+ process.exit(1);+});diff --git a/scripts/bench/temporal/corpus.json b/scripts/bench/temporal/corpus.jsonnew file mode 100644index 00000000..589c8aff--- /dev/null+++ b/scripts/bench/temporal/corpus.json@@ -0,0 +1,116 @@+{+ "name": "meridian",+ "note": "虚构公司,刻意不用真实实体:模型读过的东西量出来的是记忆,不是这条流水线。句子写得钝——一句一个事实、日期写全、名字写全——因为这个台子要量的是时间那根轴,不是抽取的运气。",+ "waves": [+ {+ "label": "wave-1",+ "note": "第一波灌完之后记一个时刻。记录轴上的所有题目都以它为界:那时我们还不知道后来的交接与更正。",+ "documents": [+ {+ "filename": "aurora-charter.md",+ "doc_time": "2023-01-10",+ "text": "# Project Aurora charter\n\nProject Aurora started on 2023-01-10 at Meridian Systems.\n\nZhang San leads Project Aurora from 2023-01-10.\n\nProject Aurora is part of the Platform Group.\n\nThe Platform Group is part of Meridian Systems.\n"+ },+ {+ "filename": "lin-zhao-offer.md",+ "doc_time": "2023-06-01",+ "text": "# Offer record\n\nLin Zhao joined Meridian Systems on 2023-06-01.\n\nLin Zhao works for Meridian Systems from 2023-06-01.\n\nLin Zhao has the job title Staff Engineer from 2023-06-01.\n\nLin Zhao has a salary of 28000 CNY from 2023-06-01.\n"+ },+ {+ "filename": "aurora-advisor-note.md",+ "doc_time": "2023-03-02",+ "text": "# Assistant note\n\nWang Wu advises Project Aurora from 2023-03-02.\n\nWang Wu works for Meridian Systems from 2020-09-01.\n"+ }+ ],+ "delete": []+ },+ {+ "label": "wave-2",+ "note": "第二波在第一个时刻之后才灌:世界轴上它们讲的是 2024 年的事,记录轴上它们是「后来才知道的」。删除那一条制造撤回。",+ "documents": [+ {+ "filename": "aurora-handover.md",+ "doc_time": "2024-07-05",+ "text": "# Handover memo\n\nLi Si leads Project Aurora from 2024-07-05.\n\nLi Si works for Meridian Systems from 2021-04-01.\n"+ },+ {+ "filename": "lin-zhao-correction.md",+ "doc_time": "2024-02-20",+ "text": "# Payroll correction\n\nLin Zhao has a salary of 32000 CNY from 2024-02-20.\n\nThe raise was approved by the Platform Group.\n"+ }+ ],+ "delete": [+ "aurora-advisor-note.md"+ ]+ }+ ],+ "axioms_note": "时态引擎的自动闭合**只按本体的声明跑**:`functional` / `inverse_functional` 永不自动推断(理由见 bootstrap_ontology.rs——它驱动的是自动改写账本)。所以这两条声明是被测配置的一部分,跑分前先建出来;`--no-declare` 跑的是「本体自己长出来」的那种库,两组的差就是这条声明值多少。",+ "classes": [+ {+ "key": "person",+ "label": "Person",+ "description": "A human being."+ },+ {+ "key": "organization",+ "label": "Organization",+ "description": "A company or a group inside one."+ },+ {+ "key": "project",+ "label": "Project",+ "description": "A named piece of work with people responsible for it."+ }+ ],+ "axioms": [+ {+ "key": "leads",+ "label": "leads",+ "kind": "relation",+ "temporal": "state",+ "inverse_functional": true,+ "description": "The person currently responsible for a project. One project has one lead at a time."+ },+ {+ "key": "works_for",+ "label": "works for",+ "kind": "relation",+ "temporal": "state",+ "description": "The organization a person is employed by."+ },+ {+ "key": "part_of",+ "label": "part of",+ "kind": "relation",+ "temporal": "state",+ "is_transitive": true,+ "description": "A group or project inside a larger organization."+ },+ {+ "key": "salary",+ "label": "salary",+ "kind": "attribute",+ "datatype": "number",+ "unit": "CNY",+ "domain": "person",+ "functional": true,+ "description": "The person's monthly salary. One value holds at a time."+ },+ {+ "key": "job_title",+ "label": "job title",+ "kind": "attribute",+ "datatype": "text",+ "domain": "person",+ "functional": true,+ "description": "The person's job title. One title holds at a time."+ },+ {+ "key": "advises",+ "label": "advises",+ "kind": "relation",+ "temporal": "state",+ "description": "A person who advises a project without being responsible for it. Several advisers are possible."+ }+ ]+}diff --git a/scripts/bench/temporal/questions.json b/scripts/bench/temporal/questions.jsonnew file mode 100644index 00000000..54f9b02f--- /dev/null+++ b/scripts/bench/temporal/questions.json@@ -0,0 +1,281 @@+{+ "note": "每道题问的是**某一个时刻**的图。`axis` 说的是哪根轴在动:world = 世界轴(那时世界是什么样,参数 at),record = 记录轴(那时我们以为世界是什么样,参数 as_of)。`as_of: \"wave-1\"` 由跑分脚本换成第一波灌完那一刻的时间戳。\n\n判分不认谓词的名字:本体在不同库里长得不一样(works_for / employed_by),而这个台子量的是时间,不是词汇。判据是「这个实体在那一刻挂着的事实里,有没有出现 expect,有没有出现 not 里的任何一个」。",+ "questions": [+ {+ "id": "aurora-lead-2023-02",+ "axis": "world",+ "at": "2023-02-01",+ "subject": "Project Aurora",+ "expect": "Zhang San",+ "not": [+ "Li Si"+ ],+ "ask": "Who led Project Aurora in February 2023?"+ },+ {+ "id": "aurora-lead-2023-06",+ "axis": "world",+ "at": "2023-06-15",+ "subject": "Project Aurora",+ "expect": "Zhang San",+ "not": [+ "Li Si"+ ],+ "ask": "Who led Project Aurora in June 2023?"+ },+ {+ "id": "aurora-lead-2024-06",+ "axis": "world",+ "at": "2024-06-30",+ "subject": "Project Aurora",+ "expect": "Zhang San",+ "not": [+ "Li Si"+ ],+ "ask": "Who led Project Aurora at the end of June 2024, days before the handover?"+ },+ {+ "id": "aurora-lead-2024-08",+ "axis": "world",+ "at": "2024-08-01",+ "subject": "Project Aurora",+ "expect": "Li Si",+ "not": [+ "Zhang San"+ ],+ "ask": "Who led Project Aurora in August 2024?"+ },+ {+ "id": "aurora-lead-2025-06",+ "axis": "world",+ "at": "2025-06-01",+ "subject": "Project Aurora",+ "expect": "Li Si",+ "not": [+ "Zhang San"+ ],+ "ask": "Who led Project Aurora in June 2025?"+ },+ {+ "id": "aurora-lead-now",+ "axis": "world",+ "at": "2026-01-01",+ "subject": "Project Aurora",+ "expect": "Li Si",+ "not": [+ "Zhang San"+ ],+ "ask": "Who leads Project Aurora in January 2026?"+ },+ {+ "id": "aurora-lead-record-wave1-at-2024-08",+ "axis": "record",+ "as_of": "wave-1",+ "at": "2024-08-01",+ "subject": "Project Aurora",+ "expect": "Zhang San",+ "not": [+ "Li Si"+ ],+ "ask": "Before the handover memo arrived, who did we think would be leading Project Aurora in August 2024?",+ "why": "两根轴的分界就在这一题:世界轴问的是 2024 年八月,而记录轴停在我们还没读到交接备忘的那一刻——那时的答案只能是张三。答成李四,就是拿今天的认知去答当时。"+ },+ {+ "id": "aurora-lead-record-wave1-now",+ "axis": "record",+ "as_of": "wave-1",+ "at": null,+ "subject": "Project Aurora",+ "expect": "Zhang San",+ "not": [+ "Li Si"+ ],+ "ask": "As of the first ingest, who was recorded as leading Project Aurora?"+ },+ {+ "id": "aurora-lead-record-now",+ "axis": "record",+ "as_of": null,+ "at": null,+ "subject": "Project Aurora",+ "expect": "Li Si",+ "not": [],+ "ask": "As we understand it today, who leads Project Aurora?"+ },+ {+ "id": "aurora-parent",+ "axis": "world",+ "at": "2023-06-01",+ "subject": "Project Aurora",+ "expect": "Platform Group",+ "not": [],+ "ask": "Which group did Project Aurora belong to in June 2023?"+ },+ {+ "id": "platform-parent",+ "axis": "world",+ "at": "2023-06-01",+ "subject": "Platform Group",+ "expect": "Meridian Systems",+ "not": [],+ "ask": "Which company was the Platform Group part of in June 2023?"+ },+ {+ "id": "lin-zhao-employer-2023-07",+ "axis": "world",+ "at": "2023-07-01",+ "subject": "Lin Zhao",+ "expect": "Meridian Systems",+ "not": [],+ "ask": "Who did Lin Zhao work for in July 2023?"+ },+ {+ "id": "lin-zhao-employer-2023-01",+ "axis": "world",+ "at": "2023-01-01",+ "subject": "Lin Zhao",+ "expect": null,+ "not": [+ "Meridian Systems"+ ],+ "ask": "Who did Lin Zhao work for in January 2023?",+ "why": "入职是六月的事。一月的答案应当是空,而不是把开区间当成「一直如此」——`valid_from` 之前不成立,正是双时态存在的理由。"+ },+ {+ "id": "lin-zhao-salary-2023-09",+ "axis": "world",+ "at": "2023-09-01",+ "subject": "Lin Zhao",+ "expect": "28000",+ "not": [+ "32000"+ ],+ "ask": "What was Lin Zhao's salary in September 2023?"+ },+ {+ "id": "lin-zhao-salary-2024-06",+ "axis": "world",+ "at": "2024-06-01",+ "subject": "Lin Zhao",+ "expect": "32000",+ "not": [+ "28000"+ ],+ "ask": "What was Lin Zhao's salary in June 2024?"+ },+ {+ "id": "lin-zhao-salary-now",+ "axis": "world",+ "at": "2026-01-01",+ "subject": "Lin Zhao",+ "expect": "32000",+ "not": [+ "28000"+ ],+ "ask": "What is Lin Zhao's salary in January 2026?"+ },+ {+ "id": "lin-zhao-salary-record-wave1",+ "axis": "record",+ "as_of": "wave-1",+ "at": null,+ "subject": "Lin Zhao",+ "expect": "28000",+ "not": [+ "32000"+ ],+ "ask": "Before the payroll correction arrived, what salary did we have on record for Lin Zhao?"+ },+ {+ "id": "lin-zhao-salary-record-wave1-at-2024-06",+ "axis": "record",+ "as_of": "wave-1",+ "at": "2024-06-01",+ "subject": "Lin Zhao",+ "expect": "28000",+ "not": [+ "32000"+ ],+ "ask": "Before the correction, what did we think Lin Zhao's June 2024 salary was?"+ },+ {+ "id": "lin-zhao-title",+ "axis": "world",+ "at": "2023-09-01",+ "subject": "Lin Zhao",+ "expect": "Staff Engineer",+ "not": [],+ "ask": "What was Lin Zhao's job title in September 2023?"+ },+ {+ "id": "adviser-record-wave1",+ "axis": "record",+ "as_of": "wave-1",+ "at": "2023-06-01",+ "subject": "Project Aurora",+ "expect": "Wang Wu",+ "not": [],+ "ask": "As of the first ingest, was Wang Wu recorded as advising Project Aurora?",+ "why": "那条未经证实的笔记当时确实在库里。记录轴回到那一刻就该看得见它——账本记的是「我们曾经这么以为」,抹掉它等于替过去修饰。"+ },+ {+ "id": "adviser-now",+ "axis": "record",+ "as_of": null,+ "at": "2023-06-01",+ "subject": "Project Aurora",+ "expect": null,+ "not": [+ "Wang Wu"+ ],+ "ask": "Do we still hold that Wang Wu advised Project Aurora in June 2023?",+ "why": "那篇文档已经删了,这条事实的出处没了。世界轴仍问 2023 年六月,而今天的认知里它不该再成立。"+ },+ {+ "id": "li-si-employer-2022",+ "axis": "world",+ "at": "2022-06-01",+ "subject": "Li Si",+ "expect": "Meridian Systems",+ "not": [],+ "ask": "Who did Li Si work for in June 2022?",+ "why": "李四 2021 年就入职了,这件事却是 2024 年的交接备忘才说出口。世界轴问 2022 年答得出来,靠的是文档写明的起点,不是它到达的时间。"+ },+ {+ "id": "li-si-employer-record-wave1",+ "axis": "record",+ "as_of": "wave-1",+ "at": "2022-06-01",+ "subject": "Li Si",+ "expect": null,+ "not": [+ "Meridian Systems"+ ],+ "ask": "As of the first ingest, what did we know about where Li Si worked in 2022?",+ "why": "上一题的反面:那时这个人还没进过我们的账本。世界轴上成立,记录轴上还不知道——两根轴各答各的。"+ },+ {+ "id": "aurora-lead-before-start",+ "axis": "world",+ "at": "2022-12-01",+ "subject": "Project Aurora",+ "expect": null,+ "not": [+ "Zhang San",+ "Li Si"+ ],+ "ask": "Who led Project Aurora in December 2022?",+ "why": "项目 2023 年一月才开始。这一题防的是把开区间读成「自古如此」。"+ },+ {+ "id": "aurora-lead-unfiltered",+ "axis": "world",+ "at": null,+ "subject": "Project Aurora",+ "expect": "Li Si",+ "not": [],+ "ask": "Who has led Project Aurora at any point?",+ "why": "`at` 不给时服务端**不过滤世界轴**——问的是「我们持有的全部」,不是「现在」。这一题把那个口径钉住:两任都该在,而不是只剩现任。"+ }+ ]+}