World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
0041 records that a name is a claim about an entity (#663)
WaylandYang4363ac339a70
diff --git a/docs/decisions/0041-a-name-is-a-claim-about-an-entity.md b/docs/decisions/0041-a-name-is-a-claim-about-an-entity.mdnew file mode 100644index 00000000..23b7ee6e--- /dev/null+++ b/docs/decisions/0041-a-name-is-a-claim-about-an-entity.md@@ -0,0 +1,110 @@+# 0041 · A name is a claim about an entity++- **Status**: decision 1 settled 2026-09-13 (names are facts) · cut 0 built: `scripts/bench/identity.mjs`, baseline on `dev` forward F1 0.43, reverse 0.54 · cut 1 implemented (#670): migration 0055 and `names` in the store, the extractor's `names`, shared-name pairs to the adjudicator; forward 0.68, reverse 0.68, the two orders agree on all 210 pairs (one run each; two cut-1 runs differed by 0.07 forward) · decisions 2 and 5 revised by cut 1 · cuts 2–4 not started+- **Written**: 2026-09-13 (conventions in the [README](README.md))+- **Related**: [0009](0009-no-type-is-a-type.md) made an undecided type an honest state, and [0016](0016-close-the-open-seams-before-cutting-new-ones.md) B3 let a declared `disjointWith` keep names apart; #270 stopped a namesake tie from being settled by candidate order and #331 let facts break it; #428 and [0025](0025-governance-reads-the-ledger-before-it-decides.md) moved duplicates through a queue an agent works; #582 and #583 made the extractor copy the words that name each side of a fact; [0037](0037-a-relation-carries-its-own-attributes.md) put attributes on edges.++> A probe is written 海洋探测器1号 in its launch notice and 海探1 in every report after it. Two engineers at one company are both 张伟. Today the probe becomes two entities with nothing in Review. The engineers become one entity, or a tie that waits for a person forever, depending on which document arrived first.++## What resolution does today++`resolve_mention` in the store resolves one mention at a time, while the extraction job writes facts:++1. **Recall is an exact string match.** Candidates are entities of the mention's type whose `canonical_name` or `aliases` equal the mention's name or one of its `recall_keys` (the name with a generic suffix stripped, from `GENERIC_SUFFIXES_CJK` and `GENERIC_WORDS_EN`). No other entity is ever looked at.+2. **A context vector decides.** `profile_embedding` is the running mean of the vectors of the chunks an entity was mentioned in. A candidate at cosine ≥ `SIM_ATTACH` (0.55) is attached; below `SIM_NEW` (0.35) a new entity is made; in between, a new entity and a pair for the adjudicator. Two same-name candidates within `SIM_TIE_MARGIN` become a pair for a person (#270), unless `corroborating_candidate` finds the name of one candidate's object as a substring of the chunk (#331).+3. **Names other than the canonical one are learned only by merging.** `merge_entities` appends the source's names to the target's `aliases`, and `revert_merge` takes them out. Nothing the extractor reads becomes a name, including a sentence that states 简称海探1.+4. Around that: `containment_reviews` queues pairs whose names contain one another, for names of four characters and more; `resolve_type_drift` handles the same name under another type; `doc_cache` in the extraction job sends every later mention without a response handle of the same type and lowercased name in a document to the first result; with no embedding model a mention attaches to the candidate with the most facts.+5. In governance, `name_shape` classifies a pair from English affixes and prepositions (`CORPORATE_SUFFIX`, " of ", "'s"), and the gate will not merge a `Version` or `Phrase` shape.++## What goes wrong, and why it is one problem++- **An alias with no bridge is a silent second entity.** 海探1 in a document that never writes the full name matches no string, is under four characters for `containment_reviews`, and leaves no review row. The comment on `containment_reviews` names this hole and leaves it open.+- **A context vector records what a passage is about. It does not record who is in it.** Two namesakes described in the same kind of document have almost the same profile and tie for good. One person mentioned in a board memo and in a technical specification drifts until a later mention falls under `SIM_ATTACH`, and splits.+- **Evidence the extractor has already produced goes unused.** The response that names 张伟 also says `张伟 worksFor 财务部`. Resolution runs before that fact is written, and facts take part only through a substring search for object names in the chunk.+- **The outcome depends on arrival order.** Resolution is greedy and never looks back: the first document seeds the profile, later ones attach to it or split from it.+- **The word lists do not travel.** `GENERIC_SUFFIXES_CJK`, `GENERIC_WORDS_EN`, `CORPORATE_SUFFIX` and the preposition list in `name_shape` cover the phrasings someone thought of, mostly English. `name_shape` cannot see the shape of a Chinese name at all, so the gate's refusal to merge a version or a phrase does not exist for Chinese names.++All of it comes from one choice: **identity is keyed on the surface string, and a topic vector chooses among the strings that match.** A namesake is one string for several things and an alias is several strings for one thing; they are the two ways that key fails.++## Decisions++### 1. A name is a fact about an entity++Every name a text uses for an entity is recorded as a fact on a system attribute `name`, with the quote it came from, the document's times, and a world validity when the text gives one ("renamed in 2021" closes the old name). `entities.canonical_name` stays as the label the interface shows, and it is also one of the entity's name facts. `entities.aliases` is retired: the migration turns each existing alias into a name fact attributed to the merge that put it there.++A fact, because a name can be wrong, corrected, superseded and taken back, which is what the ledger already does for everything else it holds. As facts, names get both clocks ("what was it called in 2019", "when did we learn that 海探1 is 海洋探测器1号"), provenance, revert, and a place in the export as `skos:altLabel`.++**A name is a value, never a node.** A name fact keeps the string in `facts.object_value` and leaves `object_id` empty, the same channel a quantity uses since #586 and #587. The canvas draws only facts with an `object_id`, and the graph's fact count counts only those, so names add no node, no edge and no count there.++**The temporal engine leaves it alone by construction.** `temporal` closes an old value and records a conflict only for state relations declared functional or inverse-functional. `name` is declared neither, because namesakes exist, so a second name never closes the first or opens a conflict.++**The price** is every reader that lists an entity's facts: the entity panel shows names in their own list, beside the facts; `entity_fact_lines` on a Review card and the `degree` used in resolution leave names out, so a name does not pass for evidence or pad a count; `entity_facts` for chat and MCP returns names marked as names; the RDF export writes them as `skos:altLabel`.++Considered: an `entity_names` table with the same columns. It keeps conflicts and reasoning untouched, and rebuilds validity, provenance, invalidation and revert beside the ledger that already has them.++### 2. The extractor reports names, and the server checks only that they are in the text++Two sources, one contract:++- **Spans.** Every fact already carries `subject_span` and `object_span`, bound to a handle whose `name` is the full name. A span that differs from that name, occurs in the quote, and is marked by the model as a name, as opposed to a pronoun or a description, becomes a name fact with the fact's quote.+- **Stated names.** Each entity in the contract gets `names`: the other names the text gives it. The server keeps a name only when it occurs verbatim in the chunk. It does not recognise 简称, 又名, aka or formerly, and does not need to.++Whether a span names the entity or refers to it is the model's call, written into the contract. The server's check is whether the string is in the text. There is no word list.++> **Revised 2026-09-13 (cut 1, #670).** One channel, not two: `names: [{ref, name, quote}]`. Spans stay a verification signal only; a model that follows rule 1b already lists the shortened form it used, and a second source would repeat the same check. The server keeps a name when the name is in its quote and the quote is in the chunk, and drops it when the same response or an earlier chunk of the document declares that string for a different entity (`name_claimed_by_another`) — one string cannot name two things, and that needs no vocabulary. The name the model writes for an entity gets the chunk as evidence when the chunk contains it. **Known gap:** a description the model reports as a name without declaring it as an entity still passes ("海探1项目" became a name of 海洋探测器1号 in every cut-1 run).++### 3. Recall has three channels, and none of them decides++Candidates for a mention come from:++1. **Names.** Current and former name facts, equal after `normalize_name`. A former name stays recallable, because older documents still use it.+2. **Name vectors.** The nearest name facts by the embedding of the name string, within the same type family, top k. This is the channel that finds 海探1 for 海洋探测器1号, and a name written in two scripts.+3. **Neighbours.** Entities that share a distinctive neighbour with the mention's facts in the same response: the same predicate, the same resolved object, and an object the other candidates do not also point at.++Recall only proposes. `recall_keys`, `GENERIC_SUFFIXES_CJK` and `GENERIC_WORDS_EN` leave once the bench shows channel 2 finds what they found.++### 4. Evidence decides, read from the ontology and both clocks++For each candidate, evidence is read from the chunk's resolved facts against the candidate's current facts:++- **For the same entity**: a shared distinctive edge (as defined in decision 3) with overlapping validity; the same value on a predicate the ontology declares inverse-functional, with overlapping validity; a name the text states for this entity (decision 2).+- **For a different entity**: different values on a predicate the ontology declares functional, both held over overlapping validity; types declared disjoint, as today. An event relation contributes no conflict, since two dated events are not two values of one slot.+- **Neither**: disagreement on a predicate that may hold several values, since a person can work at two places; and a shared name, which says nothing in either direction.++The rule: evidence against a candidate rules it out. One candidate left with evidence for it is attached. Several left with evidence gives a new entity and a pair for a person, as #270 does today. No evidence at all takes today's path: a new entity, and a pair for the adjudicator when channel 1 or 2 found the candidate. The context vector breaks ties between candidates whose evidence is equal, and does nothing else. **When unsure, split and ask**, as before.++To have edges to compare, a chunk's mentions resolve after the response is parsed: sides with one candidate and no evidence against it first, then the ambiguous sides, using their edges to the sides already resolved. `doc_cache` keyed by name is removed. Handles unify mentions within a response, and evidence unifies them across chunks.++### 5. Arrival order stops mattering++> **Revised 2026-09-13 (cut 1, #670).** The first step landed with the names themselves, because without it cut 1 made reverse order worse: the abbreviation-only documents built 海探1 first, the document stating 简称海探1 arrived last and put the name on a new full-name entity, and nothing paired the two. Now a name that another entity of a compatible type already has queues the pair for the adjudicator (`shared_name|…`), never a merge, and the adjudicator and the Review card see an entity's non-canonical names as an `also known as:` line. Without that line the adjudicator kept the pair apart at 0.90–0.95. The canonical name is not listed, so two namesakes do not appear to share evidence. Re-evaluation on a new distinctive edge, and the periodic job, remain cut 4.++When an entity gains a name fact or a distinctive edge, the pairs it now recalls are evaluated again and queued, through Review and the governance gate, never merged in place. A periodic job does the same for every group of entities that share a name. The document that states 简称海探1 therefore finds the 海探1 created a week earlier, and a pair decided on thin evidence comes back when better evidence arrives.++### 6. `name_shape` gives way to evidence++The gate refuses `Version` and `Phrase` shapes because the model says *same* for Claude and Claude 4, and for "Sam Altman's efforts" and Sam Altman. Decision 4 separates the first by a functional conflict (two release dates), and the span check of #582 keeps the second from becoming an entity. `name_shape` and `CORPORATE_SUFFIX` leave the gate once the bench shows the gate catches nothing that evidence misses.++## Not doing++- **A lexicon of abbreviations, honorifics or company forms.** Every list on this path so far has covered the phrasings its author thought of.+- **A learned identity model.** The rule in decision 4 is small enough to read, and every attach or split can name the edge or the conflict that settled it. That is what a Review card and an auditor both need.+- **Merging on name similarity.** Channel 2 finds candidates. Evidence decides.++## Measurement: cut 0++An identity bench in `scripts/bench/`, beside `recall.mjs` and `govern.mjs`: a small bilingual corpus and a truth file that groups mentions by the entity they refer to, scored as pairwise precision and recall of *same entity*, plus the number of pairs sent to people.++The corpus covers two namesakes at one company across documents; two namesakes in one document; one person across unrelated documents; a full name and its abbreviation in one document, then the abbreviation alone in another, in both arrival orders; a rename with a date; one name in Chinese and in English; and look-alikes that must stay apart (two companies' 技术中心, a product and its next version, a company and a fruit). Every run is repeated with the documents in reverse order, and the difference between the two orders is reported as a number.++`ai-timeline.duplicates.json`, the truth behind `govern.mjs`, runs alongside as a regression.++## Cuts++0. The identity bench, and a baseline on `dev`.+1. Names become facts (decision 1), the extractor reports them (decision 2), and recall reads them (decision 3, channel 1). Merge and revert move name facts; `entities.aliases` is migrated and dropped.+2. Recall channels 2 and 3.+3. Evidence decides (decision 4); mentions resolve after the response is parsed; `doc_cache` by name goes.+4. Re-evaluation (decision 5); `name_shape` out of the gate and the generic suffix lists out of recall, each when the bench says so (decision 6).++The entity panel's list of names with their sources and validity, and the evidence lines on a Review card, follow once cut 3 lands.diff --git a/docs/decisions/README.md b/docs/decisions/README.mdindex 3b6733a0..2f8e5c24 100644--- a/docs/decisions/README.md+++ b/docs/decisions/README.md@@ -64,6 +64,7 @@ The test for writing one: if someone (including us) looks at a piece of code in| 0038 | [The interface has a light side](0038-the-interface-has-a-light-side.md) | Implemented (#599) · `data-theme` on `<html>`, chosen by the reader and stored in the browser, `system` resolved to one of the two before first paint · a second token block in `styles.css` and not one rule rewritten: alpha stays, the triplet swaps · the canvas reads its colours from the tokens through one reader file and rebuilds on a switch; edges are flattened over the ground on light because the WebGL shader can only brighten · a `raw-colour` guard rule over `.ts` and `.tsx` · the entity palette is the same in both themes; a paper-tuned set is open || 0039 | [A chunk is what extraction sees](0039-a-chunk-is-what-extraction-sees.md) | Implemented, cut 1 · the parser's Markdown is read into blocks and packed by a token budget (1000, cl100k) · a table travels with its caption and header and a continuation repeats both · headings become a breadcrumb and fill `chunks.heading` · a page-break rule is not a boundary and a table it split is joined back · not measured on its own: the budget · cut 2: an external parser behind the block model, evidence that names a cell or a region || 0040 | [A chunk says where its words came from](0040-a-chunk-says-where-its-words-came-from.md) | Proposed · nothing built · a chunk carries an **origin** (`stated`, `ocr`, `transcribed`, `described`) and the model that produced it, and an **anchor** back into the original bytes (a page and region, a time range, an image inside the file) — decided before any media reader, because a transcript stored without its times can never be tied to the recording again · the packer never mixes origins in a chunk · facts from a description enter below `AUTO_CLOSE_MIN_CONFIDENCE`, so a misread chart opens a conflict instead of closing a correct fact · per-modality model settings, media reading as a resumable job, origin and anchor in the API, MCP and RDF · cuts: ledger shape, scans via Docling, recordings, descriptions, video |+| 0041 | [A name is a claim about an entity](0041-a-name-is-a-claim-about-an-entity.md) | Cut 0 built (identity bench) · cut 1 implemented (#670): names are value facts on `known_as`, the extractor reports other names, a shared name goes to the adjudicator; forward/reverse F1 0.43/0.54 → 0.68/0.68 · cuts 2–4 (name vectors and neighbours, evidence decides, re-evaluation) not started || 0042 | [The chat loop is a runner with hooks](0042-the-chat-loop-is-a-runner-with-hooks.md) | Implemented (#548) · the loop is rig's runner and every policy is a hook with a typed result · the wire stays `LlmClient` behind `RigModel` · a turn cannot end before a tool has run, `no_evidence_needed` is the exit for questions not about the base · RAG fallback only on a 400/422 to the first request with tools · an empty reply is asked again once · the skip rate is the model's (DeepSeek-V3 1–3 of 12, Qwen2.5-72B 0) and recorded, not prevented |## Not a decision recorddiff --git a/scripts/bench/corpora/identity/d00-profile.txt b/scripts/bench/corpora/identity/d00-profile.txtnew file mode 100644index 00000000..edaa2939--- /dev/null+++ b/scripts/bench/corpora/identity/d00-profile.txt@@ -0,0 +1,3 @@+东海海洋研究所简介(2020年9月10日)++东海海洋研究所位于舟山,下设财务部和工程部。财务部总监张伟负责全所的预算管理。diff --git a/scripts/bench/corpora/identity/d01-rename.txt b/scripts/bench/corpora/identity/d01-rename.txtnew file mode 100644index 00000000..d814a7be--- /dev/null+++ b/scripts/bench/corpora/identity/d01-rename.txt@@ -0,0 +1,3 @@+东海海洋研究所更名公告(2021年6月1日)++经上级主管部门批准,东海海洋研究所自2021年6月1日起更名为澜海研究院。diff --git a/scripts/bench/corpora/identity/d02-first-dive.txt b/scripts/bench/corpora/identity/d02-first-dive.txtnew file mode 100644index 00000000..6513c781--- /dev/null+++ b/scripts/bench/corpora/identity/d02-first-dive.txt@@ -0,0 +1,3 @@+海洋探测器1号首次深潜公告(2023年3月15日)++澜海研究院研制的海洋探测器1号(简称“海探1”)于2023年3月15日完成首次深潜,最大深度4500米。李娜担任海洋探测器1号的项目负责人。diff --git a/scripts/bench/corpora/identity/d03-monthly.txt b/scripts/bench/corpora/identity/d03-monthly.txtnew file mode 100644index 00000000..f9c824ea--- /dev/null+++ b/scripts/bench/corpora/identity/d03-monthly.txt@@ -0,0 +1,3 @@+海探1月度作业报告(2023年5月10日)++海探1在南海完成第三次下潜,最大深度6000米。工程部工程师张伟负责海探1的数据回传。diff --git a/scripts/bench/corpora/identity/d04-budget.txt b/scripts/bench/corpora/identity/d04-budget.txtnew file mode 100644index 00000000..0d714194--- /dev/null+++ b/scripts/bench/corpora/identity/d04-budget.txt@@ -0,0 +1,3 @@+年度预算批复(2023年6月30日)++澜海研究院财务部总监张伟批复了海探1项目的年度预算。diff --git a/scripts/bench/corpora/identity/d05-board.txt b/scripts/bench/corpora/identity/d05-board.txtnew file mode 100644index 00000000..a28219d2--- /dev/null+++ b/scripts/bench/corpora/identity/d05-board.txt@@ -0,0 +1,3 @@+理事会会议纪要(2023年9月1日)++澜海研究院理事会于2023年9月1日召开,李娜当选理事会成员。diff --git a/scripts/bench/corpora/identity/d06-spec.txt b/scripts/bench/corpora/identity/d06-spec.txtnew file mode 100644index 00000000..7028f937--- /dev/null+++ b/scripts/bench/corpora/identity/d06-spec.txt@@ -0,0 +1,3 @@+海洋探测器2号耐压舱规格(2023年10月12日)++李娜主持编写了海洋探测器2号(简称“海探2”)的耐压舱技术规格。海探2的设计深度为11000米。diff --git a/scripts/bench/corpora/identity/d07-awards.txt b/scripts/bench/corpora/identity/d07-awards.txtnew file mode 100644index 00000000..f390a3d4--- /dev/null+++ b/scripts/bench/corpora/identity/d07-awards.txt@@ -0,0 +1,3 @@+年会表彰名单(2024年1月8日)++财务部总监张伟获得年度管理奖。工程部工程师张伟获得年度技术奖。diff --git a/scripts/bench/corpora/identity/d08-english.txt b/scripts/bench/corpora/identity/d08-english.txtnew file mode 100644index 00000000..be2e76f5--- /dev/null+++ b/scripts/bench/corpora/identity/d08-english.txt@@ -0,0 +1,3 @@+Lanhai Research Institute brief (2024-02-20)++Lanhai Research Institute announced that Li Na will lead the sea trials of Haitan-2. Haitan-2 is based at the port of Sanya.diff --git a/scripts/bench/corpora/identity/d09-home-port.txt b/scripts/bench/corpora/identity/d09-home-port.txtnew file mode 100644index 00000000..6101ff5f--- /dev/null+++ b/scripts/bench/corpora/identity/d09-home-port.txt@@ -0,0 +1,3 @@+海探1母港调整通知(2024年4月2日)++自2024年4月起,海探1的母港设在三亚。diff --git a/scripts/bench/corpora/identity/d10-centres.txt b/scripts/bench/corpora/identity/d10-centres.txtnew file mode 100644index 00000000..f1cb2280--- /dev/null+++ b/scripts/bench/corpora/identity/d10-centres.txt@@ -0,0 +1,3 @@+技术合作签约(2024年5月15日)++蓝鲸集团技术中心与澜海研究院技术中心签署了声呐算法合作协议。澜海研究院技术中心隶属于澜海研究院。蓝鲸集团技术中心隶属于蓝鲸集团。diff --git a/scripts/bench/corpora/identity/d11-hq.txt b/scripts/bench/corpora/identity/d11-hq.txtnew file mode 100644index 00000000..a91b5ebe--- /dev/null+++ b/scripts/bench/corpora/identity/d11-hq.txt@@ -0,0 +1,3 @@+澜海研究院概况(2024年6月20日)++澜海研究院总部位于舟山。diff --git a/scripts/bench/identity.mjs b/scripts/bench/identity.mjsnew file mode 100644index 00000000..e3f3d41d--- /dev/null+++ b/scripts/bench/identity.mjs@@ -0,0 +1,307 @@+#!/usr/bin/env node+// 身份的测量台(0041 第 0 刀):**同一个东西有没有归成一个实体,不同的东西有没有被并成一个。**+//+// 两类错一起量:重名(两个张伟、两个技术中心、1 号与 2 号)会被错并,多名(更名、简称、+// 中英文名)会被错拆。真值是 truth/identity.json 里的锚点:「这篇文档里、引文含这段话的+// 事实、这个类型那一侧的实体,是真实世界里的哪一个」。打分只比锚点解析到的实体 id 两两+// 是否相同——**名字不参与打分**,否则别名与重名会先把打分本身搅乱。+//+// 一轮 = 新建一个库 → 种本体(冻结:自动扩本体、类型消解、治理、物化全关)→ 按顺序一篇+// 一篇灌,每篇等这个库的任务全部跑完再灌下一篇 → 打分。**入库顺序是被量的东西之一**,+// 所以同一轮正序、倒序各建一个库,报两边判得不一样的锚点对占多少。+//+// 用法:+// node scripts/bench/identity.mjs --label <名字> # 正序 + 倒序各一个新库+// node scripts/bench/identity.mjs --label <名字> --order forward+// node scripts/bench/identity.mjs --score <kb id> # 只给已有库打分+//+// 环境变量:BENCH_BASE / BENCH_EMAIL / BENCH_PASSWORD / BENCH_PSQL(同 recall.mjs)。+// 账号要能在工作区里建库。++import fs from "node:fs";+import path from "node:path";+import { execFileSync } from "node:child_process";+import { fileURLToPath } from "node:url";++const HERE = path.dirname(fileURLToPath(import.meta.url));+const BASE = process.env.BENCH_BASE || "http://localhost:18080";+const EMAIL = process.env.BENCH_EMAIL || "[email protected]";+const PASSWORD = process.env.BENCH_PASSWORD || "benchbench123";+const CORPUS = path.join(HERE, "corpora", "identity");+const truth = JSON.parse(fs.readFileSync(path.join(HERE, "truth", "identity.json"), "utf8"));++const args = Object.fromEntries(+ process.argv.slice(2).reduce((acc, cur, i, arr) => {+ if (cur.startsWith("--")) acc.push([cur.slice(2), arr[i + 1]?.startsWith("--") ? true : arr[i + 1] ?? true]);+ return acc;+ }, []),+);++function psql(sql) {+ const cmd =+ process.env.BENCH_PSQL ||+ "docker exec -e PGPASSWORD=utopia landscapebi-db-1 psql -U utopia -d utopia -tAc";+ const parts = cmd.split(" ");+ return execFileSync(parts[0], [...parts.slice(1), sql], { encoding: "utf8", maxBuffer: 64 << 20 }).trim();+}+const num = (sql) => Number(psql(sql) || 0);+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));+const stamp = () => new Date().toISOString().slice(11, 19);+const log = (...a) => console.error(`[${stamp()}]`, ...a);++let cookie = "";+async function api(method, url, body) {+ const init = { method, headers: {} };+ if (cookie) init.headers.cookie = cookie;+ if (body !== undefined) {+ init.headers["content-type"] = "application/json";+ init.body = JSON.stringify(body);+ }+ const r = await fetch(BASE + "/api/v1" + url, init);+ for (const c of r.headers.getSetCookie?.() ?? []) cookie = c.split(";")[0];+ const t = await r.text();+ if (!r.ok) throw new Error(`${method} ${url} -> ${r.status} ${t.slice(0, 300)}`);+ return t ? JSON.parse(t) : null;+}++// ---------- 等:这个库还有没有活没干完 ----------++function busy(kb) {+ // 抽取任务的 payload 只带 document_id,别的带 kb_id:两种都得认+ return num(`+ SELECT count(*) FROM jobs j+ WHERE j.status IN ('queued', 'running')+ AND (j.payload->>'kb_id' = '${kb}'+ OR j.payload->>'document_id' IN (SELECT id::text FROM documents WHERE kb_id = '${kb}'))`);+}++async function settle(kb, what, limitMs = 15 * 60 * 1000) {+ const start = Date.now();+ // 起码等一拍:刚入队的任务可能还没写进 jobs+ await sleep(1500);+ let quiet = 0;+ while (Date.now() - start < limitMs) {+ const pending = busy(kb);+ const extracting = num(+ `SELECT count(*) FROM documents WHERE kb_id = '${kb}' AND graph_status IN ('queued', 'extracting')`,+ );+ if (pending === 0 && extracting === 0) {+ // 连着两拍都空才算停:一个任务收尾时会排下一个(抽取 → 裁决)+ if (++quiet >= 2) return;+ } else quiet = 0;+ await sleep(3000);+ }+ log(`!! ${what}: 等了 ${Math.round(limitMs / 60000)} 分钟还没停,照样往下走`);+}++// ---------- 建库、种本体、灌语料 ----------++async function workspaceId() {+ // 与 run.mjs、govern.mjs 同一个取法:部署里第一个工作区+ return (await api("GET", "/workspaces"))[0].id;+}++async function buildBase(ws, name) {+ const kb = (await api("POST", `/workspaces/${ws}/kbs`, { name })).id;+ await api("PATCH", `/kbs/${kb}`, {+ auto_extend_ontology: false,+ auto_type_resolution: false,+ governance: false,+ materialize_inferences: false,+ });+ const types = {};+ for (const t of truth.ontology.types) {+ types[t.key] = (await api("POST", `/kbs/${kb}/ontology/entity-types`, t)).id;+ }+ for (const a of truth.ontology.attributes) {+ await api("POST", `/kbs/${kb}/ontology/relation-types`, {+ key: a.key, label: a.label, kind: "attribute", temporal: "state",+ datatype: a.datatype, unit: a.unit ?? "", description: a.description,+ domains: a.domains.map((d) => types[d]),+ });+ }+ for (const r of truth.ontology.relations) {+ await api("POST", `/kbs/${kb}/ontology/relation-types`, {+ key: r.key, label: r.label, kind: "relation", temporal: r.temporal, description: r.description,+ functional: !!r.functional, inverse_functional: !!r.inverse_functional,+ domains: r.domains.map((d) => types[d]), ranges: r.ranges.map((d) => types[d]),+ });+ }+ return kb;+}++async function ingestInOrder(kb, docs) {+ await settle(kb, "本体向量");+ for (const d of docs) {+ const content = fs.readFileSync(path.join(CORPUS, d.file), "utf8");+ await api("POST", `/kbs/${kb}/ingest`, { filename: d.file, content, doc_time: d.doc_time });+ await settle(kb, d.file);+ const st = psql(`SELECT graph_status FROM documents WHERE kb_id = '${kb}' AND filename = '${d.file}'`);+ log(`${d.file}: ${st}`);+ }+}++// ---------- 打分 ----------++function resolveAnchors(kb) {+ // 每条现行事实的证据:哪篇、引文、两侧解析到的**幸存者**(顺着 merged_into 走到头)与类型+ const rows = psql(`+ WITH RECURSIVE chain(id, cur) AS (+ SELECT id, id FROM entities WHERE kb_id = '${kb}'+ UNION ALL+ SELECT c.id, e.merged_into FROM chain c JOIN entities e ON e.id = c.cur+ WHERE e.merged_into IS NOT NULL+ ), surv AS (+ SELECT c.id, c.cur AS survivor FROM chain c JOIN entities e ON e.id = c.cur+ WHERE e.merged_into IS NULL+ )+ SELECT concat_ws(chr(31),+ d.filename,+ replace(replace(coalesce(fe.quote, ''), chr(10), ' '), chr(13), ' '),+ coalesce(ss.survivor::text, ''), coalesce(st.key, ''), coalesce(sv.canonical_name, ''),+ coalesce(os.survivor::text, ''), coalesce(ot.key, ''), coalesce(ov.canonical_name, ''),+ coalesce(f.object_value->>'value', ''))+ FROM facts f+ JOIN fact_evidence fe ON fe.fact_id = f.id+ JOIN documents d ON d.id = fe.document_id+ LEFT JOIN surv ss ON ss.id = f.subject_id+ LEFT JOIN entities sv ON sv.id = ss.survivor+ LEFT JOIN entity_types st ON st.id = sv.type_id+ LEFT JOIN surv os ON os.id = f.object_id+ LEFT JOIN entities ov ON ov.id = os.survivor+ LEFT JOIN entity_types ot ON ot.id = ov.type_id+ WHERE f.kb_id = '${kb}' AND f.invalidated_at IS NULL`)+ .split("\n")+ .filter(Boolean)+ .map((l) => {+ const [file, quote, sId, sType, sName, oId, oType, oName, value] = l.split("");+ return { file, quote, value, sides: { subject: [sId, sType, sName], object: [oId, oType, oName] } };+ });++ const squash = (s) => (s || "").toLowerCase().replace(/\s+/g, "");+ const out = {};+ for (const a of truth.anchors) {+ const ids = new Map(); // id -> name+ for (const r of rows) {+ if (r.file !== a.doc || !squash(r.quote).includes(squash(a.quote_has))) continue;+ for (const role of a.role ? [a.role] : ["subject", "object"]) {+ const [id, type, name] = r.sides[role];+ if (!id || type !== a.type) continue;+ // 一句话里并排好几条事实时(「位于舟山,下设财务部和工程部」),用另一侧把锚点钉在+ // 那一条上:另一侧的名字或值里得有 other_has+ if (a.other_has) {+ const other = r.sides[role === "subject" ? "object" : "subject"];+ if (!squash(`${other[2]} ${r.value}`).includes(squash(a.other_has))) continue;+ }+ ids.set(id, name);+ }+ }+ out[a.id] = [...ids.entries()];+ }+ return out;+}++function score(kb) {+ const found = resolveAnchors(kb);+ const byId = Object.fromEntries(truth.anchors.map((a) => [a.id, a]));+ const missing = truth.anchors.filter((a) => found[a.id].length === 0).map((a) => a.id);+ // 一个锚点在同一篇里解析出几个实体:那一篇之内就已经拆开了+ const splitInDoc = truth.anchors.filter((a) => found[a.id].length > 1).map((a) => a.id);++ // 两两比:一个锚点取它解析到的第一个 id;拆开的锚点另外报+ const usable = truth.anchors.filter((a) => found[a.id].length >= 1);+ const verdicts = {};+ let tp = 0, fp = 0, fn = 0, tn = 0;+ const falseMerges = [], falseSplits = [];+ for (let i = 0; i < usable.length; i++) {+ for (let j = i + 1; j < usable.length; j++) {+ const a = usable[i], b = usable[j];+ const same = found[a.id][0][0] === found[b.id][0][0];+ const truthSame = a.truth === b.truth;+ verdicts[`${a.id}~${b.id}`] = same;+ if (same && truthSame) tp++;+ else if (same && !truthSame) { fp++; falseMerges.push(`${a.id}(${a.truth}) = ${b.id}(${b.truth})`); }+ else if (!same && truthSame) { fn++; falseSplits.push(`${a.id} ≠ ${b.id} (${a.truth})`); }+ else tn++;+ }+ }+ const precision = tp + fp ? tp / (tp + fp) : 1;+ const recall = tp + fn ? tp / (tp + fn) : 1;+ const f1 = precision + recall ? (2 * precision * recall) / (precision + recall) : 0;++ // 每个真实实体落成了几个系统实体、叫什么+ const perEntity = {};+ for (const [key] of Object.entries(truth.entities)) {+ const ids = new Map();+ for (const a of usable.filter((x) => x.truth === key)) for (const [id, name] of found[a.id]) ids.set(id, name);+ perEntity[key] = { entities: ids.size, as: [...ids.values()] };+ }+ const lookalikes = truth.lookalikes.map(([x, y]) => {+ const xs = new Set(usable.filter((a) => a.truth === x).flatMap((a) => found[a.id].map(([id]) => id)));+ const shared = usable.filter((a) => a.truth === y).flatMap((a) => found[a.id].map(([id]) => id)).some((id) => xs.has(id));+ return { pair: `${x}/${y}`, merged: shared };+ });++ const reviews = psql(`+ SELECT concat_ws(chr(31), stage, why, n) FROM (+ SELECT stage, split_part(coalesce(reason, ''), '|', 1) AS why, count(*) AS n+ FROM resolution_reviews WHERE kb_id = '${kb}' AND status = 'pending'+ GROUP BY 1, 2) r ORDER BY stage, why`)+ .split("\n").filter(Boolean).map((l) => l.split("")).map(([stage, reason, n]) => ({ stage, reason, n: Number(n) }));++ return {+ kb,+ anchors: truth.anchors.length,+ missing,+ split_in_doc: splitInDoc,+ pairs: { tp, fp, fn, tn },+ precision: +precision.toFixed(3),+ recall: +recall.toFixed(3),+ f1: +f1.toFixed(3),+ false_merges: falseMerges,+ false_splits: falseSplits,+ per_entity: perEntity,+ lookalikes,+ pending_reviews: reviews,+ entities_in_base: num(`SELECT count(*) FROM entities WHERE kb_id = '${kb}' AND merged_into IS NULL`),+ verdicts,+ _byId: byId,+ };+}++function orderDisagreement(a, b) {+ const keys = Object.keys(a.verdicts).filter((k) => k in b.verdicts);+ const differ = keys.filter((k) => a.verdicts[k] !== b.verdicts[k]);+ return { compared: keys.length, differ: differ.length, rate: keys.length ? +(differ.length / keys.length).toFixed(3) : 0, pairs: differ };+}++function brief(r) {+ const { verdicts, _byId, ...rest } = r;+ return rest;+}++// ---------- 主流程 ----------++await api("POST", "/auth/login", { email: EMAIL, password: PASSWORD });++if (args.score) {+ console.log(JSON.stringify(brief(score(args.score)), null, 2));+ process.exit(0);+}++const label = args.label || "run";+const orders = args.order === "forward" ? ["forward"] : args.order === "reverse" ? ["reverse"] : ["forward", "reverse"];+const ws = await workspaceId();+const results = {};+for (const order of orders) {+ const name = `identity ${label} ${order} ${new Date().toISOString().slice(0, 16)}`;+ const kb = await buildBase(ws, name);+ log(`${order}: base ${kb}`);+ const docs = order === "forward" ? truth.docs : [...truth.docs].reverse();+ await ingestInOrder(kb, docs);+ results[order] = score(kb);+ log(`${order}: precision ${results[order].precision} recall ${results[order].recall} f1 ${results[order].f1}`);+}+const summary = { label, ...Object.fromEntries(Object.entries(results).map(([k, v]) => [k, brief(v)])) };+if (results.forward && results.reverse) summary.order_disagreement = orderDisagreement(results.forward, results.reverse);+console.log(JSON.stringify(summary, null, 2));diff --git a/scripts/bench/truth/identity.json b/scripts/bench/truth/identity.jsonnew file mode 100644index 00000000..9034fa8b--- /dev/null+++ b/scripts/bench/truth/identity.json@@ -0,0 +1,402 @@+{+ "about": "Identity bench for 0041. Documents are ingested one at a time, in `docs` order (forward) or reversed, waiting for every job of the base to finish before the next. An anchor is the entity on one side of the facts whose quote contains `quote_has` in `doc`, picked by type (and by `role` where both sides share a type). Anchors are compared pairwise by the id of the entity they resolve to, so names never take part in the scoring.",+ "ontology": {+ "types": [+ {+ "key": "person",+ "label": "人物",+ "description": "有名有姓的自然人"+ },+ {+ "key": "organization",+ "label": "机构",+ "description": "研究院、研究所、集团,以及它们下属的部门与中心"+ },+ {+ "key": "equipment",+ "label": "装备",+ "description": "有型号的探测器、潜水器等装备"+ },+ {+ "key": "place",+ "label": "地点",+ "description": "城市、海域、港口"+ }+ ],+ "attributes": [+ {+ "key": "depth",+ "label": "深度",+ "datatype": "number",+ "unit": "米",+ "domains": [+ "equipment"+ ],+ "description": "下潜或设计深度"+ }+ ],+ "relations": [+ {+ "key": "works_for",+ "label": "任职于",+ "temporal": "state",+ "domains": [+ "person"+ ],+ "ranges": [+ "organization"+ ],+ "description": "一个人在某个机构或部门任职"+ },+ {+ "key": "member_of",+ "label": "担任成员",+ "temporal": "state",+ "domains": [+ "person"+ ],+ "ranges": [+ "organization"+ ],+ "description": "一个人是某个理事会、委员会的成员"+ },+ {+ "key": "leads",+ "label": "负责",+ "temporal": "state",+ "domains": [+ "person"+ ],+ "ranges": [+ "equipment"+ ],+ "description": "一个人负责某个装备的项目、试验或技术工作"+ },+ {+ "key": "operated_by",+ "label": "所属机构",+ "temporal": "state",+ "functional": true,+ "domains": [+ "equipment"+ ],+ "ranges": [+ "organization"+ ],+ "description": "装备由哪个机构研制或运营"+ },+ {+ "key": "home_port",+ "label": "母港",+ "temporal": "state",+ "functional": true,+ "domains": [+ "equipment"+ ],+ "ranges": [+ "place"+ ],+ "description": "装备的母港"+ },+ {+ "key": "part_of",+ "label": "隶属于",+ "temporal": "state",+ "functional": true,+ "domains": [+ "organization"+ ],+ "ranges": [+ "organization"+ ],+ "description": "下属部门或中心隶属于哪个机构"+ },+ {+ "key": "located_in",+ "label": "位于",+ "temporal": "state",+ "functional": true,+ "domains": [+ "organization"+ ],+ "ranges": [+ "place"+ ],+ "description": "机构总部所在地"+ }+ ]+ },+ "docs": [+ {+ "file": "d00-profile.txt",+ "doc_time": "2020-09-10T00:00:00Z"+ },+ {+ "file": "d01-rename.txt",+ "doc_time": "2021-06-01T00:00:00Z"+ },+ {+ "file": "d02-first-dive.txt",+ "doc_time": "2023-03-15T00:00:00Z"+ },+ {+ "file": "d03-monthly.txt",+ "doc_time": "2023-05-10T00:00:00Z"+ },+ {+ "file": "d04-budget.txt",+ "doc_time": "2023-06-30T00:00:00Z"+ },+ {+ "file": "d05-board.txt",+ "doc_time": "2023-09-01T00:00:00Z"+ },+ {+ "file": "d06-spec.txt",+ "doc_time": "2023-10-12T00:00:00Z"+ },+ {+ "file": "d07-awards.txt",+ "doc_time": "2024-01-08T00:00:00Z"+ },+ {+ "file": "d08-english.txt",+ "doc_time": "2024-02-20T00:00:00Z"+ },+ {+ "file": "d09-home-port.txt",+ "doc_time": "2024-04-02T00:00:00Z"+ },+ {+ "file": "d10-centres.txt",+ "doc_time": "2024-05-15T00:00:00Z"+ },+ {+ "file": "d11-hq.txt",+ "doc_time": "2024-06-20T00:00:00Z"+ }+ ],+ "entities": {+ "INST": {+ "names": [+ "东海海洋研究所",+ "澜海研究院",+ "Lanhai Research Institute"+ ],+ "case": "former name, current name, English name"+ },+ "P1": {+ "names": [+ "海洋探测器1号",+ "海探1"+ ],+ "case": "full name and abbreviation; abbreviation alone in later documents"+ },+ "P2": {+ "names": [+ "海洋探测器2号",+ "海探2",+ "Haitan-2"+ ],+ "case": "next version of P1; abbreviation; transliteration"+ },+ "ZW_FIN": {+ "names": [+ "张伟"+ ],+ "case": "namesake of ZW_ENG, finance"+ },+ "ZW_ENG": {+ "names": [+ "张伟"+ ],+ "case": "namesake of ZW_FIN, engineering"+ },+ "LN": {+ "names": [+ "李娜",+ "Li Na"+ ],+ "case": "one person across a dive notice, board minutes, a technical spec and an English brief"+ },+ "TC_LH": {+ "names": [+ "澜海研究院技术中心"+ ],+ "case": "look-alike of TC_LJ"+ },+ "TC_LJ": {+ "names": [+ "蓝鲸集团技术中心"+ ],+ "case": "look-alike of TC_LH"+ }+ },+ "lookalikes": [+ [+ "ZW_FIN",+ "ZW_ENG"+ ],+ [+ "TC_LH",+ "TC_LJ"+ ],+ [+ "P1",+ "P2"+ ]+ ],+ "anchors": [+ {+ "id": "d00-inst",+ "doc": "d00-profile.txt",+ "quote_has": "位于舟山",+ "type": "organization",+ "truth": "INST",+ "other_has": "舟山"+ },+ {+ "id": "d00-zw",+ "doc": "d00-profile.txt",+ "quote_has": "预算管理",+ "type": "person",+ "truth": "ZW_FIN"+ },+ {+ "id": "d02-p1-depth",+ "doc": "d02-first-dive.txt",+ "quote_has": "4500",+ "type": "equipment",+ "truth": "P1"+ },+ {+ "id": "d02-ln",+ "doc": "d02-first-dive.txt",+ "quote_has": "项目负责人",+ "type": "person",+ "truth": "LN"+ },+ {+ "id": "d02-p1-lead",+ "doc": "d02-first-dive.txt",+ "quote_has": "项目负责人",+ "type": "equipment",+ "truth": "P1"+ },+ {+ "id": "d03-p1",+ "doc": "d03-monthly.txt",+ "quote_has": "6000",+ "type": "equipment",+ "truth": "P1"+ },+ {+ "id": "d03-zw",+ "doc": "d03-monthly.txt",+ "quote_has": "数据回传",+ "type": "person",+ "truth": "ZW_ENG"+ },+ {+ "id": "d04-zw",+ "doc": "d04-budget.txt",+ "quote_has": "年度预算",+ "type": "person",+ "truth": "ZW_FIN"+ },+ {+ "id": "d05-ln",+ "doc": "d05-board.txt",+ "quote_has": "理事会成员",+ "type": "person",+ "truth": "LN"+ },+ {+ "id": "d06-ln",+ "doc": "d06-spec.txt",+ "quote_has": "技术规格",+ "type": "person",+ "truth": "LN"+ },+ {+ "id": "d06-p2-lead",+ "doc": "d06-spec.txt",+ "quote_has": "技术规格",+ "type": "equipment",+ "truth": "P2"+ },+ {+ "id": "d06-p2-depth",+ "doc": "d06-spec.txt",+ "quote_has": "11000",+ "type": "equipment",+ "truth": "P2"+ },+ {+ "id": "d07-zw-fin",+ "doc": "d07-awards.txt",+ "quote_has": "管理奖",+ "type": "person",+ "truth": "ZW_FIN"+ },+ {+ "id": "d07-zw-eng",+ "doc": "d07-awards.txt",+ "quote_has": "技术奖",+ "type": "person",+ "truth": "ZW_ENG"+ },+ {+ "id": "d08-ln",+ "doc": "d08-english.txt",+ "quote_has": "sea trials",+ "type": "person",+ "truth": "LN"+ },+ {+ "id": "d08-p2",+ "doc": "d08-english.txt",+ "quote_has": "Sanya",+ "type": "equipment",+ "truth": "P2"+ },+ {+ "id": "d08-inst",+ "doc": "d08-english.txt",+ "quote_has": "Lanhai",+ "type": "organization",+ "truth": "INST"+ },+ {+ "id": "d09-p1",+ "doc": "d09-home-port.txt",+ "quote_has": "母港",+ "type": "equipment",+ "truth": "P1"+ },+ {+ "id": "d10-tc-lh",+ "doc": "d10-centres.txt",+ "quote_has": "技术中心隶属于澜海研究院",+ "type": "organization",+ "role": "subject",+ "truth": "TC_LH"+ },+ {+ "id": "d10-tc-lj",+ "doc": "d10-centres.txt",+ "quote_has": "技术中心隶属于蓝鲸集团",+ "type": "organization",+ "role": "subject",+ "truth": "TC_LJ"+ },+ {+ "id": "d11-inst",+ "doc": "d11-hq.txt",+ "quote_has": "总部位于舟山",+ "type": "organization",+ "truth": "INST",+ "other_has": "舟山"+ }+ ]+}