World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
The design has a current layer (#738)
WaylandYang0655ddbbf4cf
diff --git a/.gitignore b/.gitignoreindex ab2f1c63..aaf02783 100644--- a/.gitignore+++ b/.gitignore@@ -13,6 +13,7 @@ data/# 必须写成 /docs/* 而非 /docs——父目录被忽略时,子路径无法用 ! 重新纳入。/docs/*!/docs/decisions/+!/docs/design/!/docs/pipeline.md.claude/diff --git a/docs/decisions/README.md b/docs/decisions/README.mdindex a1344670..7d83b768 100644--- a/docs/decisions/README.md+++ b/docs/decisions/README.md@@ -1,5 +1,7 @@# Decision records+The current design, condensed per domain, is in [../design/](../design/README.md); this directory holds the reasons behind it.+Code records what was built and git records when it changed. Neither records why it was built this way and which roads turned out to be dead ends. This directory does.The test for writing one: if someone (including us) looks at a piece of code in six months and asks "why not simply…", and the answer is not in the code, there should be a record.@@ -69,6 +71,54 @@ The test for writing one: if someone (including us) looks at a piece of code in| 0044 | [A document is read in stages](0044-a-document-is-read-in-stages.md) | Proposed · nothing built · extraction becomes stages with narrow contracts: a skim that writes a document card with evidenced fields, graded mentions where descriptions never become entities, statements in the text's words with ends from the inventory, normalisation once per distinct phrase with proposals through governance, time resolved by code, identity by evidence in batch · reviews become decisions that write back · today's extractor frozen until an offline prototype of stages 0 and 3 reports on five benches || 0044 | [The ontology is a view over what documents say](0044-the-ontology-is-a-view-over-what-documents-say.md) | Proposed · nothing built · replaces the staged-reading draft · three layers: extraction writes an open graph in the documents' words (0 of 333 statements unstated in the prototype, against 4–9% of facts bound at write time), a small ontology proposed by an agent and approved by people, and a typed graph computed from the open graph on cached signatures · time mentions resolved against a document time context by code · identity across documents on deterministic evidence before the adjudicator · an errata agent reviews the typed graph || 0045 | [A time mention is resolved against its document](0045-a-time-mention-is-resolved-against-its-document.md) | Proposed · nothing built · a time expression is a mention with its words and place; the model returns shape, anchor, offset and granularity and code computes the interval; a document carries its own date, calendars and anchors across chunks, never its upload time; unresolved mentions wait for an anchor; timelines close on resolution grade instead of confidence |+| | Record | Domain | Status |+|---|---|---|---|+| 0001 | [Ontology import and governance](0001-ontology-import-and-governance.md) | ontology | partly superseded (by 0009, 0010, 0012, 0044) |+| 0002 | [Reasoning engine](0002-reasoning-engine.md) | rules | current |+| 0003 | [The ontology grows out of the corpus](0003-ontology-growth-loop.md) | ontology | partly superseded (by 0007, 0010, 0044) |+| 0004 | [Language follows the reader of each text](0004-language-and-localization.md) | interface | partly superseded (by 0044) |+| 0005 | [The alert center](0005-alert-center.md) | interface | current |+| 0006 | [Ontology scale and the extraction prompt](0006-ontology-scale-and-the-prompt.md) | extraction | superseded (by 0044) |+| 0007 | [Counting decides what becomes a relation](0007-who-decides-what-becomes-a-relation.md) | ontology | partly superseded (by 0044) |+| 0008 | [Ontology packs as the cold start](0008-ontology-packs-as-cold-start.md) | ontology | current |+| 0009 | [An undecided type stays empty](0009-no-type-is-a-type.md) | ontology | current |+| 0010 | [An unnamed relation stays empty](0010-no-relation-is-no-relation.md) | ledger | partly superseded (by 0044) |+| 0011 | [A mapping is configuration](0011-a-mapping-is-not-a-fact.md) | lakehouse-and-actions | partly superseded (by 0036, 0044) |+| 0012 | [The ontology is a contract](0012-the-ontology-is-a-contract-not-a-suggestion.md) | ontology | partly superseded (by 0044) |+| 0013 | [A source hands over its history](0013-a-source-should-hand-over-its-history.md) | sources | current |+| 0014 | [Identity from the person, scope from the token](0014-identity-from-the-person-scope-from-the-token.md) | access-and-audit | current |+| 0015 | [A recorded sentence waits for a nod](0015-recording-a-sentence-is-not-asserting-a-fact.md) | governance | current |+| 0016 | [Close the open seams before cutting new ones](0016-close-the-open-seams-before-cutting-new-ones.md) | process | partly superseded (by 0036) |+| 0017 | [A contradiction points at an error upstream](0017-a-contradiction-points-upstream.md) | governance | current |+| 0018 | [The lakehouse is one protocol away](0018-the-lakehouse-is-one-protocol-away.md) | lakehouse-and-actions | current |+| 0019 | [The second clock can be rewound](0019-the-second-clock-can-be-rewound.md) | time | current |+| 0020 | [An auditor reads it without us](0020-an-auditor-reads-it-without-us.md) | access-and-audit | current |+| 0021 | [A rule reads attributes and concludes a type](0021-a-rule-reads-attributes-and-concludes-a-type.md) | rules | current |+| 0022 | [An unknown date is not an open one](0022-an-unknown-date-is-not-an-open-one.md) | time | partly superseded (by 0045) |+| 0023 | [RSS observations are not documents](0023-rss-observations-are-not-documents.md) | sources | current |+| 0024 | [The world axis reaches the second](0024-the-world-axis-reaches-the-second.md) | time | current |+| 0025 | [Governance reads the ledger before it decides](0025-governance-reads-the-ledger-before-it-decides.md) | governance | current |+| 0026 | [A decision records why](0026-a-decision-records-why.md) | governance | current |+| 0027 | [An automatic merge is gated by what it can undo](0027-an-automatic-merge-is-gated-by-what-it-can-undo.md) | governance | current |+| 0028 | [The adjudicator looks before it asks](0028-the-adjudicator-looks-before-it-asks.md) | governance | current |+| 0029 | [A rule may say "or", once](0029-a-rule-may-say-or-once.md) | rules | current |+| 0030 | [A rule may read what a rule concluded](0030-a-rule-may-read-what-a-rule-concluded.md) | rules | current |+| 0031 | [An event holds at the moment it names](0031-an-event-holds-at-the-moment-it-names.md) | time | current |+| 0032 | [A rule computes what it concludes](0032-a-rule-computes-what-it-concludes.md) | rules | current |+| 0033 | [RSS summaries are scoped to the source being listed](0033-rss-source-summaries-are-source-scoped.md) | sources | current |+| 0034 | [An action is a declared call](0034-an-action-is-a-declared-call.md) | lakehouse-and-actions | proposed |+| 0035 | [A vector index is built by a job](0035-a-vector-index-is-built-by-a-job.md) | sources | current |+| 0036 | [Exploration aligns a schema to the ontology](0036-exploration-aligns-a-schema-to-the-ontology.md) | lakehouse-and-actions | current |+| 0037 | [A relation carries its own attributes](0037-a-relation-carries-its-own-attributes.md) | ledger | partly superseded (by 0044) |+| 0038 | [The interface has a light side](0038-the-interface-has-a-light-side.md) | interface | current |+| 0039 | [A chunk is what extraction sees](0039-a-chunk-is-what-extraction-sees.md) | sources | current |+| 0040 | [A chunk says where its words came from](0040-a-chunk-says-where-its-words-came-from.md) | sources | current |+| 0041 | [A name is a claim about an entity](0041-a-name-is-a-claim-about-an-entity.md) | identity | current |+| 0042 | [The chat loop is a runner with hooks](0042-the-chat-loop-is-a-runner-with-hooks.md) | chat-and-mcp | current |+| 0044 | [The ontology is a view over what documents say](0044-the-ontology-is-a-view-over-what-documents-say.md) | ontology | proposed |+| 0045 | [A time mention is resolved against its document](0045-a-time-mention-is-resolved-against-its-document.md) | time | proposed |++The status word is whether a later record has overtaken this one; what is built is in the record's own status line. Domains are the files of [../design/](../design/README.md), where every record is dated and the status words are defined.## Not a decision record@@ -76,4 +126,4 @@ The test for writing one: if someone (including us) looks at a piece of code in## What does not belong here-The `docs/` root is a local scratch area (`/docs/*` is git-ignored except `/docs/decisions/`). Research notes, temporary checklists and test output live there and stay out of the repository. When a draft settles into a judgment worth keeping, it moves here as a record.+The `docs/` root is a local scratch area (`/docs/*` is git-ignored except `/docs/decisions/`, `/docs/design/` and `/docs/pipeline.md`). Research notes, temporary checklists and test output live there and stay out of the repository. When a draft settles into a judgment worth keeping, it moves here as a record.diff --git a/docs/design/README.md b/docs/design/README.mdnew file mode 100644index 00000000..03b104da--- /dev/null+++ b/docs/design/README.md@@ -0,0 +1,111 @@+# The design has a current layer++`docs/design/` says what the system does today, one file per domain. A design file changes when+the design changes. `docs/decisions/` says why: each record is history, written once and never+rewritten; a change of mind is a dated revision note in place or a new record that supersedes the+old one. Read a design file first; open a record when you need the reasoning, the measurements or+the dead ends behind one sentence of it. Records are cited as [0022]; issues and pull requests as+#725.++## Domains++| File | What it covers |+|---|---|+| [ontology](ontology.md) | Classes, relations, attributes and their declarations; packs and OWL import; growth and adoption; the workbench and alignment over the open graph (0044) |+| [extraction](extraction.md) | What extraction reads and writes, the open contract, drop signals, benches and thresholds |+| [identity](identity.md) | Entities, names (0041), resolution, adjudication, merges and their undo |+| [time](time.md) | Two clocks, the world axis and its precision, unknown bounds, the temporal engine, events, time mentions (0045) |+| [ledger](ledger.md) | Facts and their layers, evidence, invalidation and supersedes, qualifiers, pending facts, purge, export |+| [governance](governance.md) | Review queues, the governor and its gates, agent decisions, nods, the queue redesign (#725) |+| [rules](rules.md) | Axioms, business rules, derived facts, proofs, contradictions |+| [sources](sources.md) | Ingestion, connectors, documents and versions, chunks, media origins (0040), embeddings |+| [lakehouse-and-actions](lakehouse-and-actions.md) | Query engines over mounted data, the semantic layer, declared actions |+| [access-and-audit](access-and-audit.md) | Roles, tokens, credentials, the audit ledger, the export an auditor reads |+| [interface](interface.md) | Language, theme, alerts, the design rules, what the browse pages show |+| [chat-and-mcp](chat-and-mcp.md) | The chat loop, tools, retrieval, `remember`, MCP |++## What changed on 2026-09-16++Cut 1 of 0044 landed (#731): the ledger holds open statements in the document's own words. A+statement is a `facts` row with `layer = 'open'` and the phrase the document used, with qualifiers+keyed by the document's role word, the time words it mentions kept verbatim with an offset, and+evidence located by character offsets in the chunk; a thing the document describes without naming+is an entity with a `description` and no name fact. The same day the user decided that extraction+writes only the open graph: memory documents take the open path and wait for their nod as open+statements (#735), the typed extraction path is deleted together with its per-base switch (#736),+the optional bound pass of 0044 decision 3 is withdrawn, and typed facts will come only from+alignment (0044 cut 2). Extraction writes nothing bound to an ontology any more.++That decision supersedes the prompt-related decisions of 0001, 0003, 0004, 0006, 0010, 0011 and+0037 (the ontology in the prompt, its budget and per-chunk retrieval, the description language the+model reads, the growth loop fed from the model's predicate words, qualifiers keyed by declared+attributes at extraction) and, for the same reason, the write-time direction correction of 0012 and+the input of 0007's counting loop. Their ledger decisions stand: a predicate may be null and display+falls back to the source's wording (0010), evidence on every fact and every drop a row (0001),+qualifiers live beside the edge (0037), an adoption rewrite is an append with undo (0003). The table+below marks these records from this note; their own status lines predate it and were not touched.++## Status words++`current`: every decision in the record holds. `partly superseded (by NNNN)`: a later record, or+the note above, overturned some of its decisions and the rest hold; the record's own revision notes+say which. `superseded (by NNNN)`: none of its decisions hold. `proposed`: the direction is accepted+and the record is not yet on `dev` (0034 has no code; 0043 sits in PR #699; 0044 and 0045 sit in+PRs #710 and #724, and 0044's cut 1 has landed ahead of the record). The record's own status line+is the source of truth for what is built; this table only adds whether a later record has overtaken+it.++## Every record++| | Date | Record | Domain | Status |+|---|---|---|---|---|+| 0001 | 2026-08-27 | [Ontology import and governance](../decisions/0001-ontology-import-and-governance.md) | ontology | partly superseded (by 0009, 0010, 0012, 0044) |+| 0002 | 2026-08-28 | [Reasoning engine](../decisions/0002-reasoning-engine.md) | rules | current |+| 0003 | 2026-08-28 | [The ontology grows out of the corpus](../decisions/0003-ontology-growth-loop.md) | ontology | partly superseded (by 0007, 0010, 0044) |+| 0004 | 2026-08-29 | [Language follows the reader of each text](../decisions/0004-language-and-localization.md) | interface | partly superseded (by 0044) |+| 0005 | 2026-08-29 | [The alert center](../decisions/0005-alert-center.md) | interface | current |+| 0006 | 2026-08-29 | [Ontology scale and the extraction prompt](../decisions/0006-ontology-scale-and-the-prompt.md) | extraction | superseded (by 0044) |+| 0007 | 2026-08-30 | [Counting decides what becomes a relation](../decisions/0007-who-decides-what-becomes-a-relation.md) | ontology | partly superseded (by 0044) |+| 0008 | 2026-08-30 | [Ontology packs as the cold start](../decisions/0008-ontology-packs-as-cold-start.md) | ontology | current |+| 0009 | 2026-08-30 | [An undecided type stays empty](../decisions/0009-no-type-is-a-type.md) | ontology | current |+| 0010 | 2026-08-30 | [An unnamed relation stays empty](../decisions/0010-no-relation-is-no-relation.md) | ledger | partly superseded (by 0044) |+| 0011 | 2026-08-31 | [A mapping is configuration](../decisions/0011-a-mapping-is-not-a-fact.md) | lakehouse-and-actions | partly superseded (by 0036, 0044) |+| 0012 | 2026-08-31 | [The ontology is a contract](../decisions/0012-the-ontology-is-a-contract-not-a-suggestion.md) | ontology | partly superseded (by 0044) |+| 0013 | 2026-08-31 | [A source hands over its history](../decisions/0013-a-source-should-hand-over-its-history.md) | sources | current |+| 0014 | 2026-09-01 | [Identity from the person, scope from the token](../decisions/0014-identity-from-the-person-scope-from-the-token.md) | access-and-audit | current |+| 0015 | 2026-09-01 | [A recorded sentence waits for a nod](../decisions/0015-recording-a-sentence-is-not-asserting-a-fact.md) | governance | current |+| 0016 | 2026-09-02 | [Close the open seams before cutting new ones](../decisions/0016-close-the-open-seams-before-cutting-new-ones.md) | process | partly superseded (by 0036) |+| 0017 | 2026-09-03 | [A contradiction points at an error upstream](../decisions/0017-a-contradiction-points-upstream.md) | governance | current |+| 0018 | 2026-09-03 | [The lakehouse is one protocol away](../decisions/0018-the-lakehouse-is-one-protocol-away.md) | lakehouse-and-actions | current |+| 0019 | 2026-09-04 | [The second clock can be rewound](../decisions/0019-the-second-clock-can-be-rewound.md) | time | current |+| 0020 | 2026-09-05 | [An auditor reads it without us](../decisions/0020-an-auditor-reads-it-without-us.md) | access-and-audit | current |+| 0021 | 2026-09-05 | [A rule reads attributes and concludes a type](../decisions/0021-a-rule-reads-attributes-and-concludes-a-type.md) | rules | current |+| 0022 | 2026-09-06 | [An unknown date is not an open one](../decisions/0022-an-unknown-date-is-not-an-open-one.md) | time | partly superseded (by 0045) |+| 0023 | 2026-09-05 | [RSS observations are not documents](../decisions/0023-rss-observations-are-not-documents.md) | sources | current |+| 0024 | 2026-09-06 | [The world axis reaches the second](../decisions/0024-the-world-axis-reaches-the-second.md) | time | current |+| 0025 | 2026-09-06 | [Governance reads the ledger before it decides](../decisions/0025-governance-reads-the-ledger-before-it-decides.md) | governance | current |+| 0026 | 2026-09-08 | [A decision records why](../decisions/0026-a-decision-records-why.md) | governance | current |+| 0027 | 2026-09-08 | [An automatic merge is gated by what it can undo](../decisions/0027-an-automatic-merge-is-gated-by-what-it-can-undo.md) | governance | current |+| 0028 | 2026-09-08 | [The adjudicator looks before it asks](../decisions/0028-the-adjudicator-looks-before-it-asks.md) | governance | current |+| 0029 | 2026-09-08 | [A rule may say "or", once](../decisions/0029-a-rule-may-say-or-once.md) | rules | current |+| 0030 | 2026-09-08 | [A rule may read what a rule concluded](../decisions/0030-a-rule-may-read-what-a-rule-concluded.md) | rules | current |+| 0031 | 2026-09-08 | [An event holds at the moment it names](../decisions/0031-an-event-holds-at-the-moment-it-names.md) | time | current |+| 0032 | 2026-09-08 | [A rule computes what it concludes](../decisions/0032-a-rule-computes-what-it-concludes.md) | rules | current |+| 0033 | 2026-09-06 | [RSS summaries are scoped to the source being listed](../decisions/0033-rss-source-summaries-are-source-scoped.md) | sources | current |+| 0034 | 2026-09-08 | [An action is a declared call](../decisions/0034-an-action-is-a-declared-call.md) | lakehouse-and-actions | proposed |+| 0035 | 2026-09-09 | [A vector index is built by a job](../decisions/0035-a-vector-index-is-built-by-a-job.md) | sources | current |+| 0036 | 2026-09-09 | [Exploration aligns a schema to the ontology](../decisions/0036-exploration-aligns-a-schema-to-the-ontology.md) | lakehouse-and-actions | current |+| 0037 | 2026-09-10 | [A relation carries its own attributes](../decisions/0037-a-relation-carries-its-own-attributes.md) | ledger | partly superseded (by 0044) |+| 0038 | 2026-09-11 | [The interface has a light side](../decisions/0038-the-interface-has-a-light-side.md) | interface | current |+| 0039 | 2026-09-13 | [A chunk is what extraction sees](../decisions/0039-a-chunk-is-what-extraction-sees.md) | sources | current |+| 0040 | 2026-09-13 | [A chunk says where its words came from](../decisions/0040-a-chunk-says-where-its-words-came-from.md) | sources | current |+| 0041 | 2026-09-13 | [A name is a claim about an entity](../decisions/0041-a-name-is-a-claim-about-an-entity.md) | identity | current |+| 0042 | 2026-09-13 | [The chat loop is a runner with hooks](../decisions/0042-the-chat-loop-is-a-runner-with-hooks.md) | chat-and-mcp | current |+| 0043 | 2026-09-14 | Every review queue is governed (in PR #699, not yet in this directory) | governance | proposed |+| 0044 | 2026-09-16 | [The ontology is a view over what documents say](../decisions/0044-the-ontology-is-a-view-over-what-documents-say.md) | ontology | proposed |+| 0045 | 2026-09-16 | [A time mention is resolved against its document](../decisions/0045-a-time-mention-is-resolved-against-its-document.md) | time | proposed |++0016 is a schedule and a convention rather than a design; its convention (the PR that implements a+record updates its status line) lives in the [decisions README](../decisions/README.md), and its+lines are tracked in the domain files they belong to. [pipeline.md](../pipeline.md) still describes+the typed path of 2026-09-02 and waits for a rewrite around the open graph.diff --git a/docs/design/access-and-audit.md b/docs/design/access-and-audit.mdnew file mode 100644index 00000000..d825b256--- /dev/null+++ b/docs/design/access-and-audit.md@@ -0,0 +1,74 @@+# Identity comes from the person and the ledger outlives them++Records: [0014] (tokens and MCP scope), [0020] (the export and the read contract), [0026] (why on+every decision), [0025] d1 (only people write precedent), [0005] d3 and d4 (visibility), [0013]+(credentials), [0034] d5 and d6 (roles for actions), [0016] E.++## What it does today++**People, workspaces, bases, roles.** A person has an account; a workspace holds bases and grants of+data sources; a base has the roles Viewer, Editor and Admin (`access::kb_role`, `require_kb`), and+`users.is_admin` sees everything [0005, 0014]. Visibility of alerts reuses the role chain+(`min_role` on the row; system alerts to admins only) [0005].++**A personal token acts as the person and can only narrow.** `personal_tokens` (prefix `utp_pat_`,+SHA-256 hash, `scope` read or write, `kb_ids` or every base, `expires_at` with 90 days as the UI+default, `last_used_at`, `revoked_at` as a trace); effective permission is the person's role+intersected with the token's scope; every MCP call re-runs authentication, scope and role in SQL and+writes an audit row `mcp.tool_called`; the tokens page shows the plaintext once beside a client+snippet [0014]. `ingest_token` on a source is a different kind: plaintext, push-only [0014].++**The audit ledger.** `audit_events` records what people did, with an actor and an `actor_label`+snapshot, kept forever, never a notification: entity retypes and renames with before and after,+review decisions with both names and the `why` a person wrote, merges and reverts, nods, mapping+decisions, tool calls, `kb.exported`; a machine's decision carries no actor and keeps the machine's+own `why` [0001, 0005, 0025, 0026]. Precedents are read from this ledger and only rows with an actor+count [0025].++**Credentials.** Source keys, warehouse connection strings and model keys are sealed at rest+(`secrets::seal`), listed in `SOURCE_SECRET_KEYS`, stripped from every response, kept when a field+is blank on edit [0013, 0034]. A token's `user_id` cascades on delete; `audit_events.actor_id` does+not, so the ledger outlives the person and the keys do not [0014].++**An auditor reads the base without us.** `GET /kbs/{id}/export?format=turtle|jsonld` streams the+base as RDF: imported classes keep their IRI, everything else is minted under a stable URN namespace+(`?base=` overrides), every fact is an `rdf:Statement` with both clocks, confidence, evidence with+origin, qualifiers and PROV-O lineage to documents and rules; a fact held and valid now is also a+plain triple [0020, 0037, 0040]. The RDF mapping is the supported external read contract; a breaking+change needs a record; MCP `structuredContent` shares the ledger UUIDs, and a documented field is+removed or renamed only with a record [0020 revised].++## Why++- **Neither the JWT nor the ingest token fits an agent**: one expires weekly and cannot be revoked,+ the other follows a source and never expires [0014].+- **The confused deputy**: an MCP client runs under someone else's prompt over untrusted content, so+ the default token is read-only on one base and writing is ticked [0014].+- **Different blast radius, different storage**: a personal token reaches warehouses outside Utopia,+ so it is hashed while the ingest token is not [0014].+- **Base-level machine tokens were refused** because attribution would become synthetic and a third+ authorisation model fails towards giving too much [0014].+- **Every call checks scope** because list filtering guards only what is visible [0014].+- **The ledger records what people did; alerts record what the system did**, with different+ retention and readers [0005].+- **Reification and standard terms** so the export opens in what the reader already runs, and+ `prov:invalidatedAtTime` means what the record axis means [0020].+- **A URN, not the Host header**, so two exports a year apart line up [0020].++## Proposed and not built++- OIDC SSO, backup and restore [0016 E, #712]. 0016 E also listed credentials encrypted at rest as+ pending; 0034, written later, records `secrets::seal` as already holding model keys, connection+ strings and source credentials, and the later record holds.+- A SPARQL endpoint over an in-memory projection; import of our own export; a download button+ [0020].+- Conflict and review state, chunk identity behind a quote, and historical proofs in the export+ [0020].+- Tokens across workspaces, possibly merging with `data_source_grants` [0014].+- A request queue for action grants [0034]; a theme and language that follow the account [0038].++## Open questions++- What evidence an external agent's fact carries and how its SQL runs are audited [0014].+- Whether 0014's status line still holds: it says `can_write` is hard-coded false, while #735+ exercises `remember` over MCP with a write token; check the code before relying on either.diff --git a/docs/design/chat-and-mcp.md b/docs/design/chat-and-mcp.mdnew file mode 100644index 00000000..57115e76--- /dev/null+++ b/docs/design/chat-and-mcp.md@@ -0,0 +1,79 @@+# The chat loop is a runner with hooks++Records: [0042] (the loop), [0014] (MCP tools and scope), [0015] (`remember` and the nod), [0020]+(the read contract), [0021] (rule tools), [0019] and [0022] (timed reads), [0035] (retrieval),+[0011] and [0036] (mappings in the prompt), [0040] (origin in results).++## What it does today++**The loop** is rig's multi-turn runner (`rig-core` / `rig-agent`, no default features); every+policy is one `AgentHook` in `api/agent.rs`: `tool_choice: required` until a tool has run, then the+budget withdraws the tools and orders an answer; a malformed call is refused with the same message+as before; a tool's UI step goes to the stream; an empty turn is asked again once; a text-only first+turn from an endpoint that ignored `required` is sent back once [0042]. A turn cannot end before a+tool has run; `no_evidence_needed` is the exit for a greeting or "make it shorter"; `STALL_NUDGE`+and `DONE` are gone [0042 d3]. The wire stays `LlmClient` behind `RigModel`: the read timeout,+error bodies, the out-of-credit against rate-limit classification and cache logging; earlier+entities become a `system` message right before the question; degradation to one-shot RAG happens+only on a 400 or 422 to the first request with tools [0042 d2].++**Tools** live once in `tools.rs` and serve chat and MCP alike: `search_chunks`, `search_docs`,+`find_entities`, `entity_facts` (names marked as names, derived rows with their rule, `as_of` and+`at` respected), `changes`, `list_rules`, `rule_matches`, `remember`; MCP results carry+`structuredContent` with ledger UUIDs and each evidence row's origin [0014, 0021, 0041, 0019,+0022, 0040]. The previous turn's tool calls are replayed so the model knows what it did, not only+what it said [0015].++**Retrieval.** Hybrid: vector recall on `chunks.embedding` through the per-dimension HNSW index with+`relaxed_order` iterative scan, plus full text in embedded Tantivy; both take `as_of`; full text is+"now" only; neither takes `at` [0035, 0019, 0022]. Confirmed mappings and a schema document reach+the prompt through retrieval; a conventions document works today where a rule would be exact+[0011, 0036].++**`remember`** writes a memory document at once; its statements go through open extraction and wait+in `pending_facts`; the assistant says the sentence is recorded and its statements will be shown+for confirmation; the card grows into the conversation on the SSE `pending` event, showing the+sentence above the statements with phrase, time words and qualifiers; a nod writes an open+statement through the extraction path [0015, #735]. Over MCP an external agent can therefore only+propose [0015 d6].++**MCP.** Streamable HTTP at `POST /api/v1/kbs/{kb_id}/mcp`, JSON responses, a personal token per+person with scope read or write and a base list, every call re-authenticated and audited [0014].+The external read contract is the RDF export; MCP fields documented in `web/src/docs/mcp.md` are+removed or renamed only with a record, and clients tolerate additions [0020].++**Measured.** DeepSeek-V3 skips the tool on 1 to 3 of 12 data questions and Qwen2.5-72B on 0;+prompt wording did not move it; the miss is recorded in `tool_exchange` and `sources` is empty,+which #547 marks on the answer [0042].++## Why++- **A heuristic judged by the model it corrects does not converge**: the "are you done" guard fired+ four times in fifteen turns and the model said DONE every time [0042].+- **The loop belongs to a library and the policy is ours**, so a policy is a typed hook and not a+ branch in an 1,100-line body [0042].+- **Termination is structural** and guarantees a decision, not a correct one [0042].+- **A first-round error is not "this endpoint cannot call tools"**; only a 400 or 422 is [0042].+- **Earlier entities as a `user` message were answered as if the user wrote them**, hence `system`+ [0042].+- **What the assistant says and what the graph gets must not differ**; the card shows the sentence so+ the person judges from something [0015].+- **The nod gate removes the confused-deputy objection to writes over MCP** [0014, 0015].+- **One tool implementation for chat and MCP**, or the two drift [0014].++## Proposed and not built++- A per-task model (`on_model_select`) and a chat model chosen per base [0042].+- Retrieval that takes `at`; a versioned full-text index [0022, 0019].+- The read-contract gaps: conflict and review state, chunk identity behind a quote [0020].+- Which protocol renders an agent's interface in an external client (A2UI or MCP's own extension)+ when the nod card is needed there [0016].+- `query_data` over MCP: what evidence an external agent's fact carries, how SQL runs are audited+ [0014].++## Open questions++- Whether the nod gate holds every single-item interactive write, such as a hand-added edge [0015].+- `search_chunks` still says results can be cited as `[n]`, which MCP clients cannot see [0014].+- Whether "on 2024-03-15" reads better than "from 2024-03-15 to 2024-03-15" in the tools' wording+ of an event [0031].diff --git a/docs/design/extraction.md b/docs/design/extraction.mdnew file mode 100644index 00000000..ec3defd3--- /dev/null+++ b/docs/design/extraction.md@@ -0,0 +1,107 @@+# Extraction writes what the document says++Extraction turns a chunk into open statements. This file says what it reads, what it writes, what it+refuses, and how it is measured. Records: [0044] (the contract), [0039] and [0040] (the chunk it+sees, in [sources](sources.md)), [0041] (names), [0001] (drop signals), [0015] (memory documents),+and the superseded typed contract of [0006], [0012], [0031], [0037]. Cut 1 landed in #731, memory+documents joined in #735, and the typed path is deleted in #736.++## What it does today++**Input.** One call per chunk, at temperature 0, with the document's filename, the things already+accepted earlier in the document, the document's opening (its first 1,500 characters, #680) and the+chunk text. No ontology, no document date, no request for dates [0044 d2, #731]. The chunk is the+whole context: what its boundary cut away cannot be recovered by the prompt, so the chunker's rules+are extraction's rules [0039].++**Output, the compact contract** (`utopia_extract::open`). `e`: things, each a name or a+description, the document's kind word, and whether it is named or described. `s`: statements, each+naming its subject and object in words, the relation phrase as written, an object name or a literal+value, qualifiers keyed by the document's role words, `when` / `ended` time words, and its own+verbatim quote. `n`: other names the document gives a thing. Statements name their sides in words+and carry their own quote because a numbered contract mixed the ids up on dense passages (misworded+10 to 14% with ids, 2 to 7% with names) [#731]. Parsing is per item: a malformed item is counted and+a truncated reply is repaired to its last complete item.++**Server checks, each a drop reason in `extraction_drops`.** Every quote must occur in the chunk+(`quote_not_in_chunk`); every name in its quote (`name_not_in_text`); a time mention only when its+words occur in the statement's own quote (`time_not_in_quote`), because the model otherwise attaches+one time to unrelated sentences; every subject must be a listed thing (`unknown_ref`); an unlisted+object becomes a literal value with a signal (`object_undeclared`); a string one entity already+claims in the document is not a name of another (`name_claimed_by_another`). Offsets are computed on+the server by locating the quote, never taken from the model. The remaining codes are+`malformed_item`, `truncated_reply` and `object_missing`; the 28 codes of the typed path went with+it [#736]. A drop is a row, never silence; the table is cleared per document when extraction starts+and shown per document in the Library, apart from Review [0001, 0005].++**What it writes** (see [ledger](ledger.md)). Open statements as `facts` rows with `layer = 'open'`,+the phrase, no predicate and no `valid_*`; `statement_qualifiers` keyed by role word;+`time_mentions` as verbatim words with an offset; evidence with `quote_start` / `quote_end`; a+described thing as an entity with `description` and no name fact; name facts on `known_as` [0041];+`attested_at` from the document's date only when that date came from the content or a dating+source, never the upload time (#714). Dedup is on (base, subject, phrase, object), which closes the+NULL-predicate dedup hole of [0010].++**What it skips.** The ontology lists and per-chunk retrieval, predicate matching, the temporal+reconcile, signature checks, type resolution and ontology bootstrap [#731, #736]. Identity still+runs per handle (`resolve_handle`, namesake reviews, the govern and adjudicate enqueues); a chunk of+origin `described` keeps its confidence ceiling [0040]. Until alignment, a base holds open+statements and no typed facts, so timelines, conflicts and signature queues stay empty.++**Memory documents** take the same path; their statements wait in `pending_facts` with phrase,+qualifiers, time words and quote span, and a nod writes an open statement [0015, #735].++## Why++- **Statements in the document's words are faithful**: 0 of 333 not stated and 2% misworded in the+ prototype, against 4 to 9% not stated when bound at write time [0044].+- **No ontology in the prompt**: binding happens per signature and the ontology grows with use; a+ 1,010-class ontology cost 108k tokens a chunk and made the model choose worse [0044, 0006].+- **No dates from the model**: arithmetic inside the model leaves no trace, and "today" resolved+ against the upload day; time is words now and an interpretation later [0045, #714].+- **Code checks structure, never vocabulary.** A name is kept because it is in the text, not because+ a list says it is a name; the same rule for quotes and time words [0041, 0044 d8].+- **A described thing is an entity** and gets no name fact, so a description never bridges two+ documents [#731].+- **Named sides and per-statement quotes** cost tokens and buy the not-stated rate [#731].+- **Every drop is a row** because an append-only ledger with evidence on every fact cannot drop+ silently [0001].+- **Temperature 0** because the endpoint default swung one paragraph between 8 and 31 statements+ [#731].++## Benches and thresholds++`scripts/bench/recall.mjs` (SEC filings, pharma, ai-timeline; Re-DocRED fetched by script and never+used as prompt examples) scores entity-pair recall; `judge_open.mjs` has a judge model read the+chunk and reports stated, misworded and not stated; `identity.mjs`, `govern.mjs`, `temporal.mjs`+and the lease bench cover the other domains. Every cut reports at least three domains, two runs per+configuration, with the judge's calibration stated; no F1 against Re-DocRED, whose gold omits true+facts [0044]. Thresholds: not stated at most 2%, entity-pair recall no lower than before, prompt+tokens per document reported. Measured at cut 1: not stated at or under 2% in 16 of 18+corpus-rounds; NVDA full documents 44 to 46 of 52 (one standard deviation about 2.7) against 43 and+45 before [#731, 0039].++## Superseded++The typed contract: the ontology laid out under a 24,000-character budget or retrieved per chunk+with ancestor completion [0006]; `[event]` / `[eternal]` marks and the zoned-time sentence [0031,+0024]; qualifiers listed after a relation and keyed by a declared attribute [0037]; direction+corrected by signature at write time [0012]; `specific_type` and `proposed_type` for type+resolution [0001]; the model's `relative` flag on a deadline [0022]; the word lists for entity names+and clauses [0012]. The ledger rules those records made stand (see [ledger](ledger.md)).++## Proposed and not built++- Alignment producing typed facts from statements [0044 cut 2]; time interpretation and the document+ time context [0045]; identity profiles [0044 d6]; a layer marker in the interface (an open row+ renders as an unnamed relation labelled by its phrase).+- A second pass asking which figures and dates are not yet in a statement (dense sentences drop the+ amount); the period column of a financial-table cell [#729].+- The errata agent over the typed graph [0044 d7].++## Open questions++- Chinese chunks get less context at 300 tokens than at 1,200 characters; unmeasured [0039].+- Whether a stated observation should raise the confidence of a fact first seen in a description+ [0040].+- Recall of facts that need more than one sentence waits for derivation rules [0044].diff --git a/docs/design/governance.md b/docs/design/governance.mdnew file mode 100644index 00000000..adcac1a8--- /dev/null+++ b/docs/design/governance.md@@ -0,0 +1,129 @@+# Governance decides from the ledger and can be undone++Records: [0025] (the governor), [0026] (why), [0027] (the impact gate), [0028] (the second look),+[0043] (every queue, in PR #699), [0015] (nods), [0017] (contradictions as a queue), [0016] C2 and+[0003] d4 (switches), [0001] P4; the queue redesign is issue #725. The identity rules are in+[identity](identity.md), the temporal actions in [time](time.md).++## What it does today++**Queues, grouped by where an item came from.** Review holds nods (`pending_facts`, placed first),+duplicates (`resolution_reviews`), conflicts (`fact_conflicts`), unconfirmed (every evidence chunk+superseded), low-confidence (`confidence < 0.75`), violations (`axiom_violations`) and defects+(`ontology_defects`), plus the Agent queue; ontology proposals sit on the ontology page and mappings+on their own page [0011, 0015, 0017, 0025]. Every human decision is a ledger event with an actor+(`review.merge`, `review.keep`, `merge.revert`, `fact.*`, `conflict.*`) and may carry a `rationale`,+asked as "What told you?", optional, written as `why` on the event [0026].++**The governor.** `knowledge_bases.governance`, on by default; the `govern` job is enqueued when the+switch turns on, after an extraction that leaves pairs, after a person's decision, and hourly for+backlog, at most one waiting per base; it re-reads the switch before every cluster [0025 d2]. A+round takes up to three clusters from the head of a first-in-first-out queue, each cluster the+pairs sharing a name or entity with its head, up to twelve, one model call per cluster, results+applied in order [0025 d3]. The prompt carries precedents from the ledger: what people decided on+these names, on either name, on this type pair, and any reverted merge; only rows with an actor+count, and a precedent quotes the person's `why` [0025 d1, 0026 d3].++**The gate** is a pure function of verdict, confidence, type conflict and precedents. History that+contradicts the verdict blocks; history that agrees lowers the bar from 0.85 to 0.75; otherwise the+agent's confidence decides, merging and keeping alike; conflicting type families never merge; a pair+people already decided is applied as they decided [0025 d4 revised]. A `same` verdict on a version+or phrase shape, or a `different` verdict on identical compatible names, goes to the second look+first [0025 d10]. Then the impact gate: a merge that would put two values of a functional predicate+on one entity at one moment, that a live derivation rests on, or whose side an assistant answer+named, is held for a person as `escalate_impact` whatever the confidence; keeps are not gated+[0027]. Whatever does not clear the gates becomes a proposal.++**The second look.** A pair the batch cannot settle gets one conversation with tools: the full facts+of a side, source passages, a substring search of the ledger, namesakes, and `consequences` (what a+merge would touch, whether the types share a family); at most six lookups, then `decide` or `defer`+with the one question a person could answer; a daily budget of loop calls per base; the trace is+stored under the row [0025 d8, 0028 d3]. With governance off the batch adjudicator sends its+unsettled pairs through the same loop, and every look is an `agent_decisions` row [0028].++**Every look is a row.** `agent_decisions` holds the pair or target, the action, the confidence, the+model's sentence, the precedents shown, the question, the trace, the calls and a status: `proposed`,+`applied`, `accepted`, `overridden`, `reverted`, `superseded` (a person decided a sibling, so the+pair returns to the queue with the new precedent); one open proposal per pair. Answers go through+the person's own decide path, so an answer is a person's decision and a precedent [0025 d5, d6].+The machine's `why` is written on its merges and keeps and is never a precedent [0026 d6]. The+governor never touches the adjudicator's verdict cache; the adjudicator reads precedents too and+keys its cache on them [0025 d7, 0026 d4].++**The fuse.** Two reverts of the agent's merges within seven days of the switch turning on turn it+off, raise `governance.tripped` and write a `kb.updated` row with the machine as actor; overrides+and merges over a keep do not count [0025 d9].++**What a person sees.** A cluster being decided is locked (stage, job and switch together); the+Overview says how many pairs wait; a duplicate card carries the proposal chip whose Merge / Keep+answers it; the Agent queue lists rows newest first with precedents and trace; a held pair says why+[0025, 0027]. No confident pair is held for a person: the one-in-ten sample was removed on+2026-09-14 because it kept a lease split [0026 d5 revised].++**Contradictions are an audit.** A derivation that collides with an assertion does not land and+becomes a `derived_contradiction` violation with a clue (stale, wrong merge, unsure extraction) and+repairs (close at a date, retract, go to duplicates, relax the ontology, accept); derived against+derived aggregates per rule pair as `rules_disagree`; a disputed fact is marked where it sits [0017].++**Switches.** `auto_extend_ontology`, `auto_type_resolution`, `materialize_inferences`, `governance`:+all default on; every automatic action is visible and reversible, and a switch governs action,+never attention [0003 d4, 0016 C2, 0002 d6 revised, 0025 d2 revised]. Nods stay with people [0015].++**Measured.** `scripts/bench/govern.mjs` against 411 hand-labelled name pairs: 97.7% decided on its+own, 96.7% agreement, 6 wrong merges (4 of them one flipping pair), 12 left for people, 13 minutes+for 589 pairs [0025 d10].++## Why++- **The ledger is the precedent and only people write it**; the agent's own rows would let it cite+ itself and grow more confident every round [0025].+- **History as reference, not prerequisite**: with precedent required a fresh base merged nothing and+ 200 pairs waited [0025 d4 revised].+- **Merging moves facts; a keep is undone by the next merge**, so only merges are gated, and a merge+ that leaves the graph is held because a revert restores the graph and not the world it was read+ into [0027].+- **A held pair is a reason, not a doubt**, and the card says so [0027].+- **A rationale is free text and optional**; a required field teaches the model that people decide+ for no reason, and a pick-list is wrong for the next base [0026].+- **The verdict cache must know the precedents** or a cached answer is one taken without them+ [0025 d7, 0026 d4].+- **Two reverts in a week** is the strongest signal the ledger has [0025 d9].+- **Written identity rules, two of them mechanical**, because the model's own decisions were wrong+ in patterned ways; a shape raises a doubt and only evidence settles it [0025 d10].+- **A contradiction points upstream**; every button is a repair, and a per-item queue needs a cap+ [0017].+- **A nod belongs to the person who said it**; an agent nodding turns a remark into an assertion+ [0015, 0043].++## Proposed and not built++- **Every queue is governed** [0043, PR #699]: the same governor walks low-confidence and stale+ facts (batches of eight) and then conflicts after its duplicate rounds, each with its own actions+ through the people's store paths (`confirm`, `reject`; `close_old`, `retime_new`, `keep_both`,+ `reject_new`); a date must be in the evidence and a stale fact is confirmed only by a quote in the+ current version; every action records its undo in `detail`; reverts of any kind count for the+ fuse. #725 splits the PR: the temporal fixes merge first, the conflict actions become the+ conflicts queue, the low-confidence handling is dropped. Cut 2 (violations, defects, mappings)+ is open.+- **Queues organised by what a decision changes** [#725]: ontology proposals; alignment (a+ signature or rule the aligner cannot settle); time anchors (a missing or conflicting document+ date, an unanchored mention); identity (pairs evidence cannot settle, merges the gate held);+ typed-fact errata; cross-document conflicts; nods. Agents handle every item; people see what an+ agent could not settle or what one decision resolves at once. Cards are one template per queue,+ assembled by code from the ledger with zero model calls at review time; actions are approve, edit+ and approve, reject, defer, make a rule; ordering by leverage, then impact, then age; metrics of+ leverage, agreement, revert rate and backlog tune the gate. Retired: the low-confidence and+ unconfirmed queues (unconfirmed becomes an automatic re-attach with an alert), mappings leave the+ Review page. Kept: `agent_decisions`, `execution_gate`, the temporal engine, name facts,+ `pending_facts`, the audit ledger. Sequence: after 0044 cuts 1 and 2, with no legacy layer.+- The errata agent on the typed graph through the gate [0044 d7].+- Decision memory as retrieval and corrections aggregated into ontology signals [0001 P4b, P4c].++## Open questions++- A budget for the batch (the loop has one) [0025]; how often the impact gate holds on a base with+ reasoning on [0027].+- The agreement rate split by `via` now that the sample is gone: nothing computes it [0026, 0028].+- Tools the loop lacks: the graph beyond direct facts, full-text search, disambiguator history+ [0025].+- Whether an agent may ever write a rule, and through what pending queue [0021].diff --git a/docs/design/identity.md b/docs/design/identity.mdnew file mode 100644index 00000000..c5eb6b8d--- /dev/null+++ b/docs/design/identity.md@@ -0,0 +1,100 @@+# Identity is decided on evidence++Records: [0041] (names and the plan), [0001] (name collisions), [0009] (untyped namesakes,+disjointness), [0016] B3, [0012] (a clause is not a name), [0019] (an entity's clock), [0025] to+[0028] for the adjudicator and governor (in [governance](governance.md)), [0044] d6 and #725 for+the proposed redesign.++## What it does today++**A name is a fact.** Every name a text uses for an entity is a value fact on the system attribute+`known_as` with its quote and both clocks; `canonical_name` stays the label and is one of the name+facts; `entities.aliases` is gone. A name is a value and never a node, so it adds no edge and no+count; `known_as` is neither functional nor inverse-functional, so a second name closes nothing and+opens no conflict [0041 d1, #670]. The extractor reports other names (`n` in the open contract) and+the server keeps one only when it occurs verbatim in its quote and the quote in the chunk; a string+another entity of the document already claims is dropped (`name_claimed_by_another`) [0041 d2]. A+described thing gets no name fact and is never recalled by name [#731].++**Resolution, one mention at a time, while extraction writes.** Recall is an exact match of the+mention's name (and its generic-suffix-stripped keys) against the name facts of entities of the same+type; a context vector (`profile_embedding`, the running mean of chunk vectors) attaches at cosine+0.55 or above, makes a new entity below 0.35, and in between makes a new entity and a pair for the+adjudicator; two same-name candidates within a tie margin go to a person unless a candidate's object+name appears in the chunk [0041, #270, #331]. A name another entity of a compatible type already+holds queues a `shared_name` pair for the adjudicator, never a merge, and the card lists an entity's+non-canonical names as "also known as" [0041 d5 revised]. Containment pairs (`Holmes` inside+`Sherlock Holmes`) queue for names of four characters and more, capped per run; after a merge the+other pending pairs on the source are redirected to the target rather than closed+([pipeline](../pipeline.md)). Declared `disjointWith`, inherited down both hierarchies, keeps+same-name entities apart ahead of every heuristic; kin classes go to Review; `CONFUSABLE_TYPE_KEYS`+is the fallback when nothing is declared [0009, 0016 B3]. Untyped same-name entities may coexist+[0009].++**Merges are reversible on both clocks.** `entity_merges` records what moved with `created_at` and+`reverted_at`; `revert_merge` moves facts, name facts and qualifiers back; a read `as_of` a moment+before a merge shows two nodes [0019, 0041]. A merge is a person's action, the adjudicator's at 0.8,+or the governor's through its gate, and an automatic merge is held for a person when it would leave+the graph (a functional contradiction at one moment, a derivation, a cited answer) [0027]. Retypes+and renames are audited with before and after; a name collision on rename is a prompt to merge, not+a 409 [0001].++**Adjudication.** Twelve pairs per call with names, types, top facts, "also known as" and precedents+from the ledger; unsettled pairs get a second look with tools; the identity rules (a version, edition+or division tail is a different thing; a phrase containing a name is not the name; a list is not its+members; a dropped qualifier, a corporate suffix, a surname or an acronym is the same thing; a+document stays one thing through its amendments) are written once and read by both prompts, and two+are mechanical: `name_shape` and `type_family` [0025 d10, 0028]. Details in+[governance](governance.md).++**Bench.** `scripts/bench/identity.mjs`: a bilingual corpus with namesakes, abbreviations in both+arrival orders, a dated rename, two scripts, and look-alikes; scored as pairwise precision and+recall of same-entity, run forward and reversed, the difference reported. Cut 1 moved forward and+reverse F1 from 0.43 and 0.54 to 0.68 and 0.68 (one run each); re-measured on DeepSeek-V3, 0.61+and 0.44 to 0.53 over three runs [0041].++## Why++- **Identity keyed on the surface string fails two ways**: a namesake is one string for several+ things, an alias is several strings for one; a topic vector records what a passage is about, not+ who is in it; the outcome depended on arrival order [0041].+- **A name is a fact because it can be wrong, corrected and dated**, and the ledger already does that+ for everything else; a separate table would rebuild validity, provenance and revert [0041].+- **The server checks presence in the text, not a word list**; every list so far covered the+ phrasings its author thought of, and `name_shape` cannot see a Chinese name [0041, 0044 d8].+- **Keep apart when unsure**: a wrong merge mixes two entities' facts, and a revert does not recall+ the derivations, violations and answers built on it meanwhile [0001, 0027].+- **Two 张伟 of one type must be storable apart**, so the name index is plain and NULL types do not+ collide [0001, 0009].+- **A shared name goes to the adjudicator, never to a merge**, because reverse arrival order put the+ full name on a new entity and nothing paired the two [0041 d5].+- **A clause is not an entity name**: judged by word count and a finite verb, since a 57-character+ court name and a 65-character clause cannot be told apart by length [0012].++## Proposed and not built++- **Cuts 2 to 4 of 0041**: name vectors and shared distinctive neighbours as recall channels;+ evidence decides (a shared distinctive edge or an inverse-functional value with overlapping+ validity attaches; a functional conflict or declared disjointness rules out; a shared name says+ nothing; the context vector only breaks ties); mentions resolved after the response is parsed and+ `doc_cache` removed; re-evaluation when a name or edge arrives; `name_shape` and the suffix lists+ leave once the bench says the gate catches nothing evidence misses.+- **Identity across documents on profiles** [0044 d6]: entities merged within a document, profiled+ by names, classes, attributes, neighbours and time span; candidates from name facts and name+ vectors across scripts; deterministic scoring with cannot-links; only undecided pairs to the+ adjudicator, which sees two profiles; constrained clustering so A≈B and B≈C never merge A and C+ against evidence; a rename is one entity with names valid at different times; a role is not an+ entity.+- **#725 retires** write-time resolution (`resolve_mention`, the thresholds, `containment_reviews`,+ `resolve_type_drift`, `doc_cache`, the no-embedding fallback), the batch adjudicator and+ `resolution_verdicts`, and the word lists; removal waits until the design passes the identity+ bench and the Linked-Re-DocRED pairs.+- The entity panel's list of names with sources and validity; evidence lines on a Review card+ [0041].++## Open questions++- Whether a description reported as a name ("海探1项目") can be kept out without a word list [0041].+- Two untyped namesakes rely on profile similarity alone [0009].+- How much of identity deterministic evidence settles before the adjudicator is needed [0044].+- Cross-base precedent for namesakes [0025].diff --git a/docs/design/interface.md b/docs/design/interface.mdnew file mode 100644index 00000000..c47fe73a--- /dev/null+++ b/docs/design/interface.md@@ -0,0 +1,82 @@+# The interface reads tokens and never guesses++Records: [0004] (language), [0005] (alerts), [0038] (theme), `web/DESIGN.md` (the six rules), with+the interface cuts named in [0017], [0019], [0021], [0025], [0031], [0037], [0040], [0041].++## What it does today++**Language follows the reader of each text.** UI strings are a per-person browser setting resolved+in one function, with a Chinese bundle typed as `typeof S` so a missing key fails compilation; the+browser language is not guessed while the bundle trails; user-reachable server errors are codes+(`AppError::Invalid` with `code`) rendered by the client, and the server produces no display text;+`label` is data, `description` follows the base's `ontology_lang`, LLM text for people takes+`locale` from the request, and extracted data stays verbatim [0004].++**Theme.** Dark by default, light by choice, system as a third, stored as `utopia.theme` in the+browser and set on `<html data-theme>` before first paint; only the token block changes, alpha+describes hierarchy and the rgb triplet is what swaps; the canvas has one reader+(`graphVisuals.ts`) that re-reads tokens and rebuilds the graph on a switch, edges flattened over+the ground on light because the WebGL shader can only brighten; a `raw-colour` guard fails CI on a+colour value outside the token blocks and the two reader files; the entity palette is the same in+both themes and identical to the server's copy [0038].++**Design rules** (`web/DESIGN.md`, enforced by `pnpm guard`): five type sizes by name, six spacing+steps, four radii by role derived from one `--radius`, colour as tokens with a status as a dot and+plain words, shadcn/ui behind a thin shell. The merge rule for interface work is look first, not+green first: four defects in the theme cut passed every check and were found by looking [0038].++**Alerts.** One row per failure, never updated; the read side folds adjacent rows of one `(kb, kind)`+into an episode; read state is per person; visibility by role; a row names one subject and carries+no display text, titles assembled by kind on the client; one global SSE stream tells clients to+refetch; classification is a pure function on error types after retries are exhausted; a bell with+a red dot and a popover; a group can carry the action that closes its loop ("Run those again"); rows+purge after 30 days [0005]. Kinds: `source.sync_failed`, `llm.unreachable`, `llm.rate_limited`,+`llm.out_of_credit`, `data_source.schema_sync_failed`, `document.needs_reader`,+`governance.tripped` [0005, 0025, 0040].++**Browse pages.** The graph draws an untyped entity in grey rather than dropping it, an edge outside+the ontology in lighter grey with "the source's wording" on hover, an unnamed relation in italics,+a contested edge in the alert colour (`--u-contest`, coral), a blocked derivation as a ghost edge+behind the Derived toggle, and derived edges gold [0009, 0010, 0017, 0002]. The entity panel lists+names apart from facts, marks derived, stale, corrected and disputed rows, shows a rule's premises+back to the passage, and rewinds derivations with the assertions [0041, 0017, 0021, 0019]. Review+cards show the original sentence above the statements, the agent's proposal chip, a held reason and+a rationale input [0015, 0025, 0026, 0027]. An open statement renders as an unnamed relation+labelled by its phrase [#731]. A page subtitle says the action the screen does, in one sentence.++## Why++- **One switch cannot serve five kinds of text** with different readers and sources; a quotation is+ not copy [0004].+- **A string left in Rust is permanently untranslatable** once the locale lives in the client+ [0004].+- **Colour belongs to data and the chrome is grey**; forty values outside the tokens each said "the+ ground is dark" [0038].+- **`prefers-color-scheme` alone fails** because the choice is the reader's, not the OS's, and two+ token sets fork within a month [0038].+- **Aggregation belongs to the view**: a stored aggregate lies as things recover; a self-healing flag+ needs every kind to define "fixed" [0005].+- **Nobody can read an alert away for someone else** [0005].+- **A bell, not a page**, because a page pulls people from their work and then nobody goes [0005].+- **A ring on the node cannot be told from a grey edge** in peripheral vision, so a contested edge+ changes colour whole [0017].+- **Capability first, interface after**: every record lands the capability and names its interface+ cut separately, and the cut is looked at on a running build before it merges.++## Proposed and not built++- A layer marker for open statements; qualifiers on the canvas label; a rule-classified entity's+ marker; an event drawn as a point; the `as_of` control on the graph page; the anchor beside a+ blank start; a rules-feed-rules page [#731, 0037, 0021, 0031, 0019, 0022, 0030].+- The origin and anchor on evidence rows, opening the page, image or recording; the reader settings+ cards [0040].+- The entity panel's names with sources and validity; evidence lines on cards [0041].+- Review cards assembled from the ledger with zero model calls, one template per queue [#725].+- A paper-tuned entity palette; a theme that follows the account; `styles.css` under the guard+ [0038]; Chinese descriptions and `navigator.language` back in `detect()` [0016 C3, 0004].++## Open questions++- Mixed-language corpora in one base, and bundle drift when English wording changes [0004].+- Whether the `as_of` control is a second slider or a mode switch [0019].+- Too many ghost edges on a large base [0017].diff --git a/docs/design/lakehouse-and-actions.md b/docs/design/lakehouse-and-actions.mdnew file mode 100644index 00000000..368d8cbc--- /dev/null+++ b/docs/design/lakehouse-and-actions.md@@ -0,0 +1,75 @@+# A base reads mounted data and acts through declared calls++Records: [0018] (query engines), [0011] (mappings), [0036] (alignment of schemas), [0034] (actions,+proposed), [0016] D, [0014] (`query_data` over MCP). The rules a definition becomes are in+[rules](rules.md).++## What it does today++**Data sources are registered at the deployment and granted to workspaces** (`data_source_grants`,+#142); a base mounts a granted source, which ingests its schema as a document for retrieval+[0011, 0018, 0036 d7]. The connection string is the only input: the scheme picks the engine+(`postgres`, the MySQL wire family since #303, `trino`, `databricks`, `snowflake`); credentials sit+in the userinfo, the path is catalog / database / schema, switches in the query string [0018].++**Four gates on a query.** Parse with the engine's dialect and admit one SELECT only; wrap a LIMIT;+a read-only session with a timeout (the HTTP family has no session, so a timeout alone); JSON Lines+out with numbers and booleans restored from the vendor's column types [0011, 0018]. Loopback and+`NO_PROXY` hosts go direct; everything else follows the proxy variables [0018]. Trino has run+against a real cluster; Databricks and Snowflake are covered by protocol replays and await one+(#241, #242); MaxCompute waits [0018].++**The semantic layer.** `concept_mappings` (concept, source, table, expr, sql, unit, summary,+status, revisions) is configuration outside the ledger with its own page, proposed by exploration+and confirmed by a person; rerunning exploration does not erase a rejection; a Viewer reads, an+Editor revises; chat reads confirmed mappings into its prompt, capped at 30 [0011]. A definition+can be written by hand (#563), and a one-page conventions document in the schema folder moved a+bench from 2 to 14 of 18 with no code [0036]. The concept exploration used to create (`Metric` and+`Dimension` entities with a SQL expression in a side table) is the wrong kind of thing and is to+retire [0036, 0009].++**Actions** are recorded and not built [0034].++## Why++- **Engines follow protocols, not product names**: Iceberg, Delta and Hive are Trino catalogs; the+ binary carries no native driver; reading Iceberg directly is another product [0018].+- **The scheme picks the engine**, so three engines add no dropdown [0018].+- **A mapping is not a fact**: it answers how a number is computed, not what exists; confirming it+ was an UPDATE on an append-only table; its fields hid in JSON [0011].+- **A concept is an attribute of a real class or a rule over attributes**, not an entity:+ exploration filed 28 column names as entities and proposed 0 of 18 usable definitions; a+ convention like "test orders do not count" had nowhere to live (17 of 18 with it, 1 of 18 without)+ [0036].+- **SQL text as a definition binds to a dialect and can name a missing column**; a tree over+ attribute ids fails at save time and renders to each dialect [0036, 0032].+- **An action is data, substituted and never scripted**, registered once for the deployment and+ granted to a base, every run a row, the reach the operator's [0034].+- **A response is never written back as a fact**: that would make a run both a side effect and an+ assertion [0034].++## Proposed and not built++- **Alignment of a schema** [0036]: exploration proposes per table the class it is a table of, its+ attributes, its foreign keys as relations, and a conversion tree per column (0032's tree plus+ cast, case and date truncation; text parsed by `sqlparser`, never stored as SQL), adopted through+ `ontology_proposals` as one thing; a definition is a rule over aligned attributes written by a+ person, with a shared convention as one rule others read; `concept_mappings` becomes rendered+ from an alignment and a rule; `Metric` / `Dimension` retire (#554, #555, #556).+- **Actions** [0034]: `actions`, `action_params` (scalar, bounded, with value sets),+ `action_grants`, `action_runs`; preview and run endpoints for the deployment and for a base; three+ roles; rules firing actions on a derived row's first appearance with a dedupe key; an agent's call+ waits for a nod.+- Mappings leave the Review page for the data-source side [#725]; `concept_mappings.evidence`+ [0016 D1]; the MaxCompute connector; Snowflake key-pair and Trino Kerberos when someone needs them+ [0018].++## Open questions++- Where a rule over aligned attributes runs: at the source, where a sum is honest, or only as+ rendered SQL for chat [0036].+- Matching a column to an existing attribute (by name, sampled values, or the model) [0036].+- How much schema to fetch from a lakehouse catalog (cap 200 tables today); Snowflake's `fixed`+ scale losing trailing zeros [0018].+- Retention of `action_runs`, retries when a rule fires, secrets in arguments, a second action kind+ [0034].diff --git a/docs/design/ledger.md b/docs/design/ledger.mdnew file mode 100644index 00000000..8763a640--- /dev/null+++ b/docs/design/ledger.md@@ -0,0 +1,105 @@+# The ledger appends and never rewrites++Records: [0001] P0 and P1, [0002] d3, [0003] d1 and d2, [0010], [0011], [0015], [0019], [0020],+[0022], [0023], [0037], [0040] d4, [0041] d1, [0044] cut 1 (#731, #735). Time columns are explained+in [time](time.md), derived facts in [rules](rules.md).++## What it does today++**A fact is a row that is never updated in place.** `facts (kb, subject_id, predicate_id NULL,+object_id | object_value, confidence, valid_*, *_precision, attested_from, attested_to, recorded_at,+invalidated_at, supersedes, end_derived, layer, phrase)`. A correction inserts a new row that+`supersedes` the old one and invalidates it; a retraction invalidates; `entity_history` reads the+chain back as "recorded as X, refined to Y, by whom" [0001, 0003]. Confirming a mapping by+`UPDATE ... confidence = 1.0` was the one violation and is why mappings left the ledger [0011].++**Two layers.** `layer = 'open'` rows are what a document says: the relation phrase in `phrase`, no+predicate, no `valid_*`, dedup on (base, subject, phrase, object), `proposed_predicate = phrase` on+the evidence so every read path that tolerates a NULL predicate labels the row by its phrase.+`layer = 'typed'` rows are facts in the ontology's vocabulary; today they are the rows written before+2026-09-16, and alignment will write new ones with `from_statement_id` [#731, 0044]. A predicate is+nullable on both: an unnamed relation is empty, `fact_surface_predicate()` shows the most frequent+source wording, the reader is told it is the source's word, and `no_predicate_still_shows.rs`+guards every read path [0010].++**Evidence.** `fact_evidence (fact, chunk NOT NULL, quote, proposed_predicate, quote_start,+quote_end)`, one row per observation; offsets are computed on the server. A chunk carries `origin`+(stated, ocr, transcribed, described), `origin_model` and an `anchor` into the original bytes, so+evidence says how the words were obtained [0040]. A fact from a described chunk is inserted below+`AUTO_CLOSE_MIN_CONFIDENCE`, so it opens a conflict instead of closing a correct fact [0040 d4].++**Beside the row.** `statement_qualifiers (fact, role, value | entity)` keyed by the document's role+word for open statements; `fact_qualifiers (fact, qualifier_type, value | entity)` keyed by a+declared attribute for typed rows, converted by the attribute's datatype, the unit read from the+text (`$4 billion` becomes 4000000000 with unit `$`) and the declared unit only when the text+carries none; a qualifier is not part of the edge's identity; every superseding writer copies+qualifiers with the evidence [0037, #731]. `time_mentions (fact, chunk, text, char_start)` hold+verbatim time words [#731]. Names are value facts on `known_as` [0041]. `entities.description`+marks a described, unnamed thing [#731].++**Derived facts live in their own table** (`derived_facts`, widened to literal objects, with+`fact_derivations` as a proof tree) and never enter `facts`; an asserted triple is never derived,+and a derivation that contradicts an assertion does not land [0002, 0021, 0030].++**Pending facts are their own table.** A memory document's statements wait in `pending_facts`+(chunk, phrase, qualifiers, time words, quote span, `proposed_by`) and are written to `facts` only+on a nod; rejections go to `rejected_facts` keyed by subject, phrase and object and block only that+phrase; a nod takes the extraction write path [0015, #735].++**The record axis.** `invalidated_at` is the only way a fact leaves; `record_axis::held_at` is the one+predicate; deleting a document invalidates the facts it was the last source of and leaves a+tombstone, so the state before is reachable `as_of` [0019]. Purge releases a document's key while+its deletion history keeps the external identity, so an old backlog cannot resurrect purged content+[0023]. Alerts keep 30 days; `audit_events` keeps forever [0005].++**Drops are rows.** `extraction_drops` records what did not land, cleared per document at the start+of extraction; signals that are not drops (`object_undeclared`) are recorded the same way so nothing+is silent [0001].++**Export.** Every fact is an `rdf:Statement` with world time (`schema:validFrom` / `validThrough`),+record time (`prov:generatedAtTime` / `invalidatedAtTime`), confidence, evidence with its origin,+qualifiers as triples on the statement node, and `prov:wasDerivedFrom` its documents; a fact+currently held and valid is also written as the plain triple [0020, 0037, 0040].++## Why++- **The change of belief is information**: a correction that overwrites loses what was believed and+ when; a superseding row keeps both and makes the second clock rewindable [0001, 0019].+- **A null predicate is honest; a relation named "related" is an assertion.** 533 facts displayed+ "related" while their real wording sat in the evidence [0010].+- **A status column on `facts` fails the wrong way**: 56 queries filter live facts by+ `invalidated_at IS NULL`, and one missed `AND nod <> 'pending'` leaks an unconfirmed fact; a+ separate table hides the queue instead [0015, 0002 d3].+- **A state encoded as a float lies twice**: it lands in the low-confidence queue and cannot express+ "in effect" [0011, 0015].+- **The edge already has the standing of a node** (an id, evidence, conflicts, a reified statement in+ the export); it lacked only a place for an attribute [0037].+- **The origin lives on the chunk** because every evidence row from one chunk shares it and the chunk+ is what extraction reads [0040].+- **An anchor not stored at ingest cannot be recovered**: a transcript without its times can never be+ tied to the recording [0040].+- **A name is a value, never a node**, so names add no edge, no count and no conflict [0041].+- **Reification, not RDF-star or named graphs**, because the file has to open in what the reader+ already runs [0020].+- **Every read path must tolerate a NULL predicate**: eleven inner joins became LEFT JOINs and the+ database caught three misses the compiler could not; SQL changes need DB tests [0010].++## Proposed and not built++- Typed rows carrying `from_statement_id`, marked implied when a rule produced them, and recomputed+ per signature [0044 cut 2].+- Two rows and a `fact_conflicts` entry when two mentions of one typed edge disagree on a qualifier+ (today the first value stays and the disagreement is a drop); an entity-valued qualifier (column+ reserved) [0037].+- A unique index or per-base lock behind `insert_fact_inner`'s read-then-write dedup, which let two+ parallel documents insert one edge twice [0037].+- Versioned proofs: `fact_derivations` is rewritten in place when a kept conclusion is reproved+ [0019, 0030].+- Conflict and review state, chunk identity behind a quote, and historical proof snapshots in the+ export [0020].++## Open questions++- Whether corroboration by a stated observation should raise a described fact's confidence [0040].+- Whether the anchor should be shown beside a blank start [0022].+- Retention of `action_runs` when actions land [0034].diff --git a/docs/design/ontology.md b/docs/design/ontology.mdnew file mode 100644index 00000000..bd900770--- /dev/null+++ b/docs/design/ontology.md@@ -0,0 +1,129 @@+# The ontology is a view over what documents say++The ontology is the vocabulary a base answers questions in: classes, relations, attributes and their+declarations. This file says where that vocabulary comes from, how it is governed, and how it binds+to what documents say. Records: [0001], [0003], [0006], [0007], [0008], [0009], [0012], [0036],+[0037], [0044]. The review side of proposals is in [governance](governance.md), the schema side of+mappings in [lakehouse-and-actions](lakehouse-and-actions.md), the extraction contract in+[extraction](extraction.md).++## What it does today++**Three layers, and "ontology" names one of them.** The open graph holds what documents say:+entities with their names and type words, statements whose relation is the document's phrase,+qualifiers, time mentions and quotes. The ontology holds object types, link types and properties,+each with a definition, examples and regression cases, plus constraints, versioned on the recorded+axis. The typed graph holds facts expressed in the ontology, computed from the open graph and marked+with the ontology version it was computed under [0044]. Since 2026-09-16 extraction writes the open+graph and nothing else (#731, #736); the ontology enters no extraction prompt; typed facts will come+only from alignment (0044 cut 2, not built). A base extracted today holds open statements and name+facts and no typed facts, so signature checks, timelines and proposals have nothing to read until+alignment lands.++**The tables.** `entity_types` and `relation_types` (`kind` is relation or attribute),+`entity_type_parents` for multiple inheritance with one primary parent for the tree view,+`relation_type_domains` / `relation_type_ranges` checked through the subclass DAG,+`entity_type_disjoint`, `inverseOf` and `subPropertyOf`, the `functional` and `inverse_functional`+flags, `temporal` (state, event, eternal) on every relation, `datatype` and `unit` on attributes,+and `relation_type_qualifiers` naming the attributes an edge of that relation may carry+[0001, 0031, 0037]. A class or relation imported from a vocabulary keeps its IRI (`UNIQUE (kb_id,+iri)`) and the export writes that IRI back; `key` (`[a-z0-9_]`, at most 40 characters) is the token+the API uses, unique per base [0001, 0020].++**Where a vocabulary comes from.** A new base is empty: no classes, no relations, and no pack unless+the person ticks one in the creation dialog [0008, 0009]. Five packs ship inside the binary+(schema.org, W3C Org, PROV-O, FOAF, IOF Core), multi-select, reconciled by a static alignment table+of about twenty rows, additive and without undo [0008]. An OWL file (Turtle or RDF/XML) is stored+verbatim in the blob store and projected into the tables by a re-runnable derivation that reports+what it did not project; a dry run precedes every import; individuals are never imported as facts+[0001]. The rest is typed on the ontology page or adopted from a proposal. The self-check of [0002]+runs after an import: eight defect kinds (a relation both symmetric and asymmetric, a subclass cycle,+an inverse that does not point back) are shown before any fact-level check, and a base without axioms+reports zero.++**A type on an entity is optional and a person's decision is final.** `entities.type_id` is+nullable; NULL means undecided, and `type_source` (extracted, human, inferred) keeps a person's+retype, including "no type", out of every automatic path [0009, 0001]. Same-name entities may coexist+across types and untyped [0009]. Type resolution (candidates from class descriptions and from the+classes of context neighbours, interleaved and never scored together; a class inside the current+subtree applied on its own, a cross-axis class sent to a person, each pair acknowledged once) runs+from the ontology page; extraction no longer enqueues it [0001, 0016, #736].++**Argument order is enforced, participation is guided.** A declared domain or range shapes+candidates and never discards a fact; argument order is the key's encoding convention, so a fact+whose subject violates the domain while the object fits is swapped by the signature and marked+`direction_corrected` [0012]. The write-time swap left with the typed extraction path; what remains+is `ontology::judge_direction` on adoption (a predicate that fits neither way is left off) and on+merge (moved facts are re-checked and reported as `signature` violations) [0012, #736]. Under+alignment a signature's direction is decided once per signature [0044].++**Growth from the corpus.** Predicate words that matched nothing accumulate as proposals with their+counts; adoption is decided by counting (`MIN_DOCS = 2`, and `MIN_SIGNALS = 3` for an LLM run),+grouped by inflectional base, named by the most frequent phrasing, folded onto an existing+equivalent (`_by` inverses included) with subject and object swapped; the waiting facts are+rewritten as superseding rows, undoable per batch; dismissal is remembered and counting goes on; no+axiom is ever set automatically [0003, 0007]. The loop reads typed rows only, so an open phrase is+never auto-adopted, and with the typed path gone it receives no new input until alignment [#731,+#736]. `Metric` and `Dimension` are to retire: exploration inserts no class without a proposal+[0036, 0009].++**Language.** `label` is data and follows no switch; `key` is never translated; `description` follows+`knowledge_bases.ontology_lang` because it was the line the model read [0004]. With no ontology in+the prompt, a description is read by people and by the aligner.++## Why++- **Keep the original, project what has a consumer.** Discarding is irreversible; a projection can+ be redone when a consumer appears, so "not expressible" becomes "not yet projected" [0001].+- **IRI is identity, key is the token.** A key derived from a label breaks on relabel and collides+ across vocabularies; an IRI is a name and never an address [0001].+- **A declaration can be wrong** (`part_of` marked functional produced 59 false conflicts), so types+ guide and never gate; order is not a claim about the world, so it is enforced, measured 57% to 4%+ violations with reversals 39 to 0 [0001, 0012].+- **A placeholder in the ontology is control flow written as vocabulary.** `concept`, `related_to`,+ `mapped_to` and then `Metric` / `Dimension` were removed on the same argument; an undecided thing+ stays empty and readable [0009, 0010, 0011, 0036].+- **Packs give direction through structure**, which prose cannot; but a pack gives types, not+ predicates (81% of facts unbound before bootstrap), costs ten points of type accuracy to omit and+ buys an ontology in the corpus's own words, so none is the default [0008].+- **Counting decides adoption** because the model skipped `runs_on` (8 documents) and adopted a+ one-document verb, and a deterministic ontology makes the bench comparable; the model keeps only+ synonym merging, and merges stay human [0007, 0003].+- **Knowledge vetoes deleting ontology**: a type with entities cannot be deleted, so an import has+ no undo, only a bulk view of empty classes (not built) [0008].+- **A qualifier definition is an attribute definition** with a relation as its domain; a parallel+ type system was refused [0037].+- **The ontology is a view.** Facts bound at write time were induced by the property list (an award+ written as a genre): 4 to 9% not stated against 0 of 333 open statements; binding after the fact+ per signature costs distinct phrasings, not documents, and a changed ontology recomputes only the+ affected signatures instead of re-extracting [0044].+- **Standard vocabularies reify events as classes**, so every pack object property is a state; a+ person changes `temporal` on the page and reads pick it up at once [0031].++## Proposed and not built++- **Alignment** (0044 cut 2): a table of signature (phrase, subject classes, object classes) to+ property and direction with confidence and ontology version; implication rules proposed by the+ aligner, approved on the workbench, executed by code with cached readings; typed rows carrying+ `from_statement_id`; recomputation per changed signature. The prototype aligner reached 14.7% and+ 12.1% of gold recall in two runs against 15.5% for the withdrawn bound pass, so the bar for cut 2+ is parity over two clean runs [0044, #729].+- **The workbench** (0044 cut 5): the ontology page fed by suggestions from the open graph (frequent+ unbound signatures, type words in use, an ontology agent reading them against competency+ questions), by imported files, and by editing; every approved element carries regression cases;+ duplicate properties merged as governance.+- **Exploration proposes an alignment per table** through `ontology_proposals`, adopted as one+ thing; `Metric` / `Dimension` retire (#554, #556) [0036].+- Filtering reified-shell relations (`Action`, `Offer`) out of pack import [0012]; Chinese+ descriptions generated per `ontology_lang` [0016 C3]; a bulk view of an import's empty classes+ [0008].++## Open questions++- What becomes of `ontology_misses`, predicate adoption, type resolution and the bootstrap job once+ alignment lands: #736 leaves them idle; #725 proposes retiring the type machinery and adoption+ from `proposed_predicate` in favour of the ontology agent.+- Competency questions for a base that has none; when the typed graph is materialised eagerly+ [0044].+- Mixed-pack accuracy, Chinese labels on an English pack, and what a 1,500-term start does to+ adoption thresholds never retuned [0008, 0007]; narrative verbs and `_by` folding [0007].diff --git a/docs/design/rules.md b/docs/design/rules.mdnew file mode 100644index 00000000..ec0a847d--- /dev/null+++ b/docs/design/rules.md@@ -0,0 +1,96 @@+# A rule concludes from readings it can name++Records: [0002] (axioms), [0021] (attribute rules), [0029] (or), [0030] (chaining), [0032]+(computed conclusions), [0017] (contradictions), [0036] d4 (definitions as rules), [0034] (the seam+to actions). Interval rules are in [time](time.md), the review side in+[governance](governance.md).++## What it does today++**Two reasoners, one materialisation job.** Axioms compiled from the ontology (transitive,+symmetric, inverseOf, subPropertyOf; functional, inverse-functional, asymmetric, irreflexive for+checking) run `derive()` over entity-to-entity edges; business rules (`attribute_rules`) run a+second pass over literal readings; both write `derived_facts` and `fact_derivations`, in a job every+`inference_interval_minutes` (60) under `materialize_inferences`, on by default; every run+recomputes the whole base [0002, 0021, 0030]. A fresh base holds only open statements, which have+no predicate, so there is nothing to derive until alignment writes typed rows [#731].++**R0 checks first and writes no facts.** Six fact-level kinds (`self_loop`, `asymmetry`, `cycle` with+its path, `functional`, `signature`, `derived_contradiction`) go to `axiom_violations`, which asks+"wrong data or wrong definition" (`fact_retracted`, `fact_closed`, `axiom_relaxed`, `accepted`);+eight ontology defect kinds go to `ontology_defects` and are shown first; "data is wrong" retracts+the fact and reruns [0002, 0017]. A functional violation is two values holding at one moment; a+succession is none [0017, #635].++**Asserted beats derived, hard.** An asserted triple is never derived; a derivation that contradicts+an assertion stays out and becomes one capped `derived_contradiction` row; two derivations that+contradict each other both stay out, aggregated per rule pair as `rules_disagree` [0002 d4, 0017].+`MAX_DERIVED_PER_PREDICATE` (20,000) truncates and reports; validity is the intersection of premise+read intervals, confidence the minimum, precision from the premise that set the bound; an empty+intersection derives nothing [0002 d8, 0022, 0024].++**A business rule** names a subject class, a conclusion (a class IRI on the builtin `is_a`, an+attribute value, or a computed expression) and conditions `(attribute, op, operand)` with ops+`> >= < <= in not_in present`, grouped: conditions in a group join with and, groups with or, one+level [0021, 0029, 0032]. A concluded typing is a derived attribute fact and never writes+`entities.type_id` [0021 d2]. An operand or a conclusion may be an expression tree over attributes+(`attr | const | add sub mul div`, depth capped), stored as JSONB, reachable through the API, the+picker not built; a missing reading or a division by zero concludes nothing and is reported [0032].++**Rules chain inside one run.** A round's conclusions rejoin the pool as readings and as derived+class membership (entering as a premise so the interval narrows to when the entity was that class);+the fixed point runs to `MAX_DEPTH` with a sorted frontier so a run is reproducible; a rule may read+its own conclusions; the previous run's `derived_facts` are never input [0030]. The proof is a tree+(`premise_fact_id` or `premise_derived_id`, one `seq`), walked by `proof()`; a kept row whose+premises changed is reproved [0030]. Retirement is free: every run builds the wanted set and+invalidates what is not in it [0030].++**Explanation.** The entity panel shows a conclusion with its rule and each premise back to its+passage; derived edges are gold behind a toggle; blocked derivations are ghost edges; MCP has+`list_rules` and `rule_matches`, read-only [0002, 0017, 0021].++## Why++- **A reasoner amplifies defects**: 185 `part_of` facts became 828 under closure, with cycles from+ extraction errors, so the engine is a checker first [0002].+- **Derived facts in their own table** because of forty queries over `facts` one knew a flag; a+ forgotten UNION hides derivations instead of mixing them in [0002 d3].+- **Rules come from the ontology or from a person, never proposed by the model** [0002 d5, 0021 d3].+- **Widen `derived_facts` rather than add a parallel table**: a typing and an attribute are special+ cases of the shape `facts` already has [0021 d1].+- **A derived typing is an overlay**: writing `type_id` would put a derivation where the asserted+ type lives and force reconstructing ground truth on retraction [0021].+- **One level of or**: a hit reports the readings that made it true, and a disjunction has no+ premises of its own; absence is not a condition because it has no premise and the base is+ open-world [0029].+- **A fixed point, not an ordering**, or whether `A → B → C` fires depends on load order; the DDL's+ objection protects "a run reads only asserted facts", which the in-memory fixed point keeps [0030].+- **Picked, not typed**, because a picker cannot name an attribute that does not exist and a text+ box grows a grammar; storing a string evaluated at run time is what is forbidden [0032].+- **Aggregation is refused** because a sum asserts a completeness the base cannot hold, its proof+ cannot explain its retirement, and it forecloses incremental maintenance [0032].+- **Repairs, not rulings**: a contradiction is stale knowledge, a misread, a wrong merge or an+ over-strict ontology, and each button goes there [0017].++## Proposed and not built++- Reading a value across a relation (`well → field → depth`); unit and datatype checks when an+ expression is written; the expression picker [0032].+- The page showing which rules feed which and a deeper proof rendering [0030].+- A definition over aligned attributes as a rule, with a shared convention as one rule others read,+ and an executor that runs it at the source where a sum is honest [0036].+- Rules firing actions on a derived row's first appearance, deduped on `(rule, derived_fact)`+ [0034]; implication rules of alignment, executed by code and marked implied [0044 d3].+- R3 incremental maintenance, deferred until a full re-derivation exceeds a threshold (10 s, a+ guess) [0002, 0016 B4].+- A canvas marker for rule-classified entities; editing a conclusion in place; an agent writing a+ rule behind a pending queue [0021].++## Open questions++- Materialise or evaluate at query time at scale [0002]; the cap of 50 per predicate for+ contradictions and per-pair `accepted` granularity [0017].+- Category conditions match strings literally across languages [0021, 0029].+- Whether a derived typing should feed extraction or alignment [0021].+- How deep a chain is worth drawing; whether a computed conclusion feeds the next rule (it should,+ untested) [0030, 0032].diff --git a/docs/design/sources.md b/docs/design/sources.mdnew file mode 100644index 00000000..76aed25e--- /dev/null+++ b/docs/design/sources.md@@ -0,0 +1,99 @@+# A source hands over its history++Records: [0013] (connectors), [0023] and [0033] (RSS), [0039] (chunks), [0040] (origin and media),+[0035] (vector index), [0036] d7 (schema documents), [0005] (sync alerts), [0022] and [0045] d3+(document dates), [0016] E.++## What it does today++**Ingest.** A file or a synced item goes into a content-addressed blob store; `documents` and+`document_versions` keep every version; `ingest_item()` owns identity (`external_key`), sha256 dedup+and versions; a document is `ready` for search once embedded while extraction queues behind it+[0001, 0013, [pipeline](../pipeline.md)]. `documents.doc_time` with `doc_time_source` (content,+source, upload_time, none) is the document's own date; upload time is recorded time and never a+document date, so an upload-dated document counts as undated for attestation and ordering+[0022, 0045, #714].++**Connectors.** GitHub issues, Jira, Notion, RSS, WebDAV, S3 / Azure / GCS, and custom push with an+`ingest_token`; the kinds come from one `SourceKind` enum that the allowlist, the dispatcher and the+web list derive from; credentials never leave the server (`SOURCE_SECRET_KEYS`) and a blank field+keeps the stored value; a truncated round says so [0013]. A ticket is one document whose body+carries its history as dated declarative sentences and a `## History` list; GitHub fetches events+per ticket; timestamps are written to the second [0013].++**RSS.** Observations are not documents: one table with baseline, candidate and no_source rows;+jobs own attempts, documents own identity; an entry without a GUID or an article link is skipped;+purge and reappearance are fenced by database time; one Readability extractor serves linked pages+[0023]. The source list aggregates only the listed source's current generation with a lateral join,+and the API carries a typed nested `rss_full_content` summary [0033].++**Parse and chunk.** The parser's Markdown is read into blocks (heading, paragraph, table with header+and rows, rule); the packer fills a token budget (300 by default, cl100k, `UTOPIA_CHUNK_TOKENS`); a+table travels with its caption and header and a continuation repeats both; a caption attaches to+every table that follows it; a row is never split; headings become a breadcrumb prefix and+`chunks.heading`; a thematic break is not a boundary and a table it split is joined back; no+overlap; chunk text is verbatim slices so a quote is found by offset; a document is re-chunked on+its next reprocess [0039].++**Origin and anchor.** `chunks.origin` (stated, ocr, transcribed, described), `origin_model`, and an+`anchor` whose shape is checked per origin (page and box; start, end and speakers; page and image or+part); the packer never mixes origins in one chunk; a described block is its own chunk with the+breadcrumb and caption. Images and recordings are recognised by header or extension and a PDF with+an empty text layer is a scan; without the reader the document fails once with `reader_needed` and+a `document.needs_reader` alert, and saving the setting queues it again [0040]. Scans and images go+to a workspace's MinerU service (`llm_settings.ocr_*`), one segment per page, the job waiting with+`Deferred` on the remote task recorded on the document; recordings go to a diarizing transcription+model (`transcribe_*`) and a transcript without speaker labels is refused; speakers are written into+the text as turns [0040 cuts 2 and 3]. Facts from a described chunk enter below the auto-close+threshold [0040 d4].++**Embedding and index.** `chunks.embedding` and `entities.profile_embedding` are `vector` with no+fixed dimension; the first write of a dimension requests a partial HNSW expression index built by a+`build_vector_index` job outside any transaction (512 MB `maintenance_work_mem`); reads write the+dimension as a literal, set `hnsw.iterative_scan = relaxed_order` with a raised scan memory, and let+the planner choose; above 2000 dimensions stays exact [0035]. Superseded chunks keep their vectors+so retrieval rewinds [0019].++**A schema document is a search corpus, not a source of facts**: a mounted database's schema is+indexed and never extracted (`sources.config.extract`) [0036 d7].++## Why++- **A source is worth connecting if it has real timestamps, can contradict itself, has a stable+ identity and holds enterprise knowledge**; the valuable part of a ticket is its timeline, and most+ systems keep it and only need to be asked [0013].+- **Sentences with dates extract to facts with a start; key-value pairs make the model guess**+ [0013].+- **Fixtures come from real responses**; hand-written JSON proves only the imagined shape [0013].+- **No provider abstraction until a third source shares a shape** [0013].+- **A feed item can vanish before acquisition and a summary is not full content**, so observations+ are retained apart from documents [0023].+- **The chunk boundary is part of the extraction contract**: 13 of 35 chunks began mid-table without+ a header and every miss was a literal; a 1,000-token budget lost (36 of 52) because the extractor's+ output does not grow with its input [0039].+- **The pipeline eats text and cannot tell a paraphrase of a chart from a sentence**; the ledger acts+ on what it is told, so a description may not close a correct fact on its own [0040].+- **Per-modality settings** because a board recording is more sensitive than chat and may stay on a+ local model [0040].+- **A job, not a migration, for the index**: the dimension is known only when vectors arrive and+ `CREATE INDEX CONCURRENTLY` cannot run inside a migration; `ivfflat` loses recall quietly [0035].+- **Truncation and partial rounds are reported**, or "sync complete" misleads [0013].++## Proposed and not built++- Descriptions of charts and photos under the ceiling (cut 4) and video (cut 5); the origin and+ anchor in the interface, opening the page, image or recording [0040].+- Feishu and Confluence connectors with a block-tree renderer and sampled versions [0013].+- An external parser behind the block model for PDF layout; evidence naming a table cell or an image+ region; real Markdown tables from the DOCX, PDF, spreadsheet and CSV parsers; header rows promoted+ by content [0039].+- Backup and restore (#712), a 100k-document benchmark (#713) [0016 E].++## Open questions++- Whether low engine confidence on an OCR or transcript segment should cap facts [0040].+- The header the parser hands over, and the budget measured on its own [0039].+- A recording's date as a `doc_time_source`; what gets embedded when a chunk holds references+ [0040].+- Recall on two large tenants of different subject matter; an old dimension's index when a+ workspace changes model [0035].diff --git a/docs/design/time.md b/docs/design/time.mdnew file mode 100644index 00000000..d5f49f63--- /dev/null+++ b/docs/design/time.md@@ -0,0 +1,105 @@+# Time has two clocks and every read says which++Records: [0019] (the record axis), [0022] (unknown bounds, anchors, the temporal engine), [0024]+(precision), [0031] (events and eternal facts), [0045] (time mentions, proposed), with [0013] d6,+[0002] d8, [0021] d4, [0027] revised, [0043] d5, [0044] d5. The columns live in [ledger](ledger.md).++## What it does today++**Two clocks, never one control.** `valid_from` / `valid_to` say when a fact held in the world;+`recorded_at` / `invalidated_at` say when the base held it. Every graph read takes `at` (world) and+`as_of` (record) separately; `at` absent means every moment, `as_of` absent means now [0019, 0022].+`record_axis::held_at(T)` and `world_axis::facts_hold_at` / `derived_hold_at` are the only places+the two predicates are spelled; write paths keep the hard-coded current-row filter because a+correction is never made as of March [0019, 0022]. Entities rewind through `entity_merges`,+derivations through `derived_at`, retrieval through superseded chunks that keep their vectors;+full-text search is "now" only [0019].++**Precision.** The world ladder runs year, month, day, hour, minute, second, plus `unknown` on an+end; a stored value is truncated to its precision by CHECK on `facts`, `pending_facts` and+`derived_facts`; one list (`WORLD_PRECISIONS`) feeds the parser, the renderers, the export and the+evaluator; a clock time without a zone is a date; world time renders in UTC, record time in the+viewer's zone [0024, 0013]. GitHub and Jira timestamps are written to the second [0013].++**An unknown bound reaches as far as the evidence.** A missing start is not "since always": the lower+bound is `valid_from` or else `attested_from`, the earliest document date among the row's evidence.+An ending with no date is not "still holds": `valid_to_precision = 'unknown'` with `attested_to`,+the date of the document that said it was over. An open end still reads as holds until told+otherwise [0022]. Anchors are copied by every superseding writer and moved only earlier.++**The temporal engine** (`temporal.rs`) works only on `state` relations declared functional or+inverse-functional. A new value closes the open one before it; an undated or dated ending closes+the open row it ends through a superseding row; a start-less row is ordered by its earliest dated+evidence (`doc_time_source` content or source only) and closes its predecessor as ended-unknown+there; ends the engine drew are marked `end_derived` and recomputed in one pass whenever a timeline+changes, so the result depends on which rows exist and not on arrival order; a relation unique on+both sides sits on two timelines under one lock; a deadline stated relative to an event is stored as+written and flagged `relative` [0022 revised, #679]. A successor below `AUTO_CLOSE_MIN_CONFIDENCE`+(0.75) may not take over: it opens a `low_confidence` conflict instead; confirming a fact changes+the value and recomputes its timelines [0040, 0043 d5]. Two values that still hold at one moment are+a conflict; a succession is not, for the consistency check and for the merge gate alike [0017,+0027 revised].++**Events and eternal facts.** `Validity::under` normalises every write by the predicate's+`temporal`: an event is one moment written at both ends (a span collapses to its start, an unknown+ending is dropped); an eternal fact has no dates; a state keeps its interval. Reads give an event+the bucket its precision names, an undated event no moment, an eternal fact every moment; a fact+with no predicate reads as a state [0031]. Rows written before the rule are read correctly and not+rewritten. The `[event]` / `[eternal]` prompt marks of 0031 d4 and the zoned-time sentence of 0024+belonged to the typed contract and are gone with it; the write rule in the store is what remains.++**Derived rows** store the interval as read: the intersection of premise read intervals, an anchored+bound with no precision, precision otherwise from the premise that set it [0022, 0024, 0002].++**Open statements carry no dates.** An open row has no `valid_*`; its time words are+`time_mentions` (verbatim text, chunk, offset) and its `attested_at` comes from the document's date+only when that date came from the content or a dating source, else null [#731]. Interpretation is+0045's and not built, so the slider shows nothing for a freshly extracted base.++## Why++- **"What did we believe in March" was unanswerable** through any shipped query while the rows were+ on disk; a hard-coded `invalidated_at IS NULL` sat at 54 read sites [0019].+- **Folding the two clocks into one slider** answers a different question than the one asked, and+ both look plausible on screen [0019].+- **A defence spread across read sites fails where one is missed**, and neither SQL nor `cargo check`+ says a word, so each predicate lives in one place with a database test in both directions+ [0019, 0022].+- **Backward continuation has no corrector**; forward continuation does (a later document, a+ person). A raise approved in a 2024 note is not placed in 2023, and a "no longer" dated 2025+ bounds the fact there [0022].+- **The document's date, not `recorded_at`**, because a back-filled corpus would put every undated+ fact in the year of ingest; and the anchor is a column because the refinement path copies evidence+ [0022].+- **The engine orders by the same anchor the reads use**, or a landlord is closed before the lease+ existed; an end the engine drew must move with the value that follows [0022, #679].+- **The ladder stops at the second** because no source states less, and a value is truncated so+ storage, display and export cannot disagree [0024].+- **An event written at both ends fails safe** in a reader that does not consult the predicate; a+ NULL end reads as "still holds" [0031].+- **Confidence is part of the timeline**: confirming a doubtful successor decides whether it takes+ over [0043].++## Proposed and not built++- **A time mention is resolved against its document** [0045]: the model returns shape (point,+ interval, as-of, duration, ending without a date), a reference (absolute, or an anchor plus offset,+ or none) and granularity; code computes the interval; a document carries its own date, calendars+ and narrative anchors from chunk to chunk; an unanchored mention waits and is recomputed when an+ anchor arrives; `attested_at` is null for an undated document (revising 0022 d3, which anchored it+ at the moment of recording; #731 already writes null on open rows); timelines close on resolution+ grade (A written, B anchored, C unresolved) and the confidence gate leaves the engine; no time+ words in code. Cuts: interpretation columns; context and code resolution (closes #714); grades;+ re-resolution and a time-anchor review queue.+- Thresholds before cut 2: 95% on absolute and 85% on anchored mentions across corpora; the lease+ bench no lower than 17 of 18 on both runs [0045].+- The `as_of` control on the graph page; the anchor shown beside a blank start; an event drawn as a+ point; retrieval that takes `at` [0019, 0022, 0031].++## Open questions++- Record-axis reads of evidence with no attestation: every moment or none [0045].+- A document with two calendars; a period-valued attribute as one statement with an interval or a+ value with a period mention [0045]; zone-less clock times and a document's own time zone [0024].+- The end convention for states: "until 2024-07" ends at the start of July today [0031].+- The histogram bucket for a sub-day fact on the canvas [0024].