Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
feat(demo): add the document-to-ontology pipeline
FABIOTESSf15c98d42755
diff --git a/demo/README.md b/demo/README.mdnew file mode 100644index 00000000..dfda60b2--- /dev/null+++ b/demo/README.md@@ -0,0 +1,106 @@+# Document-to-ontology demo pipeline++This directory holds the pipeline that turns a corpus of plain documents into+an ontology, populates it, verifies it, and scans the result for+contradictions that split along document provenance. The code here is+domain-agnostic: nothing about any particular subject is hardcoded, the+vocabulary is whatever the documents themselves turn out to be about.++The corpus that ships with this repository is public DCAT-US material, and+its provenance is recorded alongside it so every derived claim can be traced+back to the document and section it came from.++## Pipeline stages++1. **Read.** Every document in the corpus folder is loaded (`corpus_pipeline.py`,+ `ontology_from_docs.py`).+2. **Tokenise.** Sensitive values are detected and replaced with stable,+ deterministic tokens before anything reaches a model (`tokenisation.py`).+ The same value produces the same token in every document, so a token+ doubles as a join key for entity resolution without any component ever+ handling the raw value.+3. **Chunk.** Text is split on semantic boundaries, not fixed width, with+ each chunk carrying enough context to be interpreted on its own and an id+ scheme that lets a fragment be rejoined to its section (`chunker.py`).+4. **Derive per-document fragments.** Each document proposes its own classes,+ properties and disjointness axioms, independently of every other document+ (`ontology_from_docs.py`, stage DERIVE).+5. **Merge.** The per-document fragments are concatenated into one candidate+ ontology.+6. **Reconcile.** Independent derivation leaves behind competing modelling+ patterns: one document may model a distinction as a subclass partition+ while another models the same notion as a separate attribute class.+ Where both exist for the same parent, the attribute class is removed+ because it is redundant with the partition and is the one form a+ reasoner cannot check for contradictions. Status and State are exempt,+ because a status is a property of a thing over time, not a kind of+ thing (`ontology_from_docs.py`, function `reconcile`).+7. **Populate.** Instances are extracted from the documents, constrained to+ the derived and reconciled ontology so nothing gets invented+ (`ontology_from_docs.py`, stage POPULATE; `extract.py`).+8. **Verify.** A closed-world check confirms every term used in the+ extracted data was declared in the ontology, and a standalone pass+ re-checks parsing, stats, lint, enforce, vocabulary and reasoning against+ whatever is currently on disk (`ontology_from_docs.py` stage VERIFY;+ `verify.py`).+9. **Scan for contradictions.** Once claims are extracted into the graph,+ contradictions are found in three cascading tiers: structural checks the+ reasoner and SPARQL catch outright, blocked candidate pairs that compare+ only claims about the same entity from different documents, and an+ optional model adjudication pass over the small surviving set+ (`contradiction_scan.py`). Findings that split along document provenance+ are the ones worth showing anyone: two documents disagreeing about the+ same entity is a genuine contradiction in the corpus, not a broken+ extraction.++Two supporting tools sit alongside the main pipeline:++- `corpus_text.py` extracts plain text directly from the source documents,+ with no help from the ontology, so a classic chunk-retrieval baseline can+ be compared fairly against the graph-based retrieval path.+- `kpi_context_graph.py` derives a KPI context graph from the `computedFrom`,+ `appliesTo`, `dependsOnKPI` and `governedBy` triples the ontology itself+ declares, so the graph cannot drift from the model it describes. It+ answers impact questions (what depends on a changed term) and evaluates+ indicators against current data.+- `cq/run-cross-doc.py` runs a fixed set of competency questions against the+ merged store, as a repeatable check that the ontology can answer the+ questions it was built to answer.++## Derived artifacts++Everything the pipeline produces is written under `demo/derived/` and+`demo/corpus_extracted/`, and is regenerated from the corpus on every run.+None of it is checked in:++- `demo/derived/_ontology.ttl`: the merged, reconciled ontology.+- `demo/derived/_corpus_text.json`: the plain-text baseline extraction.+- `demo/corpus_extracted/*.ttl`: one knowledge graph fragment per document.+- `demo/corpus_extracted/_merged.ttl` and `_live.ttl`: the merged store+ loaded into the running engine.+- `demo/_review.jsonl`: individuals the pipeline could not type with+ confidence, queued for a human decision rather than guessed at.++## Running it++```bash+python -m pip install -r demo/requirements.txt+make demo+```++`make demo` runs the pipeline end to end against the corpus in+`demo/corpus/dcat-us`: read, tokenise, derive, merge, reconcile, populate,+verify, and scan for contradictions. Each stage can also be run on its own,+for example:++```bash+python3 demo/ontology_from_docs.py --corpus demo/corpus/dcat-us+python3 demo/contradiction_scan.py+python3 demo/kpi_context_graph.py graph+```++Run the tests with:++```bash+python -m pytest demo/tests/+```diff --git a/demo/chunker.py b/demo/chunker.pynew file mode 100644index 00000000..73999684--- /dev/null+++ b/demo/chunker.py@@ -0,0 +1,222 @@+#!/usr/bin/env python3+"""+Semantic chunking with context and a rejoin scheme.++Three decisions, all of which matter, and none of which the usual splitters do:++1. SEMANTIC BOUNDARIES, NOT FIXED WIDTH.+ Chunks are whole paragraphs inside whole sections. Small paragraphs are+ merged upward toward a target size rather than emitted alone. The point is+ not tidiness: fixed-width splitting explodes the number of chunks, and the+ cost of a pipeline like this is dominated by how many times a model gets+ called. Fewer, more meaningful chunks is both cheaper and better.++2. EVERY CHUNK CARRIES ITS CONTEXT.+ A chunk lifted out of a fifty-page report is uninterpretable on its own.+ Each record stores a `context` sentence saying which document it came from,+ what that document is about, which section it sits in, and where in that+ section it falls. This is what lets a model reason about a fragment without+ re-reading the source.++3. IDS ENCODE STRUCTURE SO CHUNKS CAN BE REJOINED.+ Ids look like DOC-042.S3.P2, and every record carries parent, prev and next.+ Retrieval returns a fragment; answering often needs the whole section. With+ a join scheme the consumer can ask for the parent or walk the siblings+ instead of hallucinating the surrounding text or over-retrieving everything.++Emitted as JSON so the records can be stored, indexed and joined by anything.+"""++from __future__ import annotations++import json+import re+from dataclasses import dataclass, asdict+from typing import Iterator++# Target chunk size in characters. Paragraphs below MIN are merged forward;+# paragraphs above MAX are split at sentence boundaries.+TARGET = 1200+MIN = 350+MAX = 2400++HEADING = re.compile(+ r"^\s*(?:#{1,6}\s+(?P<md>.+?)|(?P<num>\d+(?:\.\d+)*)\.?\s+(?P<numtitle>[A-Z].{2,80}))\s*$"+)+SENTENCE = re.compile(r"(?<=[.!?])\s+(?=[A-Z(])")+++@dataclass+class Chunk:+ id: str+ doc_id: str+ section: str+ section_title: str+ text: str+ context: str+ parent: str | None+ prev: str | None+ next: str | None++ def to_json(self) -> dict:+ return asdict(self)+++def _split_long(text: str) -> list[str]:+ """Split an oversized paragraph at sentence boundaries, not mid-sentence."""+ if len(text) <= MAX:+ return [text]+ out, cur = [], ""+ for sentence in SENTENCE.split(text):+ if cur and len(cur) + len(sentence) > TARGET:+ out.append(cur.strip())+ cur = sentence+ else:+ cur = f"{cur} {sentence}".strip()+ if cur.strip():+ out.append(cur.strip())+ return out+++def _sections(text: str) -> Iterator[tuple[str, str, list[str]]]:+ """Yield (section_number, section_title, paragraphs)."""+ lines = text.splitlines()+ num, title, buf = "0", "Preamble", []+ for line in lines:+ m = HEADING.match(line)+ if m:+ if buf:+ yield num, title, _paragraphs("\n".join(buf))+ buf = []+ if m.group("md"):+ heading = m.group("md").strip()+ lead = re.match(r"^(\d+(?:\.\d+)*)\.?\s+(.*)$", heading)+ num, title = (lead.group(1), lead.group(2)) if lead else (num, heading)+ else:+ num, title = m.group("num"), m.group("numtitle").strip()+ else:+ buf.append(line)+ if buf:+ yield num, title, _paragraphs("\n".join(buf))+++def _paragraphs(block: str) -> list[str]:+ """Paragraphs, with short ones merged forward toward the target size."""+ raw = [p.strip() for p in re.split(r"\n\s*\n", block) if p.strip()]+ merged: list[str] = []+ for p in raw:+ if merged and len(merged[-1]) < MIN:+ merged[-1] = f"{merged[-1]}\n\n{p}"+ else:+ merged.append(p)+ out: list[str] = []+ for p in merged:+ out.extend(_split_long(p))+ return [p for p in out if len(p.strip()) > 40]+++# Front-matter noise that says nothing about what a document is ABOUT.+_BOILERPLATE = re.compile(+ r"^\s*(?:\||#|\*\*Document control|SYNTHETIC|\*\*SYNTHETIC|-{3,}|\d+\.\s*(?:Purpose|Scope)\b)",+ re.I,+)+++def _summarise(text: str, limit: int = 160) -> str:+ """A one-line gist of the document, used in every chunk's context.++ Skips document-control tables, headings and demo disclaimers: they are+ present in every document and say nothing about what this one covers, so+ including them would make every chunk's context identical and useless.+ """+ lines = [l.strip() for l in text.splitlines()]+ prose = [l for l in lines if l and not _BOILERPLATE.match(l) and len(l) > 40]+ body = re.sub(r"\s+", " ", " ".join(prose[:4])).strip()+ body = re.sub(r"^\d+(?:\.\d+)*\s+", "", body)+ gist = " ".join(SENTENCE.split(body)[:2]).strip()+ return (gist[: limit - 1] + "…") if len(gist) > limit else (gist or "an internal controlled document")+++def chunk_document(text: str, doc_id: str, title: str | None = None) -> list[Chunk]:+ """Chunk one document into context-carrying, rejoinable records."""+ doc_title = title or (re.search(r"^#\s+(.+)$", text, flags=re.M).group(1)+ if re.search(r"^#\s+(.+)$", text, flags=re.M) else doc_id)+ gist = _summarise(text)++ chunks: list[Chunk] = []+ for num, sec_title, paras in _sections(text):+ total = len(paras)+ for i, para in enumerate(paras, start=1):+ cid = f"{doc_id}.S{num}.P{i}"+ context = (+ f"From {doc_id} ({doc_title}), a document about {gist} "+ f"Section {num}, {sec_title}. Paragraph {i} of {total} in this section."+ )+ chunks.append(Chunk(+ id=cid, doc_id=doc_id, section=num, section_title=sec_title,+ text=para, context=context, parent=f"{doc_id}.S{num}",+ prev=None, next=None,+ ))++ # Link siblings after the fact, so prev/next span section boundaries in+ # reading order rather than stopping at each heading.+ for i, c in enumerate(chunks):+ c.prev = chunks[i - 1].id if i > 0 else None+ c.next = chunks[i + 1].id if i < len(chunks) - 1 else None+ return chunks+++def pack(chunks: list[Chunk], budget: int = 9000) -> list[list[Chunk]]:+ """Group chunks into as few model calls as possible.++ This is the decision that controls cost. One call per chunk is what makes+ these pipelines absurdly expensive on long documents; packing to a context+ budget, while keeping each chunk's context header attached, gets the same+ information across in a fraction of the calls.+ """+ groups: list[list[Chunk]] = []+ cur: list[Chunk] = []+ size = 0+ for c in chunks:+ cost = len(c.text) + len(c.context) + 40+ if cur and size + cost > budget:+ groups.append(cur)+ cur, size = [], 0+ cur.append(c)+ size += cost+ if cur:+ groups.append(cur)+ return groups+++def render(group: list[Chunk]) -> str:+ """Render a packed group for a prompt, each chunk labelled and contextualised."""+ parts = []+ for c in group:+ parts.append(f"[{c.id}]\n({c.context})\n{c.text}")+ return "\n\n".join(parts)+++def to_jsonl(chunks: list[Chunk]) -> str:+ return "\n".join(json.dumps(c.to_json(), ensure_ascii=False) for c in chunks)+++if __name__ == "__main__":+ import pathlib+ import sys++ if len(sys.argv) < 2:+ sys.exit("usage: chunker.py <file.md> [--json]")+ path = pathlib.Path(sys.argv[1])+ doc_id = re.match(r"([A-Z]+-[0-9][0-9A-Za-z\-]*?)(?=-[a-z])", path.stem)+ cs = chunk_document(path.read_text(), doc_id.group(1) if doc_id else path.stem)+ if "--json" in sys.argv:+ print(to_jsonl(cs))+ else:+ groups = pack(cs)+ print(f"{len(cs)} chunks in {len(groups)} model calls "+ f"(a per-chunk pipeline would make {len(cs)})")+ for c in cs[:3]:+ print(f"\n[{c.id}] parent={c.parent} prev={c.prev} next={c.next}")+ print(f" context: {c.context}")+ print(f" text: {c.text[:110]}...")diff --git a/demo/contradiction_scan.py b/demo/contradiction_scan.pynew file mode 100644index 00000000..2cce9907--- /dev/null+++ b/demo/contradiction_scan.py@@ -0,0 +1,318 @@+#!/usr/bin/env python3+"""+Contradiction scanner: find the passages in a document corpus that disagree.++WHY NOT BAYESIAN OPTIMISATION+-----------------------------+The obvious framing is "search the space of chunks for the one that trips+retrieval". That invites Bayesian optimisation, and BO is the wrong tool:+it is built for optimising an expensive black-box function over a smooth,+low-dimensional, continuous space. Chunk-pair contradiction is combinatorial,+discrete, and has no useful notion of "nearby". BO would sample slowly through+a space that does not reward sampling.++It is also unnecessary. Once claims are extracted into the graph, most+contradictions stop being a search problem:++ Tier 1 STRUCTURAL the reasoner and SPARQL find them outright.+ Cost: milliseconds. Exact, with zero false positives.+ Catches disjoint-class violations, functional-property+ conflicts, and domain rule breaches.++ Tier 2 BLOCKING for everything semantic that the axioms cannot express,+ do NOT compare all pairs. Claims carry `aboutEntity`,+ so only compare claims about the SAME entity that come+ from DIFFERENT documents. This is standard blocking from+ entity resolution: it collapses O(M^2) over the corpus+ to sum(k_i^2) over per-entity groups, where k_i is+ typically single digits.++ Tier 3 ADJUDICATE run a model over the surviving handful of candidate+ pairs only. Optional, --adjudicate.++The cascade is orders of magnitude cheaper than any search-based method, and+unlike BO the first tier is exact rather than probabilistic.++Usage:+ python3 demo/contradiction_scan.py+ python3 demo/contradiction_scan.py --adjudicate # add tier 3+"""++import argparse+import json+import os+import pathlib+import subprocess+import sys+import urllib.request++ROOT = pathlib.Path(__file__).resolve().parent.parent+BIN = ROOT / "target" / "release" / "open-ontologies"+BUNDLE = ROOT / "demo" / "bundle" / "dcat-us-full.ttl"+# Any extracted document graphs are merged in too, so the scan sees the whole+# corpus rather than only the curated demo data.+EXTRA = sorted((ROOT / "demo" / "corpus_extracted").glob("*.ttl"))+LLM_BASE = os.environ.get("ONTO_LLM_BASE_URL", "http://localhost:8081/v1").rstrip("/")+LLM_KEY = os.environ.get("ONTO_LLM_API_KEY", "not-needed")++PREFIXES = """PREFIX crd: <https://w3id.org/dcat-us-demo#>+PREFIX owl: <http://www.w3.org/2002/07/owl#>+PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>+PREFIX prov: <http://www.w3.org/ns/prov#>+"""++# --- Tier 1: structural contradictions -----------------------------------++TIER1 = [+ (+ "disjoint-class",+ "An individual is typed into two classes the ontology declares disjoint",+ PREFIXES+ + """SELECT DISTINCT ?subject ?classA ?classB ?disjointA ?disjointB WHERE {+ ?subject a ?classA, ?classB .+ FILTER(STR(?classA) < STR(?classB))+ ?classA rdfs:subClassOf* ?disjointA .+ ?classB rdfs:subClassOf* ?disjointB .+ { ?disjointA owl:disjointWith ?disjointB } UNION { ?disjointB owl:disjointWith ?disjointA }+}""",+ ),+ (+ "functional-property",+ "A functional property carries two different values on the same subject",+ PREFIXES+ + """SELECT DISTINCT ?subject ?property ?valueA ?valueB WHERE {+ ?property a owl:FunctionalProperty .+ ?subject ?property ?valueA, ?valueB .+ FILTER(STR(?valueA) < STR(?valueB))+}""",+ ),+ (+ "domain-rule",+ "A candidate declares a control target the register classifies as beneficial",+ PREFIXES+ + """SELECT DISTINCT ?subject ?organism ?viaClass WHERE {+ ?subject crd:targetsOrganism ?organism .+ ?organism a ?viaClass .+ ?viaClass rdfs:subClassOf* crd:BeneficialOrganism .+}""",+ ),+]++# --- Tier 2: blocked candidate generation --------------------------------++# Blocking must be tight or it stops being blocking. Two refinements beyond+# "same entity, different documents":+# - exclude CLASS-level entities. A claim "about :Candidate" is about a+# category, not a thing, and pairing every such claim with every other+# generates noise proportional to corpus size rather than to real overlap.+# - require the two claims to share vocabulary. Claims about the same+# individual that share no significant term are rarely contradictory.+TIER2 = PREFIXES + """SELECT ?entity ?docA ?secA ?claimA ?docB ?secB ?claimB WHERE {+ ?a a crd:Claim ; crd:aboutEntity ?entity ; crd:claimText ?claimA ; crd:statedIn ?sa .+ ?b a crd:Claim ; crd:aboutEntity ?entity ; crd:claimText ?claimB ; crd:statedIn ?sb .+ FILTER(STR(?a) < STR(?b))+ FILTER NOT EXISTS { ?entity a owl:Class }+ FILTER NOT EXISTS { ?someone a ?entity }+ ?sa crd:sectionNumber ?secA . ?da crd:hasSection ?sa ; crd:docId ?docA .+ ?sb crd:sectionNumber ?secB . ?db crd:hasSection ?sb ; crd:docId ?docB .+ FILTER(?docA != ?docB)+}"""++# Total claims, for an honest reduction ratio rather than a bare count.+CLAIM_COUNT = PREFIXES + "SELECT (COUNT(?c) AS ?n) WHERE { ?c a crd:Claim }"++# Which source asserted each conflicting fact.+PROVENANCE = PREFIXES + """SELECT ?thing ?sourceLabel WHERE {+ ?thing prov:wasDerivedFrom ?source .+ ?source rdfs:label ?sourceLabel .+}"""+++def short(v):+ if not isinstance(v, str):+ return v+ if v.startswith("<") and v.endswith(">"):+ return v[1:-1].split("#")[-1]+ if v.startswith('"'):+ body = v[1:]+ for cut in ('"^^', '"@', '"'):+ if cut in body:+ return body.split(cut)[0]+ return body+ return v+++def query(sparql):+ loads = "\n".join([f"load {BUNDLE}"] + [f"load {e}" for e in EXTRA])+ batch = f"clear\n{loads}\nquery {json.dumps(' '.join(sparql.split()))}\n"+ out = subprocess.run([str(BIN), "batch", "-"], input=batch, capture_output=True, text=True).stdout.strip()+ if not out:+ return []+ try:+ rows = json.loads(out.split("\n")[-1]).get("result", {}).get("results", [])+ except Exception:+ return []+ return [{k: short(v) for k, v in r.items()} for r in rows]+++STOP = set("the a an of to in for is are be and or that this with by on at as it its "+ "must not no any all where which when from than then shall should may".split())+++def signal(a, b):+ """Cheap high-precision discriminator applied after blocking.++ Blocking on entity alone is asymptotically good but weak on a small corpus:+ a handful of hot entities appear in most documents. Two extra signals cut+ it further at negligible cost:++ - shared significant vocabulary (the claims are actually about the same+ thing, not merely tagged to the same entity)+ - a numeric disagreement (two documents stating different values for+ what reads like the same quantity is the single most common form of+ real contradiction in controlled documents)++ Returns (score, reason). Pairs scoring 0 are dropped.+ """+ import re as _re+ ta = {w for w in _re.findall(r"[a-z]{4,}", a.lower()) if w not in STOP}+ tb = {w for w in _re.findall(r"[a-z]{4,}", b.lower()) if w not in STOP}+ shared = ta & tb+ na = set(_re.findall(r"\d+(?:\.\d+)?", a))+ nb = set(_re.findall(r"\d+(?:\.\d+)?", b))+ score, why = 0, []+ if len(shared) >= 3:+ score += 1+ why.append(f"{len(shared)} shared terms")+ if na and nb and na != nb:+ score += 2+ why.append(f"numeric disagreement {sorted(na)} vs {sorted(nb)}")+ neg = {"not", "no", "must not", "never"}+ if any(n in a.lower() for n in neg) != any(n in b.lower() for n in neg):+ score += 1+ why.append("one asserts, one negates")+ return score, "; ".join(why)+++def adjudicate(entity, a, b):+ """Tier 3. Only ever called on the small blocked candidate set."""+ prompt = (+ f"Two statements about the same entity ({entity}) from different controlled documents.\n\n"+ f"A: {a}\nB: {b}\n\n"+ "Do these CONTRADICT each other? A contradiction means both cannot be true at once. "+ "Differing detail or emphasis is NOT a contradiction.\n"+ 'Answer with JSON only: {"contradiction": true|false, "reason": "<one sentence>"}'+ )+ try:+ model = json.load(+ urllib.request.urlopen(+ urllib.request.Request(f"{LLM_BASE}/models", headers={"Authorization": f"Bearer {LLM_KEY}"}),+ timeout=30,+ )+ )["data"][0]["id"]+ req = urllib.request.Request(+ f"{LLM_BASE}/chat/completions",+ data=json.dumps(+ {+ "model": model,+ "messages": [{"role": "user", "content": prompt}],+ "temperature": 0.1,+ "max_tokens": 200,+ "chat_template_kwargs": {"enable_thinking": False},+ }+ ).encode(),+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"},+ )+ text = json.load(urllib.request.urlopen(req, timeout=180))["choices"][0]["message"]["content"]+ start, end = text.find("{"), text.rfind("}")+ return json.loads(text[start : end + 1]) if start >= 0 else {"contradiction": None, "reason": "unparseable"}+ except Exception as e:+ return {"contradiction": None, "reason": f"adjudication unavailable: {e}"}+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--adjudicate", action="store_true", help="run tier 3 over blocked candidates")+ args = ap.parse_args()++ if not BIN.exists():+ sys.exit(f"engine binary not found: {BIN}")++ findings = 0++ print("=" * 72)+ print("TIER 1 structural contradictions (reasoner + SPARQL, exact)")+ print("=" * 72)+ for name, description, sparql in TIER1:+ rows = query(sparql)+ print(f"\n[{name}] {description}")+ if not rows:+ print(" none")+ continue+ findings += len(rows)+ for r in rows:+ print(f" CONFLICT {json.dumps(r)}")++ prov = query(PROVENANCE)+ if prov:+ print("\n[provenance] which upstream source asserted each fact")+ by_thing = {}+ for r in prov:+ by_thing.setdefault(r["thing"], []).append(r["sourceLabel"])+ for thing, sources in by_thing.items():+ marker = " <-- asserted by TWO sources" if len(sources) > 1 else ""+ print(f" {thing}: {', '.join(sources)}{marker}")++ print("\n" + "=" * 72)+ print("TIER 2 blocked candidate pairs (same entity, different documents)")+ print("=" * 72)+ candidates = query(TIER2)+ try:+ total = int(query(CLAIM_COUNT)[0]["n"])+ except Exception:+ total = 0+ full = total * (total - 1) // 2 if total else 0+ print(f"\nclaims in corpus: {total} full pairwise space: {full}")+ print(f"candidate pairs after blocking: {len(candidates)}"+ + (f" ({100*len(candidates)/full:.1f}% of pairwise)" if full else ""))+ scored, seen = [], set()+ for c in candidates:+ # The SPARQL yields each pair once, but different section IRIs can carry+ # the same claim text, so dedupe on the unordered text pair.+ key = frozenset((c["claimA"], c["claimB"]))+ if key in seen:+ continue+ seen.add(key)+ sc, why = signal(c["claimA"], c["claimB"])+ if sc:+ scored.append((sc, why, c))+ scored.sort(key=lambda x: -x[0])+ print(f"high-signal pairs after discriminator: {len(scored)}"+ + (f" ({100*len(scored)/full:.1f}% of pairwise)" if full else ""))+ print(f"\n{DIM if False else ''}Blocking alone is asymptotic: its advantage grows with corpus size.")+ print("On 36 claims a few hot entities dominate, so the discriminator does the")+ print("real work here. On a corpus of thousands both matter.\n")+ candidates = [c for _, _, c in scored[:12]]+ for sc, why, c in scored[:12]:+ print(f" [signal {sc}] {c['entity']} ({why})")+ print(f" {c['docA']} S{c['secA']}: {c['claimA'][:130]}")+ print(f" {c['docB']} S{c['secB']}: {c['claimB'][:130]}")+ if not candidates:+ print(" none")+ for c in candidates:+ if args.adjudicate:+ verdict = adjudicate(c["entity"], c["claimA"], c["claimB"])+ flag = "CONTRADICTION" if verdict.get("contradiction") else "consistent"+ print(f" -> {flag}: {verdict.get('reason', '')}")++ print("\n" + "=" * 72)+ print(f"tier 1 structural findings: {findings}")+ print(f"tier 2 candidate pairs : {len(candidates)}")+ print("\nCost note: tier 1 is exact and runs in milliseconds. Tier 2 compares")+ print("only claims that share an entity across documents, so the model in")+ print("tier 3 is invoked on a handful of pairs rather than on every chunk pair.")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/demo/corpus_pipeline.py b/demo/corpus_pipeline.pynew file mode 100644index 00000000..831bd086--- /dev/null+++ b/demo/corpus_pipeline.py@@ -0,0 +1,310 @@+#!/usr/bin/env python3+"""+Read a folder of Word documents, build a knowledge graph, find the+contradictions between them.++This is the showcase end to end:++ 1. READ every .docx in demo/corpus/dcat-us (text via pandoc)+ 2. EXTRACT a knowledge graph per document, constrained to the ontology+ 3. VERIFY closed-world check that no term was invented+ 4. LOAD merge every document graph into one store+ 5. SCAN find the passages that contradict each other++Nothing leaves the machine: extraction runs against whatever model endpoint is+configured, which by default is a local one.++Usage:+ python3 demo/corpus_pipeline.py+ python3 demo/corpus_pipeline.py --limit 3 # quick run+ python3 demo/corpus_pipeline.py --cached # skip extraction, reuse output+"""++import argparse+import functools+import concurrent.futures as cf+import json+import os+import pathlib+import re+import subprocess+import sys+import time+import urllib.request++ROOT = pathlib.Path(__file__).resolve().parent.parent+CORPUS = ROOT / "demo" / "corpus" / "dcat-us"+OUT = ROOT / "demo" / "corpus_extracted"+BIN = ROOT / "target" / "release" / "open-ontologies"+VOCAB = ROOT / "demo" / "bundle" / "dcat-us-vocab.ttl"+ENGINE = os.environ.get("ONTO_ENGINE", "http://127.0.0.1:8137")+LLM_BASE = os.environ.get("ONTO_LLM_BASE_URL", "http://localhost:8081/v1").rstrip("/")+LLM_KEY = os.environ.get("ONTO_LLM_API_KEY", "not-needed")++if os.environ.get("NO_COLOR"):+ BOLD = DIM = RED = GRN = YEL = OFF = ""+else:+ BOLD, DIM, RED, GRN, YEL, OFF = "\033[1m", "\033[2m", "\033[31m", "\033[32m", "\033[33m", "\033[0m"++VOCAB_HINT = """Classes: :Document :StandardOperatingProcedure :RegulatoryGuidance :TrialReport+ :Section :Claim :Control :ReferenceSource :Candidate :Organism :Pest :Pathogen+ :BeneficialOrganism :Pollinator :NaturalEnemy :FieldTrial :TargetProtein :Mutation+ :EvidenceGap :StageGate+Object properties: :hasSection :parentSection :relatedSection :statedIn :aboutEntity+ :supersedes :declaresControl :controlOver :blocksProgressionOf :documentsRiskIn+Datatype properties: :docId :classification :aclGroup :sectionNumber :claimText+ :isAutomated :isBlocking rdfs:label rdfs:comment"""++PROMPT = """Extract a knowledge graph from this controlled document.++Return ONLY Turtle. No prose, no fences.++Prefixes:+@prefix : <https://w3id.org/dcat-us-demo#> .+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .++Use ONLY these terms. Invent nothing:+{vocab}++Extract:+1. One :Document node :DOC_{safe} with :docId, rdfs:label, :classification, :aclGroup+ and its most specific document type.+2. A :Section per numbered section, IRI :SEC_{safe}_<number with dots as underscores>,+ with :sectionNumber and rdfs:label, linked by :hasSection.+3. :Claim nodes for assertions about candidate progression, organism classification,+ thresholds, or safeguards. Each with :claimText, :statedIn, and :aboutEntity where+ the subject is identifiable.+4. Where a section states a safeguard, a :Control with :isAutomated and :isBlocking.++DOCUMENT ID: {docid}++{body}+"""++PREFIXES = ("@prefix : <https://w3id.org/dcat-us-demo#> .\n"+ "@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n"+ "@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n")+++def docx_text(path):+ """Word document to plain text."""+ r = subprocess.run(["pandoc", str(path), "-t", "plain", "--wrap=none"],+ capture_output=True, text=True)+ return r.stdout+++def normalise(text):+ text = re.sub(r"```(?:turtle|ttl)?", "", text).strip()+ lines = text.split("\n")+ start = None+ for i, line in enumerate(lines):+ if line.strip().startswith(("@prefix", "@base")):+ start = i+ break+ if start is None:+ for i, line in enumerate(lines):+ if re.match(r"^\s*:\S+\s+(a|rdf:type)\s", line):+ start = i+ break+ text = "\n".join(lines[start if start is not None else 0:]).strip()+ lines = text.split("\n")+ for i in range(len(lines) - 1, -1, -1):+ if lines[i].rstrip().endswith((".", ";", ",")):+ text = "\n".join(lines[: i + 1]).strip()+ break+ if "@prefix :" not in text:+ text = PREFIXES + "\n" + text+ return text+++def call_model(prompt):+ model = os.environ.get("ONTO_LLM_MODEL", "")+ if not model:+ req = urllib.request.Request(LLM_BASE + "/models", headers={"Authorization": f"Bearer {LLM_KEY}"})+ model = json.load(urllib.request.urlopen(req, timeout=30))["data"][0]["id"]+ req = urllib.request.Request(+ LLM_BASE + "/chat/completions",+ data=json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}],+ "temperature": 0.2, "max_tokens": 3000,+ "chat_template_kwargs": {"enable_thinking": False}}).encode(),+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"})+ return json.load(urllib.request.urlopen(req, timeout=900))["choices"][0]["message"]["content"]+++def extract_one(path):+ raw = docx_text(path)+ m = re.match(r"([A-Z]+-[0-9][0-9A-Za-z\-]*?)(?=-[a-z])", path.stem)+ docid = m.group(1) if m else path.stem+ safe = docid.replace("-", "_")+ body = raw[:14000]+ for attempt in range(3):+ try:+ content = call_model(PROMPT.format(vocab=VOCAB_HINT, docid=docid, safe=safe, body=body))+ except Exception as e:+ return docid, None, f"model error: {e}"+ if re.search(r"(\b\S{1,6}\b[ .]{1,3})\1{15,}", content):+ continue+ ttl = normalise(content)+ if len(ttl) > 200:+ out = OUT / f"{docid}.ttl"+ out.write_text(f"# Extracted from {path.name} on 2026-08-10\n\n{ttl}")+ return docid, out, None+ return docid, None, "degenerate output after 3 attempts"+++def mcp(method, params, sid=[None]):+ h = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream"}+ if sid[0]:+ h["Mcp-Session-Id"] = sid[0]+ req = urllib.request.Request(ENGINE + "/mcp", headers=h, data=json.dumps(+ {"jsonrpc": "2.0", "id": 1, "method": method, "params": params}).encode())+ r = urllib.request.urlopen(req, timeout=120)+ sid[0] = r.headers.get("Mcp-Session-Id") or sid[0]+ body = r.read().decode().strip()+ if body.startswith("{"):+ return json.loads(body)+ for line in body.split("\n"):+ if line.startswith("data:"):+ try:+ fr = json.loads(line[5:].strip())+ if "result" in fr or "error" in fr:+ return fr+ except Exception:+ pass+ return {}+++def vocab_check(ttl):+ """Closed-world term check. The vocabulary must be loaded in the SAME+ session, so reload it here rather than assuming it survived."""+ try:+ mcp("initialize", {"protocolVersion": "2025-03-26", "capabilities": {},+ "clientInfo": {"name": "pipeline", "version": "1"}})+ mcp("tools/call", {"name": "onto_load", "arguments":+ {"path": str(VOCAB), "name": "dcat-us-vocab", "force_recompile": True}})+ res = mcp("tools/call", {"name": "onto_vocab_check",+ "arguments": {"data": ttl, "inline": True}})+ return json.loads(res.get("result", {}).get("content", [{}])[0].get("text", "{}"))+ except Exception as e:+ return {"conforms": None, "error": str(e)}+++def engine(cmds):+ return subprocess.run([str(BIN), "batch", "-"], input="\n".join(cmds) + "\n",+ capture_output=True, text=True).stdout.strip()+++print = functools.partial(__builtins__.print if not isinstance(__builtins__, dict) else __builtins__["print"], flush=True)+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--limit", type=int, default=0)+ ap.add_argument("--cached", action="store_true")+ ap.add_argument("--workers", type=int, default=4)+ ap.add_argument("--corpus", default=str(CORPUS),+ help="folder of documents to read")+ args = ap.parse_args()++ OUT.mkdir(exist_ok=True)+ corpus_dir = pathlib.Path(args.corpus)+ docs = sorted(corpus_dir.glob("*.docx")) or sorted(corpus_dir.glob("*.md"))+ if args.limit:+ docs = docs[: args.limit]+ if not docs:+ sys.exit(f"no .docx or .md found in {corpus_dir}")++ t0 = time.time()+ print(f"\n{BOLD}STAGE 1 READ{OFF} {len(docs)} documents from {corpus_dir.name}/")+ for d in docs:+ print(f" {DIM}{d.name}{OFF}")++ extracted = []+ if args.cached:+ extracted = sorted(OUT.glob("*.ttl"))+ print(f"\n{BOLD}STAGE 2 EXTRACT{OFF} {DIM}cached, {len(extracted)} graphs reused{OFF}")+ else:+ print(f"\n{BOLD}STAGE 2 EXTRACT{OFF} knowledge graph per document")+ print(f" {DIM}model endpoint {LLM_BASE} ({args.workers} in parallel){OFF}")+ with cf.ThreadPoolExecutor(max_workers=args.workers) as pool:+ for docid, out, err in pool.map(extract_one, docs):+ if err:+ print(f" {RED}FAIL{OFF} {docid:14s} {err}")+ else:+ print(f" {GRN}ok{OFF} {docid:14s} {out.name}")+ extracted.append(out)++ print(f"\n{BOLD}STAGE 3 VERIFY{OFF} closed-world check that no term was invented")+ ok = bad = 0+ for f in extracted:+ d = vocab_check(f.read_text())+ conforms, halluc = d.get("conforms"), d.get("hallucinated_terms", [])+ if conforms:+ ok += 1+ print(f" {GRN}conforms{OFF} {f.stem}")+ else:+ bad += 1+ print(f" {YEL}review {OFF} {f.stem} {halluc if halluc else ''}")++ print(f"\n{BOLD}STAGE 4 LOAD{OFF} merge every document graph into the live store")+ # Load through the RUNNING engine over MCP, not a throwaway CLI process.+ # Two things depend on this: the graph view renders the store that was+ # just built, and every load is recorded in the lineage trail.+ try:+ mcp("initialize", {"protocolVersion": "2025-03-26", "capabilities": {},+ "clientInfo": {"name": "pipeline", "version": "1"}})+ mcp("tools/call", {"name": "onto_clear", "arguments": {}})+ base = ROOT / "demo" / "bundle" / "dcat-us-full.ttl"+ mcp("tools/call", {"name": "onto_load", "arguments":+ {"path": str(base), "name": "corpus", "force_recompile": True}})+ merged = base.read_text().rstrip() + "\n"+ for f in extracted:+ body = "\n".join(l for l in f.read_text().splitlines()+ if not l.lstrip().startswith(("@prefix", "@base", "#"))).strip()+ if not body:+ continue+ # A block that does not terminate its last statement will glue onto+ # the next file and produce a parse error hundreds of lines away.+ if not body.endswith("."):+ body += " ."+ merged += "\n" + body + "\n"+ staged = ROOT / "demo" / "corpus_extracted" / "_merged.ttl"+ staged.write_text(merged)+ mcp("tools/call", {"name": "onto_load", "arguments":+ {"path": str(staged), "name": "corpus", "force_recompile": True}})+ # onto_load does not write lineage; a save does. Persist the merged+ # store so the ingestion appears in the lineage trail.+ mcp("tools/call", {"name": "onto_save", "arguments":+ {"path": str(ROOT / "demo" / "corpus_extracted" / "_live.ttl")}})+ res = mcp("tools/call", {"name": "onto_stats", "arguments": {}})+ st = json.loads(res.get("result", {}).get("content", [{}])[0].get("text", "{}"))+ print(f" {st.get('triples','?')} triples, {st.get('classes','?')} classes, "+ f"{st.get('individuals','?')} individuals {DIM}(loaded into the live engine){OFF}")+ except Exception as e:+ print(f" {YEL}live engine unavailable ({e}); falling back to a local load{OFF}")+ res = engine(["clear", f"load {ROOT/'demo'/'bundle'/'dcat-us-full.ttl'}"]+ + [f"load {f}" for f in extracted] + ["stats"])+ try:+ st = json.loads(res.split("\n")[-1])["result"]+ print(f" {st['triples']} triples, {st['classes']} classes, {st['individuals']} individuals")+ except Exception:+ print(f" {DIM}{res[-200:]}{OFF}")++ print(f"\n{BOLD}STAGE 5 SCAN{OFF} find passages that contradict each other")+ scan = subprocess.run([sys.executable, str(ROOT / "demo" / "contradiction_scan.py")],+ capture_output=True, text=True, cwd=ROOT).stdout+ for line in scan.splitlines():+ if "CONFLICT" in line:+ print(f" {RED}{line.strip()}{OFF}")+ elif "TWO sources" in line or "candidate pairs after blocking" in line:+ print(f" {YEL}{line.strip()}{OFF}")++ print(f"\n{BOLD}{len(docs)} documents processed in {time.time()-t0:.0f}s."+ f" {ok} graphs conformed, {bad} flagged for review.{OFF}")+ print(f"{DIM}Every fact carries the document and section it came from, and inherits"+ f"\nthat document's access group.{OFF}\n")+++if __name__ == "__main__":+ main()diff --git a/demo/corpus_text.py b/demo/corpus_text.pynew file mode 100644index 00000000..bd5e6520--- /dev/null+++ b/demo/corpus_text.py@@ -0,0 +1,62 @@+"""Extract plain document text for the baseline retriever.++The comparison in the Studio puts classic chunk retrieval beside graph+retrieval. For that comparison to mean anything, the baseline must read the+SAME documents from the SAME source, with no help from the ontology. Feeding it+anything derived would rig the result and any competent reviewer would say so.++So this reads the .docx files directly and writes plain text. The baseline+chunks that text and retrieves over it; the graph path never touches this file.++Output: demo/derived/_corpus_text.json+ { "DOC-202": {"title": ..., "text": ...}, ... }+"""++from __future__ import annotations++import json+import pathlib+import re+import zipfile++CORPUS = pathlib.Path("demo/corpus/dcat-us")+OUT = pathlib.Path("demo/derived/_corpus_text.json")+++def docx_text(path: pathlib.Path) -> str:+ """Paragraph-preserving text extraction.++ Paragraph boundaries matter: chunking on a single flattened line would+ produce chunks that straddle unrelated sections, which would make the+ baseline look worse than a real one does.+ """+ xml = zipfile.ZipFile(path).read("word/document.xml").decode("utf8", "ignore")+ xml = re.sub(r"</w:p>", "\n", xml)+ text = re.sub(r"<[^>]+>", "", xml)+ text = text.replace("&", "&").replace("<", "<").replace(">", ">")+ lines = [re.sub(r"[ \t]+", " ", ln).strip() for ln in text.split("\n")]+ return "\n".join(ln for ln in lines if ln)+++def main() -> None:+ docs: dict[str, dict[str, str]] = {}+ for path in sorted(CORPUS.glob("*.docx")):+ text = docx_text(path)+ m = re.search(r"Doc Id ([A-Z]+-\d+)", text.replace("\n", " "))+ doc_id = m.group(1) if m else path.stem+ title = ""+ t = re.search(r"Title (.+?)(?: Owner| Classification|$)", text.replace("\n", " "))+ if t:+ title = t.group(1).strip()+ docs[doc_id] = {"title": title, "text": text}++ OUT.parent.mkdir(parents=True, exist_ok=True)+ OUT.write_text(json.dumps(docs, indent=1))+ chars = sum(len(d["text"]) for d in docs.values())+ print(f"{len(docs)} documents, {chars:,} characters -> {OUT}")+ for k, v in docs.items():+ print(f" {k:10} {len(v['text']):6,} chars {v['title'][:50]}")+++if __name__ == "__main__":+ main()diff --git a/demo/cq/run-cross-doc.py b/demo/cq/run-cross-doc.pynew file mode 100644index 00000000..e77ad0af--- /dev/null+++ b/demo/cq/run-cross-doc.py@@ -0,0 +1,129 @@+#!/usr/bin/env python3+"""Run the cross-document competency questions against the full demo bundle.++These are the questions a vector retriever structurally cannot answer,+because no single passage contains the chain. Each one spans two to four+separate documents plus the structured graph.+"""++import json+import pathlib+import subprocess+import sys++ROOT = pathlib.Path(__file__).resolve().parent.parent.parent+BIN = ROOT / "target" / "release" / "open-ontologies"+BUNDLE = ROOT / "demo" / "bundle" / "dcat-us-full.ttl"++P = """PREFIX crd: <https://w3id.org/dcat-us-demo#>+PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>+"""++QUESTIONS = [+ (+ "XD-1",+ "Which controls should block SYN-4471, are they automated, and who declares them?",+ P+ + """SELECT ?docId ?control ?blocking ?automated WHERE {+ ?ctrl crd:blocksProgressionOf crd:SYN_4471 ;+ rdfs:label ?control ; crd:isBlocking ?blocking ; crd:isAutomated ?automated .+ ?doc crd:declaresControl ?ctrl ; crd:docId ?docId . }""",+ ),+ (+ "XD-2",+ "Root cause: every claim about the mis-mapped bucket or the hoverfly, with its source",+ P+ + """SELECT ?docId ?section ?claim WHERE {+ ?c a crd:Claim ; crd:claimText ?claim ; crd:statedIn ?sec ; crd:aboutEntity ?e .+ VALUES ?e { crd:AphidAssociatedArthropod crd:EpisyrphusBalteatus crd:SRC_PestScoutingFeed }+ ?sec crd:sectionNumber ?section .+ ?doc crd:hasSection ?sec ; crd:docId ?docId . }""",+ ),+ (+ "XD-3",+ "Which safeguards exist only on paper (blocking or not, but not automated)?",+ P+ + """SELECT ?docId ?control ?blocking WHERE {+ ?ctrl a crd:Control ; rdfs:label ?control ;+ crd:isAutomated false ; crd:isBlocking ?blocking .+ ?doc crd:declaresControl ?ctrl ; crd:docId ?docId . }""",+ ),+ (+ "XD-4",+ "Which reference source is superseded, and which section documents its defect?",+ P+ + """SELECT ?supersedingSource ?supersededSource ?documentingDoc ?section ?sectionLabel WHERE {+ ?winner crd:supersedes ?loser .+ BIND(REPLACE(STR(?winner), "^.*#", "") AS ?supersedingSource)+ BIND(REPLACE(STR(?loser), "^.*#", "") AS ?supersededSource)+ ?sec crd:documentsRiskIn ?loser ; crd:sectionNumber ?section ; rdfs:label ?sectionLabel .+ ?doc crd:hasSection ?sec ; crd:docId ?documentingDoc . }""",+ ),+ (+ "XD-5",+ "Cross-document section links: which section of one document bears on another?",+ P+ + """SELECT ?fromDoc ?fromSec ?toDoc ?toSec WHERE {+ ?a crd:relatedSection ?b .+ ?da crd:hasSection ?a ; crd:docId ?fromDoc . ?a crd:sectionNumber ?fromSec .+ ?db crd:hasSection ?b ; crd:docId ?toDoc . ?b crd:sectionNumber ?toSec .+ FILTER(?fromDoc != ?toDoc) }""",+ ),+ (+ "XD-6",+ "Access control: which ACL groups would a reader need to see this whole chain?",+ P+ + """SELECT DISTINCT ?docId ?classification ?acl WHERE {+ ?doc crd:docId ?docId ; crd:classification ?classification ; crd:aclGroup ?acl . }""",+ ),+]+++def clean(v):+ if not isinstance(v, str):+ return v+ if v.startswith("<") and v.endswith(">"):+ return v[1:-1].split("#")[-1]+ if v.startswith('"'):+ body = v[1:]+ for cut in ('"^^', '"@', '"'):+ if cut in body:+ body = body.split(cut)[0]+ break+ return body+ return v+++def main():+ if not BIN.exists():+ sys.exit(f"engine binary not found: {BIN}. Run cargo build --release.")+ failures = 0+ for qid, title, sparql in QUESTIONS:+ batch = f"clear\nload {BUNDLE}\nquery {json.dumps(' '.join(sparql.split()))}\n"+ out = subprocess.run(+ [str(BIN), "batch", "-"], input=batch, capture_output=True, text=True+ ).stdout.strip()+ last = out.split("\n")[-1] if out else "{}"+ try:+ res = json.loads(last).get("result", {})+ rows = res.get("results", [])+ except Exception:+ print(f"\n{qid} {title}\n PARSE FAILURE: {last[:200]}")+ failures += 1+ continue++ print(f"\n{qid} {title}")+ print(f" rows: {len(rows)}")+ if not rows:+ failures += 1+ print(" NO RESULTS")+ for row in rows:+ print(" ", {k: clean(v) for k, v in row.items()})++ print(f"\n{'-' * 60}")+ print(f"{len(QUESTIONS) - failures}/{len(QUESTIONS)} cross-document questions answered")+ return 1 if failures else 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/demo/extract.py b/demo/extract.pynew file mode 100644index 00000000..f60135e1--- /dev/null+++ b/demo/extract.py@@ -0,0 +1,326 @@+#!/usr/bin/env python3+"""+Extract a knowledge graph from the demonstration documents.++Deliberately mirrors the MCP-native convention: this script does NOT reason.+It sends each document to whatever OpenAI-compatible model endpoint is+configured, asks for Turtle constrained to the loaded vocabulary, and then+hands the result to the engine's `onto_vocab_check` for a closed-world+check of every term used.++Terms the model invents are caught there, before anything reaches the store.+That gate is the point: an LLM writes the graph, a deterministic checker+decides whether it is allowed to exist.++Usage:+ python3 demo/extract.py # extract + vocab check+ python3 demo/extract.py --dry-run # show prompts only+"""++import argparse+import json+import os+import pathlib+import re+import sys+import urllib.request++ROOT = pathlib.Path(__file__).resolve().parent.parent+DOCS = ROOT / "demo" / "documents"+OUT = ROOT / "demo" / "extracted"+ENGINE = os.environ.get("ONTO_ENGINE", "http://127.0.0.1:8137")+LLM_BASE = os.environ.get("ONTO_LLM_BASE_URL", "http://localhost:8081/v1").rstrip("/")+LLM_KEY = os.environ.get("ONTO_LLM_API_KEY", "not-needed")++VOCAB_HINT = """+Classes you may use:+ :Document :StandardOperatingProcedure :RegulatoryGuidance :TrialReport+ :Section :Claim :Control :ReferenceSource+ :Candidate :Organism :Pest :Pathogen :BeneficialOrganism :Pollinator :NaturalEnemy+ :FieldTrial :TargetProtein :Mutation :EvidenceGap :StageGate++Object properties you may use:+ :hasSection :parentSection :relatedSection :statedIn :aboutEntity+ :supersedes :declaresControl :controlOver :blocksProgressionOf :documentsRiskIn++Datatype properties you may use:+ :docId :classification :aclGroup :sectionNumber :claimText :isAutomated :isBlocking+ rdfs:label rdfs:comment+"""++PROMPT = """You are extracting a knowledge graph from a controlled document.++Return ONLY Turtle. No prose, no markdown fences, no explanation.++Use EXACTLY these prefixes:+@prefix : <https://w3id.org/dcat-us-demo#> .+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .++STRICT RULE: use ONLY the classes and properties listed below. Do not invent+any term. If something in the document does not fit the vocabulary, omit it.+{vocab}++Extract:+1. One :Document node, IRI :DOC_{docid_safe}, with :docId, rdfs:label,+ :classification, :aclGroup, and the most specific document type.+2. A :Section node per numbered section, IRI :SEC_{docid_safe}_<number with dots as underscores>,+ with :sectionNumber and rdfs:label, linked by :hasSection from the document.+3. :Claim nodes for assertions that matter to candidate progression or organism+ classification. Each with :claimText and :statedIn pointing at its section.+4. Where a section states a safeguard, a :Control node with :isAutomated and+ :isBlocking, linked with :declaresControl.+5. Where a section records a defect or limitation of a data source, link the+ section with :documentsRiskIn.++DOCUMENT ID: {docid}++DOCUMENT:+{body}+"""+++def rpc(method, params, sid=None, _i=[0]):+ _i[0] += 1+ headers = {+ "Content-Type": "application/json",+ "Accept": "application/json, text/event-stream",+ }+ if sid:+ headers["Mcp-Session-Id"] = sid+ req = urllib.request.Request(+ ENGINE + "/mcp",+ data=json.dumps(+ {"jsonrpc": "2.0", "id": _i[0], "method": method, "params": params}+ ).encode(),+ headers=headers,+ )+ r = urllib.request.urlopen(req, timeout=180)+ sid2 = r.headers.get("Mcp-Session-Id") or sid+ body = r.read().decode().strip()+ if body.startswith("{"):+ return json.loads(body), sid2+ for line in body.split("\n"):+ if line.startswith("data:"):+ try:+ fr = json.loads(line[5:].strip())+ if "result" in fr or "error" in fr:+ return fr, sid2+ except Exception:+ pass+ return None, sid2+++def tool_text(res):+ if not res:+ return ""+ return res.get("result", {}).get("content", [{}])[0].get("text", "")+++def new_session():+ _, sid = rpc(+ "initialize",+ {+ "protocolVersion": "2025-03-26",+ "capabilities": {},+ "clientInfo": {"name": "extractor", "version": "1"},+ },+ )+ return sid+++# Set once the vocabulary has been loaded, so a session reset can restore it.+_VOCAB_PATH = None+++def call_tool(name, arguments, sid):+ """Call an MCP tool, re-initialising if the session has expired.++ Extraction turns can take minutes on a local model, which is long enough+ for the server to drop the session. A stale session surfaces as a bare+ HTTP 404, so treat that as "reconnect and retry once" rather than fatal.++ A new session starts with an EMPTY store, so the vocabulary must be+ reloaded before retrying. Without this, onto_vocab_check silently reports+ "0 declared terms" and checks nothing -- it returns conforms=false with a+ warning rather than an error, which is easy to mistake for a failed+ extraction when it is actually a failed setup.+ """+ try:+ res, sid = rpc("tools/call", {"name": name, "arguments": arguments}, sid)+ return res, sid+ except urllib.error.HTTPError as e:+ if e.code != 404:+ raise+ sid = new_session()+ if _VOCAB_PATH:+ rpc("tools/call", {"name": "onto_load", "arguments": {+ "path": _VOCAB_PATH, "name": "dcat-us-vocab", "force_recompile": True}}, sid)+ res, sid = rpc("tools/call", {"name": name, "arguments": arguments}, sid)+ return res, sid+++def call_model(prompt):+ # Discover the served model id: MLX servers reject a mismatched id outright.+ model = os.environ.get("ONTO_LLM_MODEL", "")+ if not model:+ req = urllib.request.Request(+ LLM_BASE + "/models", headers={"Authorization": f"Bearer {LLM_KEY}"}+ )+ model = json.load(urllib.request.urlopen(req, timeout=30))["data"][0]["id"]++ req = urllib.request.Request(+ LLM_BASE + "/chat/completions",+ data=json.dumps(+ {+ "model": model,+ "messages": [{"role": "user", "content": prompt}],+ "temperature": 0.2,+ "top_p": 0.8,+ "max_tokens": 4096,+ "chat_template_kwargs": {"enable_thinking": False},+ }+ ).encode(),+ headers={+ "Content-Type": "application/json",+ "Authorization": f"Bearer {LLM_KEY}",+ },+ )+ resp = json.load(urllib.request.urlopen(req, timeout=600))+ return resp["choices"][0]["message"]["content"], model+++PREFIXES = (+ "@prefix : <https://w3id.org/dcat-us-demo#> .\n"+ "@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n"+ "@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n"+)+++def normalise(text):+ """Recover parseable Turtle from a chatty model response.++ Small local models reliably do three things the prompt told them not to:+ wrap output in code fences, open with a prose sentence, and omit the+ prefix block. None of that changes the extracted CONTENT, so it is+ normalised here rather than treated as a failure. Anything that survives+ still has to pass onto_vocab_check.+ """+ text = re.sub(r"```(?:turtle|ttl)?", "", text).strip()++ # Drop any prose preamble. Start at the first @prefix if there is one;+ # otherwise at the first line that is unambiguously a SUBJECT, i.e.+ # "<iri> a <type>". Matching a bare leading ":token" is wrong: predicate+ # continuation lines like ':docId "DOC-114" ;' match it too, and starting+ # there silently decapitates the statement and loses its subject.+ lines = text.split("\n")+ start = None+ for i, line in enumerate(lines):+ if line.strip().startswith(("@prefix", "@base")):+ start = i+ break+ if start is None:+ for i, line in enumerate(lines):+ if re.match(r"^\s*:\S+\s+(a|rdf:type)\s", line):+ start = i+ break+ text = "\n".join(lines[start if start is not None else 0 :]).strip()++ # Drop any trailing prose after the last statement-terminating line.+ lines = text.split("\n")+ end = len(lines)+ for i in range(len(lines) - 1, -1, -1):+ if lines[i].rstrip().endswith((".", ";", ",")):+ end = i + 1+ break+ text = "\n".join(lines[:end]).strip()++ if "@prefix :" not in text:+ text = PREFIXES + "\n" + text+ return text+++strip_fences = normalise # keep the call site stable+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--dry-run", action="store_true")+ args = ap.parse_args()++ OUT.mkdir(parents=True, exist_ok=True)+ docs = sorted(DOCS.glob("*.md"))+ if not docs:+ sys.exit("no documents found")++ # Load the vocabulary the extraction will be checked against.+ sid = None+ if not args.dry_run:+ _, sid = rpc(+ "initialize",+ {+ "protocolVersion": "2025-03-26",+ "capabilities": {},+ "clientInfo": {"name": "extractor", "version": "1"},+ },+ )+ global _VOCAB_PATH+ bundle = ROOT / "demo" / "bundle" / "dcat-us-vocab.ttl"+ _VOCAB_PATH = str(bundle)+ res, sid = call_tool(+ "onto_load",+ {"path": str(bundle), "name": "dcat-us-vocab", "force_recompile": True},+ sid,+ )+ print(f"vocabulary loaded: {tool_text(res)[:100]}")++ for doc in docs:+ raw = doc.read_text()+ m = re.match(r"^---\n(.*?)\n---\n(.*)$", raw, re.S)+ front, body = (m.group(1), m.group(2)) if m else ("", raw)+ docid = re.search(r'doc_id:\s*"?([^"\n]+)"?', front)+ docid = docid.group(1).strip() if docid else doc.stem+ safe = docid.replace("-", "_")++ prompt = PROMPT.format(+ vocab=VOCAB_HINT, docid=docid, docid_safe=safe, body=front + "\n" + body+ )+ if args.dry_run:+ print(f"--- {docid} prompt {len(prompt)} chars ---")+ continue++ print(f"\nextracting {docid} ...", flush=True)++ # Small models occasionally collapse into repetition. Detect it and+ # retry rather than writing degenerate output and calling it a graph.+ for attempt in range(1, 4):+ content, model = call_model(prompt)+ (OUT / f"{docid}.raw.attempt{attempt}.txt").write_text(content)+ if re.search(r"(\b\S{1,6}\b[ .]{1,3})\1{15,}", content):+ print(f" attempt {attempt}: degenerate repetition, retrying")+ continue+ ttl = strip_fences(content)+ if ":" in ttl and len(ttl) > 200:+ break+ print(f" attempt {attempt}: output too thin, retrying")+ else:+ print(f" GAVE UP on {docid} after 3 attempts")+ ttl = strip_fences(content)+ path = OUT / f"{docid}.ttl"+ path.write_text(+ f"# Extracted from {doc.name} on 2026-08-10\n"+ f"# Model: {model}\n"+ f"# Verified with onto_vocab_check. See demo/extracted/REPORT.md\n\n" + ttl+ )+ print(f" wrote {path.name} ({len(ttl)} chars)")++ res, sid = call_tool("onto_validate", {"input": ttl, "inline": True}, sid)+ v = tool_text(res)+ print(f" syntax : {v[:120]}")++ res, sid = call_tool("onto_vocab_check", {"data": ttl, "inline": True}, sid)+ print(f" vocab : {tool_text(res)[:400]}")+++if __name__ == "__main__":+ main()diff --git a/demo/kpi_context_graph.py b/demo/kpi_context_graph.pynew file mode 100644index 00000000..f71a768d--- /dev/null+++ b/demo/kpi_context_graph.py@@ -0,0 +1,259 @@+#!/usr/bin/env python3+"""+Derive the KPI context graph, and answer impact questions with a query.++The other pitch showed a context graph connecting ontologies, agents and 99+KPIs. It was drawn. This one is DERIVED: every edge comes from a+`computedFrom` / `appliesTo` / `dependsOnKPI` / `governedBy` triple that the+KPI itself declares. A derived graph cannot drift from the model, because it+is a rendering of the model.++Three things it does that a drawn diagram cannot:++ 1. `graph` render the context graph as Mermaid, from traversal.+ 2. `impact` given a changed IRI, list the KPIs that depend on it,+ transitively through composite KPIs. This is the question+ nobody can answer when KPIs live in a config table.+ 3. `breach` evaluate the indicators against the current data and report+ which are breached, with their governing document.++Usage:+ python3 demo/kpi_context_graph.py graph+ python3 demo/kpi_context_graph.py impact :targetsOrganism+ python3 demo/kpi_context_graph.py breach+"""++import json+import pathlib+import subprocess+import sys++ROOT = pathlib.Path(__file__).resolve().parent.parent+BIN = ROOT / "target" / "release" / "open-ontologies"+LOADS = [+ ROOT / "demo" / "bundle" / "dcat-us-full.ttl",+ ROOT / "demo" / "ontology" / "dcat-us-kpi.ttl",+]++P = """PREFIX crd: <https://w3id.org/dcat-us-demo#>+PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>+PREFIX owl: <http://www.w3.org/2002/07/owl#>+"""+++def short(v):+ if not isinstance(v, str):+ return v+ if v.startswith("<") and v.endswith(">"):+ return v[1:-1].split("#")[-1]+ if v.startswith('"'):+ body = v[1:]+ for cut in ('"^^', '"@', '"'):+ if cut in body:+ return body.split(cut)[0]+ return body+ return v+++def query(sparql):+ lines = ["clear"] + [f"load {p}" for p in LOADS]+ lines.append(f"query {json.dumps(' '.join(sparql.split()))}")+ out = subprocess.run(+ [str(BIN), "batch", "-"], input="\n".join(lines) + "\n", capture_output=True, text=True+ ).stdout.strip()+ if not out:+ return []+ try:+ rows = json.loads(out.split("\n")[-1]).get("result", {}).get("results", [])+ except Exception:+ return []+ return [{k: short(v) for k, v in r.items()} for r in rows]+++def cmd_graph():+ """Render the context graph from traversal.++ Grouped into layers and styled so it reads at a glance: what the model is+ computed FROM on the left, the indicators in the middle, what they measure+ and what mandates them on the right. Blocking indicators are highlighted,+ because "which of these stops a stage gate" is the first thing anyone asks.+ """+ inputs = query(P + "SELECT ?kpi ?input WHERE { ?kpi a/rdfs:subClassOf* crd:KPI ; crd:computedFrom ?input }")+ applies = query(P + "SELECT ?kpi ?cls WHERE { ?kpi a/rdfs:subClassOf* crd:KPI ; crd:appliesTo ?cls }")+ comps = query(P + "SELECT ?kpi ?component WHERE { ?kpi crd:dependsOnKPI ?component }")+ govs = query(P + "SELECT ?kpi ?doc WHERE { ?kpi crd:governedBy ?doc }")+ labels = {r["kpi"]: r["label"] for r in+ query(P + "SELECT ?kpi ?label WHERE { ?kpi a/rdfs:subClassOf* crd:KPI ; rdfs:label ?label }")}+ # Scope to KPIs: Controls also carry isBlocking, and would otherwise leak+ # into the indicator styling.+ blocking = {r["kpi"] for r in+ query(P + "SELECT ?kpi WHERE { ?kpi a/rdfs:subClassOf* crd:KPI ; crd:isBlocking true }")}++ kpis = sorted(labels)+ ins = sorted({r["input"] for r in inputs})+ classes = sorted({r["cls"] for r in applies})+ docs = sorted({r["doc"] for r in govs})++ def wrap(text):+ words, lines, cur = text.split(), [], ""+ for w in words:+ if len(cur) + len(w) > 18:+ lines.append(cur); cur = w+ else:+ cur = f"{cur} {w}".strip()+ if cur:+ lines.append(cur)+ return "<br/>".join(lines)++ out = ["flowchart LR"]+ out.append(' subgraph SRC["Model terms these are computed from"]')+ out.append(" direction TB")+ for n in ins:+ out.append(f' {n}["{wrap(n)}"]')+ out.append(" end")+ out.append(' subgraph IND["Indicators"]')+ out.append(" direction TB")+ for k in kpis:+ mark = " ⛔" if k in blocking else ""+ out.append(f' {k}("{wrap(labels[k])}{mark}")')+ out.append(" end")+ out.append(' subgraph TGT["What they measure / what mandates them"]')+ out.append(" direction TB")+ for n in classes:+ out.append(f' {n}["{wrap(n)}"]')+ for n in docs:+ out.append(f' {n}[["{wrap(n)}"]]')+ out.append(" end")++ for r in inputs:+ out.append(f' {r["input"]} --> {r["kpi"]}')+ for r in comps:+ out.append(f' {r["kpi"]} ==>|"composed of"| {r["component"]}')+ for r in applies:+ out.append(f' {r["kpi"]} -.-> {r["cls"]}')+ for r in govs:+ out.append(f' {r["doc"]} -.->|"mandates"| {r["kpi"]}')++ out.append(" classDef blocking fill:#f8d7da,stroke:#a4232f,stroke-width:2px,color:#5c1119")+ out.append(" classDef kpi fill:#e8eef7,stroke:#3d5a80,color:#1d2f45")+ out.append(" classDef term fill:#f4f1ea,stroke:#b0a99a,color:#3a352c")+ out.append(" classDef doc fill:#fff3cd,stroke:#a68b2c,color:#4a3c0c")+ normal = [k for k in kpis if k not in blocking]+ if normal:+ out.append(" class " + ",".join(normal) + " kpi")+ if blocking:+ out.append(" class " + ",".join(sorted(blocking)) + " blocking")+ if ins or classes:+ out.append(" class " + ",".join(ins + classes) + " term")+ if docs:+ out.append(" class " + ",".join(docs) + " doc")++ nodes = len(ins) + len(kpis) + len(classes) + len(docs)+ edges = len(inputs) + len(comps) + len(applies) + len(govs)+ print("```mermaid")+ print("\n".join(out))+ print("```")+ print(f"\n{nodes} nodes, {edges} edges. Every one derived by traversal;")+ print("none placed by hand. Blocking indicators marked.")+++def cmd_impact(changed):+ """Which KPIs depend on a changed IRI, transitively?"""+ term = changed.lstrip(":")+ direct = query(+ P+ + f"""SELECT ?kpi ?label ?blocking WHERE {{+ ?kpi crd:computedFrom crd:{term} .+ OPTIONAL {{ ?kpi rdfs:label ?label }}+ OPTIONAL {{ ?kpi crd:isBlocking ?blocking }}+}}"""+ )+ # dependsOnKPI is transitive, so the closure comes back from one query.+ downstream = query(+ P+ + f"""SELECT DISTINCT ?composite ?label WHERE {{+ ?direct crd:computedFrom crd:{term} .+ ?composite crd:dependsOnKPI ?direct .+ OPTIONAL {{ ?composite rdfs:label ?label }}+}}"""+ )++ print(f"Impact of a change to :{term}\n")+ if not direct and not downstream:+ print(" no KPI declares this as an input")+ return+ print(f" Directly computed from it ({len(direct)}):")+ for r in direct:+ flag = " [BLOCKING]" if r.get("blocking") == "true" else ""+ print(f" - {r.get('label', r['kpi'])}{flag}")+ print(f"\n Composite KPIs that inherit the change ({len(downstream)}):")+ for r in downstream:+ print(f" - {r.get('label', r['composite'])}")+ print("\n This is the question a config-table KPI registry cannot answer.")+++def cmd_breach():+ """Evaluate indicators against current data."""+ print("Indicator evaluation against the loaded graph\n")++ checks = [+ ("Field translation efficiency", 0.80, "lower value is a breach",+ P + "SELECT ?c ?v WHERE { ?c crd:fieldToScreenRatio ?v . FILTER(?v < 0.80) }"),+ ("Unexplained residual variance", 0.20, "higher value is a breach",+ P + "SELECT ?c ?v WHERE { ?c crd:unexplainedResidualVariance ?v . FILTER(?v > 0.20) }"),+ ("Binding evidence freshness", 1.0, "any stale protein is a breach",+ P + "SELECT ?p WHERE { ?p crd:affectedByMutation ?m ; crd:isStale true }"),+ ("Evidence gap closure", 1.0, "any open gap is a breach",+ P + "SELECT ?c ?g WHERE { ?c crd:hasEvidenceGap ?g }"),+ ("Trial data completeness", 1.0, "a field trial with no plot is a breach",+ P + "SELECT ?t WHERE { ?t a crd:FieldTrial . FILTER NOT EXISTS { ?t crd:hasPlot ?p } }"),+ ]+ breaches = 0+ for name, threshold, note, sparql in checks:+ rows = query(sparql)+ status = f"BREACH ({len(rows)})" if rows else "ok"+ if rows:+ breaches += 1+ print(f" [{status:>12}] {name} (threshold {threshold}, {note})")+ for r in rows:+ print(f" {json.dumps(r)}")++ safety = query(+ P+ + """SELECT ?candidate ?organism ?doc WHERE {+ ?candidate crd:targetsOrganism ?organism .+ ?organism a ?cls . ?cls rdfs:subClassOf* crd:BeneficialOrganism .+ OPTIONAL { ?kpi a crd:SafetyKPI ; crd:governedBy ?d . ?d crd:docId ?doc }+}"""+ )+ print()+ if safety:+ breaches += 1+ print(f" [ BREACH] Beneficial organism exposure [BLOCKING, threshold 0.0]")+ for r in safety:+ gov = f", mandated by {r['doc']}" if r.get("doc") else ""+ print(f" {r['candidate']} targets {r['organism']}{gov}")+ else:+ print(" [ ok] Beneficial organism exposure")++ print(f"\n {breaches} indicators breached. Stage gate readiness: FAIL")+ print(" Every breach traces to a triple, and the blocking one traces to the")+ print(" controlled document that mandates it.")+++def main():+ if not BIN.exists():+ sys.exit(f"engine binary not found: {BIN}")+ cmd = sys.argv[1] if len(sys.argv) > 1 else "graph"+ if cmd == "graph":+ cmd_graph()+ elif cmd == "impact":+ cmd_impact(sys.argv[2] if len(sys.argv) > 2 else ":targetsOrganism")+ elif cmd == "breach":+ cmd_breach()+ else:+ sys.exit(f"unknown command: {cmd}")+++if __name__ == "__main__":+ main()diff --git a/demo/ontology_from_docs.py b/demo/ontology_from_docs.pynew file mode 100644index 00000000..3e625cec--- /dev/null+++ b/demo/ontology_from_docs.py@@ -0,0 +1,1644 @@+#!/usr/bin/env python3+"""+Ontology out of text: derive the ontology FROM a document corpus, then+populate it, with PII tokenised before anything reaches a model.++This is domain-agnostic. Nothing about crop protection, biology or any other+subject is hardcoded; the vocabulary is whatever the documents turn out to be+about.++ STAGE 1 READ every document in the folder+ STAGE 2 TOKENISE detect and replace sensitive values with stable tokens+ STAGE 3 DERIVE propose classes, properties and disjointness per+ document, then merge into ONE candidate ontology+ STAGE 4 POPULATE extract instances, constrained to the derived ontology+ STAGE 5 VERIFY closed-world check that nothing was invented+ STAGE 6 SCAN reason, and report contradictions++The two-phase split is the point. A single pass either invents terms freely,+which gives an unusable model, or is pinned to a vocabulary written in advance,+which cannot describe documents nobody has read yet. Deriving first and+constraining second gives a vocabulary that came from the corpus AND a+closed-world check with teeth.++Tokenisation happens BEFORE derivation, so no model ever sees a real value.+Tokens are deterministic, which means the same value produces the same token in+every document: the token doubles as a join key for exact-match entity+resolution, achieved without any component handling the raw value.++Usage:+ python3 demo/ontology_from_docs.py --corpus demo/corpus/dcat-us+ python3 demo/ontology_from_docs.py --corpus demo/corpus/dcat-us --cached+"""++import argparse+import concurrent.futures as cf+import hashlib+import hmac+import json+import os+import pathlib+import re+import subprocess+import sys+import time+import urllib.error+import urllib.request++from rdflib import Graph, OWL, RDF, RDFS, URIRef++ROOT = pathlib.Path(__file__).resolve().parent.parent+BIN = ROOT / "target" / "release" / "open-ontologies"+ENGINE = os.environ.get("ONTO_ENGINE", "http://127.0.0.1:8137")+LLM_BASE = os.environ.get("ONTO_LLM_BASE_URL", "http://localhost:8081/v1").rstrip("/")+LLM_KEY = os.environ.get("ONTO_LLM_API_KEY", "not-needed")+NS = "https://w3id.org/dcat-us-demo#"++# Samples voted over when proposing disjointness. Odd, so a majority exists.+SAMPLES = int(os.environ.get("ONTO_DISJOINT_SAMPLES", "3"))++# Repair attempts per non-conforming graph before it goes to human review.+REPAIR_ROUNDS = int(os.environ.get("ONTO_REPAIR_ROUNDS", "3"))++# Every automated resolution decision goes on the record: what changed, in+# which document, by which rule. The Studio's resolution panel renders this+# ledger and the judgement calls in it can be reverted.+LEDGER = {"path": None}+++def record(kind, **kw):+ if LEDGER["path"]:+ with open(LEDGER["path"], "a") as fh:+ fh.write(json.dumps({"kind": kind, **kw}) + "\n")++if os.environ.get("NO_COLOR"):+ BOLD = DIM = RED = GRN = YEL = OFF = ""+else:+ BOLD, DIM, RED, GRN, YEL, OFF = "\033[1m", "\033[2m", "\033[31m", "\033[32m", "\033[33m", "\033[0m"++print = __builtins__.print if not isinstance(__builtins__, dict) else __builtins__["print"]+++# --------------------------------------------------------------------------+# Tokenisation: delegated to the swappable detector + vault module+# --------------------------------------------------------------------------++sys.path.insert(0, str(ROOT / "demo"))+from tokenisation import build as build_tokeniser # noqa: E402+++# --------------------------------------------------------------------------+# Model + engine plumbing+# --------------------------------------------------------------------------++def call_model(prompt: str, max_tokens: int = 3000, attempts: int = 4,+ temperature: float = 0.2, timeout: int = 900) -> str:+ """Call the model, backing off on overload.++ A local inference server under parallel load answers 503 or 504 rather+ than queueing. Treating that as a permanent failure loses most of a run,+ which is exactly what happened before this was added.+ """+ last = None+ for i in range(attempts):+ try:+ return _call_model_once(prompt, max_tokens, temperature, timeout)+ except urllib.error.HTTPError as e:+ if e.code not in (429, 500, 502, 503, 504):+ raise+ last = e+ except urllib.error.URLError as e:+ last = e+ time.sleep(2 ** i)+ raise RuntimeError(f"model unavailable after {attempts} attempts: {last}")+++def _call_model_once(prompt: str, max_tokens: int = 3000,+ temperature: float = 0.2, timeout: int = 900) -> str:+ model = os.environ.get("ONTO_LLM_MODEL", "")+ if not model:+ req = urllib.request.Request(LLM_BASE + "/models", headers={"Authorization": f"Bearer {LLM_KEY}"})+ model = json.load(urllib.request.urlopen(req, timeout=30))["data"][0]["id"]+ req = urllib.request.Request(+ LLM_BASE + "/chat/completions",+ data=json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}],+ "temperature": temperature, "max_tokens": max_tokens,+ "chat_template_kwargs": {"enable_thinking": False}}).encode(),+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"})+ msg = json.load(urllib.request.urlopen(req, timeout=900))["choices"][0]["message"]+ # A null content field is a valid response shape, not an error: the model+ # produced no visible text. Callers regex over this, so hand back a string.+ return msg.get("content") or ""+++def mcp(method, params, sid=[None]):+ h = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream"}+ if sid[0]:+ h["Mcp-Session-Id"] = sid[0]+ req = urllib.request.Request(ENGINE + "/mcp", headers=h, data=json.dumps(+ {"jsonrpc": "2.0", "id": 1, "method": method, "params": params}).encode())+ r = urllib.request.urlopen(req, timeout=180)+ sid[0] = r.headers.get("Mcp-Session-Id") or sid[0]+ body = r.read().decode().strip()+ if body.startswith("{"):+ return json.loads(body)+ for line in body.split("\n"):+ if line.startswith("data:"):+ try:+ fr = json.loads(line[5:].strip())+ if "result" in fr or "error" in fr:+ return fr+ except Exception:+ pass+ return {}+++def tool_text(res):+ return res.get("result", {}).get("content", [{}])[0].get("text", "") if res else ""+++def engine(cmds):+ return subprocess.run([str(BIN), "batch", "-"], input="\n".join(cmds) + "\n",+ capture_output=True, text=True).stdout.strip()+++# Declare every prefix a model might plausibly use. One undeclared prefix+# (rdf: was the culprit) aborts the parse at first use, so a 2700-line+# ontology silently loaded as 195 triples: all the classes, none of the+# properties, because classes happen to come first in the file.+PREFIXES = (f"@prefix : <{NS}> .\n"+ "@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n"+ "@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n"+ "@prefix owl: <http://www.w3.org/2002/07/owl#> .\n"+ "@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n"+ "@prefix skos: <http://www.w3.org/2004/02/skos/core#> .\n"+ "@prefix dcterms: <http://purl.org/dc/terms/> .\n")+++def is_valid_turtle(ttl: str) -> tuple[bool, str]:+ """Parse-check a fragment on its own."""+ import tempfile+ with tempfile.NamedTemporaryFile("w", suffix=".ttl", delete=False) as fh:+ fh.write(ttl)+ path = fh.name+ try:+ out = subprocess.run([str(BIN), "validate", path], capture_output=True, text=True).stdout+ return ('"ok":true' in out), out.strip()[:160]+ finally:+ pathlib.Path(path).unlink(missing_ok=True)+++# Structural vocabulary the chunker imposes on every corpus. The extractor+# reaches for it in every document because the documents really are divided+# into sections; no per-document derivation proposes it, because it is a+# property of how documents are read rather than of what they say.+STRUCTURAL = """+:Section a owl:Class ;+ rdfs:label "Section" ;+ rdfs:comment "A numbered division of a document, as identified by the chunker." .++:hasSection a owl:ObjectProperty ;+ rdfs:label "has section" ;+ rdfs:domain :Document ;+ rdfs:range :Section .++:hasSectionNumber a owl:DatatypeProperty ;+ rdfs:label "has section number" ;+ rdfs:domain :Section ;+ rdfs:range xsd:string .+"""+++def body_of(frag: str) -> str:+ """The statements of a fragment, with any prefix header removed."""+ return "\n".join(l for l in frag.splitlines()+ if not l.lstrip().startswith(("@prefix", "@base"))).strip()+++def mend(body: str) -> str:+ """Repair the syntax slips a generator makes at the end of a statement.++ A predicate list that ends `; .` or a property block whose last predicate+ keeps its separator are the two failures seen in practice. Both are+ unambiguous, so fixing them costs nothing and recovers a whole fragment+ that would otherwise be discarded along with every term it declares.+ """+ body = re.sub(r";\s*\.", " .", body) # dangling separator before the stop+ body = re.sub(r";\s*\n\s*(?=[:\w]+\s+a\s)", " .\n", body) # missing stop between subjects+ body = re.sub(r",\s*\.", " .", body)++ # A bare `disjointWith`. Turtle has no default predicate namespace, so an+ # unprefixed name is a parse error, not a lookup failure.+ body = re.sub(r"(?<![:\w])disjointWith\b", "owl:disjointWith", body)++ # A standard prefix pasted after the default one, giving `:rdfs:label`.+ # This parses, which is what makes it dangerous: it silently mints a term+ # in the local namespace that looks like the standard one, and the+ # closed-world check then correctly reports a term nobody declared.+ body = re.sub(r"(?<![\w:]):(rdf|rdfs|owl|xsd|skos|dcterms):", r"\1:", body)++ # A type assertion whose object was lost, leaving the keyword exposed.+ body = re.sub(r"\ba\s+a\b", " a", body)++ # An IRI the generator split with a space before a token:+ # ":TimePoint TOK_DATE_x a :TimePoint". Rejoin with an underscore; a bare+ # token in predicate position truncates every file it is merged into.+ body = re.sub(r"^(:\w+)[ \t]+(TOK_\w+)\b", r"\1_\2", body, flags=re.M)++ # A statement whose subject was dropped, leaving the predicate in subject+ # position. Nothing can recover the intended subject, so the statement goes+ # rather than the fragment. Only an unindented occurrence starts a+ # statement: an indented one is a continuation of a valid subject above it,+ # and removing those is what breaks an otherwise sound block. The whole+ # orphan runs to its terminating dot.+ kept, skipping = [], False+ for line in body.splitlines():+ if re.match(r"^(?:owl:)?:?disjointWith\s", line):+ skipping = True+ if skipping:+ if line.rstrip().endswith("."):+ skipping = False+ continue+ kept.append(line)+ body = "\n".join(kept)++ # A statement left unterminated before the next subject begins. The blank+ # line and the following `:Term a` are together unambiguous.+ body = re.sub(r"(?<=[\"\w>])\s*\n(\s*\n\s*)(?=:\S+\s+(?:a|rdf:type)\s)",+ r" .\n\1", body)+ return body+++def rehead(frag: str) -> str:+ """A fragment under the canonical prefix header.++ The generator writes its own header and routinely omits a prefix it then+ uses (`rdf:` above all). Validating the fragment as written therefore+ rejects it for a declaration problem rather than a modelling one. Strip+ whatever header it wrote, impose ours, and judge the statements.+ """+ return PREFIXES + "\n" + mend(body_of(frag))+++def normalise(text: str) -> str:+ text = re.sub(r"```(?:turtle|ttl)?", "", text).strip()+ lines = text.split("\n")+ start = None+ for i, line in enumerate(lines):+ if line.strip().startswith(("@prefix", "@base")):+ start = i+ break+ if start is None:+ for i, line in enumerate(lines):+ if re.match(r"^\s*:\S+\s+(a|rdf:type)\s", line):+ start = i+ break+ text = "\n".join(lines[start if start is not None else 0:]).strip()+ lines = text.split("\n")+ for i in range(len(lines) - 1, -1, -1):+ if lines[i].rstrip().endswith((".", ";", ",")):+ text = "\n".join(lines[: i + 1]).strip()+ break+ # The same mechanical repairs the vocabulary fragments get. Extracted data+ # comes from the same generator and fails in the same ways; leaving them+ # out here is why a whole graph was being discarded for one missing dot.+ body = mend(body_of(text))+ if body and not body.endswith("."):+ body += " ."+ return PREFIXES + "\n" + body+++def docx_text(path: pathlib.Path) -> str:+ if path.suffix == ".md":+ return path.read_text()+ return subprocess.run(["pandoc", str(path), "-t", "plain", "--wrap=none"],+ capture_output=True, text=True).stdout+++# --------------------------------------------------------------------------+# Stage 3: derive the vocabulary+# --------------------------------------------------------------------------++DERIVE_PROMPT = """Read this document and propose the ONTOLOGY it implies. You are+defining the vocabulary, not describing the specific facts.++Return ONLY Turtle. No prose, no code fences.++@prefix : <{ns}> .+@prefix owl: <http://www.w3.org/2002/07/owl#> .+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .++Produce:+1. owl:Class declarations for every KIND of thing the document is about. Use+ UpperCamelCase local names. Give every class rdfs:label and rdfs:comment.+2. rdfs:subClassOf where one kind is clearly a kind of another.+3. owl:ObjectProperty for relationships between things, each with rdfs:domain,+ rdfs:range and rdfs:label.+4. owl:DatatypeProperty for attributes, each with rdfs:domain, rdfs:range+ (an xsd type) and rdfs:label.+5. owl:disjointWith between any two classes that CANNOT both describe the same+ thing at the same time. Be deliberate: this is what lets a reasoner detect+ that two documents contradict each other.++Do NOT create individuals. Classes and properties only.++DOCUMENT:+{body}+"""++POPULATE_PROMPT = """Extract the FACTS from this document as instances of a+given ontology.++Return ONLY Turtle. No prose, no code fences.++@prefix : <{ns}> .+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .++Use ONLY the classes and properties in this vocabulary. Invent nothing:+{vocab}++Every class and property you write must appear VERBATIM in the list above.+Before writing any term, find it in the list and copy it exactly.++Do NOT build a property name by joining a class name to a field name. Writing+:hasChangeLogEntryChangeApplied because the subject is a :ChangeLogEntry and the+field is "applied" produces a term that does not exist. Use the property the+vocabulary already provides, even when its name is shorter or more general than+the wording in the document.++If the document asserts something the vocabulary genuinely cannot express,+OMIT it. A smaller correct graph is the goal; an invented term is a defect,+not a partial success.++Produce:+1. One :Document individual, IRI :DOC_{safe}, with rdfs:label and any+ document-control values present.+2. A :Section individual per numbered section where the vocabulary has a+ suitable class, IRI :SEC_{safe}_<number with dots as underscores>.+3. Individuals for the things the document states facts about, typed with the+ most specific class available, each with rdfs:label.++ When the document says a thing IS a kind of something, and the vocabulary+ has a class for that kind, TYPE the thing with that class. Never create an+ individual to stand for the kind itself.++ "Culture07 is maintained as an adherent line"+ correct: :Culture07 a :AdherentLine ; rdfs:label "Culture07" .+ WRONG: :Culture07 a :CellLine .+ :ADHERENT_LINE a :CellLine ; rdfs:label "Adherent line" .++ The wrong form loses the fact entirely: it records that something called+ "adherent line" exists, not that Culture07 is one.++ Give each individual EXACTLY ONE type: the single most specific class that+ fits. A registry row listing a cell line's genotype, doubling time and+ growth conditions describes ONE cell line; it does not make that cell line+ a genotype or a doubling time. Attach those as properties, never as extra+ types.++ correct: :Culture07 a :AdherentLine ;+ :hasGenotype :GENOTYPE_WT ;+ :hasDoublingTime "24h" .+ WRONG: :Culture07 a :CellLine, :CellLineGenotype, :CellLineDoublingTime .++4. Name each individual after the thing itself and nothing else. The IRI for+ Culture07 is :Culture07 in every document that mentions it. Do not prefix it+ with the document id, and do not qualify it with the context it appeared in+ (:CULTURE07_STR, :CELL_LINE_CULTURE07_REGISTRY). The same real thing must+ get the same IRI everywhere, because that is what lets facts from different+ documents meet.+5. The relationships and attribute values the document asserts.++Values that look like TOK_ followed by capitals and a hex string are tokens+standing in for redacted content. Use them verbatim as identifiers or literals.+Do not attempt to interpret or expand them.++DOCUMENT:+{body}+"""+++def derive_one(args):+ path, tokenised, docid = args+ try:+ raw = call_model(DERIVE_PROMPT.format(ns=NS, body=tokenised[:12000]))+ except Exception as e:+ return docid, None, f"model error: {e}"+ if re.search(r"(\b\S{1,6}\b[ .]{1,3})\1{15,}", raw):+ return docid, None, "degenerate output"+ return docid, normalise(raw), None+++def snap_to_vocab(body: str, declared: set[str]) -> str:+ """Snap a term to a declared term it differs from only in case.++ The generator writes `:HasCellLine` where the ontology declares+ `:hasCellLine`. Closed-world checking is right to reject it, but the intent+ is unambiguous and there is exactly one candidate, so resolving it here is+ sound rather than charitable. Anything that does not match a declared term+ under case folding is left alone: that is real invention and it must stay+ visible to the check.+ """+ fold = {}+ for d in declared:+ fold.setdefault(d.lower(), d)++ def swap(m: re.Match) -> str:+ term = m.group(1)+ target = fold.get(term.lower())+ return f":{target}" if target and target != term else m.group(0)++ return re.sub(r":(\w+)", swap, body)+++def populate_one(args):+ path, tokenised, docid, vocab = args+ safe = docid.replace("-", "_")+ try:+ raw = call_model(POPULATE_PROMPT.format(ns=NS, vocab=vocab, safe=safe, body=tokenised[:12000]))+ except Exception as e:+ return docid, None, f"model error: {e}"+ if re.search(r"(\b\S{1,6}\b[ .]{1,3})\1{15,}", raw):+ return docid, None, "degenerate output"+ declared = set(re.findall(r":(\w+)", vocab))+ return docid, snap_to_vocab(normalise(raw), declared), None+++DISJOINT_PROMPT = """Here is a list of classes from an ontology derived from a+document corpus.++Identify pairs that are MUTUALLY EXCLUSIVE: a single thing cannot be both at+the same time. Examples of the kind of pair that qualifies: a culture that is+adherent versus one in suspension; a sample that passed versus failed; a+record that is authenticated versus unauthenticated.++Return ONLY Turtle, one statement per line, no prose and no fences:++:ClassA owl:disjointWith :ClassB .++Only assert a pair when you are confident. An incorrect disjointness makes the+ontology inconsistent, which is worse than a missing one. Return nothing at all+if no pair qualifies.++CLASSES:+{classes}+"""+++def _sibling_groups(classes: list[str], cap: int = 60) -> list[list[str]]:+ """Batch the class list into groups worth asking a disjointness question about.++ Asking about all 272 classes in one call returns nothing at all: the model+ answers 20 classes with 3700 characters, 150 with 4700, and 272 with an+ empty string. That failure is silent, so the run reports zero axioms and+ looks merely unlucky.++ Grouping is by head noun rather than arbitrary slicing, because that is+ where mutual exclusivity actually lives. `AdherentLine` and+ `SuspensionLine` are candidates precisely because they are both kinds+ of `Line`; two classes with nothing in common almost never need an axiom+ between them. Small groups are pooled so a group of one is not a call.+ """+ def head(name: str) -> str:+ parts = re.findall(r"[A-Z][a-z0-9]*", name)+ return parts[-1] if parts else name++ by_head: dict[str, list[str]] = {}+ for c in classes:+ by_head.setdefault(head(c), []).append(c)++ groups, pool = [], []+ for _, members in sorted(by_head.items()):+ if len(members) >= 2:+ for i in range(0, len(members), cap):+ groups.append(sorted(members[i:i + cap]))+ else:+ pool.extend(members)+ for i in range(0, len(pool), cap):+ chunk = sorted(pool[i:i + cap])+ if len(chunk) >= 2:+ groups.append(chunk)+ return groups+++def add_disjointness(merged_path: pathlib.Path) -> int:+ """Propose disjointness across the merged vocabulary.++ Deliberately a SEPARATE pass over the merged class list rather than part of+ per-document derivation. Disjointness is a cross-document property: whether+ "adherent" and "suspension" are mutually exclusive is not visible from+ inside one document that only mentions one of them. Asking per document+ reliably produced none at all.++ Stability matters more here than coverage. These axioms decide what the+ reasoner is even capable of detecting, so an axiom set that changes run to+ run makes every downstream contradiction count unreproducible. Sampling+ once gave 57, 68, 93, 114 and 150 pairs on an identical class list. So:+ sample SAMPLES times, keep only pairs a majority agrees on, and cache the+ result against a hash of the class list. Re-running over the same+ vocabulary is then exactly reproducible, and a changed vocabulary+ invalidates the cache by construction.+ """+ ttl = merged_path.read_text()+ classes = sorted(set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", ttl)))+ if len(classes) < 2:+ return 0++ key = hashlib.sha256("\n".join(classes).encode()).hexdigest()[:16]+ cache = merged_path.parent / f"_disjoint.{key}.ttl"+ known = set(classes)++ if cache.exists():+ block = cache.read_text().strip()+ n = block.count("owl:disjointWith")+ else:+ votes: dict[tuple[str, str], int] = {}+ rounds = 0+ for group in _sibling_groups(classes):+ prompt = DISJOINT_PROMPT.format(classes="\n".join(":" + c for c in group))+ got = 0+ for _ in range(SAMPLES):+ try:+ raw = call_model(prompt, 1200, temperature=0.0)+ except Exception:+ continue+ if not raw.strip():+ continue+ got += 1+ # Within one sample a pair counts once, however often it is+ # repeated, and direction is irrelevant: disjointness is+ # symmetric.+ seen = set()+ for a, b in re.findall(r"(:\w+)\s+owl:disjointWith\s+(:\w+)", raw):+ a, b = a.lstrip(":"), b.lstrip(":")+ if a == b or a not in known or b not in known:+ continue+ seen.add(tuple(sorted((a, b))))+ for pair in seen:+ votes[pair] = votes.get(pair, 0) + 1+ rounds = max(rounds, got)+ if not rounds:+ return 0+ need = rounds // 2 + 1+ valid = sorted(p for p, v in votes.items() if v >= need)+ if not valid:+ return 0+ block = "\n".join(f":{a} owl:disjointWith :{b} ." for a, b in valid)+ cache.write_text(block + "\n")+ n = len(valid)++ ttl = re.split(r"\n# Disjointness, [^\n]*\n", ttl)[0].rstrip()+ merged_path.write_text(+ ttl + "\n\n# Disjointness, agreed by a majority of samples over the merged vocabulary\n"+ + block + "\n")+ return n+++# Suffixes that name a KIND of the parent, which is exactly what a subclass+# partition already expresses. Status/State are deliberately absent: a status+# ("in routine use", "retired") is a property of a thing over time, not a kind+# of thing, and collapsing it into the class hierarchy would be wrong.+KIND_SUFFIXES = ("Type", "Kind", "Category", "Mode", "Variant")+++def remove_subject(ttl: str, name: str) -> str:+ """Remove every statement whose subject is :name."""+ out, skipping = [], False+ for line in ttl.splitlines():+ if re.match(rf"^:{re.escape(name)}\b", line):+ skipping = True+ if skipping:+ if line.rstrip().endswith("."):+ skipping = False+ continue+ out.append(line)+ return "\n".join(out)+++def _reconcile_ttl(merged_path: pathlib.Path) -> list[str]:+ """Collapse competing modelling patterns left behind by the merge.++ Per-document derivation is independent, so one document models a mode+ as a subclass partition (:AdherentLine, :SuspensionLine) while another+ models the same notion as an attribute class (:LineType). Concatenating+ fragments keeps both, and the extractor then legitimately picks whichever+ it likes. It reliably picks the attribute class, which is the one form+ disjointness cannot reach: :Culture07Type a :LineType says nothing the+ reasoner can contradict.++ The rule: where a parent has a subclass partition (two or more+ subclasses) AND the vocabulary also declares parent+KIND_SUFFIX, the+ attribute class is redundant with the partition and loses. Remove it,+ and remove any property whose domain or range points at it, so the only+ way left to express the distinction is the one that supports reasoning.++ None of the engine's enforce packs (generic, hierarchy, value_partition)+ catches this today; it is a merge problem, not a single-ontology problem.++ Operates on the Turtle text on disk, mirroring the file-based fragment+ merge the rest of this pipeline stage uses. `reconcile()` below applies+ the identical rule to an in-memory RDF graph.+ """+ ttl = merged_path.read_text()+ classes = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", ttl))+ children: dict[str, set[str]] = {}+ for m in re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", ttl):+ children.setdefault(m.group(2), set()).add(m.group(1))++ doomed = sorted({parent + suf+ for parent, kids in children.items() if len(kids) >= 2+ for suf in KIND_SUFFIXES if parent + suf in classes})+ if not doomed:+ return []++ for name in doomed:+ ttl = remove_subject(ttl, name)+ # Properties anchored to the removed class go with it. A property+ # whose range no longer exists invites the extractor to mint an+ # untyped individual to fill it, which recreates the pattern that+ # was just removed.+ for prop in set(re.findall(+ rf":(\w+)\s+(?:a|rdf:type)\s+owl:(?:Object|Datatype)Property\b"+ rf"[^.]*?rdfs:(?:domain|range)\s+:{re.escape(name)}\b", ttl)):+ ttl = remove_subject(ttl, prop)+ ttl = re.sub(rf"^:{re.escape(name)}\s+owl:disjointWith[^\n]*\n?", "",+ ttl, flags=re.M)+ ttl = re.sub(rf"^:\w+\s+owl:disjointWith\s+:{re.escape(name)}[^\n]*\n?", "",+ ttl, flags=re.M)+ merged_path.write_text(ttl)+ return doomed+++def _split_iri(iri: URIRef) -> tuple[str, str]:+ """Namespace and local name of an IRI, splitting on the last '#' or '/'."""+ s = str(iri)+ if "#" in s:+ ns, _, local = s.rpartition("#")+ return ns + "#", local+ ns, _, local = s.rpartition("/")+ return ns + "/", local+++def reconcile(graph: Graph) -> Graph:+ """Collapse competing modelling patterns left behind by the merge.++ Per-document derivation is independent, so one document models a mode+ as a subclass partition (:AdherentLine, :SuspensionLine) while another+ models the same notion as an attribute class (:LineType). Concatenating+ fragments keeps both, and the extractor then legitimately picks whichever+ it likes. It reliably picks the attribute class, which is the one form+ disjointness cannot reach: :Culture07Type a :LineType says nothing the+ reasoner can contradict.++ The rule: where a parent has a subclass partition (two or more+ subclasses) AND the vocabulary also declares parent+KIND_SUFFIX, the+ attribute class is redundant with the partition and loses. Remove it,+ and remove any property whose domain or range points at it, so the only+ way left to express the distinction is the one that supports reasoning.+ Status and State are deliberately absent from KIND_SUFFIXES: a status is+ a property of a thing over time, not a kind of thing, so an attribute+ class named parent+Status or parent+State is genuinely an attribute and+ is spared.++ None of the engine's enforce packs (generic, hierarchy, value_partition)+ catches this today; it is a merge problem, not a single-ontology problem.++ Same rule as `_reconcile_ttl` above, expressed directly over RDF triples+ instead of the Turtle text on disk, so it can run against a graph that+ was never serialised to a file.+ """+ classes = set(graph.subjects(RDF.type, OWL.Class))+ children: dict[URIRef, set[URIRef]] = {}+ for child, parent in graph.subject_objects(RDFS.subClassOf):+ children.setdefault(parent, set()).add(child)++ doomed: set[URIRef] = set()+ for parent, kids in children.items():+ if len(kids) < 2:+ continue+ ns, name = _split_iri(parent)+ for suf in KIND_SUFFIXES:+ candidate = URIRef(ns + name + suf)+ if candidate in classes:+ doomed.add(candidate)++ for cls in doomed:+ # Properties anchored to the removed class go with it. A property+ # whose range no longer exists invites the extractor to mint an+ # untyped individual to fill it, which recreates the pattern that+ # was just removed.+ for prop in set(graph.subjects(RDFS.domain, cls)) | set(graph.subjects(RDFS.range, cls)):+ graph.remove((prop, None, None))+ # disjointWith triples referencing the removed class, either direction.+ for s, o in list(graph.subject_objects(OWL.disjointWith)):+ if s == cls or o == cls:+ graph.remove((s, OWL.disjointWith, o))+ # The class declaration itself, and everything else hanging off it.+ graph.remove((cls, None, None))++ return graph+++def _name_tokens(name: str) -> tuple[str, ...]:+ return tuple(sorted(t.lower() for t in re.findall(r"[A-Z][a-z0-9]*|^[a-z]+[0-9]*", name)+ if t.lower() != "has"))+++def mech_resolve(body: str, bad: list[str], onto_ttl: str, doc: str = "") -> str:+ """Resolve undeclared terms that map onto declared vocabulary mechanically.++ The three graphs that stall in LLM repair all fail on terms whose intent+ is recoverable without a model:++ A token equality hasConcentrationValue -> ConcentrationValue+ B unique superset hasContainmentLevel -> ContainmentLevelValue+ C has{Class} inversion S :hasCellLineGenotype O, where+ CellLineGenotype is a declared CLASS: the generator manufactured a+ forward property where the vocabulary models the inverse. Rewritten+ as O a :CellLineGenotype ; :hasCellLine S, which is the declared+ pattern every other document uses.++ Anything ambiguous (hasName with five plausible targets) is left alone+ for the LLM repair rounds and, failing those, the review queue.+ """+ classes = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", onto_ttl))+ props = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:(?:Object|Datatype)Property", onto_ttl))+ parent = dict(re.findall(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", onto_ttl))++ by_toks: dict[tuple, list[str]] = {}+ for p in props:+ by_toks.setdefault(_name_tokens(p), []).append(p)++ renames, inversions = {}, set()+ for term in {b.split("#")[-1] for b in bad}:+ if not re.fullmatch(r"\w+", term) or term in props or term in classes:+ continue+ tt = _name_tokens(term)+ if tt in by_toks and len(by_toks[tt]) == 1: # rule A+ renames[term] = by_toks[tt][0]+ continue+ sups = [p for p in props if set(tt) < set(_name_tokens(p))]+ if sups: # rule B+ m = min(len(_name_tokens(p)) for p in sups)+ best = [p for p in sups if len(_name_tokens(p)) == m]+ if len(best) == 1 and m - len(tt) <= 2:+ renames[term] = best[0]+ continue+ if term[:3].lower() == "has" and term[3:] in classes: # rule C+ inversions.add(term)++ for old, new in renames.items():+ body = re.sub(rf":{re.escape(old)}\b", f":{new}", body)+ record("renamed", doc=doc, **{"from": old, "to": new},+ how="token match against declared vocabulary")++ if inversions:+ types = {}+ subj = None+ for line in body.splitlines():+ m0 = re.match(r"^:(\w+)\b", line)+ if m0:+ subj = m0.group(1)+ mt = re.search(r"(?:^|\s)a\s+:(\w+)", line)+ if mt and subj and subj not in types:+ types[subj] = mt.group(1)++ out, extra, subj = [], [], None+ for line in body.splitlines():+ m0 = re.match(r"^:(\w+)\b", line)+ if m0:+ subj = m0.group(1)+ m = re.match(r"^(\s*):(\w+)\s+:(\w+)\s*([;.])\s*$", line)+ if m and m.group(2) in inversions and subj:+ prop, obj, term = m.group(2), m.group(3), m.group(4)+ inv, c, seen = None, types.get(subj), set()+ while c and c not in seen: # walk up to a declared inverse+ seen.add(c)+ inv = next((x for x in (f"has{c}", f"Has{c}") if x in props), None)+ if inv:+ break+ c = parent.get(c)+ stmt = f":{obj} a :{prop[3:]}"+ if inv:+ stmt += f" ;\n :{inv} :{subj}"+ extra.append(stmt + " .")+ record("inverted", doc=doc, predicate=prop, subject=subj, obj=obj,+ how=f"has{{Class}} inversion via declared :{inv}" if inv+ else "has{Class} inversion, no declared inverse")+ if term == ".": # re-terminate the statement+ for i in range(len(out) - 1, -1, -1):+ if out[i].strip():+ out[i] = re.sub(r"[;,]\s*$", " .", out[i])+ break+ continue+ out.append(line)+ body = "\n".join(out + [""] + extra)+ return body+++REFINE_PROMPT = """One document contains these excerpts about "{label}":++{snippets}++Which ONE of these categories does the document assign to "{label}"?+{options}++Reply with EXACTLY one category name from the list and nothing else.+If "{label}" is not any kind of {parent} at all, reply exactly: other+If the excerpts do not say, reply exactly: unstated+"""++GATE_PROMPT = """One document contains these excerpts mentioning "{label}":++{snippets}++Question: does "{label}" name a {parent}?++The excerpts may be registry rows, change logs, or procedures ABOUT the thing;+those still count: a registry row about a cell line is describing a cell line.+Answer no only when "{label}" names a different KIND of thing, such as a+reagent batch, a measurement, a document, or a person.++Reply exactly yes or no.+"""+++def partitions_of(ttl: str) -> dict[str, list[str]]:+ """Parents whose subclasses form a disjoint partition worth asking about."""+ kids: dict[str, set[str]] = {}+ for m in re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", ttl):+ kids.setdefault(m.group(2), set()).add(m.group(1))+ disjoint = set()+ for m in re.finditer(r":(\w+)\s+owl:disjointWith\s+:(\w+)", ttl):+ disjoint.add(frozenset(m.groups()))+ return {p: sorted(k) for p, k in kids.items()+ if len(k) >= 2 and any(frozenset((a, b)) in disjoint+ for a in k for b in k if a != b)}+++def drop_clone_spam(body: str, threshold: int = 8) -> tuple[str, int]:+ """Remove runs of numbered clone individuals (STEM_1 ... STEM_78).++ A partially degenerate generation emits dozens of individuals differing+ only in a counter. The repetition guard misses it because every line is+ unique, and the vocabulary check passes it because every term is+ declared. The counter is the tell: no real document describes 78 things+ named identically but for an integer.+ """+ stems: dict[str, int] = {}+ for m in re.finditer(r"^:(\w+?)_\d+\b", body, re.M):+ stems[m.group(1)] = stems.get(m.group(1), 0) + 1+ doomed = {s for s, c in stems.items() if c >= threshold}+ if not doomed:+ return body, 0+ kept, skipping, dropped = [], False, 0+ for line in body.splitlines():+ m = re.match(r"^:(\w+?)_\d+\b", line)+ if m and m.group(1) in doomed:+ skipping = True+ dropped += 1+ if skipping:+ if line.rstrip().endswith("."):+ skipping = False+ continue+ kept.append(line)+ return "\n".join(kept), dropped+++def refine(merged_path: pathlib.Path, instances: list[pathlib.Path],+ texts: dict[str, str]) -> int:+ """Assign partition subclasses with one closed-choice question each.++ Six rounds of instruction in the main extraction prompt failed to make+ the generator type individuals with partition subclasses; it wrote the+ safe parent class every time. So the decision is taken out of the big+ prompt entirely. For each individual typed with a bare partition parent,+ the excerpts mentioning it are shown back with the subclass names as a+ closed choice. A model that cannot follow "use the most specific class"+ inside a 16k-character extraction task answers "adherent or suspension+ or unstated" correctly, because there is nothing else to do.++ This is the LLM-Modulo shape end to end: generation is decomposed until+ each answer is small enough to verify, and the reasoner adjudicates what+ the answers jointly imply.+ """+ onto_ttl = merged_path.read_text()+ parts = partitions_of(onto_ttl)+ if not parts:+ return 0+ all_classes = sorted(set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", onto_ttl)))+ (merged_path.parent / "_review.jsonl").unlink(missing_ok=True)+ LEDGER["path"] = merged_path.parent / "_decisions.jsonl"+ LEDGER["path"].unlink(missing_ok=True)++ # Verdicts persist across runs. A queued individual stays typed with the+ # parent class, so every cached replay re-found it and re-asked the model+ # the same gate question: seconds of model time per build, answering+ # something already answered. The memo is keyed by subject and asserted+ # type; reverting a typing decision changes the type and naturally+ # re-opens the question.+ memo_path = merged_path.parent / "_refine_memo.json"+ try:+ memo = json.loads(memo_path.read_text())+ except Exception:+ memo = {}+ n = 0+ for f in instances:+ doc = f.stem.replace(".data", "")+ text = texts.get(doc, "")+ g = f.read_text()+ g, spam = drop_clone_spam(g)+ if spam:+ record("spam", doc=doc, dropped=spam,+ how="numbered clone individuals removed")+ print(f" {YEL}spam{OFF} {doc:14s} dropped {spam} numbered clone individuals")+ if text:+ for parent, kids in parts.items():+ seen_subjects: set[str] = set()+ for m in list(re.finditer(rf":(\w+)\s+a\s+:{parent}\b(?!\w)", g)):+ subj = m.group(1)+ # One subject can carry several statement blocks when a+ # document mentions it in several rows; asking (and+ # queueing) once per block produced duplicate ledger+ # entries for the same decision.+ if subj in seen_subjects:+ continue+ seen_subjects.add(subj)+ if subj in all_classes:+ # An INDIVIDUAL whose IRI is a declared class (a source+ # document emitted a literal :CellLine individual). Retyping it+ # would rewrite what looks like vocabulary. Humans only.+ with open(merged_path.parent / "_review.jsonl", "a") as rq:+ rq.write(json.dumps({+ "doc": doc, "subject": subj,+ "reason": "individual IRI collides with a declared class"}) + "\n")+ print(f" {YEL}review{OFF} {doc:14s} :{subj} "+ f"{DIM}IRI collides with a class; queued{OFF}")+ n += 1+ continue+ lm = re.search(rf":{re.escape(subj)}\b[^.]*?rdfs:label\s+\"([^\"]+)\"",+ g, re.S)+ label = lm.group(1) if lm else subj+ lines = [l.strip() for l in text.splitlines()+ if label.lower() in l.lower() and len(l.strip()) > len(label)]+ if not lines:+ continue+ mkey = f"{doc}|{subj}|{parent}"+ if memo.get(mkey) == "queued":+ entry = {"doc": doc, "subject": subj, "label": label,+ "asserted": parent,+ "reason": f"not a {parent} per gate question (memoised)"}+ with open(merged_path.parent / "_review.jsonl", "a") as rq:+ rq.write(json.dumps(entry) + "\n")+ record("queued", **entry)+ print(f" {YEL}review{OFF} {doc:14s} :{subj} "+ f"{DIM}not a {parent}; memoised verdict{OFF}")+ n += 1+ continue++ snippets = "\n".join(f"- {l[:220]}" for l in lines[:6])+ # The gate is its own binary question. Offered only as an+ # extra option inside the partition question, "other" loses+ # to pattern-matching: an FBS lot mentioned alongside+ # adherent cultures came back "AdherentLine". Asked+ # alone, "is Lot A itself a cell line?" gets the right no.+ try:+ gate = call_model(GATE_PROMPT.format(+ label=label, parent=parent, snippets=snippets),+ 16, attempts=2, temperature=0.0, timeout=60).strip().lower()+ except Exception:+ continue+ # An empty or off-script reply is NO VERDICT. Routing it+ # to the queue turned one transient null response into a+ # permanent, memoised wrong answer about the corpus's main+ # entity.+ if not gate or not (gate.startswith("yes") or gate.startswith("no")):+ continue+ anskey, pick = "", None+ if gate.startswith("yes"):+ try:+ ans = call_model(REFINE_PROMPT.format(+ label=label, parent=parent, snippets=snippets,+ options="\n".join(kids)), 30, attempts=2,+ temperature=0.0, timeout=60).strip()+ except Exception:+ continue+ anskey = ans.strip().strip(":").lower()+ pick = next((k for k in kids if k.lower() == anskey), None)+ else:+ anskey = "other"+ if pick:+ g = re.sub(rf"(:{re.escape(subj)}\b\s+a\s+):{parent}\b(?!\w)",+ rf"\g<1>:{pick}", g)+ n += 1+ record("typed", doc=doc, subject=subj,+ **{"from": parent, "to": pick},+ how="closed-choice question over the partition")+ print(f" {GRN}typed{OFF} {doc:14s} :{subj} -> :{pick}")+ elif anskey == "other":+ # The extractor mistyped it upstream. An automatic+ # second guess was tried here and reintroduced the+ # failure it was meant to fix (an FBS lot became+ # :CellLineGrowthRateComparison with :FBSLot sitting+ # in the candidate list). Wrong-type-with-confidence+ # is worse than wrong-type-in-a-queue: the type stays+ # as asserted and a human decides.+ entry = {"doc": doc, "subject": subj, "label": label,+ "asserted": parent,+ "reason": f"not a {parent} per gate question"}+ with open(merged_path.parent / "_review.jsonl", "a") as rq:+ rq.write(json.dumps(entry) + "\n")+ record("queued", **entry)+ memo[f"{doc}|{subj}|{parent}"] = "queued"+ memo_path.write_text(json.dumps(memo))+ print(f" {YEL}review{OFF} {doc:14s} :{subj} "+ f"{DIM}not a {parent}; queued for a human{OFF}")+ n += 1+ f.write_text(g)+ return n+++def split_statements(ttl: str) -> list[tuple[str, str]]:+ """(subject, statement) pairs, line-based, prefix header excluded."""+ out, subj, buf = [], None, []+ for line in ttl.splitlines():+ if not buf:+ m = re.match(r"^:(\w+)\b", line)+ if not m:+ continue+ subj = m.group(1)+ buf.append(line)+ if line.rstrip().endswith("."):+ out.append((subj, "\n".join(buf)))+ subj, buf = None, []+ return out+++def prune_to_used(onto_ttl: str, data: str) -> tuple[str, dict]:+ """The vocabulary the data actually exercises, nothing else.++ Derivation over-proposes by an order of magnitude: 10 documents produced+ 346 classes, most of them compositional one-offs+ (CellLineGrowthRateMeasurementStatisticalSignificanceAnalysis) that no+ instance ever uses. Rendering all of that buries the dozen entities the+ corpus is actually about. The FULL vocabulary stays on disk and keeps+ constraining extraction; the store the graph view and chat read carries+ only: classes with instances, their ancestors, the domain and range of+ every property the data uses, and disjointness among what remains.+ """+ # A class is anything the ontology treats as one, not only what carries an+ # explicit `a owl:Class`. The partition members are declared through+ # rdfs:subClassOf alone, and dropping them silently removed the exact+ # disjointness axiom the contradiction demo runs on.+ decl_cls = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", onto_ttl))+ decl_cls |= set(re.findall(r":(\w+)\s+rdfs:subClassOf\s+:", onto_ttl))+ for a, b in re.findall(r":(\w+)\s+owl:disjointWith\s+:(\w+)", onto_ttl):+ decl_cls.update((a, b))+ decl_props = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:(?:Object|Datatype)Property", onto_ttl))+ counts: dict[str, int] = {}+ for c in re.findall(r"\ba\s+:(\w+)", data):+ counts[c] = counts.get(c, 0) + 1+ # A class instantiated once is a description of one node, not a category.+ # The graph view draws the CLASS diagram, so every singleton is a box that+ # says nothing. Recurring classes are what the corpus is actually about;+ # singletons keep their individuals in the store but not their box.+ used_cls = {c for c, n in counts.items() if n >= 2 and c in decl_cls}+ used_props = decl_props & set(re.findall(r":(\w+)", data))++ stmts = split_statements(onto_ttl)+ by_subj = {}+ for s, t in stmts:+ by_subj.setdefault(s, t)++ keep = set(used_cls)+ for pr in used_props: # domain/range classes stay: edges render+ for dr in re.findall(r"rdfs:(?:domain|range)\s+:(\w+)", by_subj.get(pr, "")):+ if dr in decl_cls:+ keep.add(dr)++ parents: dict[str, set[str]] = {}+ for m in re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", onto_ttl):+ parents.setdefault(m.group(1), set()).add(m.group(2))+ frontier = list(keep)+ while frontier: # ancestors close the hierarchy+ for p in parents.get(frontier.pop(), ()):+ if p in decl_cls and p not in keep:+ keep.add(p)+ frontier.append(p)++ # View collapse. Two shapes of noise survive the instance-count filter:+ # specialisation chains (CellLineGrowthRateMeasurementStatistical...+ # extending CellLineGrowthRate token by token) and families of near+ # duplicate siblings (fourteen ChangeLogEntryChange* classes). A box per+ # variant hides the dozen entities the corpus is about. Chains collapse+ # into their shortest kept prefix; families keep their two most+ # instantiated members. Instance data is never touched, and partition+ # members are exempt: they are what the reasoner argues about.+ # Exempt only true partitions: disjoint SIBLINGS under a common parent.+ # The disjointness pass asserts pairs across the whole vocabulary, so+ # exempting every pair member exempts nearly everything.+ all_parents: dict[str, set[str]] = {}+ for m in re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", onto_ttl):+ all_parents.setdefault(m.group(1), set()).add(m.group(2))+ exempt = {c for a, b in re.findall(r":(\w+)\s+owl:disjointWith\s+:(\w+)", onto_ttl)+ if all_parents.get(a, set()) & all_parents.get(b, set())+ for c in (a, b)}+ ctok = {c: tuple(re.findall(r"[A-Z][a-z0-9]*", c)) for c in keep if c in decl_cls}+ hide = set()+ for a, ta in ctok.items():+ for b, tb in ctok.items():+ if a != b and len(tb) > len(ta) and tb[:len(ta)] == ta and b not in exempt:+ hide.add(b)+ families: dict[tuple, list[str]] = {}+ for c, tc in ctok.items():+ if c not in hide and len(tc) >= 3:+ families.setdefault(tc[:3], []).append(c)+ for group in families.values():+ if len(group) >= 5:+ group.sort(key=lambda c: -counts.get(c, 0))+ hide.update(c for c in group[2:] if c not in exempt)+ keep -= hide++ # Case twins and noun duplicates of the same property (hasOwner/HasOwner,+ # Title/hasTitle) render as distinct edges saying the same thing. One+ # survives in the view; the data keeps using whichever it used.+ used_props -= {p for p in used_props+ if p.startswith("Has") and "has" + p[3:] in used_props}+ used_props -= {p for p in used_props+ if not p.startswith(("has", "Has"))+ and ("has" + p in used_props or "Has" + p in used_props+ or "has" + p[0].upper() + p[1:] in used_props)}++ kept = []+ for s, t in stmts:+ if re.match(r"^:\w+\s+owl:disjointWith", t):+ other = re.search(r"owl:disjointWith\s+:(\w+)", t)+ if s in keep and other and other.group(1) in keep:+ kept.append(t)+ elif s in keep or s in used_props:+ kept.append(t)++ stats = {"classes": (len(decl_cls), len(keep & decl_cls)),+ "properties": (len(decl_props), len(used_props))}+ return PREFIXES + "\n" + "\n\n".join(kept), stats+++def sources_of(instances: list[pathlib.Path]) -> dict[tuple[str, str], set[str]]:+ """Which document asserted that a given individual has a given type.++ Provenance is what separates the two things a disjointness violation can+ mean. If one document types Culture07 as both a cell line and a doubling+ time, that is a broken extraction. If two documents disagree about which+ of two mutually exclusive classes it belongs to, that is the corpus+ contradicting itself, and it is the only kind worth showing anyone.++ Reported together they are indistinguishable, and the noisy kind buries+ the interesting kind: one over-typed registry produced 603 violations and+ not one of them was a disagreement.+ """+ out: dict[tuple[str, str], set[str]] = {}+ for f in instances:+ doc = f.stem.replace(".data", "")+ text = f.read_text()+ for m in re.finditer(r":(\w+)\s+(?:a|rdf:type)\s+([^;.]+)", text):+ for t in re.findall(r":(\w+)", m.group(2)):+ out.setdefault((m.group(1), t), set()).add(doc)+ return out+++def vocab_summary(ttl: str) -> str:+ """Vocabulary for the populate prompt, WITH definitions.++ A bare list of names invites the model to invent near-misses: given+ :CellLine it will happily write :CellLineType or :CellLineInRoutineUse+ because they sound plausible. Including each term's label and comment makes+ the existing vocabulary concrete enough to reuse instead of extend, which+ is what the closed-world check is measuring.+ """+ def described(kind: str) -> list[str]:+ out = []+ for name in sorted(set(re.findall(rf":(\w+)\s+a\s+owl:{kind}", ttl))):+ m = re.search(rf":{name}\s+a\s+owl:{kind}\b.*?rdfs:label\s+\"([^\"]+)\"", ttl, re.S)+ out.append(f" :{name}" + (f" ({m.group(1)})" if m else ""))+ return out++ return ("CLASSES (use these exact names, do not invent variants):\n"+ + "\n".join(described("Class"))+ + "\n\nOBJECT PROPERTIES:\n" + "\n".join(described("ObjectProperty"))+ + "\n\nDATATYPE PROPERTIES:\n" + "\n".join(described("DatatypeProperty")))+++REPAIR_PROMPT = """This Turtle uses terms that do not exist in the ontology. Every+term must come from the vocabulary below.++INVALID TERMS USED: {bad}++VOCABULARY:+{vocab}++Rewrite the Turtle so that every class and property is one of the vocabulary+terms above. Where an invalid term has a clear equivalent, map it. Where it does+not, DROP those statements rather than inventing a substitute. Keep everything+that was already valid unchanged.++Return ONLY Turtle, no prose and no fences.++TURTLE TO REPAIR:+{ttl}+"""+++def repair(ttl: str, bad: list[str], vocab: str) -> str | None:+ """One repair attempt against the named invalid terms.++ A closed-world check that only reports is half a control: it tells you the+ output is unusable without making it usable. Naming the specific offending+ terms and asking for a rewrite is cheap, and it converts a rejected graph+ into an accepted one without a human in the loop for the routine cases.+ """+ names = sorted({t.split("#")[-1] for t in bad})[:25]+ try:+ raw = call_model(REPAIR_PROMPT.format(bad=", ".join(names), vocab=vocab, ttl=ttl[:9000]), 3000)+ except Exception:+ return None+ out = normalise(raw)+ return out if len(out) > 200 else None+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--corpus", default=str(ROOT / "demo" / "corpus" / "dcat-us"))+ ap.add_argument("--workers", type=int, default=2,+ help="parallel model calls; local servers overload above 2")+ ap.add_argument("--cached", action="store_true")+ ap.add_argument("--reuse-derive", action="store_true",+ help="re-merge existing vocab fragments instead of re-deriving")+ ap.add_argument("--no-tokenise", action="store_true")+ args = ap.parse_args()++ corpus = pathlib.Path(args.corpus)+ out = ROOT / "demo" / "derived"+ out.mkdir(exist_ok=True)+ docs = sorted(corpus.glob("*.docx")) or sorted(corpus.glob("*.md"))+ if not docs:+ sys.exit(f"no documents in {corpus}")++ t0 = time.time()+ print(f"\n{BOLD}STAGE 1 READ{OFF} {len(docs)} documents from {corpus.name}/")++ # ---- Stage 2: tokenise -------------------------------------------------+ os.environ.setdefault("ONTO_VAULT_PATH", str(out / "_vault.json"))+ # Reading and tokenising dominate the wall clock of a cached build:+ # pandoc is a subprocess per document and the Presidio detector loads a+ # spaCy model on import, which costs seconds before any work happens.+ # Both are pure functions of the file, so cache by mtime and only build+ # the detector when at least one document actually changed.+ doc_cache_path = out / "_docs_cache.json"+ try:+ doc_cache = json.loads(doc_cache_path.read_text())+ except Exception:+ doc_cache = {}+ tokeniser = None+ prepared, total_tokens = [], 0+ dirty = False+ for d in docs:+ m = re.match(r"([A-Z]+-[0-9][0-9A-Za-z\-]*?)(?=-[a-z])", d.stem)+ docid = m.group(1) if m else d.stem+ key, mtime = str(d), d.stat().st_mtime+ hit = doc_cache.get(key)+ if not args.no_tokenise and hit and hit.get("mtime") == mtime:+ total_tokens += hit["count"]+ prepared.append((d, hit["text"], docid))+ continue+ text = docx_text(d)+ if args.no_tokenise:+ prepared.append((d, text, docid))+ continue+ if tokeniser is None:+ tokeniser = build_tokeniser()+ tok, n = tokeniser.tokenise(text)+ total_tokens += n+ prepared.append((d, tok, docid))+ doc_cache[key] = {"mtime": mtime, "text": tok, "count": n}+ dirty = True+ if tokeniser is not None:+ tokeniser.save()+ if dirty:+ doc_cache_path.write_text(json.dumps(doc_cache))+ if args.no_tokenise:+ print(f"\n{BOLD}STAGE 2 TOKENISE{OFF} {YEL}skipped (--no-tokenise){OFF}")+ else:+ print(f"\n{BOLD}STAGE 2 TOKENISE{OFF} {total_tokens} values replaced with stable tokens")+ desc = tokeniser.description if tokeniser is not None else \+ "cached detection results (documents unchanged)"+ print(f" {DIM}{desc}. No model sees a real value from here on.{OFF}")++ # ---- Stage 3: derive ---------------------------------------------------+ merged_path = out / "_ontology.ttl"+ if args.cached and merged_path.exists():+ print(f"\n{BOLD}STAGE 3 DERIVE{OFF} {DIM}cached{OFF}")+ elif args.reuse_derive and list(out.glob("*.vocab.ttl")):+ # Derivation is the expensive, stable half. Re-merging existing+ # fragments exercises the merge and everything after it without+ # paying for the model calls again.+ # Persist fragments in mended form: the align tools parse these+ # files as written, and a missing prefix makes a fragment invisible+ # to schema resolution even though the merge recovered it.+ for fp in sorted(out.glob("*.vocab.ttl")):+ fixed = rehead(fp.read_text())+ if is_valid_turtle(fixed)[0]:+ fp.write_text(fixed)+ fragments = [f.read_text() for f in sorted(out.glob("*.vocab.ttl"))]+ print(f"\n{BOLD}STAGE 3 DERIVE{OFF} reusing {len(fragments)} existing fragments")+ good, dropped = [], []+ for frag in fragments:+ ok, why = is_valid_turtle(rehead(frag))+ if ok:+ good.append(mend(body_of(frag)))+ else:+ dropped.append(why)+ if dropped:+ print(f" {YEL}{len(dropped)} fragment(s) dropped as unparseable{OFF}")+ for why in dropped[:4]:+ print(f" {DIM}{why}{OFF}")+ merged_path.write_text(PREFIXES + "\n" + STRUCTURAL + "\n" + "\n\n".join(b for b in good if b))+ ok, why = is_valid_turtle(merged_path.read_text())+ print(f" merged {len(good)} fragments, parses: {'yes' if ok else 'NO — ' + why}")+ else:+ print(f"\n{BOLD}STAGE 3 DERIVE{OFF} propose the vocabulary the documents imply")+ fragments = []+ with cf.ThreadPoolExecutor(max_workers=args.workers) as pool:+ for docid, ttl, err in pool.map(derive_one, prepared):+ if err:+ print(f" {RED}fail{OFF} {docid:14s} {err}")+ else:+ mended_ttl = rehead(ttl)+ (out / f"{docid}.vocab.ttl").write_text(+ mended_ttl if is_valid_turtle(mended_ttl)[0] else ttl)+ fragments.append(ttl)+ n = len(re.findall(r"a\s+owl:Class", ttl))+ print(f" {GRN}ok{OFF} {docid:14s} {n} classes proposed")+ # Merge: prefixes once, bodies concatenated. Duplicate declarations are+ # harmless in RDF; the store deduplicates identical triples.+ # Validate each fragment alone. One malformed fragment otherwise+ # truncates the merged file at its first bad statement, discarding+ # every valid fragment that follows it.+ good, dropped = [], []+ for frag in fragments:+ ok, why = is_valid_turtle(rehead(frag))+ if ok:+ good.append(mend(body_of(frag)))+ else:+ dropped.append(why)+ if dropped:+ print(f" {YEL}{len(dropped)} fragment(s) dropped as unparseable:{OFF}")+ for why in dropped[:3]:+ print(f" {DIM}{why}{OFF}")+ merged_path.write_text(PREFIXES + "\n" + STRUCTURAL + "\n" + "\n\n".join(b for b in good if b))+ ok, why = is_valid_turtle(merged_path.read_text())+ if not ok:+ print(f" {RED}merged ontology still does not parse: {why}{OFF}")++ doomed = _reconcile_ttl(merged_path)+ if doomed:+ print(f" {BOLD}reconciled:{OFF} removed {len(doomed)} attribute class(es) that "+ f"compete with a subclass partition: {', '.join(':' + d for d in doomed)}")++ check = engine(["clear", f"load {merged_path}", "stats"])+ try:+ st = json.loads(check.split("\n")[-1])["result"]+ print(f" {BOLD}derived ontology: {st['classes']} classes, {st['properties']} properties, "+ f"{st['triples']} triples{OFF}")+ except Exception:+ print(f" {YEL}derived ontology did not parse cleanly{OFF}")+ print(f" {DIM}{check[-200:]}{OFF}")++ n_disjoint = add_disjointness(merged_path)+ print(f" {BOLD}disjointness: {n_disjoint} mutually exclusive pairs asserted{OFF}"+ if n_disjoint else f" {YEL}disjointness: none asserted (nothing for the reasoner to catch){OFF}")++ vocab = vocab_summary(merged_path.read_text())++ # ---- Stage 4: populate -------------------------------------------------+ print(f"\n{BOLD}STAGE 4 POPULATE{OFF} extract facts, constrained to that vocabulary")+ instances = []+ if args.cached:+ instances = sorted(out.glob("*.data.ttl"))+ if not instances:+ print(f" {YEL}no cached graphs to replay. Tick 'Re-run extraction' to build them.{OFF}")+ sys.exit(2)+ print(f" {DIM}cached, {len(instances)} graphs{OFF}")+ else:+ jobs = [(d, t, i, vocab) for (d, t, i) in prepared]+ with cf.ThreadPoolExecutor(max_workers=args.workers) as pool:+ for docid, ttl, err in pool.map(populate_one, jobs):+ if err:+ print(f" {RED}fail{OFF} {docid:14s} {err}")+ else:+ f = out / f"{docid}.data.ttl"+ f.write_text(ttl)+ instances.append(f)+ print(f" {GRN}ok{OFF} {docid:14s} {f.name}")++ # ---- Stage 4b: refine --------------------------------------------------+ print(f"\n{BOLD}STAGE 4b REFINE{OFF} partition typing as closed-choice questions")+ texts = {docid: tokenised for (_p, tokenised, docid) in prepared}+ refined = refine(merged_path, instances, texts)+ if not refined:+ print(f" {DIM}nothing to refine{OFF}")++ # ---- Stage 5: verify ---------------------------------------------------+ print(f"\n{BOLD}STAGE 5 VERIFY{OFF} closed-world check against the DERIVED ontology")+ ok = bad = 0+ try:+ mcp("initialize", {"protocolVersion": "2025-03-26", "capabilities": {},+ "clientInfo": {"name": "otd", "version": "1"}})+ mcp("tools/call", {"name": "onto_load", "arguments":+ {"path": str(merged_path), "name": "derived", "force_recompile": True}})+ declared = set(re.findall(r":(\w+)\s+a\s+owl:", merged_path.read_text()))++ def check(ttl: str) -> dict:+ return json.loads(tool_text(mcp("tools/call", {+ "name": "onto_vocab_check",+ "arguments": {"data": ttl, "inline": True}})) or "{}")++ for f in instances:+ current = f.read_text()+ d = check(current)+ if d.get("conforms"):+ ok += 1+ print(f" {GRN}conforms{OFF} {f.stem}")+ continue+ if "error" in d:+ # A graph that does not parse is a different and worse failure+ # than one that uses the wrong terms. Reporting both as+ # "review" hid a broken graph behind an empty term list.+ bad += 1+ print(f" {RED}unparsed{OFF} {f.stem} {DIM}{d['error'][:90]}{OFF}")+ continue++ # Mechanical resolution first: token-equality renames, unique+ # superset renames, and has{Class} inversions need no model and+ # cannot hallucinate. The LLM repair rounds then only see what is+ # genuinely ambiguous.+ h = d.get("hallucinated_terms", [])+ start = len(h)+ mech = mech_resolve(current, h, merged_path.read_text(),+ doc=f.stem.replace(".data", ""))+ if mech != current and is_valid_turtle(mech)[0]:+ d = check(mech)+ if "error" not in d:+ current = mech+ f.write_text(current)+ h = d.get("hallucinated_terms", [])+ if d.get("conforms"):+ ok += 1+ print(f" {GRN}repaired{OFF} {f.stem} "+ f"{DIM}({start} terms resolved mechanically){OFF}")+ continue+ for _ in range(REPAIR_ROUNDS):+ fixed = repair(current, h, vocab)+ if not fixed:+ break+ fixed = snap_to_vocab(fixed, declared)+ d = check(fixed)+ if d.get("conforms"):+ f.write_text(fixed)+ current, h = fixed, []+ break+ nh = d.get("hallucinated_terms", h)+ if "error" in d or len(nh) >= len(h):+ break # not converging, stop paying for it+ current, h = fixed, nh+ if not h:+ ok += 1+ print(f" {GRN}repaired{OFF} {f.stem} {DIM}({start} invalid terms resolved){OFF}")+ else:+ f.write_text(current) # keep the partial improvement+ bad += 1+ print(f" {YEL}review {OFF} {f.stem} {DIM}{start} -> {len(h)}{OFF} "+ f"{[t.split('#')[-1] for t in h][:5]}")+ except Exception as e:+ print(f" {YEL}verification unavailable: {e}{OFF}")++ # ---- Stage 6: scan -----------------------------------------------------+ print(f"\n{BOLD}STAGE 6 SCAN{OFF} reason over the derived ontology and its instances")+ loads = ["clear", f"load {merged_path}"] + [f"load {f}" for f in instances]+ res = engine(loads + ["stats"])+ try:+ st = json.loads(res.split("\n")[-1])["result"]+ print(f" merged: {st['triples']} triples, {st['classes']} classes, {st['individuals']} individuals")+ except Exception:+ pass++ q = ("PREFIX owl: <http://www.w3.org/2002/07/owl#> "+ "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> "+ "SELECT DISTINCT ?subject ?a ?b WHERE { ?subject a ?a, ?b . "+ "FILTER(STR(?a) < STR(?b)) ?a rdfs:subClassOf* ?da . ?b rdfs:subClassOf* ?db . "+ "{ ?da owl:disjointWith ?db } UNION { ?db owl:disjointWith ?da } }")+ res = engine(loads + [f"query {json.dumps(q)}"])+ try:+ rows = json.loads(res.split("\n")[-1])["result"]["results"]+ except Exception:+ rows = []+ src = sources_of(instances)+ disagreements, defects = [], []+ for r in rows:+ v = {k: x.split("#")[-1].rstrip(">") for k, x in r.items()}+ da = src.get((v["subject"], v["a"]), set())+ db = src.get((v["subject"], v["b"]), set())+ # A disagreement needs a document that asserts one side and does not+ # assert the other. Both types coming from the same single document is+ # one extractor being wrong, not two sources in conflict.+ (disagreements if (da - db) and (db - da) else defects).append((v, da, db))++ for v, da, db in disagreements:+ print(f" {RED}CONTRADICTION{OFF} {v['subject']} is {v['a']} per "+ f"{','.join(sorted(da - db))} but {v['b']} per {','.join(sorted(db - da))}")+ if defects:+ by_doc: dict[str, int] = {}+ for _, da, db in defects:+ for d in da | db:+ by_doc[d] = by_doc.get(d, 0) + 1+ worst = sorted(by_doc.items(), key=lambda x: -x[1])[:3]+ print(f" {YEL}{len(defects)} violations within a single document{OFF} "+ f"{DIM}(over-typing, not disagreement: "+ f"{', '.join(f'{d} {n}' for d, n in worst)}){OFF}")+ if not rows:+ print(f" {DIM}no disjointness violations{OFF}")++ # ---- Stage 7: load -----------------------------------------------------+ # Everything above ran against throwaway CLI stores. The Studio graph+ # view and the chat's GraphRAG read the LIVE engine, so the session ends+ # by loading the merged ontology plus every instance graph there. Without+ # this the app renders vocabulary with zero individuals, which looks like+ # extraction never happened.+ print(f"\n{BOLD}STAGE 7 LOAD{OFF} push the result into the live engine "+ f"for the graph view and chat")+ store = merged_path.parent / "_store.ttl"+ data = "\n\n".join(body_of(f.read_text()) for f in instances)+ pruned, pstats = prune_to_used(merged_path.read_text(), data)+ c0, c1 = pstats["classes"]+ p0, p1 = pstats["properties"]+ print(f" pruned to the vocabulary the data uses: "+ f"{c0} -> {c1} classes, {p0} -> {p1} properties "+ f"{DIM}(full vocabulary stays on disk for extraction){OFF}")+ # Typing claims with provenance, in the shape GraphRAG retrieves: the+ # chat's own suggested question is "which documents disagree", and+ # without these the store knows THAT a conflict exists but not who says+ # what. One claim node per specific typing per document.+ claims, ci = [], 0+ for f in instances:+ doc = f.stem.replace(".data", "")+ for m in re.finditer(r":(\w+)\s+a\s+:(\w+)", f.read_text()):+ subj, cls = m.groups()+ if cls in ("Document", "Section"):+ continue+ ci += 1+ claims.append(+ f":CLAIM_{ci} a :Claim ;\n"+ f" :claimText \"{doc} types {subj} as {cls}\" ;\n"+ f" :aboutEntity :{subj} ;\n"+ f" :statedIn :DOC_{doc.replace('-', '_')} .\n"+ f":{subj} <http://www.w3.org/ns/prov#wasDerivedFrom> :DOC_{doc.replace('-', '_')} .\n"+ f":DOC_{doc.replace('-', '_')} rdfs:label \"{doc}\" .")+ store.write_text(pruned + "\n" + data + "\n" + "\n".join(claims))+ store_ok, why = is_valid_turtle(store.read_text())+ if not store_ok:+ # A parse error truncates the load at the bad line and the engine+ # reports the partial store without complaint: the graph view then+ # quietly shows a fraction of the data.+ print(f" {YEL}store has a syntax defect; the view may be partial: {why}{OFF}")+ try:+ mcp("tools/call", {"name": "onto_load", "arguments":+ {"path": str(store), "name": "derived", "force_recompile": True}})+ st = json.loads(tool_text(mcp("tools/call",+ {"name": "onto_stats", "arguments": {}})) or "{}")+ print(f" live store: {st.get('triples', '?')} triples, "+ f"{st.get('classes', '?')} classes, {st.get('individuals', '?')} individuals")+ except Exception as e:+ print(f" {YEL}live engine not reachable ({e}); graph view will be stale{OFF}")++ print(f"\n{BOLD}{len(docs)} documents in {time.time()-t0:.0f}s. "+ f"{ok} conformed, {bad} for review, "+ f"{len(disagreements)} contradictions across documents.{OFF}")+ if not args.no_tokenise:+ print(f"{DIM}Every value the model saw was a token. Detokenisation requires the "+ f"vault and happens only at render time.{OFF}\n")+++if __name__ == "__main__":+ try:+ main()+ except SystemExit:+ raise+ except Exception:+ # The Studio pane only shows "finished with errors" plus whatever was+ # printed. A crash with the traceback on stdout is diagnosable from+ # the pane; a bare non-zero exit is not.+ import traceback+ print("PIPELINE CRASH:")+ traceback.print_exc(file=sys.stdout)+ sys.exit(1)diff --git a/demo/requirements.txt b/demo/requirements.txtnew file mode 100644index 00000000..8177212d--- /dev/null+++ b/demo/requirements.txt@@ -0,0 +1,3 @@+rdflib>=7.0+presidio-analyzer>=2.2+pytest>=8.0diff --git a/demo/tests/test_reconcile.py b/demo/tests/test_reconcile.pynew file mode 100644index 00000000..f93d5200--- /dev/null+++ b/demo/tests/test_reconcile.py@@ -0,0 +1,28 @@+from rdflib import Graph, Namespace, RDFS, OWL, RDF++from demo.ontology_from_docs import reconcile++EX = Namespace("https://example.org/t#")+++def _graph(pairs):+ g = Graph()+ for sub, parent in pairs:+ g.add((sub, RDF.type, OWL.Class))+ g.add((sub, RDFS.subClassOf, parent))+ return g+++def test_attribute_class_is_removed_when_a_partition_exists():+ g = _graph([(EX.PublishedDataset, EX.Dataset), (EX.DraftDataset, EX.Dataset)])+ g.add((EX.DatasetType, RDF.type, OWL.Class))+ out = reconcile(g)+ assert (EX.DatasetType, RDF.type, OWL.Class) not in out+ assert (EX.PublishedDataset, RDFS.subClassOf, EX.Dataset) in out+++def test_status_is_spared_because_states_are_attributes():+ g = _graph([(EX.ActiveThing, EX.Thing), (EX.RetiredThing, EX.Thing)])+ g.add((EX.ThingStatus, RDF.type, OWL.Class))+ out = reconcile(g)+ assert (EX.ThingStatus, RDF.type, OWL.Class) in outdiff --git a/demo/tokenisation.py b/demo/tokenisation.pynew file mode 100644index 00000000..4cb72d1c--- /dev/null+++ b/demo/tokenisation.py@@ -0,0 +1,423 @@+#!/usr/bin/env python3+"""+Swappable tokenisation: detection and custody are separate concerns.++The market splits along exactly this seam, so the code should too:++ DETECTION finding the sensitive spans in free text.+ Regex (demo) | Microsoft Presidio (MIT, local) | a vendor's own.++ CUSTODY holding the mapping and controlling reversal.+ Local JSON (demo only) | Skyflow (managed) | Databunker (self-hosted).++Skyflow sells both halves in one product. Open source makes you assemble them,+which is fine as long as the assembly is a config choice rather than a rewrite.++ ONTO_DETECTOR = regex | presidio | both+ ONTO_VAULT = local | skyflow | databunker++Everything downstream depends only on `Tokeniser`, so swapping either half+changes nothing in the pipeline. That is the point: the tokeniser is an+interchangeable component, not an architectural commitment.++The one invariant every implementation must hold: **tokens are deterministic**.+The same value must produce the same token in every document, because that is+what lets a token double as a join key for exact-match entity resolution+without any component handling the raw value.+"""++from __future__ import annotations++import hashlib+import hmac+import json+import os+import pathlib+import re+from dataclasses import dataclass+from typing import Protocol+++@dataclass(frozen=True)+class Span:+ """A detected sensitive span: character offsets plus what it looks like."""+ start: int+ end: int+ kind: str+ text: str+++# --------------------------------------------------------------------------+# Detection+# --------------------------------------------------------------------------++class Detector(Protocol):+ name: str++ def detect(self, text: str) -> list[Span]:+ ...+++class RegexDetector:+ """Pattern matching. Demonstration only.++ Honest about its limits: this finds email addresses, phone-shaped strings,+ titled names and identifier-shaped tokens. It will miss bare personal names,+ addresses, dates of birth, national identifiers, and anything not in+ English. Do not present this as PII detection.+ """++ name = "regex"++ PATTERNS = [+ ("EMAIL", re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.]{2,}\b")),+ ("PHONE", re.compile(r"\b(?:\+\d{1,3}[ -]?)?(?:\(?\d{3,5}\)?[ -]?){2,3}\d{3,4}\b")),+ ("PERSON", re.compile(r"\b(?:Dr|Prof|Mr|Mrs|Ms)\.? [A-Z][a-z]+ [A-Z][a-z]+\b")),+ ("ID", re.compile(r"\b[A-Z]{2,4}-\d{4,}\b")),+ ]++ def detect(self, text: str) -> list[Span]:+ spans: list[Span] = []+ for kind, pattern in self.PATTERNS:+ for m in pattern.finditer(text):+ spans.append(Span(m.start(), m.end(), kind, m.group(0)))+ return _dedupe(spans)+++class PresidioDetector:+ """Microsoft Presidio. MIT licensed, runs locally, no egress.++ This is the component that makes tokenisation real rather than illustrative:+ it recognises a broad set of entity types across locales, which is precisely+ where pattern matching fails.++ Install: pip install presidio-analyzer && python -m spacy download en_core_web_lg+ """++ name = "presidio"++ DEFAULT_ENTITIES = [+ "PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "LOCATION",+ "DATE_TIME", "NRP", "CREDIT_CARD", "IBAN_CODE", "IP_ADDRESS",+ "MEDICAL_LICENSE", "UK_NHS", "US_SSN",+ ]++ def __init__(self, entities: list[str] | None = None, language: str = "en"):+ try:+ from presidio_analyzer import AnalyzerEngine # type: ignore+ except ImportError as e:+ raise RuntimeError(+ "presidio-analyzer is not installed. "+ "pip install presidio-analyzer && python -m spacy download en_core_web_lg"+ ) from e+ self._engine = AnalyzerEngine()+ self.entities = entities or self.DEFAULT_ENTITIES+ self.language = language++ def detect(self, text: str) -> list[Span]:+ results = self._engine.analyze(text=text, entities=self.entities, language=self.language)+ return _dedupe(+ [Span(r.start, r.end, r.entity_type, text[r.start:r.end]) for r in results]+ )+++class CompositeDetector:+ """Run several detectors and merge their spans.++ Neither half is sufficient alone, which the test output makes obvious:+ Presidio finds names, dates and locations that no pattern will catch, and+ misses domain identifiers like batch 20481 because it has no reason to know+ they are sensitive. Regex catches those and nothing else.++ Overlaps resolve to the longer span, so a Presidio PERSON covering a name+ wins over a narrower pattern hit on part of it.+ """++ def __init__(self, *detectors: Detector):+ self._detectors = [d for d in detectors if d is not None]+ self.name = "+".join(d.name for d in self._detectors)++ def detect(self, text: str) -> list[Span]:+ spans: list[Span] = []+ for d in self._detectors:+ spans.extend(d.detect(text))+ return _dedupe(spans)+++def _dedupe(spans: list[Span]) -> list[Span]:+ """Drop overlaps, preferring the longer match, and return in reverse order.++ Reverse order matters: replacements are applied back-to-front so earlier+ offsets stay valid as the string is rewritten.+ """+ ordered = sorted(spans, key=lambda s: (s.start, -(s.end - s.start)))+ kept: list[Span] = []+ last_end = -1+ for s in ordered:+ if s.start >= last_end:+ kept.append(s)+ last_end = s.end+ return sorted(kept, key=lambda s: s.start, reverse=True)+++# --------------------------------------------------------------------------+# Custody+# --------------------------------------------------------------------------++class VaultBackend(Protocol):+ name: str++ def tokenise(self, kind: str, value: str) -> str:+ ...++ def detokenise(self, token: str) -> str | None:+ ...++ def save(self) -> None:+ ...+++class LocalVault:+ """Deterministic HMAC tokens with a JSON store. DEMONSTRATION ONLY.++ Two properties are real: tokens are deterministic, and reversal needs the+ store. Everything else that makes a vault a security control is absent.++ Be explicit about the risk, because it is the opposite of obvious: the+ store is a plaintext file mapping every token back to its original value.+ If it leaks it is a single artefact containing all the sensitive data in+ the corpus. That is worse than not tokenising, because it looks like a+ control while concentrating the exposure. Never ship this.+ """++ name = "local"++ def __init__(self, key: bytes | None = None, path: pathlib.Path | None = None):+ self.key = key or os.environ.get("ONTO_VAULT_KEY", "demo-key-not-for-production").encode()+ self.path = path or pathlib.Path(os.environ.get("ONTO_VAULT_PATH", "demo/derived/_vault.json"))+ self.map: dict[str, str] = {}+ if self.path.exists():+ try:+ self.map = json.loads(self.path.read_text())+ except Exception:+ self.map = {}++ def tokenise(self, kind: str, value: str) -> str:+ digest = hmac.new(self.key, value.encode(), hashlib.sha256).hexdigest()[:12]+ token = f"TOK_{kind}_{digest}"+ self.map[token] = value+ return token++ def detokenise(self, token: str) -> str | None:+ return self.map.get(token)++ def save(self) -> None:+ self.path.parent.mkdir(parents=True, exist_ok=True)+ self.path.write_text(json.dumps(self.map, indent=1))+++class SkyflowVault:+ """Skyflow managed vault.++ Skyflow has no open-source edition; this speaks to the hosted API. Custody,+ rotation, access policy and audit are the vendor's, which is the reason to+ use it over anything here.++ Requires SKYFLOW_VAULT_URL, SKYFLOW_VAULT_ID and SKYFLOW_BEARER_TOKEN.+ Determinism must be configured on the vault's column policy: a vault issuing+ random tokens per occurrence breaks the join-key property this pipeline+ relies on. Confirm that before relying on cross-document resolution.+ """++ name = "skyflow"++ def __init__(self):+ self.url = os.environ.get("SKYFLOW_VAULT_URL", "").rstrip("/")+ self.vault_id = os.environ.get("SKYFLOW_VAULT_ID", "")+ self.token = os.environ.get("SKYFLOW_BEARER_TOKEN", "")+ self.table = os.environ.get("SKYFLOW_TABLE", "pii")+ self.column = os.environ.get("SKYFLOW_COLUMN", "value")+ missing = [n for n, v in (("SKYFLOW_VAULT_URL", self.url),+ ("SKYFLOW_VAULT_ID", self.vault_id),+ ("SKYFLOW_BEARER_TOKEN", self.token)) if not v]+ if missing:+ raise RuntimeError(f"skyflow vault needs: {', '.join(missing)}")+ self._cache: dict[str, str] = {}++ def _request(self, path: str, payload: dict) -> dict:+ import urllib.request+ req = urllib.request.Request(+ f"{self.url}/v1/vaults/{self.vault_id}/{path}",+ data=json.dumps(payload).encode(),+ headers={"Content-Type": "application/json",+ "Authorization": f"Bearer {self.token}"},+ )+ import urllib.error+ try:+ return json.load(__import__("urllib.request", fromlist=["x"]).urlopen(req, timeout=60))+ except urllib.error.HTTPError as e:+ raise RuntimeError(f"skyflow {path} failed: {e.code} {e.read()[:200]!r}") from e++ def tokenise(self, kind: str, value: str) -> str:+ if value in self._cache:+ return self._cache[value]+ res = self._request(self.table, {"records": [{"fields": {self.column: value}}],+ "tokenization": True})+ try:+ token = res["records"][0]["tokens"][self.column]+ except (KeyError, IndexError) as e:+ raise RuntimeError(f"unexpected skyflow response shape: {res}") from e+ self._cache[value] = token+ return token++ def detokenise(self, token: str) -> str | None:+ res = self._request("detokenize", {"detokenizationParameters": [{"token": token}]})+ try:+ return res["records"][0]["value"]+ except (KeyError, IndexError):+ return None++ def save(self) -> None:+ """Nothing to persist: custody is the vendor's."""+++class DatabunkerVault:+ """Databunker, self-hosted and open source.++ The middle option: real custody, in your own infrastructure, with no extra+ processor on the data map. Requires DATABUNKER_URL and DATABUNKER_TOKEN.+ """++ name = "databunker"++ def __init__(self):+ self.url = os.environ.get("DATABUNKER_URL", "").rstrip("/")+ self.token = os.environ.get("DATABUNKER_TOKEN", "")+ if not self.url or not self.token:+ raise RuntimeError("databunker vault needs DATABUNKER_URL and DATABUNKER_TOKEN")+ self._cache: dict[str, str] = {}++ def _post(self, path: str, payload: dict) -> dict:+ import urllib.request+ req = urllib.request.Request(+ f"{self.url}/v1/{path}", data=json.dumps(payload).encode(),+ headers={"Content-Type": "application/json", "X-Bunker-Token": self.token})+ return json.load(urllib.request.urlopen(req, timeout=60))++ def tokenise(self, kind: str, value: str) -> str:+ if value in self._cache:+ return self._cache[value]+ res = self._post("token", {"tokentype": kind.lower(), "record": value})+ token = res.get("token") or res.get("data", {}).get("token", "")+ if not token:+ raise RuntimeError(f"unexpected databunker response: {res}")+ self._cache[value] = token+ return token++ def detokenise(self, token: str) -> str | None:+ try:+ return self._post("token/get", {"token": token}).get("record")+ except Exception:+ return None++ def save(self) -> None:+ """Nothing to persist: custody is the server's."""+++# --------------------------------------------------------------------------+# Composition+# --------------------------------------------------------------------------++class Tokeniser:+ """A detector plus a vault. This is the only type the pipeline knows about."""++ def __init__(self, detector: Detector, vault: VaultBackend):+ self.detector = detector+ self.vault = vault++ @property+ def description(self) -> str:+ return f"{self.detector.name} detection + {self.vault.name} custody"++ def tokenise(self, text: str) -> tuple[str, int]:+ """Replace every detected span with a stable token. Returns (text, count)."""+ spans = self.detector.detect(text) # reverse order, so offsets stay valid+ for s in spans:+ token = self.vault.tokenise(s.kind, s.text)+ text = text[: s.start] + token + text[s.end:]+ return text, len(spans)++ def detokenise(self, text: str) -> str:+ """Reverse. In production this is entitlement-gated and audited."""+ for token in sorted(set(re.findall(r"TOK_[A-Z_]+_[0-9a-f]{6,}|tok_[\w-]{8,}", text)),+ key=len, reverse=True):+ value = self.vault.detokenise(token)+ if value is not None:+ text = text.replace(token, value)+ return text++ def save(self) -> None:+ self.vault.save()+++def build(detector: str | None = None, vault: str | None = None) -> Tokeniser:+ """Construct from configuration, falling back loudly rather than silently.++ A tokeniser that silently degrades to weaker detection is worse than one+ that fails, because the pipeline keeps running and the output looks fine.+ """+ # Default to the strongest detection available. `both` already degrades to+ # patterns with a warning when Presidio is absent, so defaulting to `regex`+ # bought nothing and cost real coverage: a run inheriting the default found+ # 1 sensitive value where the composite found 24, with nothing in the+ # output to indicate that detection had been weakened.+ d_name = (detector or os.environ.get("ONTO_DETECTOR", "both")).lower()+ v_name = (vault or os.environ.get("ONTO_VAULT", "local")).lower()++ def _composite():+ """Presidio plus patterns, degrading loudly if Presidio is absent.++ For the composite the run is still worth completing without Presidio,+ but silence would be dishonest: the log has to say detection was+ weakened, because the output looks identical either way. An explicit+ ONTO_DETECTOR=presidio still fails hard.+ """+ try:+ return CompositeDetector(PresidioDetector(), RegexDetector())+ except RuntimeError as e:+ import sys as _sys+ print(f"WARNING: Presidio unavailable, falling back to patterns only. "+ f"Detection is materially weaker. ({e})", file=_sys.stderr)+ return RegexDetector()++ detectors = {"regex": RegexDetector, "presidio": PresidioDetector, "both": _composite}+ vaults = {"local": LocalVault, "skyflow": SkyflowVault, "databunker": DatabunkerVault}++ if d_name not in detectors:+ raise SystemExit(f"unknown ONTO_DETECTOR={d_name}. options: {', '.join(detectors)}")+ if v_name not in vaults:+ raise SystemExit(f"unknown ONTO_VAULT={v_name}. options: {', '.join(vaults)}")++ return Tokeniser(detectors[d_name](), vaults[v_name]())+++if __name__ == "__main__":+ import sys++ piped = "" if sys.stdin.isatty() else sys.stdin.read()+ sample = piped.strip() or (+ "Dr Alice Morgan ([email protected], +44 20 7946 0958) reviewed "+ "batch 20481 on 3 March. Contact Prof Brian Hale for the audit."+ )+ t = build()+ print(f"tokeniser: {t.description}\n")+ out, n = t.tokenise(sample)+ print(f"--- tokenised ({n} spans) ---\n{out}\n")+ print(f"--- detokenised (round trip) ---\n{t.detokenise(out)}\n")+ a, _ = t.tokenise("contact [email protected]")+ b, _ = t.tokenise("write to [email protected]")+ tok_a = re.search(r"TOK_\w+", a)+ tok_b = re.search(r"TOK_\w+", b)+ same = tok_a and tok_b and tok_a.group(0) == tok_b.group(0)+ print(f"determinism (same value -> same token across documents): {'YES' if same else 'NO'}")+ t.save()diff --git a/demo/verify.py b/demo/verify.pynew file mode 100644index 00000000..4ffef134--- /dev/null+++ b/demo/verify.py@@ -0,0 +1,282 @@+#!/usr/bin/env python3+"""+Standalone verifier for the derived ontology and the graphs extracted against it.++This is stage 5 and 6 of the pipeline with nothing else attached: no model+calls, no extraction, no repair. It answers one question, repeatably and in+seconds: given what is on disk right now, does it hold up?++Six checks, in the order the repo's own workflow prescribes:++ 1. PARSE every fragment and the merged ontology are syntactically valid+ 2. STATS the merged ontology loads with the class and property counts claimed+ 3. LINT missing labels, comments, domains, ranges+ 4. ENFORCE design-pattern compliance against the generic rule pack+ 5. VOCAB closed-world check of every extracted graph against the ontology+ 6. REASON disjointness violations across the merged instance data++Exit code is the number of failed checks, so this can gate a build.+"""++from __future__ import annotations++import json+import pathlib+import sys++sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))+from ontology_from_docs import (mcp, tool_text, engine, rehead, is_valid_turtle, # noqa: E402+ sources_of, ROOT)++BOLD, DIM, RED, GRN, YEL, CYA, OFF = (+ "\033[1m", "\033[2m", "\033[31m", "\033[32m", "\033[33m", "\033[36m", "\033[0m")++DERIVED = ROOT / "demo" / "derived"+ONTOLOGY = DERIVED / "_ontology.ttl"+++def head(n: int, title: str, sub: str) -> None:+ print(f"\n{BOLD}CHECK {n} {title}{OFF} {DIM}{sub}{OFF}")+++def main() -> int:+ if not ONTOLOGY.exists():+ print(f"{RED}no derived ontology at {ONTOLOGY}{OFF}")+ return 1++ graphs = sorted(DERIVED.glob("*.data.ttl"))+ fragments = sorted(DERIVED.glob("*.vocab.ttl"))+ failures = 0++ print(f"{BOLD}Verifier{OFF} {DIM}{ONTOLOGY.relative_to(ROOT)}, "+ f"{len(fragments)} fragments, {len(graphs)} extracted graphs{OFF}")++ # ---- 1. parse ----------------------------------------------------------+ head(1, "PARSE", "syntax of every fragment and the merged ontology")+ # Judged as the merge judges them: under the canonical prefix header and+ # after the mechanical repairs. Validating the raw file instead reports a+ # missing prefix declaration as a modelling failure, which it is not.+ bad_fragments, mended = [], []+ for f in fragments:+ raw = f.read_text()+ if is_valid_turtle(raw)[0]:+ continue+ if is_valid_turtle(rehead(raw))[0]:+ mended.append(f.stem)+ else:+ bad_fragments.append((f.stem, is_valid_turtle(rehead(raw))[1]))+ if mended:+ print(f" {DIM}{len(mended)} fragment(s) recovered by reheading and mending: "+ f"{', '.join(m.replace('.vocab', '') for m in mended)}{OFF}")+ if bad_fragments:+ print(f" {YEL}{len(bad_fragments)} of {len(fragments)} fragments do not parse "+ f"and are excluded from the merge{OFF}")+ for name, why in bad_fragments:+ print(f" {DIM}{name} {why[:110]}{OFF}")+ else:+ print(f" {GRN}all {len(fragments)} fragments usable{OFF}")++ res = engine(["clear", f"load {ONTOLOGY}"])+ if '"error"' in res:+ print(f" {RED}merged ontology does not parse{OFF}")+ print(f" {DIM}{res[-220:]}{OFF}")+ failures += 1+ else:+ print(f" {GRN}merged ontology parses{OFF}")++ # ---- 2. stats ----------------------------------------------------------+ head(2, "STATS", "does the ontology load with the structure it claims")+ st = {}+ try:+ st = json.loads(engine(["clear", f"load {ONTOLOGY}", "stats"]).split("\n")[-1])["result"]+ except Exception:+ pass+ classes, props = st.get("classes", 0), st.get("properties", 0)+ print(f" {classes} classes, {props} properties "+ f"({st.get('object_properties', 0)} object, {st.get('data_properties', 0)} data), "+ f"{st.get('triples', 0)} triples")+ # A vocabulary with classes and no properties is the failure mode that a+ # truncated parse produces, and it silently poisons everything downstream.+ if classes and not props:+ print(f" {RED}classes but no properties: the parse is truncated{OFF}")+ failures += 1+ elif classes and props:+ print(f" {GRN}structure present{OFF}")++ q_disjoint = ("PREFIX owl: <http://www.w3.org/2002/07/owl#> "+ "SELECT (COUNT(*) AS ?n) WHERE { ?a owl:disjointWith ?b }")+ try:+ rows = json.loads(engine(["clear", f"load {ONTOLOGY}",+ f"query {json.dumps(q_disjoint)}"]).split("\n")[-1])["result"]["results"]+ n = rows[0].get("n", "0").split('"')[1] if rows else "0"+ print(f" {n} disjointness axioms {DIM}(what gives the reasoner something to catch){OFF}")+ except Exception:+ pass++ # Competing modelling patterns: a subclass partition and an attribute+ # class for the same notion. The extractor reliably picks the attribute+ # class, which is the one form disjointness cannot reach, so leaving both+ # in the vocabulary quietly disables contradiction detection. None of the+ # engine's enforce packs catches this; the pipeline's reconcile() removes+ # them, and this check keeps it removed.+ import re as _re+ ttl_text = ONTOLOGY.read_text()+ cls = set(_re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", ttl_text))+ kids: dict[str, set[str]] = {}+ for m in _re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", ttl_text):+ kids.setdefault(m.group(2), set()).add(m.group(1))+ from ontology_from_docs import KIND_SUFFIXES+ collisions = sorted({p + s for p, k in kids.items() if len(k) >= 2+ for s in KIND_SUFFIXES if p + s in cls})+ if collisions:+ print(f" {RED}{len(collisions)} attribute class(es) compete with a subclass "+ f"partition: {', '.join(':' + c for c in collisions)}{OFF}")+ print(f" {DIM}the extractor will use these instead of the partition, "+ f"and disjointness cannot reach them{OFF}")+ failures += 1+ else:+ print(f" {GRN}no competing modelling patterns{OFF}")++ # ---- MCP session for the tool-backed checks ----------------------------+ try:+ mcp("initialize", {"protocolVersion": "2025-03-26", "capabilities": {},+ "clientInfo": {"name": "verify", "version": "1"}})+ mcp("tools/call", {"name": "onto_load", "arguments":+ {"path": str(ONTOLOGY), "name": "derived", "force_recompile": True}})+ except Exception as e:+ print(f"\n{RED}engine unreachable: {e}{OFF}")+ print(f"{DIM}start it with: ./target/release/open-ontologies serve --port 8137{OFF}")+ return failures + 1++ # ---- 3. lint -----------------------------------------------------------+ head(3, "LINT", "labels, comments, domains, ranges")+ try:+ d = json.loads(tool_text(mcp("tools/call", {"name": "onto_lint", "arguments": {}})) or "{}")+ issues = d.get("issues", [])+ by_rule: dict[str, int] = {}+ for i in issues:+ by_rule[i.get("rule", i.get("kind", "?"))] = by_rule.get(i.get("rule", i.get("kind", "?")), 0) + 1+ if issues:+ print(f" {YEL}{len(issues)} issues{OFF}")+ for rule, n in sorted(by_rule.items(), key=lambda x: -x[1])[:8]:+ print(f" {n:4d} {rule}")+ else:+ print(f" {GRN}clean{OFF}")+ except Exception as e:+ print(f" {DIM}lint unavailable: {e}{OFF}")++ # ---- 4. enforce --------------------------------------------------------+ head(4, "ENFORCE", "design patterns, generic rule pack")+ try:+ d = json.loads(tool_text(mcp("tools/call", {"name": "onto_enforce",+ "arguments": {"rules": "generic"}})) or "{}")+ v = d.get("violations", [])+ if v:+ by_rule = {}+ for i in v:+ key = i.get("rule", "?")+ by_rule[key] = by_rule.get(key, 0) + 1+ print(f" {YEL}{len(v)} violations{OFF}")+ for rule, n in sorted(by_rule.items(), key=lambda x: -x[1])[:8]:+ print(f" {n:4d} {rule}")+ else:+ print(f" {GRN}no violations{OFF}")+ except Exception as e:+ print(f" {DIM}enforce unavailable: {e}{OFF}")++ # ---- 5. vocab check ----------------------------------------------------+ head(5, "VOCAB", "closed-world check of every extracted graph")+ conform, review = 0, []+ for f in graphs:+ try:+ d = json.loads(tool_text(mcp("tools/call", {"name": "onto_vocab_check",+ "arguments": {"data": f.read_text(),+ "inline": True}})) or "{}")+ except Exception as e:+ print(f" {RED}error{OFF} {f.stem} {e}")+ continue+ if d.get("conforms"):+ conform += 1+ print(f" {GRN}conforms{OFF} {f.stem}")+ else:+ terms = [t.split("#")[-1] for t in d.get("hallucinated_terms", [])]+ review.append((f.stem, terms))+ print(f" {YEL}review {OFF} {f.stem} {DIM}{len(terms)} undeclared: "+ f"{', '.join(terms[:5])}{'…' if len(terms) > 5 else ''}{OFF}")+ if graphs:+ pct = 100 * conform / len(graphs)+ print(f" {BOLD}{conform}/{len(graphs)} conform ({pct:.0f}%){OFF}")+ if conform == 0:+ failures += 1++ # ---- 6. reason ---------------------------------------------------------+ head(6, "REASON", "disjointness violations across the merged instance data")+ loads = ["clear", f"load {ONTOLOGY}"] + [f"load {g}" for g in graphs]+ try:+ st = json.loads(engine(loads + ["stats"]).split("\n")[-1])["result"]+ print(f" merged: {st['triples']} triples, {st['classes']} classes, "+ f"{st['individuals']} individuals")+ except Exception:+ pass++ q = ("PREFIX owl: <http://www.w3.org/2002/07/owl#> "+ "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> "+ "SELECT DISTINCT ?subject ?a ?b WHERE { ?subject a ?a, ?b . "+ "FILTER(STR(?a) < STR(?b)) ?a rdfs:subClassOf* ?da . ?b rdfs:subClassOf* ?db . "+ "{ ?da owl:disjointWith ?db } UNION { ?db owl:disjointWith ?da } }")+ try:+ rows = json.loads(engine(loads + [f"query {json.dumps(q)}"]).split("\n")[-1])["result"]["results"]+ except Exception:+ rows = []+ # Split by provenance. A violation is only a contradiction when one+ # document asserts one side and does not assert the other; both sides from+ # a single document is one extractor over-typing, and it can produce+ # hundreds of violations that mean nothing.+ src = sources_of(graphs)+ contradictions, overtyping = [], []+ for r in rows:+ v = {k: val.split("#")[-1].rstrip(">") for k, val in r.items()}+ da = src.get((v["subject"], v["a"]), set())+ db = src.get((v["subject"], v["b"]), set())+ (contradictions if (da - db) and (db - da) else overtyping).append((v, da, db))++ for v, da, db in contradictions:+ print(f" {RED}CONTRADICTION{OFF} {v['subject']} is {v['a']} per "+ f"{','.join(sorted(da - db))} but {v['b']} per {','.join(sorted(db - da))}")+ if overtyping:+ by_doc: dict[str, int] = {}+ for _, da, db in overtyping:+ for d in da | db:+ by_doc[d] = by_doc.get(d, 0) + 1+ worst = ", ".join(f"{d} {n}" for d, n in sorted(by_doc.items(), key=lambda x: -x[1])[:3])+ print(f" {YEL}{len(overtyping)} violations within a single document{OFF} "+ f"{DIM}(over-typing, not disagreement: {worst}){OFF}")+ if not rows:+ print(f" {DIM}none{OFF}")++ # A contradiction can only be found if the same real-world thing carries+ # both types. Report how far the data is from that precondition, because+ # "no conflicts" and "nothing could ever conflict" look identical otherwise.+ q_multi = ("SELECT (COUNT(DISTINCT ?s) AS ?n) WHERE "+ "{ ?s a ?t1, ?t2 . FILTER(STR(?t1) < STR(?t2)) }")+ try:+ rows2 = json.loads(engine(loads + [f"query {json.dumps(q_multi)}"]).split("\n")[-1])["result"]["results"]+ n = rows2[0].get("n", "0").split('"')[1] if rows2 else "0"+ if n == "0":+ print(f" {YEL}0 individuals carry more than one type, so no contradiction is "+ f"reachable regardless of the axioms{OFF}")+ print(f" {DIM}fix is entity resolution: the same thing must get one IRI "+ f"across documents{OFF}")+ else:+ print(f" {DIM}{n} individuals carry more than one type{OFF}")+ except Exception:+ pass++ print(f"\n{BOLD}{'PASS' if not failures else str(failures) + ' CHECK(S) FAILED'}{OFF}"+ f" {DIM}{conform}/{len(graphs)} graphs conform, "+ f"{len(contradictions)} contradictions across documents{OFF}\n")+ return failures+++if __name__ == "__main__":+ sys.exit(main())