Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

feat(demo): add the document-to-ontology pipeline

FABIOTESSf15c98d42755
diff --git a/demo/README.md b/demo/README.md
new file mode 100644
index 00000000..dfda60b2
--- /dev/null
+++ b/demo/README.md
@@ -0,0 +1,106 @@
+# Document-to-ontology demo pipeline
+
+This directory holds the pipeline that turns a corpus of plain documents into
+an ontology, populates it, verifies it, and scans the result for
+contradictions that split along document provenance. The code here is
+domain-agnostic: nothing about any particular subject is hardcoded, the
+vocabulary is whatever the documents themselves turn out to be about.
+
+The corpus that ships with this repository is public DCAT-US material, and
+its provenance is recorded alongside it so every derived claim can be traced
+back to the document and section it came from.
+
+## Pipeline stages
+
+1. **Read.** Every document in the corpus folder is loaded (`corpus_pipeline.py`,
+ `ontology_from_docs.py`).
+2. **Tokenise.** Sensitive values are detected and replaced with stable,
+ deterministic tokens before anything reaches a model (`tokenisation.py`).
+ The same value produces the same token in every document, so a token
+ doubles as a join key for entity resolution without any component ever
+ handling the raw value.
+3. **Chunk.** Text is split on semantic boundaries, not fixed width, with
+ each chunk carrying enough context to be interpreted on its own and an id
+ scheme that lets a fragment be rejoined to its section (`chunker.py`).
+4. **Derive per-document fragments.** Each document proposes its own classes,
+ properties and disjointness axioms, independently of every other document
+ (`ontology_from_docs.py`, stage DERIVE).
+5. **Merge.** The per-document fragments are concatenated into one candidate
+ ontology.
+6. **Reconcile.** Independent derivation leaves behind competing modelling
+ patterns: one document may model a distinction as a subclass partition
+ while another models the same notion as a separate attribute class.
+ Where both exist for the same parent, the attribute class is removed
+ because it is redundant with the partition and is the one form a
+ reasoner cannot check for contradictions. Status and State are exempt,
+ because a status is a property of a thing over time, not a kind of
+ thing (`ontology_from_docs.py`, function `reconcile`).
+7. **Populate.** Instances are extracted from the documents, constrained to
+ the derived and reconciled ontology so nothing gets invented
+ (`ontology_from_docs.py`, stage POPULATE; `extract.py`).
+8. **Verify.** A closed-world check confirms every term used in the
+ extracted data was declared in the ontology, and a standalone pass
+ re-checks parsing, stats, lint, enforce, vocabulary and reasoning against
+ whatever is currently on disk (`ontology_from_docs.py` stage VERIFY;
+ `verify.py`).
+9. **Scan for contradictions.** Once claims are extracted into the graph,
+ contradictions are found in three cascading tiers: structural checks the
+ reasoner and SPARQL catch outright, blocked candidate pairs that compare
+ only claims about the same entity from different documents, and an
+ optional model adjudication pass over the small surviving set
+ (`contradiction_scan.py`). Findings that split along document provenance
+ are the ones worth showing anyone: two documents disagreeing about the
+ same entity is a genuine contradiction in the corpus, not a broken
+ extraction.
+
+Two supporting tools sit alongside the main pipeline:
+
+- `corpus_text.py` extracts plain text directly from the source documents,
+ with no help from the ontology, so a classic chunk-retrieval baseline can
+ be compared fairly against the graph-based retrieval path.
+- `kpi_context_graph.py` derives a KPI context graph from the `computedFrom`,
+ `appliesTo`, `dependsOnKPI` and `governedBy` triples the ontology itself
+ declares, so the graph cannot drift from the model it describes. It
+ answers impact questions (what depends on a changed term) and evaluates
+ indicators against current data.
+- `cq/run-cross-doc.py` runs a fixed set of competency questions against the
+ merged store, as a repeatable check that the ontology can answer the
+ questions it was built to answer.
+
+## Derived artifacts
+
+Everything the pipeline produces is written under `demo/derived/` and
+`demo/corpus_extracted/`, and is regenerated from the corpus on every run.
+None of it is checked in:
+
+- `demo/derived/_ontology.ttl`: the merged, reconciled ontology.
+- `demo/derived/_corpus_text.json`: the plain-text baseline extraction.
+- `demo/corpus_extracted/*.ttl`: one knowledge graph fragment per document.
+- `demo/corpus_extracted/_merged.ttl` and `_live.ttl`: the merged store
+ loaded into the running engine.
+- `demo/_review.jsonl`: individuals the pipeline could not type with
+ confidence, queued for a human decision rather than guessed at.
+
+## Running it
+
+```bash
+python -m pip install -r demo/requirements.txt
+make demo
+```
+
+`make demo` runs the pipeline end to end against the corpus in
+`demo/corpus/dcat-us`: read, tokenise, derive, merge, reconcile, populate,
+verify, and scan for contradictions. Each stage can also be run on its own,
+for example:
+
+```bash
+python3 demo/ontology_from_docs.py --corpus demo/corpus/dcat-us
+python3 demo/contradiction_scan.py
+python3 demo/kpi_context_graph.py graph
+```
+
+Run the tests with:
+
+```bash
+python -m pytest demo/tests/
+```
diff --git a/demo/chunker.py b/demo/chunker.py
new file mode 100644
index 00000000..73999684
--- /dev/null
+++ b/demo/chunker.py
@@ -0,0 +1,222 @@
+#!/usr/bin/env python3
+"""
+Semantic chunking with context and a rejoin scheme.
+
+Three decisions, all of which matter, and none of which the usual splitters do:
+
+1. SEMANTIC BOUNDARIES, NOT FIXED WIDTH.
+ Chunks are whole paragraphs inside whole sections. Small paragraphs are
+ merged upward toward a target size rather than emitted alone. The point is
+ not tidiness: fixed-width splitting explodes the number of chunks, and the
+ cost of a pipeline like this is dominated by how many times a model gets
+ called. Fewer, more meaningful chunks is both cheaper and better.
+
+2. EVERY CHUNK CARRIES ITS CONTEXT.
+ A chunk lifted out of a fifty-page report is uninterpretable on its own.
+ Each record stores a `context` sentence saying which document it came from,
+ what that document is about, which section it sits in, and where in that
+ section it falls. This is what lets a model reason about a fragment without
+ re-reading the source.
+
+3. IDS ENCODE STRUCTURE SO CHUNKS CAN BE REJOINED.
+ Ids look like DOC-042.S3.P2, and every record carries parent, prev and next.
+ Retrieval returns a fragment; answering often needs the whole section. With
+ a join scheme the consumer can ask for the parent or walk the siblings
+ instead of hallucinating the surrounding text or over-retrieving everything.
+
+Emitted as JSON so the records can be stored, indexed and joined by anything.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass, asdict
+from typing import Iterator
+
+# Target chunk size in characters. Paragraphs below MIN are merged forward;
+# paragraphs above MAX are split at sentence boundaries.
+TARGET = 1200
+MIN = 350
+MAX = 2400
+
+HEADING = re.compile(
+ r"^\s*(?:#{1,6}\s+(?P<md>.+?)|(?P<num>\d+(?:\.\d+)*)\.?\s+(?P<numtitle>[A-Z].{2,80}))\s*$"
+)
+SENTENCE = re.compile(r"(?<=[.!?])\s+(?=[A-Z(])")
+
+
+@dataclass
+class Chunk:
+ id: str
+ doc_id: str
+ section: str
+ section_title: str
+ text: str
+ context: str
+ parent: str | None
+ prev: str | None
+ next: str | None
+
+ def to_json(self) -> dict:
+ return asdict(self)
+
+
+def _split_long(text: str) -> list[str]:
+ """Split an oversized paragraph at sentence boundaries, not mid-sentence."""
+ if len(text) <= MAX:
+ return [text]
+ out, cur = [], ""
+ for sentence in SENTENCE.split(text):
+ if cur and len(cur) + len(sentence) > TARGET:
+ out.append(cur.strip())
+ cur = sentence
+ else:
+ cur = f"{cur} {sentence}".strip()
+ if cur.strip():
+ out.append(cur.strip())
+ return out
+
+
+def _sections(text: str) -> Iterator[tuple[str, str, list[str]]]:
+ """Yield (section_number, section_title, paragraphs)."""
+ lines = text.splitlines()
+ num, title, buf = "0", "Preamble", []
+ for line in lines:
+ m = HEADING.match(line)
+ if m:
+ if buf:
+ yield num, title, _paragraphs("\n".join(buf))
+ buf = []
+ if m.group("md"):
+ heading = m.group("md").strip()
+ lead = re.match(r"^(\d+(?:\.\d+)*)\.?\s+(.*)$", heading)
+ num, title = (lead.group(1), lead.group(2)) if lead else (num, heading)
+ else:
+ num, title = m.group("num"), m.group("numtitle").strip()
+ else:
+ buf.append(line)
+ if buf:
+ yield num, title, _paragraphs("\n".join(buf))
+
+
+def _paragraphs(block: str) -> list[str]:
+ """Paragraphs, with short ones merged forward toward the target size."""
+ raw = [p.strip() for p in re.split(r"\n\s*\n", block) if p.strip()]
+ merged: list[str] = []
+ for p in raw:
+ if merged and len(merged[-1]) < MIN:
+ merged[-1] = f"{merged[-1]}\n\n{p}"
+ else:
+ merged.append(p)
+ out: list[str] = []
+ for p in merged:
+ out.extend(_split_long(p))
+ return [p for p in out if len(p.strip()) > 40]
+
+
+# Front-matter noise that says nothing about what a document is ABOUT.
+_BOILERPLATE = re.compile(
+ r"^\s*(?:\||#|\*\*Document control|SYNTHETIC|\*\*SYNTHETIC|-{3,}|\d+\.\s*(?:Purpose|Scope)\b)",
+ re.I,
+)
+
+
+def _summarise(text: str, limit: int = 160) -> str:
+ """A one-line gist of the document, used in every chunk's context.
+
+ Skips document-control tables, headings and demo disclaimers: they are
+ present in every document and say nothing about what this one covers, so
+ including them would make every chunk's context identical and useless.
+ """
+ lines = [l.strip() for l in text.splitlines()]
+ prose = [l for l in lines if l and not _BOILERPLATE.match(l) and len(l) > 40]
+ body = re.sub(r"\s+", " ", " ".join(prose[:4])).strip()
+ body = re.sub(r"^\d+(?:\.\d+)*\s+", "", body)
+ gist = " ".join(SENTENCE.split(body)[:2]).strip()
+ return (gist[: limit - 1] + "…") if len(gist) > limit else (gist or "an internal controlled document")
+
+
+def chunk_document(text: str, doc_id: str, title: str | None = None) -> list[Chunk]:
+ """Chunk one document into context-carrying, rejoinable records."""
+ doc_title = title or (re.search(r"^#\s+(.+)$", text, flags=re.M).group(1)
+ if re.search(r"^#\s+(.+)$", text, flags=re.M) else doc_id)
+ gist = _summarise(text)
+
+ chunks: list[Chunk] = []
+ for num, sec_title, paras in _sections(text):
+ total = len(paras)
+ for i, para in enumerate(paras, start=1):
+ cid = f"{doc_id}.S{num}.P{i}"
+ context = (
+ f"From {doc_id} ({doc_title}), a document about {gist} "
+ f"Section {num}, {sec_title}. Paragraph {i} of {total} in this section."
+ )
+ chunks.append(Chunk(
+ id=cid, doc_id=doc_id, section=num, section_title=sec_title,
+ text=para, context=context, parent=f"{doc_id}.S{num}",
+ prev=None, next=None,
+ ))
+
+ # Link siblings after the fact, so prev/next span section boundaries in
+ # reading order rather than stopping at each heading.
+ for i, c in enumerate(chunks):
+ c.prev = chunks[i - 1].id if i > 0 else None
+ c.next = chunks[i + 1].id if i < len(chunks) - 1 else None
+ return chunks
+
+
+def pack(chunks: list[Chunk], budget: int = 9000) -> list[list[Chunk]]:
+ """Group chunks into as few model calls as possible.
+
+ This is the decision that controls cost. One call per chunk is what makes
+ these pipelines absurdly expensive on long documents; packing to a context
+ budget, while keeping each chunk's context header attached, gets the same
+ information across in a fraction of the calls.
+ """
+ groups: list[list[Chunk]] = []
+ cur: list[Chunk] = []
+ size = 0
+ for c in chunks:
+ cost = len(c.text) + len(c.context) + 40
+ if cur and size + cost > budget:
+ groups.append(cur)
+ cur, size = [], 0
+ cur.append(c)
+ size += cost
+ if cur:
+ groups.append(cur)
+ return groups
+
+
+def render(group: list[Chunk]) -> str:
+ """Render a packed group for a prompt, each chunk labelled and contextualised."""
+ parts = []
+ for c in group:
+ parts.append(f"[{c.id}]\n({c.context})\n{c.text}")
+ return "\n\n".join(parts)
+
+
+def to_jsonl(chunks: list[Chunk]) -> str:
+ return "\n".join(json.dumps(c.to_json(), ensure_ascii=False) for c in chunks)
+
+
+if __name__ == "__main__":
+ import pathlib
+ import sys
+
+ if len(sys.argv) < 2:
+ sys.exit("usage: chunker.py <file.md> [--json]")
+ path = pathlib.Path(sys.argv[1])
+ doc_id = re.match(r"([A-Z]+-[0-9][0-9A-Za-z\-]*?)(?=-[a-z])", path.stem)
+ cs = chunk_document(path.read_text(), doc_id.group(1) if doc_id else path.stem)
+ if "--json" in sys.argv:
+ print(to_jsonl(cs))
+ else:
+ groups = pack(cs)
+ print(f"{len(cs)} chunks in {len(groups)} model calls "
+ f"(a per-chunk pipeline would make {len(cs)})")
+ for c in cs[:3]:
+ print(f"\n[{c.id}] parent={c.parent} prev={c.prev} next={c.next}")
+ print(f" context: {c.context}")
+ print(f" text: {c.text[:110]}...")
diff --git a/demo/contradiction_scan.py b/demo/contradiction_scan.py
new file mode 100644
index 00000000..2cce9907
--- /dev/null
+++ b/demo/contradiction_scan.py
@@ -0,0 +1,318 @@
+#!/usr/bin/env python3
+"""
+Contradiction scanner: find the passages in a document corpus that disagree.
+
+WHY NOT BAYESIAN OPTIMISATION
+-----------------------------
+The obvious framing is "search the space of chunks for the one that trips
+retrieval". That invites Bayesian optimisation, and BO is the wrong tool:
+it is built for optimising an expensive black-box function over a smooth,
+low-dimensional, continuous space. Chunk-pair contradiction is combinatorial,
+discrete, and has no useful notion of "nearby". BO would sample slowly through
+a space that does not reward sampling.
+
+It is also unnecessary. Once claims are extracted into the graph, most
+contradictions stop being a search problem:
+
+ Tier 1 STRUCTURAL the reasoner and SPARQL find them outright.
+ Cost: milliseconds. Exact, with zero false positives.
+ Catches disjoint-class violations, functional-property
+ conflicts, and domain rule breaches.
+
+ Tier 2 BLOCKING for everything semantic that the axioms cannot express,
+ do NOT compare all pairs. Claims carry `aboutEntity`,
+ so only compare claims about the SAME entity that come
+ from DIFFERENT documents. This is standard blocking from
+ entity resolution: it collapses O(M^2) over the corpus
+ to sum(k_i^2) over per-entity groups, where k_i is
+ typically single digits.
+
+ Tier 3 ADJUDICATE run a model over the surviving handful of candidate
+ pairs only. Optional, --adjudicate.
+
+The cascade is orders of magnitude cheaper than any search-based method, and
+unlike BO the first tier is exact rather than probabilistic.
+
+Usage:
+ python3 demo/contradiction_scan.py
+ python3 demo/contradiction_scan.py --adjudicate # add tier 3
+"""
+
+import argparse
+import json
+import os
+import pathlib
+import subprocess
+import sys
+import urllib.request
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+BIN = ROOT / "target" / "release" / "open-ontologies"
+BUNDLE = ROOT / "demo" / "bundle" / "dcat-us-full.ttl"
+# Any extracted document graphs are merged in too, so the scan sees the whole
+# corpus rather than only the curated demo data.
+EXTRA = sorted((ROOT / "demo" / "corpus_extracted").glob("*.ttl"))
+LLM_BASE = os.environ.get("ONTO_LLM_BASE_URL", "http://localhost:8081/v1").rstrip("/")
+LLM_KEY = os.environ.get("ONTO_LLM_API_KEY", "not-needed")
+
+PREFIXES = """PREFIX crd: <https://w3id.org/dcat-us-demo#>
+PREFIX owl: <http://www.w3.org/2002/07/owl#>
+PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
+PREFIX prov: <http://www.w3.org/ns/prov#>
+"""
+
+# --- Tier 1: structural contradictions -----------------------------------
+
+TIER1 = [
+ (
+ "disjoint-class",
+ "An individual is typed into two classes the ontology declares disjoint",
+ PREFIXES
+ + """SELECT DISTINCT ?subject ?classA ?classB ?disjointA ?disjointB WHERE {
+ ?subject a ?classA, ?classB .
+ FILTER(STR(?classA) < STR(?classB))
+ ?classA rdfs:subClassOf* ?disjointA .
+ ?classB rdfs:subClassOf* ?disjointB .
+ { ?disjointA owl:disjointWith ?disjointB } UNION { ?disjointB owl:disjointWith ?disjointA }
+}""",
+ ),
+ (
+ "functional-property",
+ "A functional property carries two different values on the same subject",
+ PREFIXES
+ + """SELECT DISTINCT ?subject ?property ?valueA ?valueB WHERE {
+ ?property a owl:FunctionalProperty .
+ ?subject ?property ?valueA, ?valueB .
+ FILTER(STR(?valueA) < STR(?valueB))
+}""",
+ ),
+ (
+ "domain-rule",
+ "A candidate declares a control target the register classifies as beneficial",
+ PREFIXES
+ + """SELECT DISTINCT ?subject ?organism ?viaClass WHERE {
+ ?subject crd:targetsOrganism ?organism .
+ ?organism a ?viaClass .
+ ?viaClass rdfs:subClassOf* crd:BeneficialOrganism .
+}""",
+ ),
+]
+
+# --- Tier 2: blocked candidate generation --------------------------------
+
+# Blocking must be tight or it stops being blocking. Two refinements beyond
+# "same entity, different documents":
+# - exclude CLASS-level entities. A claim "about :Candidate" is about a
+# category, not a thing, and pairing every such claim with every other
+# generates noise proportional to corpus size rather than to real overlap.
+# - require the two claims to share vocabulary. Claims about the same
+# individual that share no significant term are rarely contradictory.
+TIER2 = PREFIXES + """SELECT ?entity ?docA ?secA ?claimA ?docB ?secB ?claimB WHERE {
+ ?a a crd:Claim ; crd:aboutEntity ?entity ; crd:claimText ?claimA ; crd:statedIn ?sa .
+ ?b a crd:Claim ; crd:aboutEntity ?entity ; crd:claimText ?claimB ; crd:statedIn ?sb .
+ FILTER(STR(?a) < STR(?b))
+ FILTER NOT EXISTS { ?entity a owl:Class }
+ FILTER NOT EXISTS { ?someone a ?entity }
+ ?sa crd:sectionNumber ?secA . ?da crd:hasSection ?sa ; crd:docId ?docA .
+ ?sb crd:sectionNumber ?secB . ?db crd:hasSection ?sb ; crd:docId ?docB .
+ FILTER(?docA != ?docB)
+}"""
+
+# Total claims, for an honest reduction ratio rather than a bare count.
+CLAIM_COUNT = PREFIXES + "SELECT (COUNT(?c) AS ?n) WHERE { ?c a crd:Claim }"
+
+# Which source asserted each conflicting fact.
+PROVENANCE = PREFIXES + """SELECT ?thing ?sourceLabel WHERE {
+ ?thing prov:wasDerivedFrom ?source .
+ ?source rdfs:label ?sourceLabel .
+}"""
+
+
+def short(v):
+ if not isinstance(v, str):
+ return v
+ if v.startswith("<") and v.endswith(">"):
+ return v[1:-1].split("#")[-1]
+ if v.startswith('"'):
+ body = v[1:]
+ for cut in ('"^^', '"@', '"'):
+ if cut in body:
+ return body.split(cut)[0]
+ return body
+ return v
+
+
+def query(sparql):
+ loads = "\n".join([f"load {BUNDLE}"] + [f"load {e}" for e in EXTRA])
+ batch = f"clear\n{loads}\nquery {json.dumps(' '.join(sparql.split()))}\n"
+ out = subprocess.run([str(BIN), "batch", "-"], input=batch, capture_output=True, text=True).stdout.strip()
+ if not out:
+ return []
+ try:
+ rows = json.loads(out.split("\n")[-1]).get("result", {}).get("results", [])
+ except Exception:
+ return []
+ return [{k: short(v) for k, v in r.items()} for r in rows]
+
+
+STOP = set("the a an of to in for is are be and or that this with by on at as it its "
+ "must not no any all where which when from than then shall should may".split())
+
+
+def signal(a, b):
+ """Cheap high-precision discriminator applied after blocking.
+
+ Blocking on entity alone is asymptotically good but weak on a small corpus:
+ a handful of hot entities appear in most documents. Two extra signals cut
+ it further at negligible cost:
+
+ - shared significant vocabulary (the claims are actually about the same
+ thing, not merely tagged to the same entity)
+ - a numeric disagreement (two documents stating different values for
+ what reads like the same quantity is the single most common form of
+ real contradiction in controlled documents)
+
+ Returns (score, reason). Pairs scoring 0 are dropped.
+ """
+ import re as _re
+ ta = {w for w in _re.findall(r"[a-z]{4,}", a.lower()) if w not in STOP}
+ tb = {w for w in _re.findall(r"[a-z]{4,}", b.lower()) if w not in STOP}
+ shared = ta & tb
+ na = set(_re.findall(r"\d+(?:\.\d+)?", a))
+ nb = set(_re.findall(r"\d+(?:\.\d+)?", b))
+ score, why = 0, []
+ if len(shared) >= 3:
+ score += 1
+ why.append(f"{len(shared)} shared terms")
+ if na and nb and na != nb:
+ score += 2
+ why.append(f"numeric disagreement {sorted(na)} vs {sorted(nb)}")
+ neg = {"not", "no", "must not", "never"}
+ if any(n in a.lower() for n in neg) != any(n in b.lower() for n in neg):
+ score += 1
+ why.append("one asserts, one negates")
+ return score, "; ".join(why)
+
+
+def adjudicate(entity, a, b):
+ """Tier 3. Only ever called on the small blocked candidate set."""
+ prompt = (
+ f"Two statements about the same entity ({entity}) from different controlled documents.\n\n"
+ f"A: {a}\nB: {b}\n\n"
+ "Do these CONTRADICT each other? A contradiction means both cannot be true at once. "
+ "Differing detail or emphasis is NOT a contradiction.\n"
+ 'Answer with JSON only: {"contradiction": true|false, "reason": "<one sentence>"}'
+ )
+ try:
+ model = json.load(
+ urllib.request.urlopen(
+ urllib.request.Request(f"{LLM_BASE}/models", headers={"Authorization": f"Bearer {LLM_KEY}"}),
+ timeout=30,
+ )
+ )["data"][0]["id"]
+ req = urllib.request.Request(
+ f"{LLM_BASE}/chat/completions",
+ data=json.dumps(
+ {
+ "model": model,
+ "messages": [{"role": "user", "content": prompt}],
+ "temperature": 0.1,
+ "max_tokens": 200,
+ "chat_template_kwargs": {"enable_thinking": False},
+ }
+ ).encode(),
+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"},
+ )
+ text = json.load(urllib.request.urlopen(req, timeout=180))["choices"][0]["message"]["content"]
+ start, end = text.find("{"), text.rfind("}")
+ return json.loads(text[start : end + 1]) if start >= 0 else {"contradiction": None, "reason": "unparseable"}
+ except Exception as e:
+ return {"contradiction": None, "reason": f"adjudication unavailable: {e}"}
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--adjudicate", action="store_true", help="run tier 3 over blocked candidates")
+ args = ap.parse_args()
+
+ if not BIN.exists():
+ sys.exit(f"engine binary not found: {BIN}")
+
+ findings = 0
+
+ print("=" * 72)
+ print("TIER 1 structural contradictions (reasoner + SPARQL, exact)")
+ print("=" * 72)
+ for name, description, sparql in TIER1:
+ rows = query(sparql)
+ print(f"\n[{name}] {description}")
+ if not rows:
+ print(" none")
+ continue
+ findings += len(rows)
+ for r in rows:
+ print(f" CONFLICT {json.dumps(r)}")
+
+ prov = query(PROVENANCE)
+ if prov:
+ print("\n[provenance] which upstream source asserted each fact")
+ by_thing = {}
+ for r in prov:
+ by_thing.setdefault(r["thing"], []).append(r["sourceLabel"])
+ for thing, sources in by_thing.items():
+ marker = " <-- asserted by TWO sources" if len(sources) > 1 else ""
+ print(f" {thing}: {', '.join(sources)}{marker}")
+
+ print("\n" + "=" * 72)
+ print("TIER 2 blocked candidate pairs (same entity, different documents)")
+ print("=" * 72)
+ candidates = query(TIER2)
+ try:
+ total = int(query(CLAIM_COUNT)[0]["n"])
+ except Exception:
+ total = 0
+ full = total * (total - 1) // 2 if total else 0
+ print(f"\nclaims in corpus: {total} full pairwise space: {full}")
+ print(f"candidate pairs after blocking: {len(candidates)}"
+ + (f" ({100*len(candidates)/full:.1f}% of pairwise)" if full else ""))
+ scored, seen = [], set()
+ for c in candidates:
+ # The SPARQL yields each pair once, but different section IRIs can carry
+ # the same claim text, so dedupe on the unordered text pair.
+ key = frozenset((c["claimA"], c["claimB"]))
+ if key in seen:
+ continue
+ seen.add(key)
+ sc, why = signal(c["claimA"], c["claimB"])
+ if sc:
+ scored.append((sc, why, c))
+ scored.sort(key=lambda x: -x[0])
+ print(f"high-signal pairs after discriminator: {len(scored)}"
+ + (f" ({100*len(scored)/full:.1f}% of pairwise)" if full else ""))
+ print(f"\n{DIM if False else ''}Blocking alone is asymptotic: its advantage grows with corpus size.")
+ print("On 36 claims a few hot entities dominate, so the discriminator does the")
+ print("real work here. On a corpus of thousands both matter.\n")
+ candidates = [c for _, _, c in scored[:12]]
+ for sc, why, c in scored[:12]:
+ print(f" [signal {sc}] {c['entity']} ({why})")
+ print(f" {c['docA']} S{c['secA']}: {c['claimA'][:130]}")
+ print(f" {c['docB']} S{c['secB']}: {c['claimB'][:130]}")
+ if not candidates:
+ print(" none")
+ for c in candidates:
+ if args.adjudicate:
+ verdict = adjudicate(c["entity"], c["claimA"], c["claimB"])
+ flag = "CONTRADICTION" if verdict.get("contradiction") else "consistent"
+ print(f" -> {flag}: {verdict.get('reason', '')}")
+
+ print("\n" + "=" * 72)
+ print(f"tier 1 structural findings: {findings}")
+ print(f"tier 2 candidate pairs : {len(candidates)}")
+ print("\nCost note: tier 1 is exact and runs in milliseconds. Tier 2 compares")
+ print("only claims that share an entity across documents, so the model in")
+ print("tier 3 is invoked on a handful of pairs rather than on every chunk pair.")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/demo/corpus_pipeline.py b/demo/corpus_pipeline.py
new file mode 100644
index 00000000..831bd086
--- /dev/null
+++ b/demo/corpus_pipeline.py
@@ -0,0 +1,310 @@
+#!/usr/bin/env python3
+"""
+Read a folder of Word documents, build a knowledge graph, find the
+contradictions between them.
+
+This is the showcase end to end:
+
+ 1. READ every .docx in demo/corpus/dcat-us (text via pandoc)
+ 2. EXTRACT a knowledge graph per document, constrained to the ontology
+ 3. VERIFY closed-world check that no term was invented
+ 4. LOAD merge every document graph into one store
+ 5. SCAN find the passages that contradict each other
+
+Nothing leaves the machine: extraction runs against whatever model endpoint is
+configured, which by default is a local one.
+
+Usage:
+ python3 demo/corpus_pipeline.py
+ python3 demo/corpus_pipeline.py --limit 3 # quick run
+ python3 demo/corpus_pipeline.py --cached # skip extraction, reuse output
+"""
+
+import argparse
+import functools
+import concurrent.futures as cf
+import json
+import os
+import pathlib
+import re
+import subprocess
+import sys
+import time
+import urllib.request
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+CORPUS = ROOT / "demo" / "corpus" / "dcat-us"
+OUT = ROOT / "demo" / "corpus_extracted"
+BIN = ROOT / "target" / "release" / "open-ontologies"
+VOCAB = ROOT / "demo" / "bundle" / "dcat-us-vocab.ttl"
+ENGINE = os.environ.get("ONTO_ENGINE", "http://127.0.0.1:8137")
+LLM_BASE = os.environ.get("ONTO_LLM_BASE_URL", "http://localhost:8081/v1").rstrip("/")
+LLM_KEY = os.environ.get("ONTO_LLM_API_KEY", "not-needed")
+
+if os.environ.get("NO_COLOR"):
+ BOLD = DIM = RED = GRN = YEL = OFF = ""
+else:
+ BOLD, DIM, RED, GRN, YEL, OFF = "\033[1m", "\033[2m", "\033[31m", "\033[32m", "\033[33m", "\033[0m"
+
+VOCAB_HINT = """Classes: :Document :StandardOperatingProcedure :RegulatoryGuidance :TrialReport
+ :Section :Claim :Control :ReferenceSource :Candidate :Organism :Pest :Pathogen
+ :BeneficialOrganism :Pollinator :NaturalEnemy :FieldTrial :TargetProtein :Mutation
+ :EvidenceGap :StageGate
+Object properties: :hasSection :parentSection :relatedSection :statedIn :aboutEntity
+ :supersedes :declaresControl :controlOver :blocksProgressionOf :documentsRiskIn
+Datatype properties: :docId :classification :aclGroup :sectionNumber :claimText
+ :isAutomated :isBlocking rdfs:label rdfs:comment"""
+
+PROMPT = """Extract a knowledge graph from this controlled document.
+
+Return ONLY Turtle. No prose, no fences.
+
+Prefixes:
+@prefix : <https://w3id.org/dcat-us-demo#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+
+Use ONLY these terms. Invent nothing:
+{vocab}
+
+Extract:
+1. One :Document node :DOC_{safe} with :docId, rdfs:label, :classification, :aclGroup
+ and its most specific document type.
+2. A :Section per numbered section, IRI :SEC_{safe}_<number with dots as underscores>,
+ with :sectionNumber and rdfs:label, linked by :hasSection.
+3. :Claim nodes for assertions about candidate progression, organism classification,
+ thresholds, or safeguards. Each with :claimText, :statedIn, and :aboutEntity where
+ the subject is identifiable.
+4. Where a section states a safeguard, a :Control with :isAutomated and :isBlocking.
+
+DOCUMENT ID: {docid}
+
+{body}
+"""
+
+PREFIXES = ("@prefix : <https://w3id.org/dcat-us-demo#> .\n"
+ "@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n"
+ "@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n")
+
+
+def docx_text(path):
+ """Word document to plain text."""
+ r = subprocess.run(["pandoc", str(path), "-t", "plain", "--wrap=none"],
+ capture_output=True, text=True)
+ return r.stdout
+
+
+def normalise(text):
+ text = re.sub(r"```(?:turtle|ttl)?", "", text).strip()
+ lines = text.split("\n")
+ start = None
+ for i, line in enumerate(lines):
+ if line.strip().startswith(("@prefix", "@base")):
+ start = i
+ break
+ if start is None:
+ for i, line in enumerate(lines):
+ if re.match(r"^\s*:\S+\s+(a|rdf:type)\s", line):
+ start = i
+ break
+ text = "\n".join(lines[start if start is not None else 0:]).strip()
+ lines = text.split("\n")
+ for i in range(len(lines) - 1, -1, -1):
+ if lines[i].rstrip().endswith((".", ";", ",")):
+ text = "\n".join(lines[: i + 1]).strip()
+ break
+ if "@prefix :" not in text:
+ text = PREFIXES + "\n" + text
+ return text
+
+
+def call_model(prompt):
+ model = os.environ.get("ONTO_LLM_MODEL", "")
+ if not model:
+ req = urllib.request.Request(LLM_BASE + "/models", headers={"Authorization": f"Bearer {LLM_KEY}"})
+ model = json.load(urllib.request.urlopen(req, timeout=30))["data"][0]["id"]
+ req = urllib.request.Request(
+ LLM_BASE + "/chat/completions",
+ data=json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}],
+ "temperature": 0.2, "max_tokens": 3000,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode(),
+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"})
+ return json.load(urllib.request.urlopen(req, timeout=900))["choices"][0]["message"]["content"]
+
+
+def extract_one(path):
+ raw = docx_text(path)
+ m = re.match(r"([A-Z]+-[0-9][0-9A-Za-z\-]*?)(?=-[a-z])", path.stem)
+ docid = m.group(1) if m else path.stem
+ safe = docid.replace("-", "_")
+ body = raw[:14000]
+ for attempt in range(3):
+ try:
+ content = call_model(PROMPT.format(vocab=VOCAB_HINT, docid=docid, safe=safe, body=body))
+ except Exception as e:
+ return docid, None, f"model error: {e}"
+ if re.search(r"(\b\S{1,6}\b[ .]{1,3})\1{15,}", content):
+ continue
+ ttl = normalise(content)
+ if len(ttl) > 200:
+ out = OUT / f"{docid}.ttl"
+ out.write_text(f"# Extracted from {path.name} on 2026-08-10\n\n{ttl}")
+ return docid, out, None
+ return docid, None, "degenerate output after 3 attempts"
+
+
+def mcp(method, params, sid=[None]):
+ h = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream"}
+ if sid[0]:
+ h["Mcp-Session-Id"] = sid[0]
+ req = urllib.request.Request(ENGINE + "/mcp", headers=h, data=json.dumps(
+ {"jsonrpc": "2.0", "id": 1, "method": method, "params": params}).encode())
+ r = urllib.request.urlopen(req, timeout=120)
+ sid[0] = r.headers.get("Mcp-Session-Id") or sid[0]
+ body = r.read().decode().strip()
+ if body.startswith("{"):
+ return json.loads(body)
+ for line in body.split("\n"):
+ if line.startswith("data:"):
+ try:
+ fr = json.loads(line[5:].strip())
+ if "result" in fr or "error" in fr:
+ return fr
+ except Exception:
+ pass
+ return {}
+
+
+def vocab_check(ttl):
+ """Closed-world term check. The vocabulary must be loaded in the SAME
+ session, so reload it here rather than assuming it survived."""
+ try:
+ mcp("initialize", {"protocolVersion": "2025-03-26", "capabilities": {},
+ "clientInfo": {"name": "pipeline", "version": "1"}})
+ mcp("tools/call", {"name": "onto_load", "arguments":
+ {"path": str(VOCAB), "name": "dcat-us-vocab", "force_recompile": True}})
+ res = mcp("tools/call", {"name": "onto_vocab_check",
+ "arguments": {"data": ttl, "inline": True}})
+ return json.loads(res.get("result", {}).get("content", [{}])[0].get("text", "{}"))
+ except Exception as e:
+ return {"conforms": None, "error": str(e)}
+
+
+def engine(cmds):
+ return subprocess.run([str(BIN), "batch", "-"], input="\n".join(cmds) + "\n",
+ capture_output=True, text=True).stdout.strip()
+
+
+print = functools.partial(__builtins__.print if not isinstance(__builtins__, dict) else __builtins__["print"], flush=True)
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--limit", type=int, default=0)
+ ap.add_argument("--cached", action="store_true")
+ ap.add_argument("--workers", type=int, default=4)
+ ap.add_argument("--corpus", default=str(CORPUS),
+ help="folder of documents to read")
+ args = ap.parse_args()
+
+ OUT.mkdir(exist_ok=True)
+ corpus_dir = pathlib.Path(args.corpus)
+ docs = sorted(corpus_dir.glob("*.docx")) or sorted(corpus_dir.glob("*.md"))
+ if args.limit:
+ docs = docs[: args.limit]
+ if not docs:
+ sys.exit(f"no .docx or .md found in {corpus_dir}")
+
+ t0 = time.time()
+ print(f"\n{BOLD}STAGE 1 READ{OFF} {len(docs)} documents from {corpus_dir.name}/")
+ for d in docs:
+ print(f" {DIM}{d.name}{OFF}")
+
+ extracted = []
+ if args.cached:
+ extracted = sorted(OUT.glob("*.ttl"))
+ print(f"\n{BOLD}STAGE 2 EXTRACT{OFF} {DIM}cached, {len(extracted)} graphs reused{OFF}")
+ else:
+ print(f"\n{BOLD}STAGE 2 EXTRACT{OFF} knowledge graph per document")
+ print(f" {DIM}model endpoint {LLM_BASE} ({args.workers} in parallel){OFF}")
+ with cf.ThreadPoolExecutor(max_workers=args.workers) as pool:
+ for docid, out, err in pool.map(extract_one, docs):
+ if err:
+ print(f" {RED}FAIL{OFF} {docid:14s} {err}")
+ else:
+ print(f" {GRN}ok{OFF} {docid:14s} {out.name}")
+ extracted.append(out)
+
+ print(f"\n{BOLD}STAGE 3 VERIFY{OFF} closed-world check that no term was invented")
+ ok = bad = 0
+ for f in extracted:
+ d = vocab_check(f.read_text())
+ conforms, halluc = d.get("conforms"), d.get("hallucinated_terms", [])
+ if conforms:
+ ok += 1
+ print(f" {GRN}conforms{OFF} {f.stem}")
+ else:
+ bad += 1
+ print(f" {YEL}review {OFF} {f.stem} {halluc if halluc else ''}")
+
+ print(f"\n{BOLD}STAGE 4 LOAD{OFF} merge every document graph into the live store")
+ # Load through the RUNNING engine over MCP, not a throwaway CLI process.
+ # Two things depend on this: the graph view renders the store that was
+ # just built, and every load is recorded in the lineage trail.
+ try:
+ mcp("initialize", {"protocolVersion": "2025-03-26", "capabilities": {},
+ "clientInfo": {"name": "pipeline", "version": "1"}})
+ mcp("tools/call", {"name": "onto_clear", "arguments": {}})
+ base = ROOT / "demo" / "bundle" / "dcat-us-full.ttl"
+ mcp("tools/call", {"name": "onto_load", "arguments":
+ {"path": str(base), "name": "corpus", "force_recompile": True}})
+ merged = base.read_text().rstrip() + "\n"
+ for f in extracted:
+ body = "\n".join(l for l in f.read_text().splitlines()
+ if not l.lstrip().startswith(("@prefix", "@base", "#"))).strip()
+ if not body:
+ continue
+ # A block that does not terminate its last statement will glue onto
+ # the next file and produce a parse error hundreds of lines away.
+ if not body.endswith("."):
+ body += " ."
+ merged += "\n" + body + "\n"
+ staged = ROOT / "demo" / "corpus_extracted" / "_merged.ttl"
+ staged.write_text(merged)
+ mcp("tools/call", {"name": "onto_load", "arguments":
+ {"path": str(staged), "name": "corpus", "force_recompile": True}})
+ # onto_load does not write lineage; a save does. Persist the merged
+ # store so the ingestion appears in the lineage trail.
+ mcp("tools/call", {"name": "onto_save", "arguments":
+ {"path": str(ROOT / "demo" / "corpus_extracted" / "_live.ttl")}})
+ res = mcp("tools/call", {"name": "onto_stats", "arguments": {}})
+ st = json.loads(res.get("result", {}).get("content", [{}])[0].get("text", "{}"))
+ print(f" {st.get('triples','?')} triples, {st.get('classes','?')} classes, "
+ f"{st.get('individuals','?')} individuals {DIM}(loaded into the live engine){OFF}")
+ except Exception as e:
+ print(f" {YEL}live engine unavailable ({e}); falling back to a local load{OFF}")
+ res = engine(["clear", f"load {ROOT/'demo'/'bundle'/'dcat-us-full.ttl'}"]
+ + [f"load {f}" for f in extracted] + ["stats"])
+ try:
+ st = json.loads(res.split("\n")[-1])["result"]
+ print(f" {st['triples']} triples, {st['classes']} classes, {st['individuals']} individuals")
+ except Exception:
+ print(f" {DIM}{res[-200:]}{OFF}")
+
+ print(f"\n{BOLD}STAGE 5 SCAN{OFF} find passages that contradict each other")
+ scan = subprocess.run([sys.executable, str(ROOT / "demo" / "contradiction_scan.py")],
+ capture_output=True, text=True, cwd=ROOT).stdout
+ for line in scan.splitlines():
+ if "CONFLICT" in line:
+ print(f" {RED}{line.strip()}{OFF}")
+ elif "TWO sources" in line or "candidate pairs after blocking" in line:
+ print(f" {YEL}{line.strip()}{OFF}")
+
+ print(f"\n{BOLD}{len(docs)} documents processed in {time.time()-t0:.0f}s."
+ f" {ok} graphs conformed, {bad} flagged for review.{OFF}")
+ print(f"{DIM}Every fact carries the document and section it came from, and inherits"
+ f"\nthat document's access group.{OFF}\n")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/demo/corpus_text.py b/demo/corpus_text.py
new file mode 100644
index 00000000..bd5e6520
--- /dev/null
+++ b/demo/corpus_text.py
@@ -0,0 +1,62 @@
+"""Extract plain document text for the baseline retriever.
+
+The comparison in the Studio puts classic chunk retrieval beside graph
+retrieval. For that comparison to mean anything, the baseline must read the
+SAME documents from the SAME source, with no help from the ontology. Feeding it
+anything derived would rig the result and any competent reviewer would say so.
+
+So this reads the .docx files directly and writes plain text. The baseline
+chunks that text and retrieves over it; the graph path never touches this file.
+
+Output: demo/derived/_corpus_text.json
+ { "DOC-202": {"title": ..., "text": ...}, ... }
+"""
+
+from __future__ import annotations
+
+import json
+import pathlib
+import re
+import zipfile
+
+CORPUS = pathlib.Path("demo/corpus/dcat-us")
+OUT = pathlib.Path("demo/derived/_corpus_text.json")
+
+
+def docx_text(path: pathlib.Path) -> str:
+ """Paragraph-preserving text extraction.
+
+ Paragraph boundaries matter: chunking on a single flattened line would
+ produce chunks that straddle unrelated sections, which would make the
+ baseline look worse than a real one does.
+ """
+ xml = zipfile.ZipFile(path).read("word/document.xml").decode("utf8", "ignore")
+ xml = re.sub(r"</w:p>", "\n", xml)
+ text = re.sub(r"<[^>]+>", "", xml)
+ text = text.replace("&amp;", "&").replace("&lt;", "<").replace("&gt;", ">")
+ lines = [re.sub(r"[ \t]+", " ", ln).strip() for ln in text.split("\n")]
+ return "\n".join(ln for ln in lines if ln)
+
+
+def main() -> None:
+ docs: dict[str, dict[str, str]] = {}
+ for path in sorted(CORPUS.glob("*.docx")):
+ text = docx_text(path)
+ m = re.search(r"Doc Id ([A-Z]+-\d+)", text.replace("\n", " "))
+ doc_id = m.group(1) if m else path.stem
+ title = ""
+ t = re.search(r"Title (.+?)(?: Owner| Classification|$)", text.replace("\n", " "))
+ if t:
+ title = t.group(1).strip()
+ docs[doc_id] = {"title": title, "text": text}
+
+ OUT.parent.mkdir(parents=True, exist_ok=True)
+ OUT.write_text(json.dumps(docs, indent=1))
+ chars = sum(len(d["text"]) for d in docs.values())
+ print(f"{len(docs)} documents, {chars:,} characters -> {OUT}")
+ for k, v in docs.items():
+ print(f" {k:10} {len(v['text']):6,} chars {v['title'][:50]}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/demo/cq/run-cross-doc.py b/demo/cq/run-cross-doc.py
new file mode 100644
index 00000000..e77ad0af
--- /dev/null
+++ b/demo/cq/run-cross-doc.py
@@ -0,0 +1,129 @@
+#!/usr/bin/env python3
+"""Run the cross-document competency questions against the full demo bundle.
+
+These are the questions a vector retriever structurally cannot answer,
+because no single passage contains the chain. Each one spans two to four
+separate documents plus the structured graph.
+"""
+
+import json
+import pathlib
+import subprocess
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent.parent
+BIN = ROOT / "target" / "release" / "open-ontologies"
+BUNDLE = ROOT / "demo" / "bundle" / "dcat-us-full.ttl"
+
+P = """PREFIX crd: <https://w3id.org/dcat-us-demo#>
+PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
+"""
+
+QUESTIONS = [
+ (
+ "XD-1",
+ "Which controls should block SYN-4471, are they automated, and who declares them?",
+ P
+ + """SELECT ?docId ?control ?blocking ?automated WHERE {
+ ?ctrl crd:blocksProgressionOf crd:SYN_4471 ;
+ rdfs:label ?control ; crd:isBlocking ?blocking ; crd:isAutomated ?automated .
+ ?doc crd:declaresControl ?ctrl ; crd:docId ?docId . }""",
+ ),
+ (
+ "XD-2",
+ "Root cause: every claim about the mis-mapped bucket or the hoverfly, with its source",
+ P
+ + """SELECT ?docId ?section ?claim WHERE {
+ ?c a crd:Claim ; crd:claimText ?claim ; crd:statedIn ?sec ; crd:aboutEntity ?e .
+ VALUES ?e { crd:AphidAssociatedArthropod crd:EpisyrphusBalteatus crd:SRC_PestScoutingFeed }
+ ?sec crd:sectionNumber ?section .
+ ?doc crd:hasSection ?sec ; crd:docId ?docId . }""",
+ ),
+ (
+ "XD-3",
+ "Which safeguards exist only on paper (blocking or not, but not automated)?",
+ P
+ + """SELECT ?docId ?control ?blocking WHERE {
+ ?ctrl a crd:Control ; rdfs:label ?control ;
+ crd:isAutomated false ; crd:isBlocking ?blocking .
+ ?doc crd:declaresControl ?ctrl ; crd:docId ?docId . }""",
+ ),
+ (
+ "XD-4",
+ "Which reference source is superseded, and which section documents its defect?",
+ P
+ + """SELECT ?supersedingSource ?supersededSource ?documentingDoc ?section ?sectionLabel WHERE {
+ ?winner crd:supersedes ?loser .
+ BIND(REPLACE(STR(?winner), "^.*#", "") AS ?supersedingSource)
+ BIND(REPLACE(STR(?loser), "^.*#", "") AS ?supersededSource)
+ ?sec crd:documentsRiskIn ?loser ; crd:sectionNumber ?section ; rdfs:label ?sectionLabel .
+ ?doc crd:hasSection ?sec ; crd:docId ?documentingDoc . }""",
+ ),
+ (
+ "XD-5",
+ "Cross-document section links: which section of one document bears on another?",
+ P
+ + """SELECT ?fromDoc ?fromSec ?toDoc ?toSec WHERE {
+ ?a crd:relatedSection ?b .
+ ?da crd:hasSection ?a ; crd:docId ?fromDoc . ?a crd:sectionNumber ?fromSec .
+ ?db crd:hasSection ?b ; crd:docId ?toDoc . ?b crd:sectionNumber ?toSec .
+ FILTER(?fromDoc != ?toDoc) }""",
+ ),
+ (
+ "XD-6",
+ "Access control: which ACL groups would a reader need to see this whole chain?",
+ P
+ + """SELECT DISTINCT ?docId ?classification ?acl WHERE {
+ ?doc crd:docId ?docId ; crd:classification ?classification ; crd:aclGroup ?acl . }""",
+ ),
+]
+
+
+def clean(v):
+ if not isinstance(v, str):
+ return v
+ if v.startswith("<") and v.endswith(">"):
+ return v[1:-1].split("#")[-1]
+ if v.startswith('"'):
+ body = v[1:]
+ for cut in ('"^^', '"@', '"'):
+ if cut in body:
+ body = body.split(cut)[0]
+ break
+ return body
+ return v
+
+
+def main():
+ if not BIN.exists():
+ sys.exit(f"engine binary not found: {BIN}. Run cargo build --release.")
+ failures = 0
+ for qid, title, sparql in QUESTIONS:
+ batch = f"clear\nload {BUNDLE}\nquery {json.dumps(' '.join(sparql.split()))}\n"
+ out = subprocess.run(
+ [str(BIN), "batch", "-"], input=batch, capture_output=True, text=True
+ ).stdout.strip()
+ last = out.split("\n")[-1] if out else "{}"
+ try:
+ res = json.loads(last).get("result", {})
+ rows = res.get("results", [])
+ except Exception:
+ print(f"\n{qid} {title}\n PARSE FAILURE: {last[:200]}")
+ failures += 1
+ continue
+
+ print(f"\n{qid} {title}")
+ print(f" rows: {len(rows)}")
+ if not rows:
+ failures += 1
+ print(" NO RESULTS")
+ for row in rows:
+ print(" ", {k: clean(v) for k, v in row.items()})
+
+ print(f"\n{'-' * 60}")
+ print(f"{len(QUESTIONS) - failures}/{len(QUESTIONS)} cross-document questions answered")
+ return 1 if failures else 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/demo/extract.py b/demo/extract.py
new file mode 100644
index 00000000..f60135e1
--- /dev/null
+++ b/demo/extract.py
@@ -0,0 +1,326 @@
+#!/usr/bin/env python3
+"""
+Extract a knowledge graph from the demonstration documents.
+
+Deliberately mirrors the MCP-native convention: this script does NOT reason.
+It sends each document to whatever OpenAI-compatible model endpoint is
+configured, asks for Turtle constrained to the loaded vocabulary, and then
+hands the result to the engine's `onto_vocab_check` for a closed-world
+check of every term used.
+
+Terms the model invents are caught there, before anything reaches the store.
+That gate is the point: an LLM writes the graph, a deterministic checker
+decides whether it is allowed to exist.
+
+Usage:
+ python3 demo/extract.py # extract + vocab check
+ python3 demo/extract.py --dry-run # show prompts only
+"""
+
+import argparse
+import json
+import os
+import pathlib
+import re
+import sys
+import urllib.request
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+DOCS = ROOT / "demo" / "documents"
+OUT = ROOT / "demo" / "extracted"
+ENGINE = os.environ.get("ONTO_ENGINE", "http://127.0.0.1:8137")
+LLM_BASE = os.environ.get("ONTO_LLM_BASE_URL", "http://localhost:8081/v1").rstrip("/")
+LLM_KEY = os.environ.get("ONTO_LLM_API_KEY", "not-needed")
+
+VOCAB_HINT = """
+Classes you may use:
+ :Document :StandardOperatingProcedure :RegulatoryGuidance :TrialReport
+ :Section :Claim :Control :ReferenceSource
+ :Candidate :Organism :Pest :Pathogen :BeneficialOrganism :Pollinator :NaturalEnemy
+ :FieldTrial :TargetProtein :Mutation :EvidenceGap :StageGate
+
+Object properties you may use:
+ :hasSection :parentSection :relatedSection :statedIn :aboutEntity
+ :supersedes :declaresControl :controlOver :blocksProgressionOf :documentsRiskIn
+
+Datatype properties you may use:
+ :docId :classification :aclGroup :sectionNumber :claimText :isAutomated :isBlocking
+ rdfs:label rdfs:comment
+"""
+
+PROMPT = """You are extracting a knowledge graph from a controlled document.
+
+Return ONLY Turtle. No prose, no markdown fences, no explanation.
+
+Use EXACTLY these prefixes:
+@prefix : <https://w3id.org/dcat-us-demo#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+
+STRICT RULE: use ONLY the classes and properties listed below. Do not invent
+any term. If something in the document does not fit the vocabulary, omit it.
+{vocab}
+
+Extract:
+1. One :Document node, IRI :DOC_{docid_safe}, with :docId, rdfs:label,
+ :classification, :aclGroup, and the most specific document type.
+2. A :Section node per numbered section, IRI :SEC_{docid_safe}_<number with dots as underscores>,
+ with :sectionNumber and rdfs:label, linked by :hasSection from the document.
+3. :Claim nodes for assertions that matter to candidate progression or organism
+ classification. Each with :claimText and :statedIn pointing at its section.
+4. Where a section states a safeguard, a :Control node with :isAutomated and
+ :isBlocking, linked with :declaresControl.
+5. Where a section records a defect or limitation of a data source, link the
+ section with :documentsRiskIn.
+
+DOCUMENT ID: {docid}
+
+DOCUMENT:
+{body}
+"""
+
+
+def rpc(method, params, sid=None, _i=[0]):
+ _i[0] += 1
+ headers = {
+ "Content-Type": "application/json",
+ "Accept": "application/json, text/event-stream",
+ }
+ if sid:
+ headers["Mcp-Session-Id"] = sid
+ req = urllib.request.Request(
+ ENGINE + "/mcp",
+ data=json.dumps(
+ {"jsonrpc": "2.0", "id": _i[0], "method": method, "params": params}
+ ).encode(),
+ headers=headers,
+ )
+ r = urllib.request.urlopen(req, timeout=180)
+ sid2 = r.headers.get("Mcp-Session-Id") or sid
+ body = r.read().decode().strip()
+ if body.startswith("{"):
+ return json.loads(body), sid2
+ for line in body.split("\n"):
+ if line.startswith("data:"):
+ try:
+ fr = json.loads(line[5:].strip())
+ if "result" in fr or "error" in fr:
+ return fr, sid2
+ except Exception:
+ pass
+ return None, sid2
+
+
+def tool_text(res):
+ if not res:
+ return ""
+ return res.get("result", {}).get("content", [{}])[0].get("text", "")
+
+
+def new_session():
+ _, sid = rpc(
+ "initialize",
+ {
+ "protocolVersion": "2025-03-26",
+ "capabilities": {},
+ "clientInfo": {"name": "extractor", "version": "1"},
+ },
+ )
+ return sid
+
+
+# Set once the vocabulary has been loaded, so a session reset can restore it.
+_VOCAB_PATH = None
+
+
+def call_tool(name, arguments, sid):
+ """Call an MCP tool, re-initialising if the session has expired.
+
+ Extraction turns can take minutes on a local model, which is long enough
+ for the server to drop the session. A stale session surfaces as a bare
+ HTTP 404, so treat that as "reconnect and retry once" rather than fatal.
+
+ A new session starts with an EMPTY store, so the vocabulary must be
+ reloaded before retrying. Without this, onto_vocab_check silently reports
+ "0 declared terms" and checks nothing -- it returns conforms=false with a
+ warning rather than an error, which is easy to mistake for a failed
+ extraction when it is actually a failed setup.
+ """
+ try:
+ res, sid = rpc("tools/call", {"name": name, "arguments": arguments}, sid)
+ return res, sid
+ except urllib.error.HTTPError as e:
+ if e.code != 404:
+ raise
+ sid = new_session()
+ if _VOCAB_PATH:
+ rpc("tools/call", {"name": "onto_load", "arguments": {
+ "path": _VOCAB_PATH, "name": "dcat-us-vocab", "force_recompile": True}}, sid)
+ res, sid = rpc("tools/call", {"name": name, "arguments": arguments}, sid)
+ return res, sid
+
+
+def call_model(prompt):
+ # Discover the served model id: MLX servers reject a mismatched id outright.
+ model = os.environ.get("ONTO_LLM_MODEL", "")
+ if not model:
+ req = urllib.request.Request(
+ LLM_BASE + "/models", headers={"Authorization": f"Bearer {LLM_KEY}"}
+ )
+ model = json.load(urllib.request.urlopen(req, timeout=30))["data"][0]["id"]
+
+ req = urllib.request.Request(
+ LLM_BASE + "/chat/completions",
+ data=json.dumps(
+ {
+ "model": model,
+ "messages": [{"role": "user", "content": prompt}],
+ "temperature": 0.2,
+ "top_p": 0.8,
+ "max_tokens": 4096,
+ "chat_template_kwargs": {"enable_thinking": False},
+ }
+ ).encode(),
+ headers={
+ "Content-Type": "application/json",
+ "Authorization": f"Bearer {LLM_KEY}",
+ },
+ )
+ resp = json.load(urllib.request.urlopen(req, timeout=600))
+ return resp["choices"][0]["message"]["content"], model
+
+
+PREFIXES = (
+ "@prefix : <https://w3id.org/dcat-us-demo#> .\n"
+ "@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n"
+ "@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n"
+)
+
+
+def normalise(text):
+ """Recover parseable Turtle from a chatty model response.
+
+ Small local models reliably do three things the prompt told them not to:
+ wrap output in code fences, open with a prose sentence, and omit the
+ prefix block. None of that changes the extracted CONTENT, so it is
+ normalised here rather than treated as a failure. Anything that survives
+ still has to pass onto_vocab_check.
+ """
+ text = re.sub(r"```(?:turtle|ttl)?", "", text).strip()
+
+ # Drop any prose preamble. Start at the first @prefix if there is one;
+ # otherwise at the first line that is unambiguously a SUBJECT, i.e.
+ # "<iri> a <type>". Matching a bare leading ":token" is wrong: predicate
+ # continuation lines like ':docId "DOC-114" ;' match it too, and starting
+ # there silently decapitates the statement and loses its subject.
+ lines = text.split("\n")
+ start = None
+ for i, line in enumerate(lines):
+ if line.strip().startswith(("@prefix", "@base")):
+ start = i
+ break
+ if start is None:
+ for i, line in enumerate(lines):
+ if re.match(r"^\s*:\S+\s+(a|rdf:type)\s", line):
+ start = i
+ break
+ text = "\n".join(lines[start if start is not None else 0 :]).strip()
+
+ # Drop any trailing prose after the last statement-terminating line.
+ lines = text.split("\n")
+ end = len(lines)
+ for i in range(len(lines) - 1, -1, -1):
+ if lines[i].rstrip().endswith((".", ";", ",")):
+ end = i + 1
+ break
+ text = "\n".join(lines[:end]).strip()
+
+ if "@prefix :" not in text:
+ text = PREFIXES + "\n" + text
+ return text
+
+
+strip_fences = normalise # keep the call site stable
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--dry-run", action="store_true")
+ args = ap.parse_args()
+
+ OUT.mkdir(parents=True, exist_ok=True)
+ docs = sorted(DOCS.glob("*.md"))
+ if not docs:
+ sys.exit("no documents found")
+
+ # Load the vocabulary the extraction will be checked against.
+ sid = None
+ if not args.dry_run:
+ _, sid = rpc(
+ "initialize",
+ {
+ "protocolVersion": "2025-03-26",
+ "capabilities": {},
+ "clientInfo": {"name": "extractor", "version": "1"},
+ },
+ )
+ global _VOCAB_PATH
+ bundle = ROOT / "demo" / "bundle" / "dcat-us-vocab.ttl"
+ _VOCAB_PATH = str(bundle)
+ res, sid = call_tool(
+ "onto_load",
+ {"path": str(bundle), "name": "dcat-us-vocab", "force_recompile": True},
+ sid,
+ )
+ print(f"vocabulary loaded: {tool_text(res)[:100]}")
+
+ for doc in docs:
+ raw = doc.read_text()
+ m = re.match(r"^---\n(.*?)\n---\n(.*)$", raw, re.S)
+ front, body = (m.group(1), m.group(2)) if m else ("", raw)
+ docid = re.search(r'doc_id:\s*"?([^"\n]+)"?', front)
+ docid = docid.group(1).strip() if docid else doc.stem
+ safe = docid.replace("-", "_")
+
+ prompt = PROMPT.format(
+ vocab=VOCAB_HINT, docid=docid, docid_safe=safe, body=front + "\n" + body
+ )
+ if args.dry_run:
+ print(f"--- {docid} prompt {len(prompt)} chars ---")
+ continue
+
+ print(f"\nextracting {docid} ...", flush=True)
+
+ # Small models occasionally collapse into repetition. Detect it and
+ # retry rather than writing degenerate output and calling it a graph.
+ for attempt in range(1, 4):
+ content, model = call_model(prompt)
+ (OUT / f"{docid}.raw.attempt{attempt}.txt").write_text(content)
+ if re.search(r"(\b\S{1,6}\b[ .]{1,3})\1{15,}", content):
+ print(f" attempt {attempt}: degenerate repetition, retrying")
+ continue
+ ttl = strip_fences(content)
+ if ":" in ttl and len(ttl) > 200:
+ break
+ print(f" attempt {attempt}: output too thin, retrying")
+ else:
+ print(f" GAVE UP on {docid} after 3 attempts")
+ ttl = strip_fences(content)
+ path = OUT / f"{docid}.ttl"
+ path.write_text(
+ f"# Extracted from {doc.name} on 2026-08-10\n"
+ f"# Model: {model}\n"
+ f"# Verified with onto_vocab_check. See demo/extracted/REPORT.md\n\n" + ttl
+ )
+ print(f" wrote {path.name} ({len(ttl)} chars)")
+
+ res, sid = call_tool("onto_validate", {"input": ttl, "inline": True}, sid)
+ v = tool_text(res)
+ print(f" syntax : {v[:120]}")
+
+ res, sid = call_tool("onto_vocab_check", {"data": ttl, "inline": True}, sid)
+ print(f" vocab : {tool_text(res)[:400]}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/demo/kpi_context_graph.py b/demo/kpi_context_graph.py
new file mode 100644
index 00000000..f71a768d
--- /dev/null
+++ b/demo/kpi_context_graph.py
@@ -0,0 +1,259 @@
+#!/usr/bin/env python3
+"""
+Derive the KPI context graph, and answer impact questions with a query.
+
+The other pitch showed a context graph connecting ontologies, agents and 99
+KPIs. It was drawn. This one is DERIVED: every edge comes from a
+`computedFrom` / `appliesTo` / `dependsOnKPI` / `governedBy` triple that the
+KPI itself declares. A derived graph cannot drift from the model, because it
+is a rendering of the model.
+
+Three things it does that a drawn diagram cannot:
+
+ 1. `graph` render the context graph as Mermaid, from traversal.
+ 2. `impact` given a changed IRI, list the KPIs that depend on it,
+ transitively through composite KPIs. This is the question
+ nobody can answer when KPIs live in a config table.
+ 3. `breach` evaluate the indicators against the current data and report
+ which are breached, with their governing document.
+
+Usage:
+ python3 demo/kpi_context_graph.py graph
+ python3 demo/kpi_context_graph.py impact :targetsOrganism
+ python3 demo/kpi_context_graph.py breach
+"""
+
+import json
+import pathlib
+import subprocess
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+BIN = ROOT / "target" / "release" / "open-ontologies"
+LOADS = [
+ ROOT / "demo" / "bundle" / "dcat-us-full.ttl",
+ ROOT / "demo" / "ontology" / "dcat-us-kpi.ttl",
+]
+
+P = """PREFIX crd: <https://w3id.org/dcat-us-demo#>
+PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
+PREFIX owl: <http://www.w3.org/2002/07/owl#>
+"""
+
+
+def short(v):
+ if not isinstance(v, str):
+ return v
+ if v.startswith("<") and v.endswith(">"):
+ return v[1:-1].split("#")[-1]
+ if v.startswith('"'):
+ body = v[1:]
+ for cut in ('"^^', '"@', '"'):
+ if cut in body:
+ return body.split(cut)[0]
+ return body
+ return v
+
+
+def query(sparql):
+ lines = ["clear"] + [f"load {p}" for p in LOADS]
+ lines.append(f"query {json.dumps(' '.join(sparql.split()))}")
+ out = subprocess.run(
+ [str(BIN), "batch", "-"], input="\n".join(lines) + "\n", capture_output=True, text=True
+ ).stdout.strip()
+ if not out:
+ return []
+ try:
+ rows = json.loads(out.split("\n")[-1]).get("result", {}).get("results", [])
+ except Exception:
+ return []
+ return [{k: short(v) for k, v in r.items()} for r in rows]
+
+
+def cmd_graph():
+ """Render the context graph from traversal.
+
+ Grouped into layers and styled so it reads at a glance: what the model is
+ computed FROM on the left, the indicators in the middle, what they measure
+ and what mandates them on the right. Blocking indicators are highlighted,
+ because "which of these stops a stage gate" is the first thing anyone asks.
+ """
+ inputs = query(P + "SELECT ?kpi ?input WHERE { ?kpi a/rdfs:subClassOf* crd:KPI ; crd:computedFrom ?input }")
+ applies = query(P + "SELECT ?kpi ?cls WHERE { ?kpi a/rdfs:subClassOf* crd:KPI ; crd:appliesTo ?cls }")
+ comps = query(P + "SELECT ?kpi ?component WHERE { ?kpi crd:dependsOnKPI ?component }")
+ govs = query(P + "SELECT ?kpi ?doc WHERE { ?kpi crd:governedBy ?doc }")
+ labels = {r["kpi"]: r["label"] for r in
+ query(P + "SELECT ?kpi ?label WHERE { ?kpi a/rdfs:subClassOf* crd:KPI ; rdfs:label ?label }")}
+ # Scope to KPIs: Controls also carry isBlocking, and would otherwise leak
+ # into the indicator styling.
+ blocking = {r["kpi"] for r in
+ query(P + "SELECT ?kpi WHERE { ?kpi a/rdfs:subClassOf* crd:KPI ; crd:isBlocking true }")}
+
+ kpis = sorted(labels)
+ ins = sorted({r["input"] for r in inputs})
+ classes = sorted({r["cls"] for r in applies})
+ docs = sorted({r["doc"] for r in govs})
+
+ def wrap(text):
+ words, lines, cur = text.split(), [], ""
+ for w in words:
+ if len(cur) + len(w) > 18:
+ lines.append(cur); cur = w
+ else:
+ cur = f"{cur} {w}".strip()
+ if cur:
+ lines.append(cur)
+ return "<br/>".join(lines)
+
+ out = ["flowchart LR"]
+ out.append(' subgraph SRC["Model terms these are computed from"]')
+ out.append(" direction TB")
+ for n in ins:
+ out.append(f' {n}["{wrap(n)}"]')
+ out.append(" end")
+ out.append(' subgraph IND["Indicators"]')
+ out.append(" direction TB")
+ for k in kpis:
+ mark = " ⛔" if k in blocking else ""
+ out.append(f' {k}("{wrap(labels[k])}{mark}")')
+ out.append(" end")
+ out.append(' subgraph TGT["What they measure / what mandates them"]')
+ out.append(" direction TB")
+ for n in classes:
+ out.append(f' {n}["{wrap(n)}"]')
+ for n in docs:
+ out.append(f' {n}[["{wrap(n)}"]]')
+ out.append(" end")
+
+ for r in inputs:
+ out.append(f' {r["input"]} --> {r["kpi"]}')
+ for r in comps:
+ out.append(f' {r["kpi"]} ==>|"composed of"| {r["component"]}')
+ for r in applies:
+ out.append(f' {r["kpi"]} -.-> {r["cls"]}')
+ for r in govs:
+ out.append(f' {r["doc"]} -.->|"mandates"| {r["kpi"]}')
+
+ out.append(" classDef blocking fill:#f8d7da,stroke:#a4232f,stroke-width:2px,color:#5c1119")
+ out.append(" classDef kpi fill:#e8eef7,stroke:#3d5a80,color:#1d2f45")
+ out.append(" classDef term fill:#f4f1ea,stroke:#b0a99a,color:#3a352c")
+ out.append(" classDef doc fill:#fff3cd,stroke:#a68b2c,color:#4a3c0c")
+ normal = [k for k in kpis if k not in blocking]
+ if normal:
+ out.append(" class " + ",".join(normal) + " kpi")
+ if blocking:
+ out.append(" class " + ",".join(sorted(blocking)) + " blocking")
+ if ins or classes:
+ out.append(" class " + ",".join(ins + classes) + " term")
+ if docs:
+ out.append(" class " + ",".join(docs) + " doc")
+
+ nodes = len(ins) + len(kpis) + len(classes) + len(docs)
+ edges = len(inputs) + len(comps) + len(applies) + len(govs)
+ print("```mermaid")
+ print("\n".join(out))
+ print("```")
+ print(f"\n{nodes} nodes, {edges} edges. Every one derived by traversal;")
+ print("none placed by hand. Blocking indicators marked.")
+
+
+def cmd_impact(changed):
+ """Which KPIs depend on a changed IRI, transitively?"""
+ term = changed.lstrip(":")
+ direct = query(
+ P
+ + f"""SELECT ?kpi ?label ?blocking WHERE {{
+ ?kpi crd:computedFrom crd:{term} .
+ OPTIONAL {{ ?kpi rdfs:label ?label }}
+ OPTIONAL {{ ?kpi crd:isBlocking ?blocking }}
+}}"""
+ )
+ # dependsOnKPI is transitive, so the closure comes back from one query.
+ downstream = query(
+ P
+ + f"""SELECT DISTINCT ?composite ?label WHERE {{
+ ?direct crd:computedFrom crd:{term} .
+ ?composite crd:dependsOnKPI ?direct .
+ OPTIONAL {{ ?composite rdfs:label ?label }}
+}}"""
+ )
+
+ print(f"Impact of a change to :{term}\n")
+ if not direct and not downstream:
+ print(" no KPI declares this as an input")
+ return
+ print(f" Directly computed from it ({len(direct)}):")
+ for r in direct:
+ flag = " [BLOCKING]" if r.get("blocking") == "true" else ""
+ print(f" - {r.get('label', r['kpi'])}{flag}")
+ print(f"\n Composite KPIs that inherit the change ({len(downstream)}):")
+ for r in downstream:
+ print(f" - {r.get('label', r['composite'])}")
+ print("\n This is the question a config-table KPI registry cannot answer.")
+
+
+def cmd_breach():
+ """Evaluate indicators against current data."""
+ print("Indicator evaluation against the loaded graph\n")
+
+ checks = [
+ ("Field translation efficiency", 0.80, "lower value is a breach",
+ P + "SELECT ?c ?v WHERE { ?c crd:fieldToScreenRatio ?v . FILTER(?v < 0.80) }"),
+ ("Unexplained residual variance", 0.20, "higher value is a breach",
+ P + "SELECT ?c ?v WHERE { ?c crd:unexplainedResidualVariance ?v . FILTER(?v > 0.20) }"),
+ ("Binding evidence freshness", 1.0, "any stale protein is a breach",
+ P + "SELECT ?p WHERE { ?p crd:affectedByMutation ?m ; crd:isStale true }"),
+ ("Evidence gap closure", 1.0, "any open gap is a breach",
+ P + "SELECT ?c ?g WHERE { ?c crd:hasEvidenceGap ?g }"),
+ ("Trial data completeness", 1.0, "a field trial with no plot is a breach",
+ P + "SELECT ?t WHERE { ?t a crd:FieldTrial . FILTER NOT EXISTS { ?t crd:hasPlot ?p } }"),
+ ]
+ breaches = 0
+ for name, threshold, note, sparql in checks:
+ rows = query(sparql)
+ status = f"BREACH ({len(rows)})" if rows else "ok"
+ if rows:
+ breaches += 1
+ print(f" [{status:>12}] {name} (threshold {threshold}, {note})")
+ for r in rows:
+ print(f" {json.dumps(r)}")
+
+ safety = query(
+ P
+ + """SELECT ?candidate ?organism ?doc WHERE {
+ ?candidate crd:targetsOrganism ?organism .
+ ?organism a ?cls . ?cls rdfs:subClassOf* crd:BeneficialOrganism .
+ OPTIONAL { ?kpi a crd:SafetyKPI ; crd:governedBy ?d . ?d crd:docId ?doc }
+}"""
+ )
+ print()
+ if safety:
+ breaches += 1
+ print(f" [ BREACH] Beneficial organism exposure [BLOCKING, threshold 0.0]")
+ for r in safety:
+ gov = f", mandated by {r['doc']}" if r.get("doc") else ""
+ print(f" {r['candidate']} targets {r['organism']}{gov}")
+ else:
+ print(" [ ok] Beneficial organism exposure")
+
+ print(f"\n {breaches} indicators breached. Stage gate readiness: FAIL")
+ print(" Every breach traces to a triple, and the blocking one traces to the")
+ print(" controlled document that mandates it.")
+
+
+def main():
+ if not BIN.exists():
+ sys.exit(f"engine binary not found: {BIN}")
+ cmd = sys.argv[1] if len(sys.argv) > 1 else "graph"
+ if cmd == "graph":
+ cmd_graph()
+ elif cmd == "impact":
+ cmd_impact(sys.argv[2] if len(sys.argv) > 2 else ":targetsOrganism")
+ elif cmd == "breach":
+ cmd_breach()
+ else:
+ sys.exit(f"unknown command: {cmd}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/demo/ontology_from_docs.py b/demo/ontology_from_docs.py
new file mode 100644
index 00000000..3e625cec
--- /dev/null
+++ b/demo/ontology_from_docs.py
@@ -0,0 +1,1644 @@
+#!/usr/bin/env python3
+"""
+Ontology out of text: derive the ontology FROM a document corpus, then
+populate it, with PII tokenised before anything reaches a model.
+
+This is domain-agnostic. Nothing about crop protection, biology or any other
+subject is hardcoded; the vocabulary is whatever the documents turn out to be
+about.
+
+ STAGE 1 READ every document in the folder
+ STAGE 2 TOKENISE detect and replace sensitive values with stable tokens
+ STAGE 3 DERIVE propose classes, properties and disjointness per
+ document, then merge into ONE candidate ontology
+ STAGE 4 POPULATE extract instances, constrained to the derived ontology
+ STAGE 5 VERIFY closed-world check that nothing was invented
+ STAGE 6 SCAN reason, and report contradictions
+
+The two-phase split is the point. A single pass either invents terms freely,
+which gives an unusable model, or is pinned to a vocabulary written in advance,
+which cannot describe documents nobody has read yet. Deriving first and
+constraining second gives a vocabulary that came from the corpus AND a
+closed-world check with teeth.
+
+Tokenisation happens BEFORE derivation, so no model ever sees a real value.
+Tokens are deterministic, which means the same value produces the same token in
+every document: the token doubles as a join key for exact-match entity
+resolution, achieved without any component handling the raw value.
+
+Usage:
+ python3 demo/ontology_from_docs.py --corpus demo/corpus/dcat-us
+ python3 demo/ontology_from_docs.py --corpus demo/corpus/dcat-us --cached
+"""
+
+import argparse
+import concurrent.futures as cf
+import hashlib
+import hmac
+import json
+import os
+import pathlib
+import re
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+
+from rdflib import Graph, OWL, RDF, RDFS, URIRef
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+BIN = ROOT / "target" / "release" / "open-ontologies"
+ENGINE = os.environ.get("ONTO_ENGINE", "http://127.0.0.1:8137")
+LLM_BASE = os.environ.get("ONTO_LLM_BASE_URL", "http://localhost:8081/v1").rstrip("/")
+LLM_KEY = os.environ.get("ONTO_LLM_API_KEY", "not-needed")
+NS = "https://w3id.org/dcat-us-demo#"
+
+# Samples voted over when proposing disjointness. Odd, so a majority exists.
+SAMPLES = int(os.environ.get("ONTO_DISJOINT_SAMPLES", "3"))
+
+# Repair attempts per non-conforming graph before it goes to human review.
+REPAIR_ROUNDS = int(os.environ.get("ONTO_REPAIR_ROUNDS", "3"))
+
+# Every automated resolution decision goes on the record: what changed, in
+# which document, by which rule. The Studio's resolution panel renders this
+# ledger and the judgement calls in it can be reverted.
+LEDGER = {"path": None}
+
+
+def record(kind, **kw):
+ if LEDGER["path"]:
+ with open(LEDGER["path"], "a") as fh:
+ fh.write(json.dumps({"kind": kind, **kw}) + "\n")
+
+if os.environ.get("NO_COLOR"):
+ BOLD = DIM = RED = GRN = YEL = OFF = ""
+else:
+ BOLD, DIM, RED, GRN, YEL, OFF = "\033[1m", "\033[2m", "\033[31m", "\033[32m", "\033[33m", "\033[0m"
+
+print = __builtins__.print if not isinstance(__builtins__, dict) else __builtins__["print"]
+
+
+# --------------------------------------------------------------------------
+# Tokenisation: delegated to the swappable detector + vault module
+# --------------------------------------------------------------------------
+
+sys.path.insert(0, str(ROOT / "demo"))
+from tokenisation import build as build_tokeniser # noqa: E402
+
+
+# --------------------------------------------------------------------------
+# Model + engine plumbing
+# --------------------------------------------------------------------------
+
+def call_model(prompt: str, max_tokens: int = 3000, attempts: int = 4,
+ temperature: float = 0.2, timeout: int = 900) -> str:
+ """Call the model, backing off on overload.
+
+ A local inference server under parallel load answers 503 or 504 rather
+ than queueing. Treating that as a permanent failure loses most of a run,
+ which is exactly what happened before this was added.
+ """
+ last = None
+ for i in range(attempts):
+ try:
+ return _call_model_once(prompt, max_tokens, temperature, timeout)
+ except urllib.error.HTTPError as e:
+ if e.code not in (429, 500, 502, 503, 504):
+ raise
+ last = e
+ except urllib.error.URLError as e:
+ last = e
+ time.sleep(2 ** i)
+ raise RuntimeError(f"model unavailable after {attempts} attempts: {last}")
+
+
+def _call_model_once(prompt: str, max_tokens: int = 3000,
+ temperature: float = 0.2, timeout: int = 900) -> str:
+ model = os.environ.get("ONTO_LLM_MODEL", "")
+ if not model:
+ req = urllib.request.Request(LLM_BASE + "/models", headers={"Authorization": f"Bearer {LLM_KEY}"})
+ model = json.load(urllib.request.urlopen(req, timeout=30))["data"][0]["id"]
+ req = urllib.request.Request(
+ LLM_BASE + "/chat/completions",
+ data=json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}],
+ "temperature": temperature, "max_tokens": max_tokens,
+ "chat_template_kwargs": {"enable_thinking": False}}).encode(),
+ headers={"Content-Type": "application/json", "Authorization": f"Bearer {LLM_KEY}"})
+ msg = json.load(urllib.request.urlopen(req, timeout=900))["choices"][0]["message"]
+ # A null content field is a valid response shape, not an error: the model
+ # produced no visible text. Callers regex over this, so hand back a string.
+ return msg.get("content") or ""
+
+
+def mcp(method, params, sid=[None]):
+ h = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream"}
+ if sid[0]:
+ h["Mcp-Session-Id"] = sid[0]
+ req = urllib.request.Request(ENGINE + "/mcp", headers=h, data=json.dumps(
+ {"jsonrpc": "2.0", "id": 1, "method": method, "params": params}).encode())
+ r = urllib.request.urlopen(req, timeout=180)
+ sid[0] = r.headers.get("Mcp-Session-Id") or sid[0]
+ body = r.read().decode().strip()
+ if body.startswith("{"):
+ return json.loads(body)
+ for line in body.split("\n"):
+ if line.startswith("data:"):
+ try:
+ fr = json.loads(line[5:].strip())
+ if "result" in fr or "error" in fr:
+ return fr
+ except Exception:
+ pass
+ return {}
+
+
+def tool_text(res):
+ return res.get("result", {}).get("content", [{}])[0].get("text", "") if res else ""
+
+
+def engine(cmds):
+ return subprocess.run([str(BIN), "batch", "-"], input="\n".join(cmds) + "\n",
+ capture_output=True, text=True).stdout.strip()
+
+
+# Declare every prefix a model might plausibly use. One undeclared prefix
+# (rdf: was the culprit) aborts the parse at first use, so a 2700-line
+# ontology silently loaded as 195 triples: all the classes, none of the
+# properties, because classes happen to come first in the file.
+PREFIXES = (f"@prefix : <{NS}> .\n"
+ "@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n"
+ "@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n"
+ "@prefix owl: <http://www.w3.org/2002/07/owl#> .\n"
+ "@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n"
+ "@prefix skos: <http://www.w3.org/2004/02/skos/core#> .\n"
+ "@prefix dcterms: <http://purl.org/dc/terms/> .\n")
+
+
+def is_valid_turtle(ttl: str) -> tuple[bool, str]:
+ """Parse-check a fragment on its own."""
+ import tempfile
+ with tempfile.NamedTemporaryFile("w", suffix=".ttl", delete=False) as fh:
+ fh.write(ttl)
+ path = fh.name
+ try:
+ out = subprocess.run([str(BIN), "validate", path], capture_output=True, text=True).stdout
+ return ('"ok":true' in out), out.strip()[:160]
+ finally:
+ pathlib.Path(path).unlink(missing_ok=True)
+
+
+# Structural vocabulary the chunker imposes on every corpus. The extractor
+# reaches for it in every document because the documents really are divided
+# into sections; no per-document derivation proposes it, because it is a
+# property of how documents are read rather than of what they say.
+STRUCTURAL = """
+:Section a owl:Class ;
+ rdfs:label "Section" ;
+ rdfs:comment "A numbered division of a document, as identified by the chunker." .
+
+:hasSection a owl:ObjectProperty ;
+ rdfs:label "has section" ;
+ rdfs:domain :Document ;
+ rdfs:range :Section .
+
+:hasSectionNumber a owl:DatatypeProperty ;
+ rdfs:label "has section number" ;
+ rdfs:domain :Section ;
+ rdfs:range xsd:string .
+"""
+
+
+def body_of(frag: str) -> str:
+ """The statements of a fragment, with any prefix header removed."""
+ return "\n".join(l for l in frag.splitlines()
+ if not l.lstrip().startswith(("@prefix", "@base"))).strip()
+
+
+def mend(body: str) -> str:
+ """Repair the syntax slips a generator makes at the end of a statement.
+
+ A predicate list that ends `; .` or a property block whose last predicate
+ keeps its separator are the two failures seen in practice. Both are
+ unambiguous, so fixing them costs nothing and recovers a whole fragment
+ that would otherwise be discarded along with every term it declares.
+ """
+ body = re.sub(r";\s*\.", " .", body) # dangling separator before the stop
+ body = re.sub(r";\s*\n\s*(?=[:\w]+\s+a\s)", " .\n", body) # missing stop between subjects
+ body = re.sub(r",\s*\.", " .", body)
+
+ # A bare `disjointWith`. Turtle has no default predicate namespace, so an
+ # unprefixed name is a parse error, not a lookup failure.
+ body = re.sub(r"(?<![:\w])disjointWith\b", "owl:disjointWith", body)
+
+ # A standard prefix pasted after the default one, giving `:rdfs:label`.
+ # This parses, which is what makes it dangerous: it silently mints a term
+ # in the local namespace that looks like the standard one, and the
+ # closed-world check then correctly reports a term nobody declared.
+ body = re.sub(r"(?<![\w:]):(rdf|rdfs|owl|xsd|skos|dcterms):", r"\1:", body)
+
+ # A type assertion whose object was lost, leaving the keyword exposed.
+ body = re.sub(r"\ba\s+a\b", " a", body)
+
+ # An IRI the generator split with a space before a token:
+ # ":TimePoint TOK_DATE_x a :TimePoint". Rejoin with an underscore; a bare
+ # token in predicate position truncates every file it is merged into.
+ body = re.sub(r"^(:\w+)[ \t]+(TOK_\w+)\b", r"\1_\2", body, flags=re.M)
+
+ # A statement whose subject was dropped, leaving the predicate in subject
+ # position. Nothing can recover the intended subject, so the statement goes
+ # rather than the fragment. Only an unindented occurrence starts a
+ # statement: an indented one is a continuation of a valid subject above it,
+ # and removing those is what breaks an otherwise sound block. The whole
+ # orphan runs to its terminating dot.
+ kept, skipping = [], False
+ for line in body.splitlines():
+ if re.match(r"^(?:owl:)?:?disjointWith\s", line):
+ skipping = True
+ if skipping:
+ if line.rstrip().endswith("."):
+ skipping = False
+ continue
+ kept.append(line)
+ body = "\n".join(kept)
+
+ # A statement left unterminated before the next subject begins. The blank
+ # line and the following `:Term a` are together unambiguous.
+ body = re.sub(r"(?<=[\"\w>])\s*\n(\s*\n\s*)(?=:\S+\s+(?:a|rdf:type)\s)",
+ r" .\n\1", body)
+ return body
+
+
+def rehead(frag: str) -> str:
+ """A fragment under the canonical prefix header.
+
+ The generator writes its own header and routinely omits a prefix it then
+ uses (`rdf:` above all). Validating the fragment as written therefore
+ rejects it for a declaration problem rather than a modelling one. Strip
+ whatever header it wrote, impose ours, and judge the statements.
+ """
+ return PREFIXES + "\n" + mend(body_of(frag))
+
+
+def normalise(text: str) -> str:
+ text = re.sub(r"```(?:turtle|ttl)?", "", text).strip()
+ lines = text.split("\n")
+ start = None
+ for i, line in enumerate(lines):
+ if line.strip().startswith(("@prefix", "@base")):
+ start = i
+ break
+ if start is None:
+ for i, line in enumerate(lines):
+ if re.match(r"^\s*:\S+\s+(a|rdf:type)\s", line):
+ start = i
+ break
+ text = "\n".join(lines[start if start is not None else 0:]).strip()
+ lines = text.split("\n")
+ for i in range(len(lines) - 1, -1, -1):
+ if lines[i].rstrip().endswith((".", ";", ",")):
+ text = "\n".join(lines[: i + 1]).strip()
+ break
+ # The same mechanical repairs the vocabulary fragments get. Extracted data
+ # comes from the same generator and fails in the same ways; leaving them
+ # out here is why a whole graph was being discarded for one missing dot.
+ body = mend(body_of(text))
+ if body and not body.endswith("."):
+ body += " ."
+ return PREFIXES + "\n" + body
+
+
+def docx_text(path: pathlib.Path) -> str:
+ if path.suffix == ".md":
+ return path.read_text()
+ return subprocess.run(["pandoc", str(path), "-t", "plain", "--wrap=none"],
+ capture_output=True, text=True).stdout
+
+
+# --------------------------------------------------------------------------
+# Stage 3: derive the vocabulary
+# --------------------------------------------------------------------------
+
+DERIVE_PROMPT = """Read this document and propose the ONTOLOGY it implies. You are
+defining the vocabulary, not describing the specific facts.
+
+Return ONLY Turtle. No prose, no code fences.
+
+@prefix : <{ns}> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+
+Produce:
+1. owl:Class declarations for every KIND of thing the document is about. Use
+ UpperCamelCase local names. Give every class rdfs:label and rdfs:comment.
+2. rdfs:subClassOf where one kind is clearly a kind of another.
+3. owl:ObjectProperty for relationships between things, each with rdfs:domain,
+ rdfs:range and rdfs:label.
+4. owl:DatatypeProperty for attributes, each with rdfs:domain, rdfs:range
+ (an xsd type) and rdfs:label.
+5. owl:disjointWith between any two classes that CANNOT both describe the same
+ thing at the same time. Be deliberate: this is what lets a reasoner detect
+ that two documents contradict each other.
+
+Do NOT create individuals. Classes and properties only.
+
+DOCUMENT:
+{body}
+"""
+
+POPULATE_PROMPT = """Extract the FACTS from this document as instances of a
+given ontology.
+
+Return ONLY Turtle. No prose, no code fences.
+
+@prefix : <{ns}> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+
+Use ONLY the classes and properties in this vocabulary. Invent nothing:
+{vocab}
+
+Every class and property you write must appear VERBATIM in the list above.
+Before writing any term, find it in the list and copy it exactly.
+
+Do NOT build a property name by joining a class name to a field name. Writing
+:hasChangeLogEntryChangeApplied because the subject is a :ChangeLogEntry and the
+field is "applied" produces a term that does not exist. Use the property the
+vocabulary already provides, even when its name is shorter or more general than
+the wording in the document.
+
+If the document asserts something the vocabulary genuinely cannot express,
+OMIT it. A smaller correct graph is the goal; an invented term is a defect,
+not a partial success.
+
+Produce:
+1. One :Document individual, IRI :DOC_{safe}, with rdfs:label and any
+ document-control values present.
+2. A :Section individual per numbered section where the vocabulary has a
+ suitable class, IRI :SEC_{safe}_<number with dots as underscores>.
+3. Individuals for the things the document states facts about, typed with the
+ most specific class available, each with rdfs:label.
+
+ When the document says a thing IS a kind of something, and the vocabulary
+ has a class for that kind, TYPE the thing with that class. Never create an
+ individual to stand for the kind itself.
+
+ "Culture07 is maintained as an adherent line"
+ correct: :Culture07 a :AdherentLine ; rdfs:label "Culture07" .
+ WRONG: :Culture07 a :CellLine .
+ :ADHERENT_LINE a :CellLine ; rdfs:label "Adherent line" .
+
+ The wrong form loses the fact entirely: it records that something called
+ "adherent line" exists, not that Culture07 is one.
+
+ Give each individual EXACTLY ONE type: the single most specific class that
+ fits. A registry row listing a cell line's genotype, doubling time and
+ growth conditions describes ONE cell line; it does not make that cell line
+ a genotype or a doubling time. Attach those as properties, never as extra
+ types.
+
+ correct: :Culture07 a :AdherentLine ;
+ :hasGenotype :GENOTYPE_WT ;
+ :hasDoublingTime "24h" .
+ WRONG: :Culture07 a :CellLine, :CellLineGenotype, :CellLineDoublingTime .
+
+4. Name each individual after the thing itself and nothing else. The IRI for
+ Culture07 is :Culture07 in every document that mentions it. Do not prefix it
+ with the document id, and do not qualify it with the context it appeared in
+ (:CULTURE07_STR, :CELL_LINE_CULTURE07_REGISTRY). The same real thing must
+ get the same IRI everywhere, because that is what lets facts from different
+ documents meet.
+5. The relationships and attribute values the document asserts.
+
+Values that look like TOK_ followed by capitals and a hex string are tokens
+standing in for redacted content. Use them verbatim as identifiers or literals.
+Do not attempt to interpret or expand them.
+
+DOCUMENT:
+{body}
+"""
+
+
+def derive_one(args):
+ path, tokenised, docid = args
+ try:
+ raw = call_model(DERIVE_PROMPT.format(ns=NS, body=tokenised[:12000]))
+ except Exception as e:
+ return docid, None, f"model error: {e}"
+ if re.search(r"(\b\S{1,6}\b[ .]{1,3})\1{15,}", raw):
+ return docid, None, "degenerate output"
+ return docid, normalise(raw), None
+
+
+def snap_to_vocab(body: str, declared: set[str]) -> str:
+ """Snap a term to a declared term it differs from only in case.
+
+ The generator writes `:HasCellLine` where the ontology declares
+ `:hasCellLine`. Closed-world checking is right to reject it, but the intent
+ is unambiguous and there is exactly one candidate, so resolving it here is
+ sound rather than charitable. Anything that does not match a declared term
+ under case folding is left alone: that is real invention and it must stay
+ visible to the check.
+ """
+ fold = {}
+ for d in declared:
+ fold.setdefault(d.lower(), d)
+
+ def swap(m: re.Match) -> str:
+ term = m.group(1)
+ target = fold.get(term.lower())
+ return f":{target}" if target and target != term else m.group(0)
+
+ return re.sub(r":(\w+)", swap, body)
+
+
+def populate_one(args):
+ path, tokenised, docid, vocab = args
+ safe = docid.replace("-", "_")
+ try:
+ raw = call_model(POPULATE_PROMPT.format(ns=NS, vocab=vocab, safe=safe, body=tokenised[:12000]))
+ except Exception as e:
+ return docid, None, f"model error: {e}"
+ if re.search(r"(\b\S{1,6}\b[ .]{1,3})\1{15,}", raw):
+ return docid, None, "degenerate output"
+ declared = set(re.findall(r":(\w+)", vocab))
+ return docid, snap_to_vocab(normalise(raw), declared), None
+
+
+DISJOINT_PROMPT = """Here is a list of classes from an ontology derived from a
+document corpus.
+
+Identify pairs that are MUTUALLY EXCLUSIVE: a single thing cannot be both at
+the same time. Examples of the kind of pair that qualifies: a culture that is
+adherent versus one in suspension; a sample that passed versus failed; a
+record that is authenticated versus unauthenticated.
+
+Return ONLY Turtle, one statement per line, no prose and no fences:
+
+:ClassA owl:disjointWith :ClassB .
+
+Only assert a pair when you are confident. An incorrect disjointness makes the
+ontology inconsistent, which is worse than a missing one. Return nothing at all
+if no pair qualifies.
+
+CLASSES:
+{classes}
+"""
+
+
+def _sibling_groups(classes: list[str], cap: int = 60) -> list[list[str]]:
+ """Batch the class list into groups worth asking a disjointness question about.
+
+ Asking about all 272 classes in one call returns nothing at all: the model
+ answers 20 classes with 3700 characters, 150 with 4700, and 272 with an
+ empty string. That failure is silent, so the run reports zero axioms and
+ looks merely unlucky.
+
+ Grouping is by head noun rather than arbitrary slicing, because that is
+ where mutual exclusivity actually lives. `AdherentLine` and
+ `SuspensionLine` are candidates precisely because they are both kinds
+ of `Line`; two classes with nothing in common almost never need an axiom
+ between them. Small groups are pooled so a group of one is not a call.
+ """
+ def head(name: str) -> str:
+ parts = re.findall(r"[A-Z][a-z0-9]*", name)
+ return parts[-1] if parts else name
+
+ by_head: dict[str, list[str]] = {}
+ for c in classes:
+ by_head.setdefault(head(c), []).append(c)
+
+ groups, pool = [], []
+ for _, members in sorted(by_head.items()):
+ if len(members) >= 2:
+ for i in range(0, len(members), cap):
+ groups.append(sorted(members[i:i + cap]))
+ else:
+ pool.extend(members)
+ for i in range(0, len(pool), cap):
+ chunk = sorted(pool[i:i + cap])
+ if len(chunk) >= 2:
+ groups.append(chunk)
+ return groups
+
+
+def add_disjointness(merged_path: pathlib.Path) -> int:
+ """Propose disjointness across the merged vocabulary.
+
+ Deliberately a SEPARATE pass over the merged class list rather than part of
+ per-document derivation. Disjointness is a cross-document property: whether
+ "adherent" and "suspension" are mutually exclusive is not visible from
+ inside one document that only mentions one of them. Asking per document
+ reliably produced none at all.
+
+ Stability matters more here than coverage. These axioms decide what the
+ reasoner is even capable of detecting, so an axiom set that changes run to
+ run makes every downstream contradiction count unreproducible. Sampling
+ once gave 57, 68, 93, 114 and 150 pairs on an identical class list. So:
+ sample SAMPLES times, keep only pairs a majority agrees on, and cache the
+ result against a hash of the class list. Re-running over the same
+ vocabulary is then exactly reproducible, and a changed vocabulary
+ invalidates the cache by construction.
+ """
+ ttl = merged_path.read_text()
+ classes = sorted(set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", ttl)))
+ if len(classes) < 2:
+ return 0
+
+ key = hashlib.sha256("\n".join(classes).encode()).hexdigest()[:16]
+ cache = merged_path.parent / f"_disjoint.{key}.ttl"
+ known = set(classes)
+
+ if cache.exists():
+ block = cache.read_text().strip()
+ n = block.count("owl:disjointWith")
+ else:
+ votes: dict[tuple[str, str], int] = {}
+ rounds = 0
+ for group in _sibling_groups(classes):
+ prompt = DISJOINT_PROMPT.format(classes="\n".join(":" + c for c in group))
+ got = 0
+ for _ in range(SAMPLES):
+ try:
+ raw = call_model(prompt, 1200, temperature=0.0)
+ except Exception:
+ continue
+ if not raw.strip():
+ continue
+ got += 1
+ # Within one sample a pair counts once, however often it is
+ # repeated, and direction is irrelevant: disjointness is
+ # symmetric.
+ seen = set()
+ for a, b in re.findall(r"(:\w+)\s+owl:disjointWith\s+(:\w+)", raw):
+ a, b = a.lstrip(":"), b.lstrip(":")
+ if a == b or a not in known or b not in known:
+ continue
+ seen.add(tuple(sorted((a, b))))
+ for pair in seen:
+ votes[pair] = votes.get(pair, 0) + 1
+ rounds = max(rounds, got)
+ if not rounds:
+ return 0
+ need = rounds // 2 + 1
+ valid = sorted(p for p, v in votes.items() if v >= need)
+ if not valid:
+ return 0
+ block = "\n".join(f":{a} owl:disjointWith :{b} ." for a, b in valid)
+ cache.write_text(block + "\n")
+ n = len(valid)
+
+ ttl = re.split(r"\n# Disjointness, [^\n]*\n", ttl)[0].rstrip()
+ merged_path.write_text(
+ ttl + "\n\n# Disjointness, agreed by a majority of samples over the merged vocabulary\n"
+ + block + "\n")
+ return n
+
+
+# Suffixes that name a KIND of the parent, which is exactly what a subclass
+# partition already expresses. Status/State are deliberately absent: a status
+# ("in routine use", "retired") is a property of a thing over time, not a kind
+# of thing, and collapsing it into the class hierarchy would be wrong.
+KIND_SUFFIXES = ("Type", "Kind", "Category", "Mode", "Variant")
+
+
+def remove_subject(ttl: str, name: str) -> str:
+ """Remove every statement whose subject is :name."""
+ out, skipping = [], False
+ for line in ttl.splitlines():
+ if re.match(rf"^:{re.escape(name)}\b", line):
+ skipping = True
+ if skipping:
+ if line.rstrip().endswith("."):
+ skipping = False
+ continue
+ out.append(line)
+ return "\n".join(out)
+
+
+def _reconcile_ttl(merged_path: pathlib.Path) -> list[str]:
+ """Collapse competing modelling patterns left behind by the merge.
+
+ Per-document derivation is independent, so one document models a mode
+ as a subclass partition (:AdherentLine, :SuspensionLine) while another
+ models the same notion as an attribute class (:LineType). Concatenating
+ fragments keeps both, and the extractor then legitimately picks whichever
+ it likes. It reliably picks the attribute class, which is the one form
+ disjointness cannot reach: :Culture07Type a :LineType says nothing the
+ reasoner can contradict.
+
+ The rule: where a parent has a subclass partition (two or more
+ subclasses) AND the vocabulary also declares parent+KIND_SUFFIX, the
+ attribute class is redundant with the partition and loses. Remove it,
+ and remove any property whose domain or range points at it, so the only
+ way left to express the distinction is the one that supports reasoning.
+
+ None of the engine's enforce packs (generic, hierarchy, value_partition)
+ catches this today; it is a merge problem, not a single-ontology problem.
+
+ Operates on the Turtle text on disk, mirroring the file-based fragment
+ merge the rest of this pipeline stage uses. `reconcile()` below applies
+ the identical rule to an in-memory RDF graph.
+ """
+ ttl = merged_path.read_text()
+ classes = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", ttl))
+ children: dict[str, set[str]] = {}
+ for m in re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", ttl):
+ children.setdefault(m.group(2), set()).add(m.group(1))
+
+ doomed = sorted({parent + suf
+ for parent, kids in children.items() if len(kids) >= 2
+ for suf in KIND_SUFFIXES if parent + suf in classes})
+ if not doomed:
+ return []
+
+ for name in doomed:
+ ttl = remove_subject(ttl, name)
+ # Properties anchored to the removed class go with it. A property
+ # whose range no longer exists invites the extractor to mint an
+ # untyped individual to fill it, which recreates the pattern that
+ # was just removed.
+ for prop in set(re.findall(
+ rf":(\w+)\s+(?:a|rdf:type)\s+owl:(?:Object|Datatype)Property\b"
+ rf"[^.]*?rdfs:(?:domain|range)\s+:{re.escape(name)}\b", ttl)):
+ ttl = remove_subject(ttl, prop)
+ ttl = re.sub(rf"^:{re.escape(name)}\s+owl:disjointWith[^\n]*\n?", "",
+ ttl, flags=re.M)
+ ttl = re.sub(rf"^:\w+\s+owl:disjointWith\s+:{re.escape(name)}[^\n]*\n?", "",
+ ttl, flags=re.M)
+ merged_path.write_text(ttl)
+ return doomed
+
+
+def _split_iri(iri: URIRef) -> tuple[str, str]:
+ """Namespace and local name of an IRI, splitting on the last '#' or '/'."""
+ s = str(iri)
+ if "#" in s:
+ ns, _, local = s.rpartition("#")
+ return ns + "#", local
+ ns, _, local = s.rpartition("/")
+ return ns + "/", local
+
+
+def reconcile(graph: Graph) -> Graph:
+ """Collapse competing modelling patterns left behind by the merge.
+
+ Per-document derivation is independent, so one document models a mode
+ as a subclass partition (:AdherentLine, :SuspensionLine) while another
+ models the same notion as an attribute class (:LineType). Concatenating
+ fragments keeps both, and the extractor then legitimately picks whichever
+ it likes. It reliably picks the attribute class, which is the one form
+ disjointness cannot reach: :Culture07Type a :LineType says nothing the
+ reasoner can contradict.
+
+ The rule: where a parent has a subclass partition (two or more
+ subclasses) AND the vocabulary also declares parent+KIND_SUFFIX, the
+ attribute class is redundant with the partition and loses. Remove it,
+ and remove any property whose domain or range points at it, so the only
+ way left to express the distinction is the one that supports reasoning.
+ Status and State are deliberately absent from KIND_SUFFIXES: a status is
+ a property of a thing over time, not a kind of thing, so an attribute
+ class named parent+Status or parent+State is genuinely an attribute and
+ is spared.
+
+ None of the engine's enforce packs (generic, hierarchy, value_partition)
+ catches this today; it is a merge problem, not a single-ontology problem.
+
+ Same rule as `_reconcile_ttl` above, expressed directly over RDF triples
+ instead of the Turtle text on disk, so it can run against a graph that
+ was never serialised to a file.
+ """
+ classes = set(graph.subjects(RDF.type, OWL.Class))
+ children: dict[URIRef, set[URIRef]] = {}
+ for child, parent in graph.subject_objects(RDFS.subClassOf):
+ children.setdefault(parent, set()).add(child)
+
+ doomed: set[URIRef] = set()
+ for parent, kids in children.items():
+ if len(kids) < 2:
+ continue
+ ns, name = _split_iri(parent)
+ for suf in KIND_SUFFIXES:
+ candidate = URIRef(ns + name + suf)
+ if candidate in classes:
+ doomed.add(candidate)
+
+ for cls in doomed:
+ # Properties anchored to the removed class go with it. A property
+ # whose range no longer exists invites the extractor to mint an
+ # untyped individual to fill it, which recreates the pattern that
+ # was just removed.
+ for prop in set(graph.subjects(RDFS.domain, cls)) | set(graph.subjects(RDFS.range, cls)):
+ graph.remove((prop, None, None))
+ # disjointWith triples referencing the removed class, either direction.
+ for s, o in list(graph.subject_objects(OWL.disjointWith)):
+ if s == cls or o == cls:
+ graph.remove((s, OWL.disjointWith, o))
+ # The class declaration itself, and everything else hanging off it.
+ graph.remove((cls, None, None))
+
+ return graph
+
+
+def _name_tokens(name: str) -> tuple[str, ...]:
+ return tuple(sorted(t.lower() for t in re.findall(r"[A-Z][a-z0-9]*|^[a-z]+[0-9]*", name)
+ if t.lower() != "has"))
+
+
+def mech_resolve(body: str, bad: list[str], onto_ttl: str, doc: str = "") -> str:
+ """Resolve undeclared terms that map onto declared vocabulary mechanically.
+
+ The three graphs that stall in LLM repair all fail on terms whose intent
+ is recoverable without a model:
+
+ A token equality hasConcentrationValue -> ConcentrationValue
+ B unique superset hasContainmentLevel -> ContainmentLevelValue
+ C has{Class} inversion S :hasCellLineGenotype O, where
+ CellLineGenotype is a declared CLASS: the generator manufactured a
+ forward property where the vocabulary models the inverse. Rewritten
+ as O a :CellLineGenotype ; :hasCellLine S, which is the declared
+ pattern every other document uses.
+
+ Anything ambiguous (hasName with five plausible targets) is left alone
+ for the LLM repair rounds and, failing those, the review queue.
+ """
+ classes = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", onto_ttl))
+ props = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:(?:Object|Datatype)Property", onto_ttl))
+ parent = dict(re.findall(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", onto_ttl))
+
+ by_toks: dict[tuple, list[str]] = {}
+ for p in props:
+ by_toks.setdefault(_name_tokens(p), []).append(p)
+
+ renames, inversions = {}, set()
+ for term in {b.split("#")[-1] for b in bad}:
+ if not re.fullmatch(r"\w+", term) or term in props or term in classes:
+ continue
+ tt = _name_tokens(term)
+ if tt in by_toks and len(by_toks[tt]) == 1: # rule A
+ renames[term] = by_toks[tt][0]
+ continue
+ sups = [p for p in props if set(tt) < set(_name_tokens(p))]
+ if sups: # rule B
+ m = min(len(_name_tokens(p)) for p in sups)
+ best = [p for p in sups if len(_name_tokens(p)) == m]
+ if len(best) == 1 and m - len(tt) <= 2:
+ renames[term] = best[0]
+ continue
+ if term[:3].lower() == "has" and term[3:] in classes: # rule C
+ inversions.add(term)
+
+ for old, new in renames.items():
+ body = re.sub(rf":{re.escape(old)}\b", f":{new}", body)
+ record("renamed", doc=doc, **{"from": old, "to": new},
+ how="token match against declared vocabulary")
+
+ if inversions:
+ types = {}
+ subj = None
+ for line in body.splitlines():
+ m0 = re.match(r"^:(\w+)\b", line)
+ if m0:
+ subj = m0.group(1)
+ mt = re.search(r"(?:^|\s)a\s+:(\w+)", line)
+ if mt and subj and subj not in types:
+ types[subj] = mt.group(1)
+
+ out, extra, subj = [], [], None
+ for line in body.splitlines():
+ m0 = re.match(r"^:(\w+)\b", line)
+ if m0:
+ subj = m0.group(1)
+ m = re.match(r"^(\s*):(\w+)\s+:(\w+)\s*([;.])\s*$", line)
+ if m and m.group(2) in inversions and subj:
+ prop, obj, term = m.group(2), m.group(3), m.group(4)
+ inv, c, seen = None, types.get(subj), set()
+ while c and c not in seen: # walk up to a declared inverse
+ seen.add(c)
+ inv = next((x for x in (f"has{c}", f"Has{c}") if x in props), None)
+ if inv:
+ break
+ c = parent.get(c)
+ stmt = f":{obj} a :{prop[3:]}"
+ if inv:
+ stmt += f" ;\n :{inv} :{subj}"
+ extra.append(stmt + " .")
+ record("inverted", doc=doc, predicate=prop, subject=subj, obj=obj,
+ how=f"has{{Class}} inversion via declared :{inv}" if inv
+ else "has{Class} inversion, no declared inverse")
+ if term == ".": # re-terminate the statement
+ for i in range(len(out) - 1, -1, -1):
+ if out[i].strip():
+ out[i] = re.sub(r"[;,]\s*$", " .", out[i])
+ break
+ continue
+ out.append(line)
+ body = "\n".join(out + [""] + extra)
+ return body
+
+
+REFINE_PROMPT = """One document contains these excerpts about "{label}":
+
+{snippets}
+
+Which ONE of these categories does the document assign to "{label}"?
+{options}
+
+Reply with EXACTLY one category name from the list and nothing else.
+If "{label}" is not any kind of {parent} at all, reply exactly: other
+If the excerpts do not say, reply exactly: unstated
+"""
+
+GATE_PROMPT = """One document contains these excerpts mentioning "{label}":
+
+{snippets}
+
+Question: does "{label}" name a {parent}?
+
+The excerpts may be registry rows, change logs, or procedures ABOUT the thing;
+those still count: a registry row about a cell line is describing a cell line.
+Answer no only when "{label}" names a different KIND of thing, such as a
+reagent batch, a measurement, a document, or a person.
+
+Reply exactly yes or no.
+"""
+
+
+def partitions_of(ttl: str) -> dict[str, list[str]]:
+ """Parents whose subclasses form a disjoint partition worth asking about."""
+ kids: dict[str, set[str]] = {}
+ for m in re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", ttl):
+ kids.setdefault(m.group(2), set()).add(m.group(1))
+ disjoint = set()
+ for m in re.finditer(r":(\w+)\s+owl:disjointWith\s+:(\w+)", ttl):
+ disjoint.add(frozenset(m.groups()))
+ return {p: sorted(k) for p, k in kids.items()
+ if len(k) >= 2 and any(frozenset((a, b)) in disjoint
+ for a in k for b in k if a != b)}
+
+
+def drop_clone_spam(body: str, threshold: int = 8) -> tuple[str, int]:
+ """Remove runs of numbered clone individuals (STEM_1 ... STEM_78).
+
+ A partially degenerate generation emits dozens of individuals differing
+ only in a counter. The repetition guard misses it because every line is
+ unique, and the vocabulary check passes it because every term is
+ declared. The counter is the tell: no real document describes 78 things
+ named identically but for an integer.
+ """
+ stems: dict[str, int] = {}
+ for m in re.finditer(r"^:(\w+?)_\d+\b", body, re.M):
+ stems[m.group(1)] = stems.get(m.group(1), 0) + 1
+ doomed = {s for s, c in stems.items() if c >= threshold}
+ if not doomed:
+ return body, 0
+ kept, skipping, dropped = [], False, 0
+ for line in body.splitlines():
+ m = re.match(r"^:(\w+?)_\d+\b", line)
+ if m and m.group(1) in doomed:
+ skipping = True
+ dropped += 1
+ if skipping:
+ if line.rstrip().endswith("."):
+ skipping = False
+ continue
+ kept.append(line)
+ return "\n".join(kept), dropped
+
+
+def refine(merged_path: pathlib.Path, instances: list[pathlib.Path],
+ texts: dict[str, str]) -> int:
+ """Assign partition subclasses with one closed-choice question each.
+
+ Six rounds of instruction in the main extraction prompt failed to make
+ the generator type individuals with partition subclasses; it wrote the
+ safe parent class every time. So the decision is taken out of the big
+ prompt entirely. For each individual typed with a bare partition parent,
+ the excerpts mentioning it are shown back with the subclass names as a
+ closed choice. A model that cannot follow "use the most specific class"
+ inside a 16k-character extraction task answers "adherent or suspension
+ or unstated" correctly, because there is nothing else to do.
+
+ This is the LLM-Modulo shape end to end: generation is decomposed until
+ each answer is small enough to verify, and the reasoner adjudicates what
+ the answers jointly imply.
+ """
+ onto_ttl = merged_path.read_text()
+ parts = partitions_of(onto_ttl)
+ if not parts:
+ return 0
+ all_classes = sorted(set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", onto_ttl)))
+ (merged_path.parent / "_review.jsonl").unlink(missing_ok=True)
+ LEDGER["path"] = merged_path.parent / "_decisions.jsonl"
+ LEDGER["path"].unlink(missing_ok=True)
+
+ # Verdicts persist across runs. A queued individual stays typed with the
+ # parent class, so every cached replay re-found it and re-asked the model
+ # the same gate question: seconds of model time per build, answering
+ # something already answered. The memo is keyed by subject and asserted
+ # type; reverting a typing decision changes the type and naturally
+ # re-opens the question.
+ memo_path = merged_path.parent / "_refine_memo.json"
+ try:
+ memo = json.loads(memo_path.read_text())
+ except Exception:
+ memo = {}
+ n = 0
+ for f in instances:
+ doc = f.stem.replace(".data", "")
+ text = texts.get(doc, "")
+ g = f.read_text()
+ g, spam = drop_clone_spam(g)
+ if spam:
+ record("spam", doc=doc, dropped=spam,
+ how="numbered clone individuals removed")
+ print(f" {YEL}spam{OFF} {doc:14s} dropped {spam} numbered clone individuals")
+ if text:
+ for parent, kids in parts.items():
+ seen_subjects: set[str] = set()
+ for m in list(re.finditer(rf":(\w+)\s+a\s+:{parent}\b(?!\w)", g)):
+ subj = m.group(1)
+ # One subject can carry several statement blocks when a
+ # document mentions it in several rows; asking (and
+ # queueing) once per block produced duplicate ledger
+ # entries for the same decision.
+ if subj in seen_subjects:
+ continue
+ seen_subjects.add(subj)
+ if subj in all_classes:
+ # An INDIVIDUAL whose IRI is a declared class (a source
+ # document emitted a literal :CellLine individual). Retyping it
+ # would rewrite what looks like vocabulary. Humans only.
+ with open(merged_path.parent / "_review.jsonl", "a") as rq:
+ rq.write(json.dumps({
+ "doc": doc, "subject": subj,
+ "reason": "individual IRI collides with a declared class"}) + "\n")
+ print(f" {YEL}review{OFF} {doc:14s} :{subj} "
+ f"{DIM}IRI collides with a class; queued{OFF}")
+ n += 1
+ continue
+ lm = re.search(rf":{re.escape(subj)}\b[^.]*?rdfs:label\s+\"([^\"]+)\"",
+ g, re.S)
+ label = lm.group(1) if lm else subj
+ lines = [l.strip() for l in text.splitlines()
+ if label.lower() in l.lower() and len(l.strip()) > len(label)]
+ if not lines:
+ continue
+ mkey = f"{doc}|{subj}|{parent}"
+ if memo.get(mkey) == "queued":
+ entry = {"doc": doc, "subject": subj, "label": label,
+ "asserted": parent,
+ "reason": f"not a {parent} per gate question (memoised)"}
+ with open(merged_path.parent / "_review.jsonl", "a") as rq:
+ rq.write(json.dumps(entry) + "\n")
+ record("queued", **entry)
+ print(f" {YEL}review{OFF} {doc:14s} :{subj} "
+ f"{DIM}not a {parent}; memoised verdict{OFF}")
+ n += 1
+ continue
+
+ snippets = "\n".join(f"- {l[:220]}" for l in lines[:6])
+ # The gate is its own binary question. Offered only as an
+ # extra option inside the partition question, "other" loses
+ # to pattern-matching: an FBS lot mentioned alongside
+ # adherent cultures came back "AdherentLine". Asked
+ # alone, "is Lot A itself a cell line?" gets the right no.
+ try:
+ gate = call_model(GATE_PROMPT.format(
+ label=label, parent=parent, snippets=snippets),
+ 16, attempts=2, temperature=0.0, timeout=60).strip().lower()
+ except Exception:
+ continue
+ # An empty or off-script reply is NO VERDICT. Routing it
+ # to the queue turned one transient null response into a
+ # permanent, memoised wrong answer about the corpus's main
+ # entity.
+ if not gate or not (gate.startswith("yes") or gate.startswith("no")):
+ continue
+ anskey, pick = "", None
+ if gate.startswith("yes"):
+ try:
+ ans = call_model(REFINE_PROMPT.format(
+ label=label, parent=parent, snippets=snippets,
+ options="\n".join(kids)), 30, attempts=2,
+ temperature=0.0, timeout=60).strip()
+ except Exception:
+ continue
+ anskey = ans.strip().strip(":").lower()
+ pick = next((k for k in kids if k.lower() == anskey), None)
+ else:
+ anskey = "other"
+ if pick:
+ g = re.sub(rf"(:{re.escape(subj)}\b\s+a\s+):{parent}\b(?!\w)",
+ rf"\g<1>:{pick}", g)
+ n += 1
+ record("typed", doc=doc, subject=subj,
+ **{"from": parent, "to": pick},
+ how="closed-choice question over the partition")
+ print(f" {GRN}typed{OFF} {doc:14s} :{subj} -> :{pick}")
+ elif anskey == "other":
+ # The extractor mistyped it upstream. An automatic
+ # second guess was tried here and reintroduced the
+ # failure it was meant to fix (an FBS lot became
+ # :CellLineGrowthRateComparison with :FBSLot sitting
+ # in the candidate list). Wrong-type-with-confidence
+ # is worse than wrong-type-in-a-queue: the type stays
+ # as asserted and a human decides.
+ entry = {"doc": doc, "subject": subj, "label": label,
+ "asserted": parent,
+ "reason": f"not a {parent} per gate question"}
+ with open(merged_path.parent / "_review.jsonl", "a") as rq:
+ rq.write(json.dumps(entry) + "\n")
+ record("queued", **entry)
+ memo[f"{doc}|{subj}|{parent}"] = "queued"
+ memo_path.write_text(json.dumps(memo))
+ print(f" {YEL}review{OFF} {doc:14s} :{subj} "
+ f"{DIM}not a {parent}; queued for a human{OFF}")
+ n += 1
+ f.write_text(g)
+ return n
+
+
+def split_statements(ttl: str) -> list[tuple[str, str]]:
+ """(subject, statement) pairs, line-based, prefix header excluded."""
+ out, subj, buf = [], None, []
+ for line in ttl.splitlines():
+ if not buf:
+ m = re.match(r"^:(\w+)\b", line)
+ if not m:
+ continue
+ subj = m.group(1)
+ buf.append(line)
+ if line.rstrip().endswith("."):
+ out.append((subj, "\n".join(buf)))
+ subj, buf = None, []
+ return out
+
+
+def prune_to_used(onto_ttl: str, data: str) -> tuple[str, dict]:
+ """The vocabulary the data actually exercises, nothing else.
+
+ Derivation over-proposes by an order of magnitude: 10 documents produced
+ 346 classes, most of them compositional one-offs
+ (CellLineGrowthRateMeasurementStatisticalSignificanceAnalysis) that no
+ instance ever uses. Rendering all of that buries the dozen entities the
+ corpus is actually about. The FULL vocabulary stays on disk and keeps
+ constraining extraction; the store the graph view and chat read carries
+ only: classes with instances, their ancestors, the domain and range of
+ every property the data uses, and disjointness among what remains.
+ """
+ # A class is anything the ontology treats as one, not only what carries an
+ # explicit `a owl:Class`. The partition members are declared through
+ # rdfs:subClassOf alone, and dropping them silently removed the exact
+ # disjointness axiom the contradiction demo runs on.
+ decl_cls = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", onto_ttl))
+ decl_cls |= set(re.findall(r":(\w+)\s+rdfs:subClassOf\s+:", onto_ttl))
+ for a, b in re.findall(r":(\w+)\s+owl:disjointWith\s+:(\w+)", onto_ttl):
+ decl_cls.update((a, b))
+ decl_props = set(re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:(?:Object|Datatype)Property", onto_ttl))
+ counts: dict[str, int] = {}
+ for c in re.findall(r"\ba\s+:(\w+)", data):
+ counts[c] = counts.get(c, 0) + 1
+ # A class instantiated once is a description of one node, not a category.
+ # The graph view draws the CLASS diagram, so every singleton is a box that
+ # says nothing. Recurring classes are what the corpus is actually about;
+ # singletons keep their individuals in the store but not their box.
+ used_cls = {c for c, n in counts.items() if n >= 2 and c in decl_cls}
+ used_props = decl_props & set(re.findall(r":(\w+)", data))
+
+ stmts = split_statements(onto_ttl)
+ by_subj = {}
+ for s, t in stmts:
+ by_subj.setdefault(s, t)
+
+ keep = set(used_cls)
+ for pr in used_props: # domain/range classes stay: edges render
+ for dr in re.findall(r"rdfs:(?:domain|range)\s+:(\w+)", by_subj.get(pr, "")):
+ if dr in decl_cls:
+ keep.add(dr)
+
+ parents: dict[str, set[str]] = {}
+ for m in re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", onto_ttl):
+ parents.setdefault(m.group(1), set()).add(m.group(2))
+ frontier = list(keep)
+ while frontier: # ancestors close the hierarchy
+ for p in parents.get(frontier.pop(), ()):
+ if p in decl_cls and p not in keep:
+ keep.add(p)
+ frontier.append(p)
+
+ # View collapse. Two shapes of noise survive the instance-count filter:
+ # specialisation chains (CellLineGrowthRateMeasurementStatistical...
+ # extending CellLineGrowthRate token by token) and families of near
+ # duplicate siblings (fourteen ChangeLogEntryChange* classes). A box per
+ # variant hides the dozen entities the corpus is about. Chains collapse
+ # into their shortest kept prefix; families keep their two most
+ # instantiated members. Instance data is never touched, and partition
+ # members are exempt: they are what the reasoner argues about.
+ # Exempt only true partitions: disjoint SIBLINGS under a common parent.
+ # The disjointness pass asserts pairs across the whole vocabulary, so
+ # exempting every pair member exempts nearly everything.
+ all_parents: dict[str, set[str]] = {}
+ for m in re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", onto_ttl):
+ all_parents.setdefault(m.group(1), set()).add(m.group(2))
+ exempt = {c for a, b in re.findall(r":(\w+)\s+owl:disjointWith\s+:(\w+)", onto_ttl)
+ if all_parents.get(a, set()) & all_parents.get(b, set())
+ for c in (a, b)}
+ ctok = {c: tuple(re.findall(r"[A-Z][a-z0-9]*", c)) for c in keep if c in decl_cls}
+ hide = set()
+ for a, ta in ctok.items():
+ for b, tb in ctok.items():
+ if a != b and len(tb) > len(ta) and tb[:len(ta)] == ta and b not in exempt:
+ hide.add(b)
+ families: dict[tuple, list[str]] = {}
+ for c, tc in ctok.items():
+ if c not in hide and len(tc) >= 3:
+ families.setdefault(tc[:3], []).append(c)
+ for group in families.values():
+ if len(group) >= 5:
+ group.sort(key=lambda c: -counts.get(c, 0))
+ hide.update(c for c in group[2:] if c not in exempt)
+ keep -= hide
+
+ # Case twins and noun duplicates of the same property (hasOwner/HasOwner,
+ # Title/hasTitle) render as distinct edges saying the same thing. One
+ # survives in the view; the data keeps using whichever it used.
+ used_props -= {p for p in used_props
+ if p.startswith("Has") and "has" + p[3:] in used_props}
+ used_props -= {p for p in used_props
+ if not p.startswith(("has", "Has"))
+ and ("has" + p in used_props or "Has" + p in used_props
+ or "has" + p[0].upper() + p[1:] in used_props)}
+
+ kept = []
+ for s, t in stmts:
+ if re.match(r"^:\w+\s+owl:disjointWith", t):
+ other = re.search(r"owl:disjointWith\s+:(\w+)", t)
+ if s in keep and other and other.group(1) in keep:
+ kept.append(t)
+ elif s in keep or s in used_props:
+ kept.append(t)
+
+ stats = {"classes": (len(decl_cls), len(keep & decl_cls)),
+ "properties": (len(decl_props), len(used_props))}
+ return PREFIXES + "\n" + "\n\n".join(kept), stats
+
+
+def sources_of(instances: list[pathlib.Path]) -> dict[tuple[str, str], set[str]]:
+ """Which document asserted that a given individual has a given type.
+
+ Provenance is what separates the two things a disjointness violation can
+ mean. If one document types Culture07 as both a cell line and a doubling
+ time, that is a broken extraction. If two documents disagree about which
+ of two mutually exclusive classes it belongs to, that is the corpus
+ contradicting itself, and it is the only kind worth showing anyone.
+
+ Reported together they are indistinguishable, and the noisy kind buries
+ the interesting kind: one over-typed registry produced 603 violations and
+ not one of them was a disagreement.
+ """
+ out: dict[tuple[str, str], set[str]] = {}
+ for f in instances:
+ doc = f.stem.replace(".data", "")
+ text = f.read_text()
+ for m in re.finditer(r":(\w+)\s+(?:a|rdf:type)\s+([^;.]+)", text):
+ for t in re.findall(r":(\w+)", m.group(2)):
+ out.setdefault((m.group(1), t), set()).add(doc)
+ return out
+
+
+def vocab_summary(ttl: str) -> str:
+ """Vocabulary for the populate prompt, WITH definitions.
+
+ A bare list of names invites the model to invent near-misses: given
+ :CellLine it will happily write :CellLineType or :CellLineInRoutineUse
+ because they sound plausible. Including each term's label and comment makes
+ the existing vocabulary concrete enough to reuse instead of extend, which
+ is what the closed-world check is measuring.
+ """
+ def described(kind: str) -> list[str]:
+ out = []
+ for name in sorted(set(re.findall(rf":(\w+)\s+a\s+owl:{kind}", ttl))):
+ m = re.search(rf":{name}\s+a\s+owl:{kind}\b.*?rdfs:label\s+\"([^\"]+)\"", ttl, re.S)
+ out.append(f" :{name}" + (f" ({m.group(1)})" if m else ""))
+ return out
+
+ return ("CLASSES (use these exact names, do not invent variants):\n"
+ + "\n".join(described("Class"))
+ + "\n\nOBJECT PROPERTIES:\n" + "\n".join(described("ObjectProperty"))
+ + "\n\nDATATYPE PROPERTIES:\n" + "\n".join(described("DatatypeProperty")))
+
+
+REPAIR_PROMPT = """This Turtle uses terms that do not exist in the ontology. Every
+term must come from the vocabulary below.
+
+INVALID TERMS USED: {bad}
+
+VOCABULARY:
+{vocab}
+
+Rewrite the Turtle so that every class and property is one of the vocabulary
+terms above. Where an invalid term has a clear equivalent, map it. Where it does
+not, DROP those statements rather than inventing a substitute. Keep everything
+that was already valid unchanged.
+
+Return ONLY Turtle, no prose and no fences.
+
+TURTLE TO REPAIR:
+{ttl}
+"""
+
+
+def repair(ttl: str, bad: list[str], vocab: str) -> str | None:
+ """One repair attempt against the named invalid terms.
+
+ A closed-world check that only reports is half a control: it tells you the
+ output is unusable without making it usable. Naming the specific offending
+ terms and asking for a rewrite is cheap, and it converts a rejected graph
+ into an accepted one without a human in the loop for the routine cases.
+ """
+ names = sorted({t.split("#")[-1] for t in bad})[:25]
+ try:
+ raw = call_model(REPAIR_PROMPT.format(bad=", ".join(names), vocab=vocab, ttl=ttl[:9000]), 3000)
+ except Exception:
+ return None
+ out = normalise(raw)
+ return out if len(out) > 200 else None
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--corpus", default=str(ROOT / "demo" / "corpus" / "dcat-us"))
+ ap.add_argument("--workers", type=int, default=2,
+ help="parallel model calls; local servers overload above 2")
+ ap.add_argument("--cached", action="store_true")
+ ap.add_argument("--reuse-derive", action="store_true",
+ help="re-merge existing vocab fragments instead of re-deriving")
+ ap.add_argument("--no-tokenise", action="store_true")
+ args = ap.parse_args()
+
+ corpus = pathlib.Path(args.corpus)
+ out = ROOT / "demo" / "derived"
+ out.mkdir(exist_ok=True)
+ docs = sorted(corpus.glob("*.docx")) or sorted(corpus.glob("*.md"))
+ if not docs:
+ sys.exit(f"no documents in {corpus}")
+
+ t0 = time.time()
+ print(f"\n{BOLD}STAGE 1 READ{OFF} {len(docs)} documents from {corpus.name}/")
+
+ # ---- Stage 2: tokenise -------------------------------------------------
+ os.environ.setdefault("ONTO_VAULT_PATH", str(out / "_vault.json"))
+ # Reading and tokenising dominate the wall clock of a cached build:
+ # pandoc is a subprocess per document and the Presidio detector loads a
+ # spaCy model on import, which costs seconds before any work happens.
+ # Both are pure functions of the file, so cache by mtime and only build
+ # the detector when at least one document actually changed.
+ doc_cache_path = out / "_docs_cache.json"
+ try:
+ doc_cache = json.loads(doc_cache_path.read_text())
+ except Exception:
+ doc_cache = {}
+ tokeniser = None
+ prepared, total_tokens = [], 0
+ dirty = False
+ for d in docs:
+ m = re.match(r"([A-Z]+-[0-9][0-9A-Za-z\-]*?)(?=-[a-z])", d.stem)
+ docid = m.group(1) if m else d.stem
+ key, mtime = str(d), d.stat().st_mtime
+ hit = doc_cache.get(key)
+ if not args.no_tokenise and hit and hit.get("mtime") == mtime:
+ total_tokens += hit["count"]
+ prepared.append((d, hit["text"], docid))
+ continue
+ text = docx_text(d)
+ if args.no_tokenise:
+ prepared.append((d, text, docid))
+ continue
+ if tokeniser is None:
+ tokeniser = build_tokeniser()
+ tok, n = tokeniser.tokenise(text)
+ total_tokens += n
+ prepared.append((d, tok, docid))
+ doc_cache[key] = {"mtime": mtime, "text": tok, "count": n}
+ dirty = True
+ if tokeniser is not None:
+ tokeniser.save()
+ if dirty:
+ doc_cache_path.write_text(json.dumps(doc_cache))
+ if args.no_tokenise:
+ print(f"\n{BOLD}STAGE 2 TOKENISE{OFF} {YEL}skipped (--no-tokenise){OFF}")
+ else:
+ print(f"\n{BOLD}STAGE 2 TOKENISE{OFF} {total_tokens} values replaced with stable tokens")
+ desc = tokeniser.description if tokeniser is not None else \
+ "cached detection results (documents unchanged)"
+ print(f" {DIM}{desc}. No model sees a real value from here on.{OFF}")
+
+ # ---- Stage 3: derive ---------------------------------------------------
+ merged_path = out / "_ontology.ttl"
+ if args.cached and merged_path.exists():
+ print(f"\n{BOLD}STAGE 3 DERIVE{OFF} {DIM}cached{OFF}")
+ elif args.reuse_derive and list(out.glob("*.vocab.ttl")):
+ # Derivation is the expensive, stable half. Re-merging existing
+ # fragments exercises the merge and everything after it without
+ # paying for the model calls again.
+ # Persist fragments in mended form: the align tools parse these
+ # files as written, and a missing prefix makes a fragment invisible
+ # to schema resolution even though the merge recovered it.
+ for fp in sorted(out.glob("*.vocab.ttl")):
+ fixed = rehead(fp.read_text())
+ if is_valid_turtle(fixed)[0]:
+ fp.write_text(fixed)
+ fragments = [f.read_text() for f in sorted(out.glob("*.vocab.ttl"))]
+ print(f"\n{BOLD}STAGE 3 DERIVE{OFF} reusing {len(fragments)} existing fragments")
+ good, dropped = [], []
+ for frag in fragments:
+ ok, why = is_valid_turtle(rehead(frag))
+ if ok:
+ good.append(mend(body_of(frag)))
+ else:
+ dropped.append(why)
+ if dropped:
+ print(f" {YEL}{len(dropped)} fragment(s) dropped as unparseable{OFF}")
+ for why in dropped[:4]:
+ print(f" {DIM}{why}{OFF}")
+ merged_path.write_text(PREFIXES + "\n" + STRUCTURAL + "\n" + "\n\n".join(b for b in good if b))
+ ok, why = is_valid_turtle(merged_path.read_text())
+ print(f" merged {len(good)} fragments, parses: {'yes' if ok else 'NO — ' + why}")
+ else:
+ print(f"\n{BOLD}STAGE 3 DERIVE{OFF} propose the vocabulary the documents imply")
+ fragments = []
+ with cf.ThreadPoolExecutor(max_workers=args.workers) as pool:
+ for docid, ttl, err in pool.map(derive_one, prepared):
+ if err:
+ print(f" {RED}fail{OFF} {docid:14s} {err}")
+ else:
+ mended_ttl = rehead(ttl)
+ (out / f"{docid}.vocab.ttl").write_text(
+ mended_ttl if is_valid_turtle(mended_ttl)[0] else ttl)
+ fragments.append(ttl)
+ n = len(re.findall(r"a\s+owl:Class", ttl))
+ print(f" {GRN}ok{OFF} {docid:14s} {n} classes proposed")
+ # Merge: prefixes once, bodies concatenated. Duplicate declarations are
+ # harmless in RDF; the store deduplicates identical triples.
+ # Validate each fragment alone. One malformed fragment otherwise
+ # truncates the merged file at its first bad statement, discarding
+ # every valid fragment that follows it.
+ good, dropped = [], []
+ for frag in fragments:
+ ok, why = is_valid_turtle(rehead(frag))
+ if ok:
+ good.append(mend(body_of(frag)))
+ else:
+ dropped.append(why)
+ if dropped:
+ print(f" {YEL}{len(dropped)} fragment(s) dropped as unparseable:{OFF}")
+ for why in dropped[:3]:
+ print(f" {DIM}{why}{OFF}")
+ merged_path.write_text(PREFIXES + "\n" + STRUCTURAL + "\n" + "\n\n".join(b for b in good if b))
+ ok, why = is_valid_turtle(merged_path.read_text())
+ if not ok:
+ print(f" {RED}merged ontology still does not parse: {why}{OFF}")
+
+ doomed = _reconcile_ttl(merged_path)
+ if doomed:
+ print(f" {BOLD}reconciled:{OFF} removed {len(doomed)} attribute class(es) that "
+ f"compete with a subclass partition: {', '.join(':' + d for d in doomed)}")
+
+ check = engine(["clear", f"load {merged_path}", "stats"])
+ try:
+ st = json.loads(check.split("\n")[-1])["result"]
+ print(f" {BOLD}derived ontology: {st['classes']} classes, {st['properties']} properties, "
+ f"{st['triples']} triples{OFF}")
+ except Exception:
+ print(f" {YEL}derived ontology did not parse cleanly{OFF}")
+ print(f" {DIM}{check[-200:]}{OFF}")
+
+ n_disjoint = add_disjointness(merged_path)
+ print(f" {BOLD}disjointness: {n_disjoint} mutually exclusive pairs asserted{OFF}"
+ if n_disjoint else f" {YEL}disjointness: none asserted (nothing for the reasoner to catch){OFF}")
+
+ vocab = vocab_summary(merged_path.read_text())
+
+ # ---- Stage 4: populate -------------------------------------------------
+ print(f"\n{BOLD}STAGE 4 POPULATE{OFF} extract facts, constrained to that vocabulary")
+ instances = []
+ if args.cached:
+ instances = sorted(out.glob("*.data.ttl"))
+ if not instances:
+ print(f" {YEL}no cached graphs to replay. Tick 'Re-run extraction' to build them.{OFF}")
+ sys.exit(2)
+ print(f" {DIM}cached, {len(instances)} graphs{OFF}")
+ else:
+ jobs = [(d, t, i, vocab) for (d, t, i) in prepared]
+ with cf.ThreadPoolExecutor(max_workers=args.workers) as pool:
+ for docid, ttl, err in pool.map(populate_one, jobs):
+ if err:
+ print(f" {RED}fail{OFF} {docid:14s} {err}")
+ else:
+ f = out / f"{docid}.data.ttl"
+ f.write_text(ttl)
+ instances.append(f)
+ print(f" {GRN}ok{OFF} {docid:14s} {f.name}")
+
+ # ---- Stage 4b: refine --------------------------------------------------
+ print(f"\n{BOLD}STAGE 4b REFINE{OFF} partition typing as closed-choice questions")
+ texts = {docid: tokenised for (_p, tokenised, docid) in prepared}
+ refined = refine(merged_path, instances, texts)
+ if not refined:
+ print(f" {DIM}nothing to refine{OFF}")
+
+ # ---- Stage 5: verify ---------------------------------------------------
+ print(f"\n{BOLD}STAGE 5 VERIFY{OFF} closed-world check against the DERIVED ontology")
+ ok = bad = 0
+ try:
+ mcp("initialize", {"protocolVersion": "2025-03-26", "capabilities": {},
+ "clientInfo": {"name": "otd", "version": "1"}})
+ mcp("tools/call", {"name": "onto_load", "arguments":
+ {"path": str(merged_path), "name": "derived", "force_recompile": True}})
+ declared = set(re.findall(r":(\w+)\s+a\s+owl:", merged_path.read_text()))
+
+ def check(ttl: str) -> dict:
+ return json.loads(tool_text(mcp("tools/call", {
+ "name": "onto_vocab_check",
+ "arguments": {"data": ttl, "inline": True}})) or "{}")
+
+ for f in instances:
+ current = f.read_text()
+ d = check(current)
+ if d.get("conforms"):
+ ok += 1
+ print(f" {GRN}conforms{OFF} {f.stem}")
+ continue
+ if "error" in d:
+ # A graph that does not parse is a different and worse failure
+ # than one that uses the wrong terms. Reporting both as
+ # "review" hid a broken graph behind an empty term list.
+ bad += 1
+ print(f" {RED}unparsed{OFF} {f.stem} {DIM}{d['error'][:90]}{OFF}")
+ continue
+
+ # Mechanical resolution first: token-equality renames, unique
+ # superset renames, and has{Class} inversions need no model and
+ # cannot hallucinate. The LLM repair rounds then only see what is
+ # genuinely ambiguous.
+ h = d.get("hallucinated_terms", [])
+ start = len(h)
+ mech = mech_resolve(current, h, merged_path.read_text(),
+ doc=f.stem.replace(".data", ""))
+ if mech != current and is_valid_turtle(mech)[0]:
+ d = check(mech)
+ if "error" not in d:
+ current = mech
+ f.write_text(current)
+ h = d.get("hallucinated_terms", [])
+ if d.get("conforms"):
+ ok += 1
+ print(f" {GRN}repaired{OFF} {f.stem} "
+ f"{DIM}({start} terms resolved mechanically){OFF}")
+ continue
+ for _ in range(REPAIR_ROUNDS):
+ fixed = repair(current, h, vocab)
+ if not fixed:
+ break
+ fixed = snap_to_vocab(fixed, declared)
+ d = check(fixed)
+ if d.get("conforms"):
+ f.write_text(fixed)
+ current, h = fixed, []
+ break
+ nh = d.get("hallucinated_terms", h)
+ if "error" in d or len(nh) >= len(h):
+ break # not converging, stop paying for it
+ current, h = fixed, nh
+ if not h:
+ ok += 1
+ print(f" {GRN}repaired{OFF} {f.stem} {DIM}({start} invalid terms resolved){OFF}")
+ else:
+ f.write_text(current) # keep the partial improvement
+ bad += 1
+ print(f" {YEL}review {OFF} {f.stem} {DIM}{start} -> {len(h)}{OFF} "
+ f"{[t.split('#')[-1] for t in h][:5]}")
+ except Exception as e:
+ print(f" {YEL}verification unavailable: {e}{OFF}")
+
+ # ---- Stage 6: scan -----------------------------------------------------
+ print(f"\n{BOLD}STAGE 6 SCAN{OFF} reason over the derived ontology and its instances")
+ loads = ["clear", f"load {merged_path}"] + [f"load {f}" for f in instances]
+ res = engine(loads + ["stats"])
+ try:
+ st = json.loads(res.split("\n")[-1])["result"]
+ print(f" merged: {st['triples']} triples, {st['classes']} classes, {st['individuals']} individuals")
+ except Exception:
+ pass
+
+ q = ("PREFIX owl: <http://www.w3.org/2002/07/owl#> "
+ "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> "
+ "SELECT DISTINCT ?subject ?a ?b WHERE { ?subject a ?a, ?b . "
+ "FILTER(STR(?a) < STR(?b)) ?a rdfs:subClassOf* ?da . ?b rdfs:subClassOf* ?db . "
+ "{ ?da owl:disjointWith ?db } UNION { ?db owl:disjointWith ?da } }")
+ res = engine(loads + [f"query {json.dumps(q)}"])
+ try:
+ rows = json.loads(res.split("\n")[-1])["result"]["results"]
+ except Exception:
+ rows = []
+ src = sources_of(instances)
+ disagreements, defects = [], []
+ for r in rows:
+ v = {k: x.split("#")[-1].rstrip(">") for k, x in r.items()}
+ da = src.get((v["subject"], v["a"]), set())
+ db = src.get((v["subject"], v["b"]), set())
+ # A disagreement needs a document that asserts one side and does not
+ # assert the other. Both types coming from the same single document is
+ # one extractor being wrong, not two sources in conflict.
+ (disagreements if (da - db) and (db - da) else defects).append((v, da, db))
+
+ for v, da, db in disagreements:
+ print(f" {RED}CONTRADICTION{OFF} {v['subject']} is {v['a']} per "
+ f"{','.join(sorted(da - db))} but {v['b']} per {','.join(sorted(db - da))}")
+ if defects:
+ by_doc: dict[str, int] = {}
+ for _, da, db in defects:
+ for d in da | db:
+ by_doc[d] = by_doc.get(d, 0) + 1
+ worst = sorted(by_doc.items(), key=lambda x: -x[1])[:3]
+ print(f" {YEL}{len(defects)} violations within a single document{OFF} "
+ f"{DIM}(over-typing, not disagreement: "
+ f"{', '.join(f'{d} {n}' for d, n in worst)}){OFF}")
+ if not rows:
+ print(f" {DIM}no disjointness violations{OFF}")
+
+ # ---- Stage 7: load -----------------------------------------------------
+ # Everything above ran against throwaway CLI stores. The Studio graph
+ # view and the chat's GraphRAG read the LIVE engine, so the session ends
+ # by loading the merged ontology plus every instance graph there. Without
+ # this the app renders vocabulary with zero individuals, which looks like
+ # extraction never happened.
+ print(f"\n{BOLD}STAGE 7 LOAD{OFF} push the result into the live engine "
+ f"for the graph view and chat")
+ store = merged_path.parent / "_store.ttl"
+ data = "\n\n".join(body_of(f.read_text()) for f in instances)
+ pruned, pstats = prune_to_used(merged_path.read_text(), data)
+ c0, c1 = pstats["classes"]
+ p0, p1 = pstats["properties"]
+ print(f" pruned to the vocabulary the data uses: "
+ f"{c0} -> {c1} classes, {p0} -> {p1} properties "
+ f"{DIM}(full vocabulary stays on disk for extraction){OFF}")
+ # Typing claims with provenance, in the shape GraphRAG retrieves: the
+ # chat's own suggested question is "which documents disagree", and
+ # without these the store knows THAT a conflict exists but not who says
+ # what. One claim node per specific typing per document.
+ claims, ci = [], 0
+ for f in instances:
+ doc = f.stem.replace(".data", "")
+ for m in re.finditer(r":(\w+)\s+a\s+:(\w+)", f.read_text()):
+ subj, cls = m.groups()
+ if cls in ("Document", "Section"):
+ continue
+ ci += 1
+ claims.append(
+ f":CLAIM_{ci} a :Claim ;\n"
+ f" :claimText \"{doc} types {subj} as {cls}\" ;\n"
+ f" :aboutEntity :{subj} ;\n"
+ f" :statedIn :DOC_{doc.replace('-', '_')} .\n"
+ f":{subj} <http://www.w3.org/ns/prov#wasDerivedFrom> :DOC_{doc.replace('-', '_')} .\n"
+ f":DOC_{doc.replace('-', '_')} rdfs:label \"{doc}\" .")
+ store.write_text(pruned + "\n" + data + "\n" + "\n".join(claims))
+ store_ok, why = is_valid_turtle(store.read_text())
+ if not store_ok:
+ # A parse error truncates the load at the bad line and the engine
+ # reports the partial store without complaint: the graph view then
+ # quietly shows a fraction of the data.
+ print(f" {YEL}store has a syntax defect; the view may be partial: {why}{OFF}")
+ try:
+ mcp("tools/call", {"name": "onto_load", "arguments":
+ {"path": str(store), "name": "derived", "force_recompile": True}})
+ st = json.loads(tool_text(mcp("tools/call",
+ {"name": "onto_stats", "arguments": {}})) or "{}")
+ print(f" live store: {st.get('triples', '?')} triples, "
+ f"{st.get('classes', '?')} classes, {st.get('individuals', '?')} individuals")
+ except Exception as e:
+ print(f" {YEL}live engine not reachable ({e}); graph view will be stale{OFF}")
+
+ print(f"\n{BOLD}{len(docs)} documents in {time.time()-t0:.0f}s. "
+ f"{ok} conformed, {bad} for review, "
+ f"{len(disagreements)} contradictions across documents.{OFF}")
+ if not args.no_tokenise:
+ print(f"{DIM}Every value the model saw was a token. Detokenisation requires the "
+ f"vault and happens only at render time.{OFF}\n")
+
+
+if __name__ == "__main__":
+ try:
+ main()
+ except SystemExit:
+ raise
+ except Exception:
+ # The Studio pane only shows "finished with errors" plus whatever was
+ # printed. A crash with the traceback on stdout is diagnosable from
+ # the pane; a bare non-zero exit is not.
+ import traceback
+ print("PIPELINE CRASH:")
+ traceback.print_exc(file=sys.stdout)
+ sys.exit(1)
diff --git a/demo/requirements.txt b/demo/requirements.txt
new file mode 100644
index 00000000..8177212d
--- /dev/null
+++ b/demo/requirements.txt
@@ -0,0 +1,3 @@
+rdflib>=7.0
+presidio-analyzer>=2.2
+pytest>=8.0
diff --git a/demo/tests/test_reconcile.py b/demo/tests/test_reconcile.py
new file mode 100644
index 00000000..f93d5200
--- /dev/null
+++ b/demo/tests/test_reconcile.py
@@ -0,0 +1,28 @@
+from rdflib import Graph, Namespace, RDFS, OWL, RDF
+
+from demo.ontology_from_docs import reconcile
+
+EX = Namespace("https://example.org/t#")
+
+
+def _graph(pairs):
+ g = Graph()
+ for sub, parent in pairs:
+ g.add((sub, RDF.type, OWL.Class))
+ g.add((sub, RDFS.subClassOf, parent))
+ return g
+
+
+def test_attribute_class_is_removed_when_a_partition_exists():
+ g = _graph([(EX.PublishedDataset, EX.Dataset), (EX.DraftDataset, EX.Dataset)])
+ g.add((EX.DatasetType, RDF.type, OWL.Class))
+ out = reconcile(g)
+ assert (EX.DatasetType, RDF.type, OWL.Class) not in out
+ assert (EX.PublishedDataset, RDFS.subClassOf, EX.Dataset) in out
+
+
+def test_status_is_spared_because_states_are_attributes():
+ g = _graph([(EX.ActiveThing, EX.Thing), (EX.RetiredThing, EX.Thing)])
+ g.add((EX.ThingStatus, RDF.type, OWL.Class))
+ out = reconcile(g)
+ assert (EX.ThingStatus, RDF.type, OWL.Class) in out
diff --git a/demo/tokenisation.py b/demo/tokenisation.py
new file mode 100644
index 00000000..4cb72d1c
--- /dev/null
+++ b/demo/tokenisation.py
@@ -0,0 +1,423 @@
+#!/usr/bin/env python3
+"""
+Swappable tokenisation: detection and custody are separate concerns.
+
+The market splits along exactly this seam, so the code should too:
+
+ DETECTION finding the sensitive spans in free text.
+ Regex (demo) | Microsoft Presidio (MIT, local) | a vendor's own.
+
+ CUSTODY holding the mapping and controlling reversal.
+ Local JSON (demo only) | Skyflow (managed) | Databunker (self-hosted).
+
+Skyflow sells both halves in one product. Open source makes you assemble them,
+which is fine as long as the assembly is a config choice rather than a rewrite.
+
+ ONTO_DETECTOR = regex | presidio | both
+ ONTO_VAULT = local | skyflow | databunker
+
+Everything downstream depends only on `Tokeniser`, so swapping either half
+changes nothing in the pipeline. That is the point: the tokeniser is an
+interchangeable component, not an architectural commitment.
+
+The one invariant every implementation must hold: **tokens are deterministic**.
+The same value must produce the same token in every document, because that is
+what lets a token double as a join key for exact-match entity resolution
+without any component handling the raw value.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import hmac
+import json
+import os
+import pathlib
+import re
+from dataclasses import dataclass
+from typing import Protocol
+
+
+@dataclass(frozen=True)
+class Span:
+ """A detected sensitive span: character offsets plus what it looks like."""
+ start: int
+ end: int
+ kind: str
+ text: str
+
+
+# --------------------------------------------------------------------------
+# Detection
+# --------------------------------------------------------------------------
+
+class Detector(Protocol):
+ name: str
+
+ def detect(self, text: str) -> list[Span]:
+ ...
+
+
+class RegexDetector:
+ """Pattern matching. Demonstration only.
+
+ Honest about its limits: this finds email addresses, phone-shaped strings,
+ titled names and identifier-shaped tokens. It will miss bare personal names,
+ addresses, dates of birth, national identifiers, and anything not in
+ English. Do not present this as PII detection.
+ """
+
+ name = "regex"
+
+ PATTERNS = [
+ ("EMAIL", re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.]{2,}\b")),
+ ("PHONE", re.compile(r"\b(?:\+\d{1,3}[ -]?)?(?:\(?\d{3,5}\)?[ -]?){2,3}\d{3,4}\b")),
+ ("PERSON", re.compile(r"\b(?:Dr|Prof|Mr|Mrs|Ms)\.? [A-Z][a-z]+ [A-Z][a-z]+\b")),
+ ("ID", re.compile(r"\b[A-Z]{2,4}-\d{4,}\b")),
+ ]
+
+ def detect(self, text: str) -> list[Span]:
+ spans: list[Span] = []
+ for kind, pattern in self.PATTERNS:
+ for m in pattern.finditer(text):
+ spans.append(Span(m.start(), m.end(), kind, m.group(0)))
+ return _dedupe(spans)
+
+
+class PresidioDetector:
+ """Microsoft Presidio. MIT licensed, runs locally, no egress.
+
+ This is the component that makes tokenisation real rather than illustrative:
+ it recognises a broad set of entity types across locales, which is precisely
+ where pattern matching fails.
+
+ Install: pip install presidio-analyzer && python -m spacy download en_core_web_lg
+ """
+
+ name = "presidio"
+
+ DEFAULT_ENTITIES = [
+ "PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "LOCATION",
+ "DATE_TIME", "NRP", "CREDIT_CARD", "IBAN_CODE", "IP_ADDRESS",
+ "MEDICAL_LICENSE", "UK_NHS", "US_SSN",
+ ]
+
+ def __init__(self, entities: list[str] | None = None, language: str = "en"):
+ try:
+ from presidio_analyzer import AnalyzerEngine # type: ignore
+ except ImportError as e:
+ raise RuntimeError(
+ "presidio-analyzer is not installed. "
+ "pip install presidio-analyzer && python -m spacy download en_core_web_lg"
+ ) from e
+ self._engine = AnalyzerEngine()
+ self.entities = entities or self.DEFAULT_ENTITIES
+ self.language = language
+
+ def detect(self, text: str) -> list[Span]:
+ results = self._engine.analyze(text=text, entities=self.entities, language=self.language)
+ return _dedupe(
+ [Span(r.start, r.end, r.entity_type, text[r.start:r.end]) for r in results]
+ )
+
+
+class CompositeDetector:
+ """Run several detectors and merge their spans.
+
+ Neither half is sufficient alone, which the test output makes obvious:
+ Presidio finds names, dates and locations that no pattern will catch, and
+ misses domain identifiers like batch 20481 because it has no reason to know
+ they are sensitive. Regex catches those and nothing else.
+
+ Overlaps resolve to the longer span, so a Presidio PERSON covering a name
+ wins over a narrower pattern hit on part of it.
+ """
+
+ def __init__(self, *detectors: Detector):
+ self._detectors = [d for d in detectors if d is not None]
+ self.name = "+".join(d.name for d in self._detectors)
+
+ def detect(self, text: str) -> list[Span]:
+ spans: list[Span] = []
+ for d in self._detectors:
+ spans.extend(d.detect(text))
+ return _dedupe(spans)
+
+
+def _dedupe(spans: list[Span]) -> list[Span]:
+ """Drop overlaps, preferring the longer match, and return in reverse order.
+
+ Reverse order matters: replacements are applied back-to-front so earlier
+ offsets stay valid as the string is rewritten.
+ """
+ ordered = sorted(spans, key=lambda s: (s.start, -(s.end - s.start)))
+ kept: list[Span] = []
+ last_end = -1
+ for s in ordered:
+ if s.start >= last_end:
+ kept.append(s)
+ last_end = s.end
+ return sorted(kept, key=lambda s: s.start, reverse=True)
+
+
+# --------------------------------------------------------------------------
+# Custody
+# --------------------------------------------------------------------------
+
+class VaultBackend(Protocol):
+ name: str
+
+ def tokenise(self, kind: str, value: str) -> str:
+ ...
+
+ def detokenise(self, token: str) -> str | None:
+ ...
+
+ def save(self) -> None:
+ ...
+
+
+class LocalVault:
+ """Deterministic HMAC tokens with a JSON store. DEMONSTRATION ONLY.
+
+ Two properties are real: tokens are deterministic, and reversal needs the
+ store. Everything else that makes a vault a security control is absent.
+
+ Be explicit about the risk, because it is the opposite of obvious: the
+ store is a plaintext file mapping every token back to its original value.
+ If it leaks it is a single artefact containing all the sensitive data in
+ the corpus. That is worse than not tokenising, because it looks like a
+ control while concentrating the exposure. Never ship this.
+ """
+
+ name = "local"
+
+ def __init__(self, key: bytes | None = None, path: pathlib.Path | None = None):
+ self.key = key or os.environ.get("ONTO_VAULT_KEY", "demo-key-not-for-production").encode()
+ self.path = path or pathlib.Path(os.environ.get("ONTO_VAULT_PATH", "demo/derived/_vault.json"))
+ self.map: dict[str, str] = {}
+ if self.path.exists():
+ try:
+ self.map = json.loads(self.path.read_text())
+ except Exception:
+ self.map = {}
+
+ def tokenise(self, kind: str, value: str) -> str:
+ digest = hmac.new(self.key, value.encode(), hashlib.sha256).hexdigest()[:12]
+ token = f"TOK_{kind}_{digest}"
+ self.map[token] = value
+ return token
+
+ def detokenise(self, token: str) -> str | None:
+ return self.map.get(token)
+
+ def save(self) -> None:
+ self.path.parent.mkdir(parents=True, exist_ok=True)
+ self.path.write_text(json.dumps(self.map, indent=1))
+
+
+class SkyflowVault:
+ """Skyflow managed vault.
+
+ Skyflow has no open-source edition; this speaks to the hosted API. Custody,
+ rotation, access policy and audit are the vendor's, which is the reason to
+ use it over anything here.
+
+ Requires SKYFLOW_VAULT_URL, SKYFLOW_VAULT_ID and SKYFLOW_BEARER_TOKEN.
+ Determinism must be configured on the vault's column policy: a vault issuing
+ random tokens per occurrence breaks the join-key property this pipeline
+ relies on. Confirm that before relying on cross-document resolution.
+ """
+
+ name = "skyflow"
+
+ def __init__(self):
+ self.url = os.environ.get("SKYFLOW_VAULT_URL", "").rstrip("/")
+ self.vault_id = os.environ.get("SKYFLOW_VAULT_ID", "")
+ self.token = os.environ.get("SKYFLOW_BEARER_TOKEN", "")
+ self.table = os.environ.get("SKYFLOW_TABLE", "pii")
+ self.column = os.environ.get("SKYFLOW_COLUMN", "value")
+ missing = [n for n, v in (("SKYFLOW_VAULT_URL", self.url),
+ ("SKYFLOW_VAULT_ID", self.vault_id),
+ ("SKYFLOW_BEARER_TOKEN", self.token)) if not v]
+ if missing:
+ raise RuntimeError(f"skyflow vault needs: {', '.join(missing)}")
+ self._cache: dict[str, str] = {}
+
+ def _request(self, path: str, payload: dict) -> dict:
+ import urllib.request
+ req = urllib.request.Request(
+ f"{self.url}/v1/vaults/{self.vault_id}/{path}",
+ data=json.dumps(payload).encode(),
+ headers={"Content-Type": "application/json",
+ "Authorization": f"Bearer {self.token}"},
+ )
+ import urllib.error
+ try:
+ return json.load(__import__("urllib.request", fromlist=["x"]).urlopen(req, timeout=60))
+ except urllib.error.HTTPError as e:
+ raise RuntimeError(f"skyflow {path} failed: {e.code} {e.read()[:200]!r}") from e
+
+ def tokenise(self, kind: str, value: str) -> str:
+ if value in self._cache:
+ return self._cache[value]
+ res = self._request(self.table, {"records": [{"fields": {self.column: value}}],
+ "tokenization": True})
+ try:
+ token = res["records"][0]["tokens"][self.column]
+ except (KeyError, IndexError) as e:
+ raise RuntimeError(f"unexpected skyflow response shape: {res}") from e
+ self._cache[value] = token
+ return token
+
+ def detokenise(self, token: str) -> str | None:
+ res = self._request("detokenize", {"detokenizationParameters": [{"token": token}]})
+ try:
+ return res["records"][0]["value"]
+ except (KeyError, IndexError):
+ return None
+
+ def save(self) -> None:
+ """Nothing to persist: custody is the vendor's."""
+
+
+class DatabunkerVault:
+ """Databunker, self-hosted and open source.
+
+ The middle option: real custody, in your own infrastructure, with no extra
+ processor on the data map. Requires DATABUNKER_URL and DATABUNKER_TOKEN.
+ """
+
+ name = "databunker"
+
+ def __init__(self):
+ self.url = os.environ.get("DATABUNKER_URL", "").rstrip("/")
+ self.token = os.environ.get("DATABUNKER_TOKEN", "")
+ if not self.url or not self.token:
+ raise RuntimeError("databunker vault needs DATABUNKER_URL and DATABUNKER_TOKEN")
+ self._cache: dict[str, str] = {}
+
+ def _post(self, path: str, payload: dict) -> dict:
+ import urllib.request
+ req = urllib.request.Request(
+ f"{self.url}/v1/{path}", data=json.dumps(payload).encode(),
+ headers={"Content-Type": "application/json", "X-Bunker-Token": self.token})
+ return json.load(urllib.request.urlopen(req, timeout=60))
+
+ def tokenise(self, kind: str, value: str) -> str:
+ if value in self._cache:
+ return self._cache[value]
+ res = self._post("token", {"tokentype": kind.lower(), "record": value})
+ token = res.get("token") or res.get("data", {}).get("token", "")
+ if not token:
+ raise RuntimeError(f"unexpected databunker response: {res}")
+ self._cache[value] = token
+ return token
+
+ def detokenise(self, token: str) -> str | None:
+ try:
+ return self._post("token/get", {"token": token}).get("record")
+ except Exception:
+ return None
+
+ def save(self) -> None:
+ """Nothing to persist: custody is the server's."""
+
+
+# --------------------------------------------------------------------------
+# Composition
+# --------------------------------------------------------------------------
+
+class Tokeniser:
+ """A detector plus a vault. This is the only type the pipeline knows about."""
+
+ def __init__(self, detector: Detector, vault: VaultBackend):
+ self.detector = detector
+ self.vault = vault
+
+ @property
+ def description(self) -> str:
+ return f"{self.detector.name} detection + {self.vault.name} custody"
+
+ def tokenise(self, text: str) -> tuple[str, int]:
+ """Replace every detected span with a stable token. Returns (text, count)."""
+ spans = self.detector.detect(text) # reverse order, so offsets stay valid
+ for s in spans:
+ token = self.vault.tokenise(s.kind, s.text)
+ text = text[: s.start] + token + text[s.end:]
+ return text, len(spans)
+
+ def detokenise(self, text: str) -> str:
+ """Reverse. In production this is entitlement-gated and audited."""
+ for token in sorted(set(re.findall(r"TOK_[A-Z_]+_[0-9a-f]{6,}|tok_[\w-]{8,}", text)),
+ key=len, reverse=True):
+ value = self.vault.detokenise(token)
+ if value is not None:
+ text = text.replace(token, value)
+ return text
+
+ def save(self) -> None:
+ self.vault.save()
+
+
+def build(detector: str | None = None, vault: str | None = None) -> Tokeniser:
+ """Construct from configuration, falling back loudly rather than silently.
+
+ A tokeniser that silently degrades to weaker detection is worse than one
+ that fails, because the pipeline keeps running and the output looks fine.
+ """
+ # Default to the strongest detection available. `both` already degrades to
+ # patterns with a warning when Presidio is absent, so defaulting to `regex`
+ # bought nothing and cost real coverage: a run inheriting the default found
+ # 1 sensitive value where the composite found 24, with nothing in the
+ # output to indicate that detection had been weakened.
+ d_name = (detector or os.environ.get("ONTO_DETECTOR", "both")).lower()
+ v_name = (vault or os.environ.get("ONTO_VAULT", "local")).lower()
+
+ def _composite():
+ """Presidio plus patterns, degrading loudly if Presidio is absent.
+
+ For the composite the run is still worth completing without Presidio,
+ but silence would be dishonest: the log has to say detection was
+ weakened, because the output looks identical either way. An explicit
+ ONTO_DETECTOR=presidio still fails hard.
+ """
+ try:
+ return CompositeDetector(PresidioDetector(), RegexDetector())
+ except RuntimeError as e:
+ import sys as _sys
+ print(f"WARNING: Presidio unavailable, falling back to patterns only. "
+ f"Detection is materially weaker. ({e})", file=_sys.stderr)
+ return RegexDetector()
+
+ detectors = {"regex": RegexDetector, "presidio": PresidioDetector, "both": _composite}
+ vaults = {"local": LocalVault, "skyflow": SkyflowVault, "databunker": DatabunkerVault}
+
+ if d_name not in detectors:
+ raise SystemExit(f"unknown ONTO_DETECTOR={d_name}. options: {', '.join(detectors)}")
+ if v_name not in vaults:
+ raise SystemExit(f"unknown ONTO_VAULT={v_name}. options: {', '.join(vaults)}")
+
+ return Tokeniser(detectors[d_name](), vaults[v_name]())
+
+
+if __name__ == "__main__":
+ import sys
+
+ piped = "" if sys.stdin.isatty() else sys.stdin.read()
+ sample = piped.strip() or (
+ "Dr Alice Morgan ([email protected], +44 20 7946 0958) reviewed "
+ "batch 20481 on 3 March. Contact Prof Brian Hale for the audit."
+ )
+ t = build()
+ print(f"tokeniser: {t.description}\n")
+ out, n = t.tokenise(sample)
+ print(f"--- tokenised ({n} spans) ---\n{out}\n")
+ print(f"--- detokenised (round trip) ---\n{t.detokenise(out)}\n")
+ a, _ = t.tokenise("contact [email protected]")
+ b, _ = t.tokenise("write to [email protected]")
+ tok_a = re.search(r"TOK_\w+", a)
+ tok_b = re.search(r"TOK_\w+", b)
+ same = tok_a and tok_b and tok_a.group(0) == tok_b.group(0)
+ print(f"determinism (same value -> same token across documents): {'YES' if same else 'NO'}")
+ t.save()
diff --git a/demo/verify.py b/demo/verify.py
new file mode 100644
index 00000000..4ffef134
--- /dev/null
+++ b/demo/verify.py
@@ -0,0 +1,282 @@
+#!/usr/bin/env python3
+"""
+Standalone verifier for the derived ontology and the graphs extracted against it.
+
+This is stage 5 and 6 of the pipeline with nothing else attached: no model
+calls, no extraction, no repair. It answers one question, repeatably and in
+seconds: given what is on disk right now, does it hold up?
+
+Six checks, in the order the repo's own workflow prescribes:
+
+ 1. PARSE every fragment and the merged ontology are syntactically valid
+ 2. STATS the merged ontology loads with the class and property counts claimed
+ 3. LINT missing labels, comments, domains, ranges
+ 4. ENFORCE design-pattern compliance against the generic rule pack
+ 5. VOCAB closed-world check of every extracted graph against the ontology
+ 6. REASON disjointness violations across the merged instance data
+
+Exit code is the number of failed checks, so this can gate a build.
+"""
+
+from __future__ import annotations
+
+import json
+import pathlib
+import sys
+
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+from ontology_from_docs import (mcp, tool_text, engine, rehead, is_valid_turtle, # noqa: E402
+ sources_of, ROOT)
+
+BOLD, DIM, RED, GRN, YEL, CYA, OFF = (
+ "\033[1m", "\033[2m", "\033[31m", "\033[32m", "\033[33m", "\033[36m", "\033[0m")
+
+DERIVED = ROOT / "demo" / "derived"
+ONTOLOGY = DERIVED / "_ontology.ttl"
+
+
+def head(n: int, title: str, sub: str) -> None:
+ print(f"\n{BOLD}CHECK {n} {title}{OFF} {DIM}{sub}{OFF}")
+
+
+def main() -> int:
+ if not ONTOLOGY.exists():
+ print(f"{RED}no derived ontology at {ONTOLOGY}{OFF}")
+ return 1
+
+ graphs = sorted(DERIVED.glob("*.data.ttl"))
+ fragments = sorted(DERIVED.glob("*.vocab.ttl"))
+ failures = 0
+
+ print(f"{BOLD}Verifier{OFF} {DIM}{ONTOLOGY.relative_to(ROOT)}, "
+ f"{len(fragments)} fragments, {len(graphs)} extracted graphs{OFF}")
+
+ # ---- 1. parse ----------------------------------------------------------
+ head(1, "PARSE", "syntax of every fragment and the merged ontology")
+ # Judged as the merge judges them: under the canonical prefix header and
+ # after the mechanical repairs. Validating the raw file instead reports a
+ # missing prefix declaration as a modelling failure, which it is not.
+ bad_fragments, mended = [], []
+ for f in fragments:
+ raw = f.read_text()
+ if is_valid_turtle(raw)[0]:
+ continue
+ if is_valid_turtle(rehead(raw))[0]:
+ mended.append(f.stem)
+ else:
+ bad_fragments.append((f.stem, is_valid_turtle(rehead(raw))[1]))
+ if mended:
+ print(f" {DIM}{len(mended)} fragment(s) recovered by reheading and mending: "
+ f"{', '.join(m.replace('.vocab', '') for m in mended)}{OFF}")
+ if bad_fragments:
+ print(f" {YEL}{len(bad_fragments)} of {len(fragments)} fragments do not parse "
+ f"and are excluded from the merge{OFF}")
+ for name, why in bad_fragments:
+ print(f" {DIM}{name} {why[:110]}{OFF}")
+ else:
+ print(f" {GRN}all {len(fragments)} fragments usable{OFF}")
+
+ res = engine(["clear", f"load {ONTOLOGY}"])
+ if '"error"' in res:
+ print(f" {RED}merged ontology does not parse{OFF}")
+ print(f" {DIM}{res[-220:]}{OFF}")
+ failures += 1
+ else:
+ print(f" {GRN}merged ontology parses{OFF}")
+
+ # ---- 2. stats ----------------------------------------------------------
+ head(2, "STATS", "does the ontology load with the structure it claims")
+ st = {}
+ try:
+ st = json.loads(engine(["clear", f"load {ONTOLOGY}", "stats"]).split("\n")[-1])["result"]
+ except Exception:
+ pass
+ classes, props = st.get("classes", 0), st.get("properties", 0)
+ print(f" {classes} classes, {props} properties "
+ f"({st.get('object_properties', 0)} object, {st.get('data_properties', 0)} data), "
+ f"{st.get('triples', 0)} triples")
+ # A vocabulary with classes and no properties is the failure mode that a
+ # truncated parse produces, and it silently poisons everything downstream.
+ if classes and not props:
+ print(f" {RED}classes but no properties: the parse is truncated{OFF}")
+ failures += 1
+ elif classes and props:
+ print(f" {GRN}structure present{OFF}")
+
+ q_disjoint = ("PREFIX owl: <http://www.w3.org/2002/07/owl#> "
+ "SELECT (COUNT(*) AS ?n) WHERE { ?a owl:disjointWith ?b }")
+ try:
+ rows = json.loads(engine(["clear", f"load {ONTOLOGY}",
+ f"query {json.dumps(q_disjoint)}"]).split("\n")[-1])["result"]["results"]
+ n = rows[0].get("n", "0").split('"')[1] if rows else "0"
+ print(f" {n} disjointness axioms {DIM}(what gives the reasoner something to catch){OFF}")
+ except Exception:
+ pass
+
+ # Competing modelling patterns: a subclass partition and an attribute
+ # class for the same notion. The extractor reliably picks the attribute
+ # class, which is the one form disjointness cannot reach, so leaving both
+ # in the vocabulary quietly disables contradiction detection. None of the
+ # engine's enforce packs catches this; the pipeline's reconcile() removes
+ # them, and this check keeps it removed.
+ import re as _re
+ ttl_text = ONTOLOGY.read_text()
+ cls = set(_re.findall(r":(\w+)\s+(?:a|rdf:type)\s+owl:Class", ttl_text))
+ kids: dict[str, set[str]] = {}
+ for m in _re.finditer(r":(\w+)\s+rdfs:subClassOf\s+:(\w+)", ttl_text):
+ kids.setdefault(m.group(2), set()).add(m.group(1))
+ from ontology_from_docs import KIND_SUFFIXES
+ collisions = sorted({p + s for p, k in kids.items() if len(k) >= 2
+ for s in KIND_SUFFIXES if p + s in cls})
+ if collisions:
+ print(f" {RED}{len(collisions)} attribute class(es) compete with a subclass "
+ f"partition: {', '.join(':' + c for c in collisions)}{OFF}")
+ print(f" {DIM}the extractor will use these instead of the partition, "
+ f"and disjointness cannot reach them{OFF}")
+ failures += 1
+ else:
+ print(f" {GRN}no competing modelling patterns{OFF}")
+
+ # ---- MCP session for the tool-backed checks ----------------------------
+ try:
+ mcp("initialize", {"protocolVersion": "2025-03-26", "capabilities": {},
+ "clientInfo": {"name": "verify", "version": "1"}})
+ mcp("tools/call", {"name": "onto_load", "arguments":
+ {"path": str(ONTOLOGY), "name": "derived", "force_recompile": True}})
+ except Exception as e:
+ print(f"\n{RED}engine unreachable: {e}{OFF}")
+ print(f"{DIM}start it with: ./target/release/open-ontologies serve --port 8137{OFF}")
+ return failures + 1
+
+ # ---- 3. lint -----------------------------------------------------------
+ head(3, "LINT", "labels, comments, domains, ranges")
+ try:
+ d = json.loads(tool_text(mcp("tools/call", {"name": "onto_lint", "arguments": {}})) or "{}")
+ issues = d.get("issues", [])
+ by_rule: dict[str, int] = {}
+ for i in issues:
+ by_rule[i.get("rule", i.get("kind", "?"))] = by_rule.get(i.get("rule", i.get("kind", "?")), 0) + 1
+ if issues:
+ print(f" {YEL}{len(issues)} issues{OFF}")
+ for rule, n in sorted(by_rule.items(), key=lambda x: -x[1])[:8]:
+ print(f" {n:4d} {rule}")
+ else:
+ print(f" {GRN}clean{OFF}")
+ except Exception as e:
+ print(f" {DIM}lint unavailable: {e}{OFF}")
+
+ # ---- 4. enforce --------------------------------------------------------
+ head(4, "ENFORCE", "design patterns, generic rule pack")
+ try:
+ d = json.loads(tool_text(mcp("tools/call", {"name": "onto_enforce",
+ "arguments": {"rules": "generic"}})) or "{}")
+ v = d.get("violations", [])
+ if v:
+ by_rule = {}
+ for i in v:
+ key = i.get("rule", "?")
+ by_rule[key] = by_rule.get(key, 0) + 1
+ print(f" {YEL}{len(v)} violations{OFF}")
+ for rule, n in sorted(by_rule.items(), key=lambda x: -x[1])[:8]:
+ print(f" {n:4d} {rule}")
+ else:
+ print(f" {GRN}no violations{OFF}")
+ except Exception as e:
+ print(f" {DIM}enforce unavailable: {e}{OFF}")
+
+ # ---- 5. vocab check ----------------------------------------------------
+ head(5, "VOCAB", "closed-world check of every extracted graph")
+ conform, review = 0, []
+ for f in graphs:
+ try:
+ d = json.loads(tool_text(mcp("tools/call", {"name": "onto_vocab_check",
+ "arguments": {"data": f.read_text(),
+ "inline": True}})) or "{}")
+ except Exception as e:
+ print(f" {RED}error{OFF} {f.stem} {e}")
+ continue
+ if d.get("conforms"):
+ conform += 1
+ print(f" {GRN}conforms{OFF} {f.stem}")
+ else:
+ terms = [t.split("#")[-1] for t in d.get("hallucinated_terms", [])]
+ review.append((f.stem, terms))
+ print(f" {YEL}review {OFF} {f.stem} {DIM}{len(terms)} undeclared: "
+ f"{', '.join(terms[:5])}{'…' if len(terms) > 5 else ''}{OFF}")
+ if graphs:
+ pct = 100 * conform / len(graphs)
+ print(f" {BOLD}{conform}/{len(graphs)} conform ({pct:.0f}%){OFF}")
+ if conform == 0:
+ failures += 1
+
+ # ---- 6. reason ---------------------------------------------------------
+ head(6, "REASON", "disjointness violations across the merged instance data")
+ loads = ["clear", f"load {ONTOLOGY}"] + [f"load {g}" for g in graphs]
+ try:
+ st = json.loads(engine(loads + ["stats"]).split("\n")[-1])["result"]
+ print(f" merged: {st['triples']} triples, {st['classes']} classes, "
+ f"{st['individuals']} individuals")
+ except Exception:
+ pass
+
+ q = ("PREFIX owl: <http://www.w3.org/2002/07/owl#> "
+ "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> "
+ "SELECT DISTINCT ?subject ?a ?b WHERE { ?subject a ?a, ?b . "
+ "FILTER(STR(?a) < STR(?b)) ?a rdfs:subClassOf* ?da . ?b rdfs:subClassOf* ?db . "
+ "{ ?da owl:disjointWith ?db } UNION { ?db owl:disjointWith ?da } }")
+ try:
+ rows = json.loads(engine(loads + [f"query {json.dumps(q)}"]).split("\n")[-1])["result"]["results"]
+ except Exception:
+ rows = []
+ # Split by provenance. A violation is only a contradiction when one
+ # document asserts one side and does not assert the other; both sides from
+ # a single document is one extractor over-typing, and it can produce
+ # hundreds of violations that mean nothing.
+ src = sources_of(graphs)
+ contradictions, overtyping = [], []
+ for r in rows:
+ v = {k: val.split("#")[-1].rstrip(">") for k, val in r.items()}
+ da = src.get((v["subject"], v["a"]), set())
+ db = src.get((v["subject"], v["b"]), set())
+ (contradictions if (da - db) and (db - da) else overtyping).append((v, da, db))
+
+ for v, da, db in contradictions:
+ print(f" {RED}CONTRADICTION{OFF} {v['subject']} is {v['a']} per "
+ f"{','.join(sorted(da - db))} but {v['b']} per {','.join(sorted(db - da))}")
+ if overtyping:
+ by_doc: dict[str, int] = {}
+ for _, da, db in overtyping:
+ for d in da | db:
+ by_doc[d] = by_doc.get(d, 0) + 1
+ worst = ", ".join(f"{d} {n}" for d, n in sorted(by_doc.items(), key=lambda x: -x[1])[:3])
+ print(f" {YEL}{len(overtyping)} violations within a single document{OFF} "
+ f"{DIM}(over-typing, not disagreement: {worst}){OFF}")
+ if not rows:
+ print(f" {DIM}none{OFF}")
+
+ # A contradiction can only be found if the same real-world thing carries
+ # both types. Report how far the data is from that precondition, because
+ # "no conflicts" and "nothing could ever conflict" look identical otherwise.
+ q_multi = ("SELECT (COUNT(DISTINCT ?s) AS ?n) WHERE "
+ "{ ?s a ?t1, ?t2 . FILTER(STR(?t1) < STR(?t2)) }")
+ try:
+ rows2 = json.loads(engine(loads + [f"query {json.dumps(q_multi)}"]).split("\n")[-1])["result"]["results"]
+ n = rows2[0].get("n", "0").split('"')[1] if rows2 else "0"
+ if n == "0":
+ print(f" {YEL}0 individuals carry more than one type, so no contradiction is "
+ f"reachable regardless of the axioms{OFF}")
+ print(f" {DIM}fix is entity resolution: the same thing must get one IRI "
+ f"across documents{OFF}")
+ else:
+ print(f" {DIM}{n} individuals carry more than one type{OFF}")
+ except Exception:
+ pass
+
+ print(f"\n{BOLD}{'PASS' if not failures else str(failures) + ' CHECK(S) FAILED'}{OFF}"
+ f" {DIM}{conform}/{len(graphs)} graphs conform, "
+ f"{len(contradictions)} contradictions across documents{OFF}\n")
+ return failures
+
+
+if __name__ == "__main__":
+ sys.exit(main())