Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
case-study: onto-correctness-bench (the open-world hole, measured)
FABIOTESS6c5cbed7ef59
diff --git a/case-studies/onto-correctness-bench/.gitignore b/case-studies/onto-correctness-bench/.gitignorenew file mode 100644index 00000000..16d0b757--- /dev/null+++ b/case-studies/onto-correctness-bench/.gitignore@@ -0,0 +1,4 @@+.venv/+# Large, re-fetchable vocabularies (run-demo.sh downloads them). IES4 lives in the parent repo.+data/*.owl+data/*.ttldiff --git a/case-studies/onto-correctness-bench/BUILD_REPORT.md b/case-studies/onto-correctness-bench/BUILD_REPORT.mdnew file mode 100644index 00000000..66dfa0dd--- /dev/null+++ b/case-studies/onto-correctness-bench/BUILD_REPORT.md@@ -0,0 +1,92 @@+# BUILD_REPORT: onto-correctness-bench++Scrupulously honest record of what was fetched, what was computed, how the corpus was+constructed, and the limits of the claim. Nothing here is hand-tuned; every number in+`results/` is produced by `src/bench.py` on a fixed seed.++## What was fetched (real, public)++| Source | URL | Local | Triples parsed |+|---|---|---|---|+| schema.org (current) | https://schema.org/version/latest/schemaorg-current-https.ttl | `data/schemaorg.ttl` | 17,949 |+| PATO (Phenotype And Trait Ontology) | http://purl.obolibrary.org/obo/pato.owl | `data/pato.owl` | 259,922 |+| RO (Relation Ontology) | http://purl.obolibrary.org/obo/ro.owl | `data/ro.owl` | 11,640 |+| IES4 (UK Information Exchange Standard) | bundled in parent repo | `../../benchmark/reference/ies4.ttl` | 3,976 |++Fetched 2026-07-20. PATO and RO are combined into one "OBO (PATO+RO)" configuration+because PATO is class-heavy and declares almost no object properties of its own (it+imports RO/BFO); records type instances as PATO qualities linked by RO relations, which+is how OBO instance data is actually shaped.++## What was computed++For each vocabulary:+1. **Vocabulary extraction**: declared classes (`owl:Class`/`rdfs:Class`) and properties+ (`owl:ObjectProperty`/`owl:DatatypeProperty`/`owl:AnnotationProperty`/`rdf:Property`)+ whose IRI falls in the ontology's own *policed* namespace(s). Counts in `results/`.+2. **Corpus**: 100 clean + 100 hallucinated small "record" graphs. Every record is an+ instance minted under `https://ex.tesseract.academy/inst/` (never policed), typed with+ a real declared class and given 3 triples on real declared properties.+3. **Hallucinated records** are the clean record **plus** 1–2 *extra* fabricated terms.+ The real properties are all still present, so every record satisfies the SHACL shapes;+ the fabricated term is an unconstrained extra.+4. **SHACL shapes**: authored per record in the ordinary, non-closed style: one+ `sh:NodeShape` with `sh:targetClass <realClass>` and a `sh:property [ sh:path <realProp>;+ sh:minCount 1 ]` for each real property used. Validated with pySHACL+ (`inference="none"`, `advanced=False`).+5. **Closed-world gate**: a direct reimplementation of the `onto_vocab_check` principle:+ collect every predicate and every `rdf:type` object used in the data; flag any whose IRI+ sits in a policed namespace but is not in the declared set.++## How fabricated terms were constructed (so you can judge fairness)++The fabricated terms are *plausible-but-nonexistent*, not random strings a typo-checker+would catch:++- **Readable-name vocabularies (schema.org, IES4):** a real local name is mutated into a+ natural synonym or affixed variant (`priceRange` → `priceBracket`, `Offer` →+ `MerchandiseOffer`, `<name>` → `<name>Spec`), then **confirmed absent** from the declared+ vocabulary before use. These are the terms an LLM invents by analogy.+- **OBO (opaque IDs):** a well-formed OBO id in a policed prefix (`PATO_…`, `RO_…`,+ `BFO_…`) that is **confirmed undeclared**. This is the "LLM cites a real-looking but wrong+ ontology id" failure mode.++Every hallucinated record is guaranteed to receive at least one fabricated term, so the+per-graph catch-rate denominator is honest (no empty "hallucinated" graphs).++## The result, stated precisely++- SHACL false-pass rate = P(`conforms=true` | graph contains ≥1 fabricated term) = **100%**+ on all three vocabularies (300/300 graphs). This is *structural*, not a tuning artifact:+ SHACL Core has no shape targeting the fabricated term, so it is never inspected.+- Closed-world catch rate = P(gate flags ≥1 fabricated term | graph has one) = **100%**+ (300/300), with **0%** false positives on the 300 clean graphs.+- Term-level recall: closed-world **100%** of 418 injected terms; SHACL **0%** (it never+ flags a term by IRI-existence, because that is not what it checks).++## Limits of the claim (do not overstate)++1. **This is a demonstration of a structural property, not a surprising ML finding.** The+ closed-world gate is *defined* to flag undeclared policed-namespace terms, and the fakes+ are exactly that. The contribution is measuring, on real vocabularies at scale, that+ (a) ordinary SHACL practice cannot catch this class of error at all, and (b) the gate is+ precise (0 FP). The 100%s are the point, not a leaderboard score to beat.+2. **`sh:closed` can catch extra *predicates*** if you enumerate every allowed predicate on+ every shape and accept the maintenance and import-breakage cost. It still does not police+ `rdf:type` class values. We benchmarked ordinary open shapes because that is what teams+ deploy; a `sh:closed` comparison is a fair next addition.+3. **Only vocabulary *existence* is checked here.** A model that uses a *real* term in a+ *semantically impossible* place (right symbol, wrong denotation) is not caught by this+ gate. That is the third stage in the Track-A design (certified denotation against a+ world model) and is not measured in this repo yet.+4. **Corpus is synthetic-but-grounded.** The vocabularies are real; the record graphs are+ generated, not harvested from real LLM output. A follow-up should replay actual+ LLM-generated RDF (e.g. the IES4 fine-tune's outputs) through the same gate, the shipped+ `fabsssss/qwen3-coder-30b-a3b-ies4` model reports term-confusion 0%→88.6% and+ hallucination 0.937→0.010 under exactly this kind of closed-world check.++## Reproducibility++`./run-demo.sh` fetches the vocabularies and reruns `src/bench.py`. Deterministic: the+corpus uses a SHA-256-based PRNG seeded on (vocabulary, record index, role), so results are+identical across machines. rdflib 7.6.0, pyshacl (current) at build time.diff --git a/case-studies/onto-correctness-bench/README.md b/case-studies/onto-correctness-bench/README.mdnew file mode 100644index 00000000..a81b8295--- /dev/null+++ b/case-studies/onto-correctness-bench/README.md@@ -0,0 +1,97 @@+# onto-correctness-bench++**The open-world hole, measured: why SHACL cannot catch a hallucinated ontology term, and what closes it.**++A reproducible benchmark showing that the field's default RDF validator, **SHACL, is+open-world** and therefore structurally blind to the most common failure mode of+LLM-generated knowledge graphs, a *plausible-but-nonexistent* class or property. It+measures the gap on **three real public vocabularies** and shows that a **closed-world+vocabulary gate** (the [`onto_vocab_check`](../../README.md) principle from Open+Ontologies) catches exactly what SHACL admits.++This is the correctness layer that Encode / ARIA Challengescape items+[#101](https://encode-challengescape.pillar.vc) ("AI systems built on the Semantic Web+lack formal guarantees of correctness and safety") and #108 ("shared representations+connecting domain concepts to model internals") are asking for.++## The result++Deterministic run of [`src/bench.py`](src/bench.py), 100 clean + 100 hallucinated+record graphs per vocabulary, no hand-entered numbers (full table in+[`results/SUMMARY.md`](results/SUMMARY.md), raw data in+[`results/results.json`](results/results.json)):++| Vocabulary | Ontology triples | Declared classes / props | Fabricated terms | **SHACL false-pass** | **Closed-world catch** | CW false-positive |+|---|--:|--:|--:|--:|--:|--:|+| schema.org | 17,949 | 933 / 1,521 | 136 | **100%** | **100%** | 0% |+| IES4 | 3,976 | 510 / 204 | 134 | **100%** | **100%** | 0% |+| OBO (PATO+RO) | 270,126 | 2,889 / 759 | 148 | **100%** | **100%** | 0% |++**Across 3 real vocabularies and 418 fabricated terms in 300 graphs, open-world SHACL+reported `conforms=true` on every single graph containing a fabricated term (300/300).+The closed-world gate flagged one in 300/300, with zero false positives on clean graphs.**++## See it on one concrete record++[`examples/hallucinated.ttl`](examples/hallucinated.ttl) is a valid-looking `schema:Offer`+with two terms that **do not exist** in schema.org, the class `schema:MerchandiseOffer`+and the predicate `schema:priceBracket`:++```+SHACL conforms=True flags: [] <- admits both fakes+closed-world flags: [MerchandiseOffer, priceBracket] <- catches both+```++SHACL conforms because no shape *targets* those terms, so it never looks at them. That is+not a bug in pySHACL; it is the open-world semantics of SHACL Core. An LLM that emits a+fabricated term therefore sails through the exact validator teams rely on to trust it.++## Why this is the ownable gap++- **SHACL validates shapes, not vocabulary.** It reports violations only for triples a+ shape constrains. A predicate or `rdf:type` class with no shape is silently ignored.+- **`sh:closed` is not the fix.** Closed shapes catch *extra* predicates only if you+ enumerate every allowed predicate on *every* shape in advance, do not police `rdf:type`+ class values, and break the moment a record legitimately uses an imported vocabulary.+ It is per-shape bookkeeping, not a vocabulary guarantee.+- **The closed-world gate is one check:** every predicate and every `rdf:type` class whose+ IRI lives in a *policed* namespace must be **declared** in the loaded ontology. Standard+ `rdf`/`rdfs`/`owl`/`xsd`/`sh` vocabulary and your own instance IRIs are never policed, so+ the false-positive rate on clean data is 0%.++This is the primitive already shipped as a native Rust MCP tool, `onto_vocab_check`, in the+parent [Open Ontologies](../../README.md) server. This benchmark quantifies, on real+vocabularies, why it is not optional for any pipeline that lets a model author RDF.++## Reproduce++```bash+./run-demo.sh # fetches the 3 vocabularies, runs the deterministic benchmark+```++Requires Python 3.10+. Downloads: schema.org (current Turtle), PATO and RO from the+OBO Foundry PURLs. IES4 is bundled in the parent repo at+[`benchmark/reference/ies4.ttl`](../../benchmark/reference/ies4.ttl).++## Honest scope++See [`BUILD_REPORT.md`](BUILD_REPORT.md) for exactly what was fetched and computed, the+construction of the fabricated terms, and the limits of the claim. In short: this+demonstrates a *structural* property (SHACL cannot see undeclared terms; a closed-world+gate can), measured at scale on real vocabularies. It is not a claim that closed-world+checking replaces SHACL, the two are complementary. It does not measure whether a *real*+value is used in a *semantically wrong* place (that is the next gate: certified denotation).++---++### Built by Tesseract Academy++We build the correctness and assurance layer for AI-generated knowledge graphs and+ontologies. If you are deploying LLMs that author or extend RDF/OWL, in science, defence,+health, or the enterprise, and need to guarantee that every emitted term actually exists,+we can help.++**Get the closed-world gate running on your ontology, or talk to us about an assurance+review:** [gov.tesseract.academy](https://gov.tesseract.academy) · [email protected]++Part of the [Open Ontologies](../../README.md) project · MIT licensed · real data, real numbers.diff --git a/case-studies/onto-correctness-bench/docs/the-open-world-hole.md b/case-studies/onto-correctness-bench/docs/the-open-world-hole.mdnew file mode 100644index 00000000..8516faf0--- /dev/null+++ b/case-studies/onto-correctness-bench/docs/the-open-world-hole.md@@ -0,0 +1,133 @@+# The Open-World Hole: Why SHACL Cannot Catch a Hallucinated Ontology Term (and What Closes It)++*Tesseract Academy · Open Ontologies case study · all numbers reproducible from [onto-correctness-bench](../README.md)*++Ask a large language model to emit RDF and it will, most of the time, give you+syntactically perfect Turtle. It will also, some of the time, give you a term that does+not exist: a `schema:priceBracket` where it meant `schema:priceRange`, a+`schema:MerchandiseOffer` that was never a class, an OBO id off by a digit. The triple+parses. It reads correctly. And it is referentially fake.++The reflex is to reach for SHACL. SHACL is the W3C validation language for RDF; it is what+serious knowledge-graph teams run before they trust generated data. So here is the+uncomfortable result, measured on three real vocabularies: **SHACL does not catch this. Not+some of the time. None of the time.**++## The measurement++We took three real, public vocabularies of very different shapes: schema.org (the web's+vocabulary, 933 classes and 1,521 properties), IES4 (the UK government's Information+Exchange Standard, 510 classes and 204 properties), and a combination of PATO and RO from+the OBO Foundry (2,889 classes and 759 properties, from 270,126 triples). For each we+generated 100 clean record graphs using only real, declared terms, and 100 "hallucinated"+graphs that are identical except for one or two extra fabricated terms, each confirmed+absent from the vocabulary but named plausibly enough that a model would emit it. Then we+validated every graph two ways: with ordinary hand-authored SHACL shapes, and with a+closed-world vocabulary gate.++| Vocabulary | Fabricated terms | SHACL false-pass rate | Closed-world catch rate | CW false-positive |+|---|--:|--:|--:|--:|+| schema.org | 136 | **100%** | **100%** | 0% |+| IES4 | 134 | **100%** | **100%** | 0% |+| OBO (PATO+RO) | 148 | **100%** | **100%** | 0% |++Across 418 fabricated terms in 300 graphs, SHACL reported `conforms=true` on every single+graph that contained a fabricated term. The closed-world gate flagged one in all 300, and+raised zero false alarms on the 300 clean graphs. The full run is deterministic and+[reproducible in one command](../run-demo.sh).++## This is not a bug. It is the semantics.++It would be easy to read that 100% as a defect in some SHACL engine. It is not. It is the+open-world assumption working exactly as specified. SHACL, as Holger Knublauch and Dimitris+Kontokostas set out in the [W3C Recommendation](https://www.w3.org/TR/shacl/), validates a+data graph against *shapes*. A shape targets some nodes and constrains some of their+properties. If a triple uses a predicate no shape mentions, or an `rdf:type` no shape+targets, SHACL has nothing to say about it, and silence means conformance. The validator is+not asking "does this term exist?" It is asking "do the terms I was told to check satisfy+their constraints?" A fabricated term is, by construction, one nobody told it to check.++Consider one concrete record from the benchmark, a `schema:Offer` carrying two terms that+do not exist in schema.org:++```turtle+ex:offer1 a schema:Offer , schema:MerchandiseOffer ; # fabricated class+ schema:priceCurrency "GBP" ;+ schema:price "49.00" ;+ schema:priceBracket "mid" . # fabricated predicate+```++Run it against a normal `schema:Offer` shape that requires a price and a currency, and it+conforms. Both real constraints are satisfied; the two fakes are simply extra triples the+shape never looks at. The closed-world gate, asked instead "is every schema.org term used+here actually declared in schema.org?", returns `MerchandiseOffer` and `priceBracket`. Same+data, two questions, and only one of them is the question you actually care about when a+model wrote the graph.++## "Just use sh:closed"++The informed objection is that SHACL has `sh:closed`, which rejects triples whose predicates+are not in an allowed list. It does, and it helps, but it is not the guarantee people think+it is. Closed shapes catch extra *predicates* only if you enumerate every permitted+predicate on every shape, in advance, and keep that list current. They do not police the+values of `rdf:type` against the vocabulary. And they break the moment a record legitimately+uses a term from an imported ontology you did not list, which in the OBO world (where PATO+imports RO imports BFO) is constant. `sh:closed` is per-shape bookkeeping. What generated+data needs is a vocabulary guarantee: not "did this node stay inside the property list I+wrote for it", but "does every term in this graph denote something the ontology actually+defines".++That is a different check, and it belongs at a different place in the pipeline. Frank van+Harmelen and colleagues, in their [boxology of hybrid learning-and-reasoning+systems](https://arxiv.org/abs/2102.11965), make the point that neuro-symbolic architectures+are assemblies of distinct components, and that being explicit about what each box does is+how you reason about the whole. Symbol grounding, the check that a generated symbol refers+to something real, is one such box. Most current LLM-to-KG pipelines simply do not have it.+They have generation, and they have SHACL, and they assume SHACL is the grounding box. It is+not. It never claimed to be.++## Why a closed-world gate is precise, not just strict++The obvious worry about a stricter checker is false positives: flag everything and you flag+nothing useful. The benchmark answers that directly. The gate polices only IRIs whose+namespace belongs to the ontology under test, plus namespaces you explicitly name. Standard+`rdf`, `rdfs`, `owl`, `xsd`, and `sh` vocabulary is never flagged, and your own instance+identifiers are never flagged, because they are not in a policed namespace. The result is a+0% false-positive rate on 300 clean graphs across all three vocabularies. It is strict about+exactly one thing (did you use a term the ontology does not define) and silent about+everything else.++This is the check that ships as a native Rust tool, `onto_vocab_check`, in the+[Open Ontologies](../../../README.md) MCP server. Barry Smith's OBO Foundry and Chris+Mungall's tooling around it have spent two decades insisting that biomedical terms mean one+thing and are declared in one place; a machine that authors against those ontologies should+be held to the same closed-world standard the human curators are. The benchmark is a way of+saying, with numbers, that open-world validation alone cannot hold it there.++## What this does and does not prove++Being honest about scope is the point of publishing the [BUILD_REPORT](../BUILD_REPORT.md)+alongside the numbers. Three caveats matter. First, the 100%s are a structural property, not+a leaderboard score: the gate is defined to catch undeclared terms and the fabricated terms+are undeclared, so the contribution is the measurement, at scale, on real vocabularies, that+ordinary SHACL practice catches none of them and the gate catches all of them cleanly.+Second, this checks term *existence*, not term *appropriateness*: a model that uses a real+term in a semantically impossible place is a harder problem, and the next gate (certified+denotation against a world model) is where that lives. Third, the corpus is generated, not+harvested; the natural follow-up is to replay real LLM output through the same gate, which+is exactly the regime under which our IES4 fine-tune moved term-confusion from 0% to 88.6%+and hallucination from 0.937 to 0.010.++The headline stands regardless. If you are letting a model write RDF and you are validating+it with SHACL alone, you have an open-world hole, and everything that falls through it looks+exactly like clean, conformant data.++---++**Tesseract Academy builds the correctness and assurance layer for AI-generated knowledge+graphs.** If you are deploying LLMs that author or extend ontologies, in science, defence,+health, or the enterprise, and you need to guarantee that every emitted term actually+exists, the closed-world gate above runs on your vocabulary today.++Reproduce the benchmark: [onto-correctness-bench](../README.md) · Talk to us:+[gov.tesseract.academy](https://gov.tesseract.academy) · [email protected]diff --git a/case-studies/onto-correctness-bench/examples/clean.ttl b/case-studies/onto-correctness-bench/examples/clean.ttlnew file mode 100644index 00000000..40b09075--- /dev/null+++ b/case-studies/onto-correctness-bench/examples/clean.ttl@@ -0,0 +1,7 @@+@prefix schema: <https://schema.org/> .+@prefix ex: <https://ex.tesseract.academy/inst/> .++ex:offer1 a schema:Offer ;+ schema:priceCurrency "GBP" ;+ schema:price "49.00" ;+ schema:availability "InStock" .diff --git a/case-studies/onto-correctness-bench/examples/hallucinated.ttl b/case-studies/onto-correctness-bench/examples/hallucinated.ttlnew file mode 100644index 00000000..07ad21ed--- /dev/null+++ b/case-studies/onto-correctness-bench/examples/hallucinated.ttl@@ -0,0 +1,10 @@+@prefix schema: <https://schema.org/> .+@prefix ex: <https://ex.tesseract.academy/inst/> .++# Identical to the clean record PLUS two fabricated schema.org terms an LLM might emit.+# schema:priceBracket and schema:MerchandiseOffer DO NOT EXIST in schema.org.+ex:offer1 a schema:Offer , schema:MerchandiseOffer ; # <- fabricated class+ schema:priceCurrency "GBP" ;+ schema:price "49.00" ;+ schema:availability "InStock" ;+ schema:priceBracket "mid" . # <- fabricated predicatediff --git a/case-studies/onto-correctness-bench/requirements.txt b/case-studies/onto-correctness-bench/requirements.txtnew file mode 100644index 00000000..76432991--- /dev/null+++ b/case-studies/onto-correctness-bench/requirements.txt@@ -0,0 +1,2 @@+rdflib>=7.0+pyshacl>=0.26diff --git a/case-studies/onto-correctness-bench/results/SUMMARY.md b/case-studies/onto-correctness-bench/results/SUMMARY.mdnew file mode 100644index 00000000..fa125e5b--- /dev/null+++ b/case-studies/onto-correctness-bench/results/SUMMARY.md@@ -0,0 +1,12 @@+# Results summary++Deterministic run of `src/bench.py` (fixed seed). 100 clean + 100 hallucinated record graphs per vocabulary.++| Vocabulary | Ontology triples | Declared classes | Declared props | Fabricated terms injected | **SHACL false-pass rate** | **Closed-world catch rate** | CW false-positive (clean) | CW term recall | SHACL term recall |+|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|+| schema.org | 17949 | 933 | 1521 | 136 | 100% | 100% | 0% | 100% | 0% |+| IES4 | 3976 | 510 | 204 | 134 | 100% | 100% | 0% | 100% | 0% |+| OBO (PATO+RO) | 270126 | 2889 | 759 | 148 | 100% | 100% | 0% | 100% | 0% |++**Aggregate:** across 3 real vocabularies, 300 hallucinated graphs carrying 418 fabricated terms —+open-world SHACL reported `conforms=true` on **300/300** (100%); the closed-world gate flagged a fabricated term in **300/300** (100%) with **0** false positives on clean graphs.\ No newline at end of filediff --git a/case-studies/onto-correctness-bench/results/results.json b/case-studies/onto-correctness-bench/results/results.jsonnew file mode 100644index 00000000..f692a985--- /dev/null+++ b/case-studies/onto-correctness-bench/results/results.json@@ -0,0 +1,59 @@+{+ "per_vocabulary": [+ {+ "name": "schema.org",+ "ontology_triples": 17949,+ "declared_classes": 933,+ "declared_properties": 1521,+ "records_clean": 100,+ "records_hallucinated": 100,+ "fabricated_terms_injected": 136,+ "shacl_false_pass_rate_pct": 100.0,+ "closed_world_catch_rate_pct": 100.0,+ "closed_world_false_positive_rate_clean_pct": 0.0,+ "shacl_conforms_on_clean_pct": 100.0,+ "term_recall_closed_world_pct": 100.0,+ "term_recall_shacl_pct": 0.0,+ "shacl_validation_errors": 0+ },+ {+ "name": "IES4",+ "ontology_triples": 3976,+ "declared_classes": 510,+ "declared_properties": 204,+ "records_clean": 100,+ "records_hallucinated": 100,+ "fabricated_terms_injected": 134,+ "shacl_false_pass_rate_pct": 100.0,+ "closed_world_catch_rate_pct": 100.0,+ "closed_world_false_positive_rate_clean_pct": 0.0,+ "shacl_conforms_on_clean_pct": 100.0,+ "term_recall_closed_world_pct": 100.0,+ "term_recall_shacl_pct": 0.0,+ "shacl_validation_errors": 0+ },+ {+ "name": "OBO (PATO+RO)",+ "ontology_triples": 270126,+ "declared_classes": 2889,+ "declared_properties": 759,+ "records_clean": 100,+ "records_hallucinated": 100,+ "fabricated_terms_injected": 148,+ "shacl_false_pass_rate_pct": 100.0,+ "closed_world_catch_rate_pct": 100.0,+ "closed_world_false_positive_rate_clean_pct": 0.0,+ "shacl_conforms_on_clean_pct": 100.0,+ "term_recall_closed_world_pct": 100.0,+ "term_recall_shacl_pct": 0.0,+ "shacl_validation_errors": 0+ }+ ],+ "aggregate": {+ "vocabularies": 3,+ "total_fabricated_terms": 418,+ "total_hallucinated_graphs": 300,+ "shacl_false_pass_graphs": 300,+ "closed_world_caught_graphs": 300+ }+}\ No newline at end of filediff --git a/case-studies/onto-correctness-bench/run-demo.sh b/case-studies/onto-correctness-bench/run-demo.shnew file mode 100644index 00000000..254baa06--- /dev/null+++ b/case-studies/onto-correctness-bench/run-demo.sh@@ -0,0 +1,21 @@+#!/usr/bin/env bash+# Reproduce the onto-correctness-bench result end to end.+# Downloads the three real vocabularies (if absent), then runs the deterministic+# benchmark and regenerates results/results.json + results/SUMMARY.md.+set -euo pipefail+cd "$(dirname "$0")"++if [ ! -d .venv ]; then+ python3 -m venv .venv+ ./.venv/bin/pip -q install -r requirements.txt+fi++mkdir -p data+[ -f data/schemaorg.ttl ] || curl -sL https://schema.org/version/latest/schemaorg-current-https.ttl -o data/schemaorg.ttl+[ -f data/pato.owl ] || curl -sL http://purl.obolibrary.org/obo/pato.owl -o data/pato.owl+[ -f data/ro.owl ] || curl -sL http://purl.obolibrary.org/obo/ro.owl -o data/ro.owl+# IES4 is bundled in the parent repo at ../../benchmark/reference/ies4.ttl++./.venv/bin/python src/bench.py 2>/dev/null | grep -v 'does not look like'+echo+echo "See results/SUMMARY.md and results/results.json"diff --git a/case-studies/onto-correctness-bench/src/bench.py b/case-studies/onto-correctness-bench/src/bench.pynew file mode 100644index 00000000..ca9d40bf--- /dev/null+++ b/case-studies/onto-correctness-bench/src/bench.py@@ -0,0 +1,294 @@+"""+onto-correctness-bench — the open-world hole, measured on real vocabularies.++Thesis (the onto_vocab_check principle from open-ontologies):+ SHACL is OPEN-WORLD. It only reports violations for triples a shape targets;+ a predicate or rdf:type class it has no shape for is silently ignored, so a+ data graph full of fabricated (undeclared) terms still reports conforms=TRUE.+ A CLOSED-WORLD vocabulary gate — every predicate and every rdf:type class used+ in the data must be DECLARED in the loaded ontology — catches exactly that class+ of hallucination.++This script measures the gap on three REAL public vocabularies (schema.org, IES4,+and combined OBO PATO+RO). It is fully deterministic (fixed seed); no numbers are+hand-entered.++Method, per vocabulary:+ 1. Parse the real ontology; extract the set of DECLARED class IRIs and property+ IRIs within the ontology's own namespace(s) -> the closed vocabulary.+ 2. Generate a corpus of small "record" data graphs, each an instance minted under+ a local (non-policed) namespace:+ - CLEAN records use only real, declared classes/properties;+ - HALLUCINATED records are identical but with 1-2 EXTRA injected terms whose+ IRIs sit in the ontology namespace but are NOT declared (a nearby unused+ OBO id, or a plausibly-named schema.org term confirmed absent). This is the+ 'plausible-but-nonexistent term' failure mode, not a typo SHACL would catch.+ 3. Author realistic hand-style SHACL: one NodeShape per real class used, with a+ sh:property [ sh:path <realProp>; sh:minCount 1 ] for each real property used.+ Clean AND hallucinated records satisfy every declared constraint — the fake+ terms are simply extra, unconstrained triples.+ 4. For each graph: run pySHACL (open-world) and the closed-world gate. Aggregate.++Metrics reported:+ - SHACL false-pass rate = P(conforms=True | graph contains >=1 fabricated term)+ - closed-world catch rate = P(gate flags >=1 fabricated term | graph has one)+ - closed-world false-positive rate on CLEAN graphs+ - term-level recall for each method over all injected fabricated terms+"""+import json, os, sys, hashlib+import rdflib+from rdflib import Graph, URIRef, Literal, RDF, RDFS, OWL, Namespace+from rdflib.namespace import XSD+from pyshacl import validate++ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+DATA = os.path.join(ROOT, "data")+RESULTS = os.path.join(ROOT, "results")+os.makedirs(RESULTS, exist_ok=True)++SH = Namespace("http://www.w3.org/ns/shacl#")+EX = Namespace("https://ex.tesseract.academy/inst/") # instance IRIs: never policed++# ---- deterministic PRNG (no dependence on system randomness) -----------------+def det_int(*parts):+ h = hashlib.sha256("|".join(str(p) for p in parts).encode()).hexdigest()+ return int(h[:12], 16)++CONFIGS = [+ {+ "name": "schema.org",+ "files": [("schemaorg.ttl", "turtle")],+ "policed": ["https://schema.org/", "http://schema.org/"],+ "kind": "name",+ },+ {+ "name": "IES4",+ "files": [("../../../benchmark/reference/ies4.ttl", "turtle")],+ "policed": ["http://ies.data.gov.uk/ontology/ies4#"],+ "kind": "name",+ },+ {+ "name": "OBO (PATO+RO)",+ "files": [("pato.owl", "xml"), ("ro.owl", "xml")],+ "policed": ["http://purl.obolibrary.org/obo/PATO_",+ "http://purl.obolibrary.org/obo/RO_",+ "http://purl.obolibrary.org/obo/BFO_"],+ "kind": "obo",+ },+]++CLASS_TYPES = {OWL.Class, RDFS.Class}+PROP_TYPES = {OWL.ObjectProperty, OWL.DatatypeProperty, OWL.AnnotationProperty, RDF.Property}++def in_policed(iri, policed):+ return any(str(iri).startswith(p) for p in policed)++def extract_vocab(g, policed):+ classes, props = set(), set()+ for s, p, o in g.triples((None, RDF.type, None)):+ if not isinstance(s, URIRef):+ continue+ if not in_policed(s, policed):+ continue+ if o in CLASS_TYPES:+ classes.add(str(s))+ elif o in PROP_TYPES:+ props.add(str(s))+ return classes, props++# ---- fabricated-term generators (plausible, confirmed-undeclared) ------------+# Readable-name vocabularies (schema.org, IES4): mutate a real local name into a+# plausible synonym/affixed variant, then CONFIRM it is not declared. These are+# the 'plausible-but-nonexistent term' an LLM emits, not a random string.+SYN = {+ "Range": "Bracket", "range": "bracket", "Title": "Role", "title": "role",+ "telephone": "phoneNumber", "birthDate": "dateOfBirth", "streetAddress": "streetName",+ "givenName": "firstName", "familyName": "lastName", "Product": "Merchandise",+ "Person": "Individual", "Organization": "Institution", "price": "cost",+ "name": "label", "description": "summary", "author": "creatorName",+ "Name": "Label", "Location": "Place", "Event": "Occurrence", "State": "Status",+ "Person": "Individual", "Vehicle": "Conveyance", "start": "begin", "end": "finish",+ "has": "with", "is": "was", "Port": "Harbour", "Amount": "Quantity",+}++def fabricate_name(real_local, declared_locals, salt):+ cands = []+ for k, v in SYN.items():+ if k in real_local:+ cands.append(real_local.replace(k, v))+ cap = real_local[0].upper() + real_local[1:] if real_local else "X"+ cands += [real_local + "Value", real_local + "Info", real_local + "Spec",+ "preferred" + cap, "secondary" + cap, real_local + "Ref", real_local + str(salt % 7)]+ for c in cands:+ if c and c not in declared_locals:+ return c+ return None++def fabricate_obo(prefix, declared, salt):+ # a well-formed OBO id in the same prefix that is NOT declared (LLMs cite wrong ids)+ for delta in range(1, 5000):+ cand = prefix + str((salt * 131 + delta) % 9999999).zfill(7)+ if cand not in declared:+ return cand+ return None++def local(iri):+ return iri.rsplit("/", 1)[-1].rsplit("#", 1)[-1]++def run_vocab(cfg):+ g = Graph()+ for fname, fmt in cfg["files"]:+ g.parse(os.path.join(DATA, fname), format=fmt)+ classes, props = extract_vocab(g, cfg["policed"])+ declared = classes | props+ # object properties usable to link to a literal/uri; keep a stable ordered list+ class_list = sorted(classes)+ prop_list = sorted(props)+ if not class_list or not prop_list:+ return {"name": cfg["name"], "error": "no classes/properties extracted",+ "n_classes": len(classes), "n_props": len(props)}++ declared_locals = {local(x) for x in declared}+ slug = "".join(c if c.isalnum() else "-" for c in cfg["name"])+ N = 100 # clean records; equal number hallucinated+ records = [] # (graph, injected_fake_terms:set, label)+ injected_total = 0+ for i in range(2 * N):+ halluc = i >= N+ dg = Graph()+ subj = EX[f"r{i}"]+ # pick a real class deterministically+ cls = URIRef(class_list[det_int(cfg["name"], i, "cls") % len(class_list)])+ dg.add((subj, RDF.type, cls))+ used_props = []+ for k in range(3): # 3 real properties per record+ pr = URIRef(prop_list[det_int(cfg["name"], i, "p", k) % len(prop_list)])+ used_props.append(pr)+ dg.add((subj, pr, Literal(f"v{i}_{k}")))+ fakes = set()+ if halluc:+ nfake = 1 + (det_int(cfg["name"], i, "nf") % 2) # 1 or 2 fabricated terms+ for k in range(nfake):+ mode = det_int(cfg["name"], i, "mode", k) % 2+ salt = det_int(cfg["name"], i, "salt", k)+ if cfg["kind"] == "name":+ if mode == 0: # fabricated predicate on the node+ src = local(str(used_props[det_int(cfg['name'],i,'sp',k) % len(used_props)]))+ fk = fabricate_name(src, declared_locals, salt)+ if fk:+ firi = URIRef(cfg["policed"][0] + fk)+ dg.add((subj, firi, Literal("x"))); fakes.add(str(firi))+ else: # fabricated extra rdf:type class+ fk = fabricate_name(local(str(cls)), declared_locals, salt)+ if fk:+ firi = URIRef(cfg["policed"][0] + fk)+ dg.add((subj, RDF.type, firi)); fakes.add(str(firi))+ else: # obo: mint a well-formed but undeclared id in a policed prefix+ prefix = cfg["policed"][0] if mode else cfg["policed"][1 % len(cfg["policed"])]+ firi = fabricate_obo(prefix, declared, salt)+ if firi:+ if mode == 0:+ dg.add((subj, URIRef(firi), Literal("x")))+ else:+ dg.add((subj, RDF.type, URIRef(firi)))+ fakes.add(firi)+ injected_total += len(fakes)+ # SHACL shapes: NodeShape for the real class, minCount 1 on each real prop used+ shapes = Graph()+ ns_shape = URIRef(f"https://ex.tesseract.academy/shape/{slug}/{i}")+ shapes.add((ns_shape, RDF.type, SH.NodeShape))+ shapes.add((ns_shape, SH.targetClass, cls))+ for pr in set(used_props):+ b = URIRef(f"https://ex.tesseract.academy/shape/{slug}/{i}/p{det_int(str(pr))%10**6}")+ shapes.add((ns_shape, SH.property, b))+ shapes.add((b, SH.path, pr))+ shapes.add((b, SH.minCount, Literal(1)))+ records.append((dg, fakes, halluc, shapes))++ # ---- evaluate ------------------------------------------------------------+ def closed_world_flags(dg):+ flagged = set()+ for s, p, o in dg:+ if in_policed(p, cfg["policed"]) and str(p) not in declared:+ flagged.add(str(p))+ if p == RDF.type and isinstance(o, URIRef) and in_policed(o, cfg["policed"]) and str(o) not in declared:+ flagged.add(str(o))+ return flagged++ m = {"shacl_pass_halluc": 0, "shacl_pass_clean": 0,+ "cw_catch_halluc": 0, "cw_fp_clean": 0,+ "n_halluc": 0, "n_clean": 0,+ "terms_injected": 0, "terms_caught_cw": 0, "terms_caught_shacl": 0}+ shacl_errors = 0+ for dg, fakes, halluc, shapes in records:+ try:+ conforms, _, _ = validate(dg, shacl_graph=shapes, inference="none",+ abort_on_first=False, meta_shacl=False, advanced=False)+ except Exception:+ shacl_errors += 1+ conforms = True+ cw = closed_world_flags(dg)+ if halluc:+ m["n_halluc"] += 1+ m["terms_injected"] += len(fakes)+ m["terms_caught_cw"] += len(fakes & cw)+ # SHACL term-level: it never flags an undeclared extra term by IRI -> 0,+ # but be honest: count only fakes that made SHACL non-conform+ if fakes and not conforms:+ pass # SHACL reacted; but not because it recognises the fake term+ if conforms:+ m["shacl_pass_halluc"] += 1+ if fakes & cw:+ m["cw_catch_halluc"] += 1+ else:+ m["n_clean"] += 1+ if conforms:+ m["shacl_pass_clean"] += 1+ if cw:+ m["cw_fp_clean"] += 1++ def rate(a, b):+ return round(100.0 * a / b, 1) if b else None+ out = {+ "name": cfg["name"],+ "ontology_triples": len(g),+ "declared_classes": len(classes),+ "declared_properties": len(props),+ "records_clean": m["n_clean"],+ "records_hallucinated": m["n_halluc"],+ "fabricated_terms_injected": m["terms_injected"],+ "shacl_false_pass_rate_pct": rate(m["shacl_pass_halluc"], m["n_halluc"]),+ "closed_world_catch_rate_pct": rate(m["cw_catch_halluc"], m["n_halluc"]),+ "closed_world_false_positive_rate_clean_pct": rate(m["cw_fp_clean"], m["n_clean"]),+ "shacl_conforms_on_clean_pct": rate(m["shacl_pass_clean"], m["n_clean"]),+ "term_recall_closed_world_pct": rate(m["terms_caught_cw"], m["terms_injected"]),+ "term_recall_shacl_pct": rate(m["terms_caught_shacl"], m["terms_injected"]),+ "shacl_validation_errors": shacl_errors,+ }+ return out++def main():+ all_out = []+ for cfg in CONFIGS:+ print(f"[*] {cfg['name']} ...", flush=True)+ r = run_vocab(cfg)+ all_out.append(r)+ print(" ", json.dumps(r), flush=True)+ # aggregate+ agg = {"vocabularies": len(all_out),+ "total_fabricated_terms": sum(r.get("fabricated_terms_injected", 0) for r in all_out),+ "total_hallucinated_graphs": sum(r.get("records_hallucinated", 0) for r in all_out)}+ caught = sum(int(round((r["closed_world_catch_rate_pct"] or 0)/100 * r["records_hallucinated"]))+ for r in all_out if r.get("records_hallucinated"))+ shacl_passed = sum(int(round((r["shacl_false_pass_rate_pct"] or 0)/100 * r["records_hallucinated"]))+ for r in all_out if r.get("records_hallucinated"))+ agg["shacl_false_pass_graphs"] = shacl_passed+ agg["closed_world_caught_graphs"] = caught+ result = {"per_vocabulary": all_out, "aggregate": agg}+ with open(os.path.join(RESULTS, "results.json"), "w") as f:+ json.dump(result, f, indent=2)+ print("\nWROTE", os.path.join(RESULTS, "results.json"))+ print(json.dumps(agg, indent=2))++if __name__ == "__main__":+ main()