Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

fix(demo): make demo-verify runnable on a fresh clone with no model

FABIOTESS10bdb994578c
diff --git a/Makefile b/Makefile
index 610d35d8..62bd4fdd 100644
--- a/Makefile
+++ b/Makefile
@@ -1,4 +1,4 @@
-.PHONY: build test lint audit bench bench-pizza bench-ontoaxiom bench-mushroom bench-vision bench-reasoner bench-oaei docker clean demo demo-verify
+.PHONY: build test lint audit bench bench-pizza bench-ontoaxiom bench-mushroom bench-vision bench-reasoner bench-oaei docker clean demo demo-verify demo-verify-pipeline
# ─── Development ─────────────────────────────────────────────────────────────
@@ -90,11 +90,31 @@ demo:
python3 demo/bundle_fixtures.py --in demo/precomputed --out demo/precomputed/bundle.json
cd demo/precomputed && shasum -a 256 *.json | sort > MANIFEST.sha256
+# demo-verify is the command a sceptic runs: it must work on a fresh clone,
+# with no model server, no engine process, and no prior `make demo` run.
+# demo/verify.py needs demo/derived/_ontology.ttl (written by `make demo`,
+# which calls a local model) and a running engine over MCP, neither of which
+# a fresh clone has, so it is NOT part of this target -- see
+# demo-verify-pipeline below. Everything demo-verify does runs the
+# conformance script and its tests first (dcat_conformance.py has no network
+# access and no model call of its own), so the published numbers in
+# demo/precomputed/findings.json are re-derived, not merely checked as bytes
+# that have not moved; the checksum check after it additionally guards every
+# other precomputed artifact against silent drift.
demo-verify:
- python3 demo/verify.py
- cd demo/precomputed && shasum -a 256 -c MANIFEST.sha256
python3 demo/dcat_conformance.py
python3 -m pytest demo/tests/test_dcat_conformance.py -q
+ cd demo/precomputed && shasum -a 256 -c MANIFEST.sha256
+
+# demo-verify-pipeline re-checks the document-to-ontology pipeline itself
+# (parse, stats, lint, enforce, vocab, reason) against whatever is currently
+# under demo/derived/. Unlike demo-verify, this needs state that only `make
+# demo` produces (demo/derived/_ontology.ttl) and a running engine reachable
+# over MCP -- ontology_from_docs.py's call_model() also expects a local model
+# server at ONTO_LLM_BASE_URL (default http://localhost:8081/v1). It is a
+# separate target precisely so a fresh clone can still run demo-verify.
+demo-verify-pipeline:
+ python3 demo/verify.py
# ─── Cleanup ─────────────────────────────────────────────────────────────────
diff --git a/demo/README.md b/demo/README.md
index 0ad86683..c233346a 100644
--- a/demo/README.md
+++ b/demo/README.md
@@ -90,8 +90,26 @@ make demo
`make demo` runs the pipeline end to end against the corpus in
`demo/corpus/dcat-us`: read, tokenise, derive, merge, reconcile, populate,
-verify, and scan for contradictions. Each stage can also be run on its own,
-for example:
+verify. It does not scan for contradictions as part of this target;
+`contradiction_scan.py` is a separate script, targets a different bundle
+layout than this corpus uses, and is not wired into `make demo` (see the
+Makefile's own comment on the `demo` target). What `make demo` does call --
+`ontology_from_docs.py`'s own Stage 6 and `verify.py`'s Check 6 -- both search
+for disjointness contradictions and find none on this corpus by construction;
+see "The conformance finding" below for what this demonstration actually
+reports instead.
+
+`make demo` calls a local model server for the DERIVE, POPULATE and REFINE
+stages: `demo/ontology_from_docs.py` talks to `ONTO_LLM_BASE_URL` (default
+`http://localhost:8081/v1`) and to the engine binary over MCP at `ONTO_ENGINE`
+(default `http://127.0.0.1:8137`). Neither is started for you; without both
+reachable, `make demo` fails partway through STAGE 3/4/5. `make demo-verify`
+needs neither: it re-derives the published conformance figures from
+committed inputs only, so it is the command to run on a fresh clone.
+`make demo-verify-pipeline` re-checks the pipeline's own parse/stats/lint/
+enforce/vocab/reason state under `demo/derived/`, and does need both a prior
+`make demo` run and the model server above; it is a separate target for
+exactly that reason. Each stage can also be run on its own, for example:
```bash
python3 demo/ontology_from_docs.py --corpus demo/corpus/dcat-us
@@ -113,13 +131,14 @@ pipeline's contradiction scanner detects provenance-split typing conflicts
disagreement is not that shape: it is a README's conformance claim
contradicted by the artifacts published alongside it. That finding is
established by a validator, not the model, and lives in
-`demo/dcat_conformance.py`. `make demo-verify` runs it (and its tests) as
-part of the verify target, so the command a sceptic runs re-derives the
-figures below rather than only checking that committed bytes have not moved.
+`demo/dcat_conformance.py`. `make demo-verify` runs it (and its tests) FIRST,
+before checking any committed bytes, so the command a sceptic runs on a fresh
+clone re-derives the figures below rather than only checking that they have
+not moved -- and it needs no model and no prior state to do so.
The script reads only files committed under `demo/corpus/dcat-us/`: the full
GSA/dcat-us `jsonschema/definitions/` and `jsonschema/examples/` tree
-(`demo/corpus/dcat-us/jsonschema/`, vendored separately from the seven
+(`demo/corpus/dcat-us/jsonschema/`, vendored separately from the eight
documents the pipeline above reads), `recovered-shapes.ttl`, and
`recovered-context.jsonld`. Pull request 120 deleted two files, not one:
the SHACL shapes and the profile's only published JSON-LD `@context`
@@ -165,3 +184,15 @@ which alone moves the real-context count by several more violations if
corrected. `demo/tests/test_dcat_conformance.py::
test_shacl_violation_counts_disagree_across_methods` pins the disagreement,
not a winner.
+
+An earlier, already-public figure for the same deleted-shapes-plus-binding
+measurement is 287 (`case-studies/dcat-us-binding/README.md`, a separate,
+standalone repository's pySHACL run against its own generated binding). That
+binding's triple/predicate/DCAT/empty-file counts match this script's
+`observed` binding exactly (1,510 / 123 / 228 / 10), but its violation count
+does not: 287 there against 272 here, on the same shapes and the same bound
+corpus. The gap is investigated, not adjusted: it is not severity filtering
+and the two runs reach the identical 228 focus nodes, so the cause is still
+open. Combined with this script's own three internal methods (178 / 272 /
+147), that is four legitimate-looking counts for what sounds like one
+question, and none of the four should be quoted as settled.
diff --git a/demo/verify.py b/demo/verify.py
index dbeb9bd3..250416eb 100644
--- a/demo/verify.py
+++ b/demo/verify.py
@@ -277,12 +277,12 @@ def main() -> int:
f"{len(contradictions)} contradictions across documents{OFF}")
print(f"{DIM}CHECK 6 only re-derives disjointness contradictions in the pipeline ontology "
f"above; it is not wired to this script's exit code, and finds none here by "
- f"construction (see docs/superpowers/plans/2026-08-24-studio-public-port.md, "
- f"'PIVOT, 25 August 2026'). The findings this demonstration actually reports live "
- f"in demo/precomputed/findings.json, computed independently by "
- f"demo/dcat_conformance.py and re-derived every time 'make demo-verify' runs -- "
- f"see its own output above, or run 'python3 -m pytest demo/tests/"
- f"test_dcat_conformance.py' directly.{OFF}\n")
+ f"construction (see .superpowers/sdd/progress.md, 'PIVOT, 25 August 2026'). The "
+ f"findings this demonstration actually reports live in "
+ f"demo/precomputed/findings.json, computed independently by "
+ f"demo/dcat_conformance.py. This script (verify.py) does not run that check --"
+ f"run 'make demo-verify' (no model, no engine, works on a fresh clone) or "
+ f"'python3 -m pytest demo/tests/test_dcat_conformance.py' directly to see it.{OFF}\n")
return failures