Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
SHACL differential exited 0 when it measured nothing and its oracle was unpinned; exit 2 NOT MEASURED, plant controls, pin pyshacl, arbitrate on W3C (#320)
FABIOTESS2e8025ec6266
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex b49affa5..a6c22294 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -252,14 +252,32 @@ jobs:# test that returns early: it reports assurance that did not happen.## The oracle exits 1 on a FALSE_CLEAN, meaning we said conforms where pyshacl- # said not, which is the one failure this validator must never have. That is- # what gates here. Disagreements in the other direction and undetermined- # verdicts are printed and uploaded, not failed on, because they are a work- # list rather than a defect.+ # said not, which is the one failure this validator must never have, and on+ # a planted control it misclassifies. It exits 2, NOT MEASURED, when there is+ # no oracle, the wrong oracle, nothing discovered, fewer pairs run than+ # discovered, or no pair compared. Any non-zero exit fails this job.+ # Disagreements in the other direction and undetermined verdicts are printed+ # and uploaded, not failed on, because they are a work list rather than a+ # defect.+ #+ # Before #320 the script exited 0 on everything but a FALSE_CLEAN, and this+ # job was green on a run that compared nothing: measured 24 Oct 2026, the+ # case-studies corpus yields 4 pairs and all 4 are UNDETERMINED. The W3C suite+ # is therefore run in the same invocation, where each disagreement is+ # arbitrated against the Working Group's expected report (73 of 117 pairs+ # compared on that date).+ #+ # pyshacl and rdflib are pinned, and the script refuses (exit 2) any other+ # pyshacl than PYSHACL_VERSION, because an unpinned oracle makes a pyshacl+ # release look like a change in this engine. The versions and the sha256 of+ # every input are in the uploaded JSON.shacl-differential:timeout-minutes: 30name: SHACL differential vs pyshaclruns-on: ubuntu-latest+ env:+ PYSHACL_VERSION: "0.40.1"+ RDFLIB_VERSION: "7.6.0"steps:- uses: actions/checkout@v4- uses: dtolnay/rust-toolchain@stable@@ -276,14 +294,16 @@ jobs:- uses: actions/setup-python@v5with:python-version: '3.11'- - name: Install pyshacl- run: pip install pyshacl+ - name: Install the pinned oracle+ run: pip install "pyshacl==${PYSHACL_VERSION}" "rdflib==${RDFLIB_VERSION}"- name: Build the binary the oracle drivesrun: cargo build --release- - name: Run the differential over the case-study corpus+ - name: Run the differential over the case-study corpus and the W3C suiterun: |python3 tools/shacl_differential.py \--discover case-studies \+ --w3c tests/w3c-shacl \+ --pyshacl-version "${PYSHACL_VERSION}" \--json shacl-differential.json- name: Upload the reportif: always()@@ -417,6 +437,7 @@ jobs:tests/test_horn_rule_table.py \tests/test_shacl.py \tests/test_shacl_vacuous.py \+ tests/test_shacl_differential_harness.py \tests/test_align.py \tests/test_vocab_check.pydiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8d3d900c..72d7c636 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -23,6 +23,23 @@ All notable changes to Open Ontologies are documented here.the engine did not read `sh:severity`, had been false since #206 and is gone.A planted control mutates one PASS case's expected report in each of the sixfields and requires each copy to be rejected by name.+- **`tools/shacl_differential.py` can no longer go green on nothing**+ (reported by @noahgift in #320). It exited 0 on everything but a FALSE_CLEAN,+ including no cases, no pyshacl and no comparisons, and the CI job's+ case-studies corpus yields 4 pairs that all come back UNDETERMINED. It now+ exits 2 (NOT MEASURED) for a missing or unpinned oracle, a missing engine,+ empty discovery, fewer pairs run than discovered, or no pair compared; plants+ a must-agree control and four mutated copies of pyshacl's report on every run+ and exits 1 if any is misclassified; keys results on (focus, path, severity,+ source shape, component) instead of (focus, path); records pyshacl, rdflib,+ the engine binary's sha256 and every input's sha256; and with `--w3c`+ arbitrates each disagreement against the W3C expected report. CI pins+ pyshacl 0.40.1 and rdflib 7.6.0 and runs the corpus and the W3C suite+ together: 117 pairs, 73 compared, 0 blocking, with 12 W3C-confirmed+ FALSE_CLEANs reported as KNOWN because the conformance ratchet already+ holds each as FAIL. `tools/fol_differential.py`+ (every goal undetermined) and `tools/horn_differential.py` (every case not+ run) had the same exit-0-on-nothing and now exit 2.## [2.1.0] - 2026-10-23diff --git a/docs/ci-gates.md b/docs/ci-gates.mdindex 173256fa..c5359b21 100644--- a/docs/ci-gates.md+++ b/docs/ci-gates.md@@ -8,8 +8,8 @@ over the same documents — ran its preflight, found neither of the two tools itneeds, skipped every test in the file, and exited 0.This page is the answer to "does a green tick mean this ran?", so that nobody has-to work it out again from six workflow files and the thirty-four test files that-can skip — twenty-five Rust, nine Python. It is a statement about CI+to work it out again from six workflow files and the test files that can skip —+twenty-five Rust, fourteen Python. It is a statement about CIconfiguration, which changes. Its Rust half is now checked by a test rather thanby hand (`tests/ci_gate_coverage_test.rs`), because the by-hand version wentstale exactly where it mattered; the Python half still carries a date at the@@ -40,7 +40,17 @@ Two tool scripts mirror the variable under their own names because they are CLIsrather than test suites, and say so where they define it:`HORN_DIFF_REQUIRE_TOOLS=1` in `tools/horn_differential.py` and`FOL_DIFF_REQUIRE_ATP=1` in `tools/fol_differential.py`. Both exit 2 rather than-panicking.+panicking. Both also exit 2, flag or no flag, when the tools ARE present and the+run compared nothing: every case NOT_RUN in the first, every goal UNDETERMINED+or none asked in the second (#320).++`tools/shacl_differential.py` has no opt-out flag, because its CI job always+provides the oracle. It exits 2, NOT MEASURED, when pyshacl is missing or is not+the pinned version, when the engine binary is missing, when discovery finds+nothing, when fewer pairs ran than were discovered, and when no pair yielded a+comparison; it exits 1 when one of its planted controls is misclassified. Before+#320 all of those exited 0, and the `shacl-differential` job was green over a+corpus whose 4 pairs were all UNDETERMINED.## The table@@ -118,6 +128,11 @@ path to make strict. `module_extract_test.rs` reads| `test_horn_rule_table.py` | `tests/fixtures/horn/` + git | `python` job | **strict** || `test_shacl.py` | `pyshacl` (the `shacl` extra) | `python` job installs it | **strict** || `test_shacl_vacuous.py` | `pyshacl` | `python` job | **strict** |+| `test_shacl_differential_harness.py` | `pyshacl` | `python` job | **strict** |+| `test_shacl_falsy_terms.py` | `pyshacl` | `python` job installs it | **runs, not strict**: not in the strict leg, so a skip would pass |+| `test_shacl_targeted_shapes.py` | `pyshacl` | `python` job installs it | **runs, not strict** |+| `test_align_failed_insertion.py` | `hnswlib` | `python` job installs it | **runs, not strict** |+| `test_horn_checker_relative.py` | a symlink to an executable in a temp directory | any POSIX runner | **runs, not strict** || `test_align.py` | `hnswlib` (the `align` extra) | `python` job installs it | **strict** || `test_vocab_check.py` | `pyoxigraph` (a base dependency) | always present | **strict** || `test_dataframe.py` | `fenic` | **nothing** — `fenic` is in no extra and no dependency list | **skips** |@@ -154,7 +169,15 @@ The other seven files under `python/tests/` have no skip path.is absent or the tree is not a checkout, which on a GitHub runner after`actions/checkout` cannot happen. Measured on the current tree: the eightstrict Python files are 67 passed, 1 skipped, and the one skip is the- `oo_opt_in` twelve-minute differential.+ `oo_opt_in` twelve-minute differential. The ninth,+ `test_shacl_differential_harness.py`, adds 9 passed (measured 24 October+ 2026 against pyshacl 0.40.1).+- **Four Python files can skip and are in no strict leg**:+ `test_shacl_falsy_terms.py`, `test_shacl_targeted_shapes.py`,+ `test_align_failed_insertion.py` and `test_horn_checker_relative.py`. The+ `python` job provides what each needs, so they run there today, but a skip in+ any of them would pass. They were missing from this page until 24 October+ 2026, when the count below was re-taken and came to fourteen rather than nine.- **The `features/breadth` job runs no tests** (`test: false`); it is`cargo check` plus clippy across all features. That is deliberate and is not askip, but it does mean an all-features build is type-checked and never run.@@ -207,13 +230,13 @@ property is that CI cannot skip in silence, not that a laptop cannot skip at allThe Python half is still by hand:```bash-# every Python file that can skip: 11 hits. conftest.py is the mechanism and+# every Python file that can skip: 16 hits. conftest.py is the mechanism and# not a test, and test_conftest_collection_skip.py tests it: it writes a-# skipping module into a temp directory and cannot skip itself. So 9, and the-# table has 9 rows+# skipping module into a temp directory and cannot skip itself. So 14, and the+# table has 14 rowsgrep -rln 'pytest.skip\|importorskip\|skipif' python/tests/*.py# every leg CI makes strictgrep -rn 'OO_REQUIRE_FIXTURES=1' .github/workflows/```-Last checked against `.github/workflows/` on 15 September 2026.+Last checked against `.github/workflows/` on 24 October 2026.diff --git a/python/tests/test_shacl_differential_harness.py b/python/tests/test_shacl_differential_harness.pynew file mode 100644index 00000000..a7595e8c--- /dev/null+++ b/python/tests/test_shacl_differential_harness.py@@ -0,0 +1,227 @@+"""`tools/shacl_differential.py` cannot go green on nothing, and its planted+controls bite (#320).++Before #320 the script exited 0 whenever it found no FALSE_CLEAN, which it also+did when it found no cases, when pyshacl was not installed (every pair came back+ERROR from the import inside the run), and when every pair was capped out. The+CI job "SHACL differential vs pyshacl" was therefore green on a run that+compared nothing. Each test below fails against that version of the script.++The engine here is a stand-in: a script that answers `load` and `shacl` the way+the release binary does, by running pyshacl and printing the result in the+engine's JSON shape. A faithful stand-in is the right fixture for testing the+HARNESS, because every disagreement it shows is then the harness's own doing,+and it keeps this file from needing a release build. The real binary is what+the CI job drives.+"""++import importlib.util+import json+import os+import pathlib+import subprocess+import sys+import textwrap++import pytest++pytest.importorskip("pyshacl")++REPO = pathlib.Path(__file__).resolve().parents[2]+TOOL = REPO / "tools" / "shacl_differential.py"++FAKE_ENGINE = textwrap.dedent(+ '''\+ import json, pathlib, sys+ from pyshacl import validate+ from rdflib import BNode, Graph, Namespace, URIRef++ SH = Namespace("http://www.w3.org/ns/shacl#")+ args = sys.argv[1:]+ store = pathlib.Path(args[args.index("--data-dir") + 1])+ cmd, target = args[args.index("--data-dir") + 2], args[args.index("--data-dir") + 3]+ if cmd == "load":+ with open(store / "loaded.txt", "a") as fh:+ fh.write(target + "\\n")+ sys.exit(0)+ data = Graph()+ for f in (store / "loaded.txt").read_text().split():+ data.parse(f)+ shapes = Graph().parse(target)+ conforms, results, _ = validate(data, shacl_graph=shapes)++ def term(t):+ if t is None:+ return None+ if isinstance(t, BNode):+ return "_:" + str(t)+ return str(t) if isinstance(t, URIRef) else t.n3()++ violations = []+ for r in results.subjects(SH.resultSeverity, None):+ shape = results.value(r, SH.sourceShape)+ parent = next(iter(shapes.subjects(SH.property, shape)), None)+ violations.append({+ "focus_node": term(results.value(r, SH.focusNode)),+ "result_path": term(results.value(r, SH.resultPath)),+ "severity": str(results.value(r, SH.resultSeverity)).rsplit("#", 1)[-1],+ "source_shape": term(shape),+ "node_shape": term(parent) if parent is not None else term(shape),+ "source_constraint_component": term(results.value(r, SH.sourceConstraintComponent)),+ })+ print(json.dumps({"conforms": conforms, "violations": violations}))+ '''+)++DATA = """\+@prefix ex: <http://example.org/t#> .+ex:a a ex:Thing .+"""+SHAPES = """\+@prefix ex: <http://example.org/t#> .+@prefix sh: <http://www.w3.org/ns/shacl#> .+ex:S a sh:NodeShape ; sh:targetClass ex:Thing ; sh:property [ sh:path ex:p ; sh:minCount 1 ] .+"""+++def engine(tmp_path):+ exe = tmp_path / "fake-engine"+ exe.write_text(f"#!{sys.executable}\n" + FAKE_ENGINE)+ exe.chmod(0o755)+ return exe+++def corpus(tmp_path):+ root = tmp_path / "corpus"+ (root / "one" / "shacl").mkdir(parents=True)+ (root / "one" / "data").mkdir()+ (root / "one" / "shacl" / "s.ttl").write_text(SHAPES)+ (root / "one" / "data" / "d.ttl").write_text(DATA)+ return root+++def run(args, engine, extra_env=None):+ env = dict(os.environ, OPEN_ONTOLOGIES_BIN=str(engine), **(extra_env or {}))+ return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, text=True, env=env)+++def load_tool(monkeypatch, engine):+ monkeypatch.setenv("OPEN_ONTOLOGIES_BIN", str(engine))+ spec = importlib.util.spec_from_file_location("shacl_differential_under_test", TOOL)+ mod = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(mod)+ return mod+++def test_a_faithful_engine_is_green_and_the_receipt_is_written(engine, corpus, tmp_path):+ out = tmp_path / "r.json"+ p = run(["--discover", str(corpus), "--json", str(out)], engine)+ assert p.returncode == 0, p.stdout + p.stderr+ r = json.loads(out.read_text())+ assert r["status"] == "GREEN"+ assert r["counts"] == {"AGREE": 1}+ import pyshacl++ assert r["receipt"]["pyshacl"] == pyshacl.__version__+ assert r["receipt"]["engine"]["sha256"]+ assert sorted(r["receipt"]["inputs"]) == sorted(+ str(p) for p in [corpus / "one" / "shacl" / "s.ttl", corpus / "one" / "data" / "d.ttl"]+ )+ assert all(len(h) == 64 for h in r["receipt"]["inputs"].values())+ assert [c["ok"] for c in r["controls"]] == [True] * 5, r["controls"]+++def test_empty_discovery_is_not_measured(engine, tmp_path):+ empty = tmp_path / "empty"+ empty.mkdir()+ p = run(["--discover", str(empty)], engine)+ assert p.returncode == 2, p.stdout + p.stderr+ assert "NOT MEASURED" in p.stdout+++def test_missing_pyshacl_is_not_measured(engine, corpus, tmp_path):+ # A package named pyshacl that refuses to import, first on the path: the+ # interpreter sees pyshacl as absent, exactly as on a runner without it.+ stub = tmp_path / "stub" / "pyshacl"+ stub.mkdir(parents=True)+ (stub / "__init__.py").write_text("raise ImportError('pyshacl withheld by the test')\n")+ p = run(["--discover", str(corpus)], engine, {"PYTHONPATH": str(stub.parent)})+ assert p.returncode == 2, p.stdout + p.stderr+ assert "NOT MEASURED" in p.stdout and "pyshacl" in p.stdout+++def test_a_pin_mismatch_is_not_measured(engine, corpus):+ p = run(["--discover", str(corpus), "--pyshacl-version", "0.0.0"], engine)+ assert p.returncode == 2, p.stdout + p.stderr+++def test_fewer_run_than_discovered_is_not_measured(engine, corpus):+ p = run(["--discover", str(corpus)], engine, {"SHACL_DIFF_MAX_BYTES": "1"})+ assert p.returncode == 2, p.stdout + p.stderr+ assert "were not run" in p.stdout+++def test_a_comparator_blind_to_results_fails_on_its_controls(engine, corpus, monkeypatch, capsys):+ """A classifier that compares the verdict and nothing else would call every+ corpus pair here AGREE. The planted controls are what catch it."""+ mod = load_tool(monkeypatch, engine)+ real = mod.classify++ def verdict_only(ours, theirs):+ verdict, detail = real(ours, theirs)+ return ("AGREE", detail) if verdict == "PARTIAL" else (verdict, detail)++ monkeypatch.setattr(mod, "classify", verdict_only)+ assert mod.main(["--discover", str(corpus)]) == 1+ out = capsys.readouterr().out+ for name in ("one-result-dropped", "severity-changed", "source-shape-changed", "component-changed"):+ assert f"control `{name}` came out AGREE" in out, out+++def test_a_comparator_that_never_agrees_fails_on_its_controls(engine, corpus, monkeypatch, capsys):+ mod = load_tool(monkeypatch, engine)+ monkeypatch.setattr(mod, "classify", lambda ours, theirs: ("PARTIAL", "always"))+ assert mod.main(["--discover", str(corpus)]) == 1+ assert "control `must-agree` came out PARTIAL" in capsys.readouterr().out+++def test_w3c_arbitration_names_the_wrong_side(engine, monkeypatch):+ """`arbitrate` is what turns a disagreement on a W3C test into a verdict+ about one engine. Driven directly, on hand-built reports, so each branch is+ shown to be reachable and to say what it claims."""+ mod = load_tool(monkeypatch, engine)+ right = {"conforms": False, "keys": {("f", None, "sev", "shape", "comp")}}+ wrong = {"conforms": True, "keys": set()}++ # Both engines agree and the Working Group does not: the blind spot of a+ # two-engine differential, so it blocks.+ assert mod.arbitrate("AGREE", "", wrong, wrong, right, None)[0] == "AGREE_AGAINST_W3C"+ assert "AGREE_AGAINST_W3C" in mod.BLOCKING+ # We match the expected report and pyshacl does not.+ assert mod.arbitrate("FALSE_ALARM", "", right, wrong, right, None)[0] == "ORACLE_WRONG"+ # pyshacl is right and we are clean where we should not be: blocking unless+ # the conformance ratchet already holds this test as failing.+ assert mod.arbitrate("FALSE_CLEAN", "", wrong, right, right, "PASS")[0] == "FALSE_CLEAN"+ assert mod.arbitrate("FALSE_CLEAN", "", wrong, right, right, None)[0] == "FALSE_CLEAN"+ assert mod.arbitrate("FALSE_CLEAN", "", wrong, right, right, "FAIL")[0] == "KNOWN_FALSE_CLEAN"+ assert "KNOWN_FALSE_CLEAN" not in mod.BLOCKING+++def test_horn_differential_is_not_measured_when_every_case_is_capped(tmp_path, monkeypatch, capsys):+ """The same hole in tools/horn_differential.py: its zero-cases check ran+ before the size cap, so a corpus capped out entirely compared nothing and+ exited 0. The preflight is stubbed because the hole is in what happens+ after it."""+ (tmp_path / "a.ttl").write_text(DATA)+ spec = importlib.util.spec_from_file_location(+ "horn_differential_under_test", REPO / "tools" / "horn_differential.py"+ )+ mod = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(mod)+ monkeypatch.setattr(mod, "preflight", lambda: [])+ argv = ["horn_differential.py", "--root", str(tmp_path), "--max-bytes", "0", "--no-coverage"]+ monkeypatch.setattr(sys, "argv", argv)+ assert mod.main() == 2+ assert "compared nothing" in capsys.readouterr().outdiff --git a/tests/w3c_shacl_conformance_test.rs b/tests/w3c_shacl_conformance_test.rsindex 09b8ac60..cc3d3fef 100644--- a/tests/w3c_shacl_conformance_test.rs+++ b/tests/w3c_shacl_conformance_test.rs@@ -107,6 +107,10 @@//!//! That figure is for the gated column as it stood before #319, with every//! blank source shape collapsed to `_:`; the script still applies that lens.+//! Per-case arbitration of this engine against pyshacl on the W3C expected+//! reports, with severity and the re-keyed source shape in the key, is+//! `tools/shacl_differential.py --w3c tests/w3c-shacl`, which the+//! `shacl-differential` CI job runs.//!//! # Four buckets, because this engine has three answers//!diff --git a/tools/fol_differential.py b/tools/fol_differential.pyindex 2adbf878..180ebb36 100644--- a/tools/fol_differential.py+++ b/tools/fol_differential.py@@ -642,7 +642,17 @@ def main():"counts": counts, "proof_counts": proof_counts}, fh, indent=1)# A conclusion the engine claims and the ATP refutes is the one result that- # must fail a pipeline. An undetermined run is not a pass and not a failure.+ # must fail a pipeline. An undetermined row is not a pass and not a failure,+ # but a run in which EVERY goal is undetermined, or no goal was asked, put+ # nothing to the prover that came back with an answer, and used to exit 0+ # exactly like a run that agreed on everything (#320). It exits 2, as a+ # missing prover does under FOL_DIFF_REQUIRE_ATP=1.+ if all(r["verdict"] == "UNDETERMINED" for r in rows):+ print(+ f"\nNOT MEASURED: {len(rows)} goal(s) asked and none answered. This is not "+ "agreement; it exits 2."+ )+ return 2rejected = sum(n for k, n in proof_counts.items() if k.startswith("PROOF_REJECTED"))if rejected:print(diff --git a/tools/horn_differential.py b/tools/horn_differential.pyindex a1069f20..d4dcd0ea 100644--- a/tools/horn_differential.py+++ b/tools/horn_differential.py@@ -855,6 +855,13 @@ def main() -> int:else:print(f"{row['verdict']:<20} {row['name']} ({row['detail']})", flush=True)shutil.rmtree(specdir, ignore_errors=True)+ # The empty-corpus check above runs before the size cap and the workers, so+ # a corpus whose every case was capped, timed out or crashed reached the+ # exit at the bottom with zero rows and returned 0 (#320). Nothing compared+ # is not agreement.+ if not rows:+ print(f"all {len(cases)} case(s) were NOT_RUN: the differential compared nothing", flush=True)+ return 2counts = {}for r in rows:diff --git a/tools/shacl_differential.py b/tools/shacl_differential.pyindex b13594cd..12c38451 100644--- a/tools/shacl_differential.py+++ b/tools/shacl_differential.py@@ -2,33 +2,86 @@"""Differential oracle: open-ontologies SHACL against pyshacl, over real shapes.A validator is trusted on evidence, not on its own test suite. This runs both-engines over the same (data, shapes) pairs and compares three things in order of+engines over the same (data, shapes) pairs and compares them in order ofseverity:1. FALSE CLEAN - we say conforms, pyshacl says not. The one failure mode thevalidator must not have. Any of these is a stop-the-line bug.2. FALSE ALARM - we say not conforms, pyshacl says conforms. Noise that makesa gate untrustworthy in the other direction.- 3. UNDETERMINED - we return null because something was skipped. Honest, but a+ 3. PARTIAL - the verdicts agree and the result sets do not.+ 4. UNDETERMINED - we return null because something was skipped. Honest, but agap: the shapes graph got no verdict. These are the work list.Agreement on the verdict is not enough, so where both engines give a verdict the-violation sets are compared as (focus_node, path) pairs.+result sets are compared on (focus node, result path, severity, source shape,+source constraint component). A blank-node source shape is keyed by the IRI of+the node shape holding it under sh:property plus its sh:path, the same key+tests/w3c_shacl_conformance_test.rs uses, because a blank label is not portable+between two parses, let alone two engines. sh:value is NOT in the key: on the+default path the engine does not emit it yet (#318), so including it would turn+every case with a violation into PARTIAL and hide every other difference behind+that one.++Exit codes, after the CRUX harness rule (aprender, contracts/crux-shacl-jena-v1):++ 0 GREEN every planted control classified as planted, and no FALSE_CLEAN+ 1 RED a FALSE_CLEAN, an AGREE_AGAINST_W3C, or a misclassified control+ 2 NOT MEASURED no pyshacl, the wrong pyshacl, no engine, nothing discovered,+ fewer pairs run than discovered, or no pair yielding a+ comparison. Never green: before #320 every one of these exited+ 0, so a run that compared nothing looked like a run that found+ nothing.++Planted controls run on EVERY invocation, before the corpus, so a run is never+trusted on a comparator nobody has seen disagree. One must-agree pair goes+through both real engines and must come out AGREE. Copies of pyshacl's report on+it are then mutated (one result dropped; one result's severity, source shape and+component changed in turn) and each copy must come out PARTIAL or FALSE_CLEAN.++W3C arbitration. With --w3c, the cases are the vendored W3C SHACL suite and each+carries the Working Group's expected report. When the two engines disagree the+expected report says which one is wrong: we = W3C != pyshacl is ORACLE_WRONG+(GREEN, recorded), we = pyshacl != W3C is AGREE_AGAINST_W3C (RED, because two+engines agreeing on a wrong answer is the case a two-engine differential is+blind to). A FALSE_CLEAN the W3C report confirms, on a test the conformance+ratchet already records as failing, is KNOWN_FALSE_CLEAN and does not block+(see `arbitrate`). Entries expecting sht:Failure carry no report and are listed+as EXCLUDED. The Working Group's own `sh:value` is dropped from the key for the+same reason as above.++Why the CI job passes --w3c as well as --discover case-studies: measured on 24+Oct 2026, discovery over case-studies/ finds 4 pairs and all 4 come back+UNDETERMINED (two skip sh:xone / sh:not / sh:or, two match no focus node), so+on its own the corpus compares nothing, and before #320 that run exited 0 and+was the whole job.++Every run writes a receipt into the JSON: the pyshacl and rdflib versions, the+engine binary's sha256, and the sha256 of every input file, so a drift in the+oracle and a change in the engine are told apart by a diff.Usage:python3 tools/shacl_differential.py CORPUS.json [--json out.json]+ python3 tools/shacl_differential.py --discover case-studies [--json out.json]+ python3 tools/shacl_differential.py --w3c tests/w3c-shacl [--json out.json]+ python3 tools/shacl_differential.py --discover case-studies --w3c tests/w3c-shacl-CORPUS.json is a list of {"name", "data": [ttl...], "shapes": ttl} objects, or-pass --discover ROOT to build one from a tree laid out as */shacl/*.ttl or+The sources combine; the exit rules apply to the whole run.++CORPUS.json is a list of {"name", "data": [ttl...], "shapes": ttl} objects.+--discover ROOT builds one from a tree laid out as */shacl/*.ttl or*/shapes/*.ttl with data in */graph/*.ttl, */data/*.ttl or */examples/*.ttl.+--pyshacl-version V makes any other installed version NOT MEASURED.Nothing is capped silently: every pair excluded for size or timeout is listed in-the report with the reason.+the report with the reason, and makes the run NOT MEASURED."""import argparse+import hashlibimport jsonimport osimport pathlib+import platformimport subprocessimport sysimport tempfile@@ -43,12 +96,177 @@ BIN = os.environ.get(MAX_BYTES = int(os.environ.get("SHACL_DIFF_MAX_BYTES", 40 * 1024 * 1024))TIMEOUT = int(os.environ.get("SHACL_DIFF_TIMEOUT", 300))+GREEN, RED, NOT_MEASURED = 0, 1, 2+SH = "http://www.w3.org/ns/shacl#"+RDF_NS = "http://www.w3.org/1999/02/22-rdf-syntax-ns#"+W3C_BASE = "https://w3c.github.io/data-shapes/data-shapes-test-suite/tests/"+# Verdicts that are a comparison. UNDETERMINED and ERROR are not, and a run made+# only of them measured nothing.+MEASURED = {"AGREE", "PARTIAL", "FALSE_CLEAN", "KNOWN_FALSE_CLEAN", "FALSE_ALARM", "ORACLE_WRONG",+ "AGREE_AGAINST_W3C"}+BLOCKING = ("FALSE_CLEAN", "AGREE_AGAINST_W3C")++CONTROL_DATA = """\+@prefix ex: <http://example.org/oo-control#> .+ex:alice a ex:Person ; ex:age "forty" .+ex:bob a ex:Person ; ex:name "Bob" ; ex:age 40 .+"""+# Two results, chosen to exercise every widened key field: a BLANK property+# shape (keyed by parent + path), an IRI property shape, and a non-default+# severity.+CONTROL_SHAPES = """\+@prefix ex: <http://example.org/oo-control#> .+@prefix sh: <http://www.w3.org/ns/shacl#> .+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .+ex:PersonShape a sh:NodeShape ;+ sh:targetClass ex:Person ;+ sh:property [ sh:path ex:name ; sh:minCount 1 ] ;+ sh:property ex:AgeShape .+ex:AgeShape a sh:PropertyShape ;+ sh:path ex:age ;+ sh:datatype xsd:integer ;+ sh:severity sh:Warning .+"""+++def sha256(path):+ h = hashlib.sha256()+ with open(path, "rb") as fh:+ for chunk in iter(lambda: fh.read(1 << 20), b""):+ h.update(chunk)+ return h.hexdigest()+++def oracle_versions():+ """pyshacl and rdflib versions, or None when pyshacl cannot be imported."""+ try:+ import pyshacl+ import rdflib+ except ImportError:+ return None+ return {"pyshacl": pyshacl.__version__, "rdflib": rdflib.__version__}+++# ── keys ─────────────────────────────────────────────────────────────────────+++def shape_key(parent, path):+ return f"{parent} sh:property [sh:path {path}]"+++def severity_iri(s):+ """The engine reports the local name of sh:severity; read a bare name as+ being in the SHACL namespace, as tests/w3c_shacl_conformance_test.rs does."""+ if not s:+ return None+ return s if ":" in s else SH + s+++def our_term(s):+ """The engine prints IRIs bare and literals in N-Triples form; blank nodes+ are not portable, so they collapse."""+ if s is None or s == "":+ return None+ return "_:" if s.startswith("_:") else s+++def ours_keys(report):+ out = set()+ for v in report.get("violations", []) or []:+ # Every violation passes through `attribute()`, which mirrors `path`+ # into `result_path`, and a sh:sparql result populates only+ # `result_path`, so reading it first is what makes SPARQL-based+ # results comparable at all.+ path = our_term(v.get("result_path") or v.get("path"))+ shape = v.get("source_shape")+ if shape and shape.startswith("_:"):+ parent = v.get("node_shape")+ shape = shape_key(parent, path) if parent and not parent.startswith("_:") and path else "_:"+ out.add((+ our_term(v.get("focus_node")),+ path,+ severity_iri(v.get("severity")),+ our_term(shape),+ our_term(v.get("source_constraint_component")),+ ))+ return out+++def rdf_term(t):+ from rdflib import BNode, URIRef++ if t is None:+ return None+ if isinstance(t, BNode):+ return "_:"+ if isinstance(t, URIRef):+ return str(t)+ return t.n3()+++def path_form(graph, t, depth=0):+ """Structural rendering of a SHACL path, the same canonical form the Rust+ harness uses: bare IRI, ^IRI, (a/b), (a|b), (p)* (p)+ (p)?. A well-formed+ path is an acyclic tree, so equal renderings mean isomorphic paths."""+ from rdflib import BNode, Namespace+ from rdflib.namespace import RDF++ sh = Namespace(SH)+ if not isinstance(t, BNode):+ return rdf_term(t)+ if depth > 32:+ return "(path too deep)"+ sub = lambda x: path_form(graph, x, depth + 1) # noqa: E731+ inv = graph.value(t, sh.inversePath)+ if inv is not None:+ return "^" + sub(inv)+ alt = graph.value(t, sh.alternativePath)+ if alt is not None:+ return "(" + "|".join(sub(x) for x in graph.items(alt)) + ")"+ for pred, op in (("zeroOrMorePath", "*"), ("oneOrMorePath", "+"), ("zeroOrOnePath", "?")):+ x = graph.value(t, sh[pred])+ if x is not None:+ return f"({sub(x)}){op}"+ if graph.value(t, RDF.first) is not None:+ return "(" + "/".join(sub(x) for x in graph.items(t)) + ")"+ return "_:"+++def rdf_keys(results_graph, results, shapes_graph):+ """Keys for rdflib sh:result nodes. `shapes_graph` resolves a blank source+ shape to its parent; pyshacl reports the shapes graph's own blank node."""+ from rdflib import BNode, Namespace, URIRef++ sh = Namespace(SH)+ out = set()+ for r in results:+ p = results_graph.value(r, sh.resultPath)+ path = None+ if p is not None:+ path = path_form(results_graph if (p, None, None) in results_graph else shapes_graph, p)+ shape = results_graph.value(r, sh.sourceShape)+ if isinstance(shape, BNode):+ parents = list(shapes_graph.subjects(sh.property, shape))+ spath = shapes_graph.value(shape, sh.path)+ if len(parents) == 1 and isinstance(parents[0], URIRef) and spath is not None:+ shape_s = shape_key(str(parents[0]), path_form(shapes_graph, spath))+ else:+ shape_s = "_:"+ else:+ shape_s = rdf_term(shape)+ out.add((+ rdf_term(results_graph.value(r, sh.focusNode)),+ path,+ rdf_term(results_graph.value(r, sh.resultSeverity)),+ shape_s,+ rdf_term(results_graph.value(r, sh.sourceConstraintComponent)),+ ))+ return out+++# ── engines ──────────────────────────────────────────────────────────────────+-# Every violation passes through `attribute()`, which mirrors `path` into-# `result_path` and is the only field a sh:sparql result populates, so reading-# `result_path` first is what makes SPARQL-based results comparable at all.-# Keyed on `path` alone, two engines that agreed exactly printed PARTIAL for-# every constraint that binds ?path, which would have masked a real difference.def run_open_ontologies(data_files, shapes_file):with tempfile.TemporaryDirectory() as d:env = dict(os.environ, OPEN_ONTOLOGIES_STORAGE_MODE="persistent")@@ -64,40 +282,44 @@ def run_open_ontologies(data_files, shapes_file):capture_output=True, text=True, env=env, timeout=TIMEOUT,)try:- return json.loads(r.stdout)+ report = json.loads(r.stdout)except json.JSONDecodeError:return {"error": f"unparseable report: {(r.stdout + r.stderr).strip()[:300]}"}+ report["keys"] = ours_keys(report)+ return report-def run_pyshacl(data_files, shapes_file):+def run_pyshacl(data_files, shapes_file, originals=None):+ """`originals` maps a rewritten N-Triples file back to (Turtle source,+ base). pyshacl is handed the source: measured on the W3C suite, the+ N-Triples rewrite drops the prefix bindings pyshacl resolves `sh:prefixes`+ through, and `sparql/node/prefixes-001` became a pyshacl ERROR that the+ original file does not cause."""from pyshacl import validate- from rdflib import Graph+ from rdflib import Graph, Namespace++ sh = Namespace(SH)+ originals = originals or {}++ def parse(graph, f):+ src, base = originals.get(str(f), (str(f), None))+ return graph.parse(src, format="turtle", publicID=base) if base else graph.parse(src)data = Graph()for f in data_files:- data.parse(str(f))- shapes = Graph().parse(str(shapes_file))+ parse(data, f)+ shapes = parse(Graph(), shapes_file)conforms, results_graph, _ = validate(data, shacl_graph=shapes)- violations = set()- from rdflib.namespace import Namespace- SH = Namespace("http://www.w3.org/ns/shacl#")# Every severity, not just sh:Violation. Collecting only Violations here# compared pyshacl's violations against our results at ALL severities, and# reported the difference as our defect: four sh:Warning results in the# enterprise-knowledge vertical showed up as four extra violations we had# invented. An oracle that manufactures findings is worse than no oracle.- for result in set(results_graph.subjects(SH.resultSeverity, None)):- focus = results_graph.value(result, SH.focusNode)- path = results_graph.value(result, SH.resultPath)- violations.add((str(focus), str(path) if path else None))- return {"conforms": conforms, "violations": violations}+ results = set(results_graph.subjects(sh.resultSeverity, None))+ return {"conforms": conforms, "keys": rdf_keys(results_graph, results, shapes)}-def ours_violation_set(report):- out = set()- for v in report.get("violations", []) or []:- out.add((str(v.get("focus_node")), v.get("result_path") or v.get("path")))- return out+# ── classification ───────────────────────────────────────────────────────────def classify(ours, theirs):@@ -115,19 +337,91 @@ def classify(ours, theirs):return "UNDETERMINED", ours.get("warning", "no verdict")if our_conforms is True and their_conforms is False:- return "FALSE_CLEAN", f"pyshacl found {len(theirs['violations'])} violation(s), we found none"+ return "FALSE_CLEAN", f"pyshacl found {len(theirs['keys'])} result(s), we found none"if our_conforms is False and their_conforms is True:- return "FALSE_ALARM", f"we reported {len(ours_violation_set(ours))} violation(s), pyshacl found none"+ return "FALSE_ALARM", f"we reported {len(ours['keys'])} result(s), pyshacl found none"- ours_v, theirs_v = ours_violation_set(ours), theirs["violations"]- missed, extra = theirs_v - ours_v, ours_v - theirs_v+ missed, extra = theirs["keys"] - ours["keys"], ours["keys"] - theirs["keys"]if missed or extra:return "PARTIAL", f"{len(missed)} missed, {len(extra)} extra (verdict agrees)"- return "AGREE", f"{len(ours_v)} violation(s), identical sets"+ return "AGREE", f"{len(ours['keys'])} result(s), identical sets"+++def same(a, b):+ return a["conforms"] == b["conforms"] and a["keys"] == b["keys"]+++def arbitrate(verdict, detail, ours, theirs, expected, baseline):+ """Let the W3C expected report decide a disagreement, or flag an agreement+ it contradicts. Only called where the engine gave a verdict.++ A FALSE_CLEAN the W3C report confirms, on a test the conformance ratchet+ (tests/w3c_shacl_baseline.json) does not record as PASS, is KNOWN_FALSE_CLEAN:+ reported on every run, and not blocking here, because the `w3c-shacl` job+ already fails if that FAIL count rises or any recorded PASS regresses.+ Measured 24 Oct 2026: 12 of the 113 comparable W3C tests, every one a+ `conforms: true` where the suite expects false, and every one already FAIL+ in that baseline. On a test the baseline records as PASS it stays FALSE_CLEAN+ and blocks, since that is a regression the ratchet has not seen."""+ if verdict == "AGREE":+ if not same(theirs, expected):+ return "AGREE_AGAINST_W3C", "both engines agree and the W3C expected report does not"+ return verdict, detail+ if same(ours, expected):+ return "ORACLE_WRONG", f"W3C sides with us against pyshacl ({verdict}: {detail})"+ side = "W3C sides with pyshacl" if same(theirs, expected) else "W3C matches neither engine"+ if verdict == "FALSE_CLEAN" and baseline not in (None, "PASS"):+ return "KNOWN_FALSE_CLEAN", f"{detail}; {side}; held by the w3c-shacl ratchet as {baseline}"+ return verdict, f"{detail}; {side}"+++# ── planted controls ─────────────────────────────────────────────────────────+++def run_controls(workdir):+ """Plant the controls and return (rows, failures). A failure is a control+ whose classification is not the one it was planted to have."""+ data = pathlib.Path(workdir) / "control-data.ttl"+ shapes = pathlib.Path(workdir) / "control-shapes.ttl"+ data.write_text(CONTROL_DATA)+ shapes.write_text(CONTROL_SHAPES)+ ours = run_open_ontologies([data], shapes)+ theirs = run_pyshacl([data], shapes)++ rows, failures = [], []++ def plant(name, planted_theirs, allowed):+ verdict, detail = classify(ours, planted_theirs)+ ok = verdict in allowed+ rows.append({"control": name, "verdict": verdict, "expected": sorted(allowed),+ "detail": detail, "ok": ok})+ if not ok:+ failures.append(f"control `{name}` came out {verdict}, planted to be "+ f"{' or '.join(sorted(allowed))}: {detail}")++ plant("must-agree", theirs, {"AGREE"})+ keys = sorted(theirs["keys"], key=lambda k: tuple(x or "" for x in k))+ if len(keys) < 2:+ # Without two results the mutations below cannot be planted at all, and+ # a control that is not planted is not a control.+ failures.append(f"control pair yielded {len(keys)} pyshacl result(s), need 2 to plant mutations")+ return rows, failures+ victim = keys[0]+ plant("one-result-dropped", {**theirs, "keys": set(keys[1:])}, {"PARTIAL", "FALSE_CLEAN"})+ for field, i in (("severity", 2), ("source-shape", 3), ("component", 4)):+ changed = list(victim)+ changed[i] = f"{changed[i]}-planted"+ plant(f"{field}-changed", {**theirs, "keys": set(keys[1:]) | {tuple(changed)}}, {"PARTIAL"})+ return rows, failures+++# ── discovery ────────────────────────────────────────────────────────────────def discover(root):cases, root = [], pathlib.Path(root)+ if not root.is_dir():+ return casesfor repo in sorted(p for p in root.iterdir() if p.is_dir()):shapes = sorted(list((repo / "shacl").glob("*.ttl")) + list((repo / "shapes").glob("*.ttl")))if not shapes:@@ -143,57 +437,216 @@ def discover(root):return cases-def main():+def discover_w3c(suite, workdir):+ """Walk the vendored W3C suite from its root manifest, the walk+ tests/w3c_shacl_conformance_test.rs does. Each case's graphs are re-written+ as N-Triples with the suite's published base, because every file uses+ relative IRIs and `open-ontologies shacl` takes no base IRI. Entries whose+ mf:result is sht:Failure carry no report to compare and are returned as+ excluded, by name."""+ from rdflib import Graph, Namespace+ from rdflib.namespace import RDF++ mf = Namespace("http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#")+ sht = Namespace("http://www.w3.org/ns/shacl-test#")+ sh = Namespace(SH)+ root = os.path.realpath(suite)+ if not os.path.isfile(os.path.join(root, "manifest.ttl")):+ return [], []++ def base_of(p):+ return W3C_BASE + os.path.relpath(p, root)++ def path_of(iri):+ return os.path.join(root, str(iri)[len(W3C_BASE):])++ def load(p):+ return Graph().parse(p, format="turtle", publicID=base_of(p))++ # The conformance ratchet's per-test outcomes. A W3C test it does not record+ # as PASS is an engine defect that ratchet already holds (its FAIL count may+ # not rise), so a FALSE_CLEAN there is KNOWN rather than new; see arbitrate.+ baseline_file = pathlib.Path(root).parent / "w3c_shacl_baseline.json"+ baseline = json.loads(baseline_file.read_text())["outcomes"] if baseline_file.is_file() else {}++ cases, excluded, seen = [], [], set()+ out = pathlib.Path(workdir) / "w3c"++ def absolute(p):+ dest = out / (os.path.relpath(p, root) + ".nt")+ if not dest.exists():+ dest.parent.mkdir(parents=True, exist_ok=True)+ load(p).serialize(destination=str(dest), format="nt")+ return str(dest)++ def walk(path):+ path = os.path.realpath(path)+ if path in seen:+ return+ seen.add(path)+ g = load(path)+ name = os.path.relpath(path, root)[: -len(".ttl")]+ for manifest in g.subjects(RDF.type, mf.Manifest):+ for entries in g.objects(manifest, mf.entries):+ for entry in g.items(entries):+ if (entry, RDF.type, sht.Validate) not in g:+ continue+ result = g.value(entry, mf.result)+ if result == sht.Failure:+ excluded.append({"name": "w3c/" + name, "reason": "expects sht:Failure, not a report"})+ continue+ action = g.value(entry, mf.action)+ data = path_of(g.value(action, sht.dataGraph))+ shapes = path_of(g.value(action, sht.shapesGraph))+ expected = {+ "conforms": bool(g.value(result, sh.conforms).toPython()),+ "keys": rdf_keys(g, set(g.objects(result, sh.result)), g),+ }+ nt_data, nt_shapes = absolute(data), absolute(shapes)+ cases.append({"name": "w3c/" + name, "data": [nt_data], "shapes": nt_shapes,+ "originals": {nt_data: (data, base_of(data)),+ nt_shapes: (shapes, base_of(shapes))},+ "inputs": sorted({path, data, shapes}), "expected": expected,+ "baseline": baseline.get(name)})+ for inc in sorted(g.objects(None, mf.include)):+ walk(path_of(inc))++ walk(os.path.join(root, "manifest.ttl"))+ cases.sort(key=lambda c: c["name"])+ return cases, excluded+++# ── main ─────────────────────────────────────────────────────────────────────+++def not_measured(why, report, json_path):+ print(f"\nNOT MEASURED: {why}\nThis is not agreement between the engines; it is the absence "+ "of a comparison, and it exits 2.", flush=True)+ report.update({"status": "NOT_MEASURED", "exit": NOT_MEASURED, "why": why})+ if json_path:+ with open(json_path, "w") as fh:+ json.dump(report, fh, indent=1, default=sorted)+ return NOT_MEASURED+++def main(argv=None):ap = argparse.ArgumentParser()ap.add_argument("corpus", nargs="?")ap.add_argument("--discover")+ ap.add_argument("--w3c")ap.add_argument("--json")ap.add_argument("--limit", type=int)- args = ap.parse_args()-- cases = discover(args.discover) if args.discover else json.load(open(args.corpus))- if args.limit:- cases = cases[: args.limit]-- rows, skipped_pairs = [], []- for case in cases:- data = [f for f in case["data"] if os.path.exists(f)]- size = sum(os.path.getsize(f) for f in data) + os.path.getsize(case["shapes"])- if size > MAX_BYTES:- reason = f"{size/1e6:.0f} MB over the {MAX_BYTES/1e6:.0f} MB cap"- skipped_pairs.append({"name": case["name"], "reason": reason})- print(f"{'NOT_RUN':<13} {case['name']}\n {reason}", flush=True)- continue- try:- ours = run_open_ontologies(data, case["shapes"])- theirs = run_pyshacl(data, case["shapes"])- verdict, detail = classify(ours, theirs)- except subprocess.TimeoutExpired:- reason = f"timed out after {TIMEOUT}s"- skipped_pairs.append({"name": case["name"], "reason": reason})- print(f"{'NOT_RUN':<13} {case['name']}\n {reason}", flush=True)- continue- except Exception as exc: # a parse failure in either engine is a result, not a crash- verdict, detail = "ERROR", f"{type(exc).__name__}: {exc}"[:300]- rows.append({"name": case["name"], "verdict": verdict, "detail": detail})- print(f"{verdict:<13} {case['name']}\n {detail}", flush=True)+ ap.add_argument("--pyshacl-version")+ args = ap.parse_args(argv)++ report = {"receipt": {"python": platform.python_version(), "engine": {"path": BIN}}}+ versions = oracle_versions()+ if versions is None:+ return not_measured("pyshacl is not importable; there is no oracle", report, args.json)+ report["receipt"].update(versions)+ print(f"oracle: pyshacl {versions['pyshacl']}, rdflib {versions['rdflib']}", flush=True)+ if args.pyshacl_version and versions["pyshacl"] != args.pyshacl_version:+ return not_measured(f"pyshacl {versions['pyshacl']} is installed and {args.pyshacl_version} "+ "is pinned; a different oracle is a different measurement", report, args.json)+ if not os.path.isfile(BIN):+ return not_measured(f"no engine binary at {BIN}", report, args.json)+ report["receipt"]["engine"]["sha256"] = sha256(BIN)++ with tempfile.TemporaryDirectory() as workdir:+ if not (args.w3c or args.discover or args.corpus):+ ap.error("give a CORPUS.json, --discover ROOT and/or --w3c SUITE")+ cases, excluded = [], []+ if args.corpus:+ cases += json.load(open(args.corpus))+ if args.discover:+ cases += discover(args.discover)+ if args.w3c:+ w3c_cases, excluded = discover_w3c(args.w3c, workdir)+ cases += w3c_cases+ if args.limit:+ cases = cases[: args.limit]+ report["excluded"] = excluded+ for e in excluded:+ print(f"{'EXCLUDED':<17} {e['name']}\n {e['reason']}", flush=True)++ inputs = set()+ for case in cases:+ inputs.update(case.get("inputs") or (list(case["data"]) + [case["shapes"]]))+ report["receipt"]["inputs"] = {f: sha256(f) for f in sorted(inputs) if os.path.isfile(f)}+ report["discovered"] = len(cases)++ controls, control_failures = run_controls(workdir)+ report["controls"] = controls+ for c in controls:+ print(f"CONTROL {'ok ' if c['ok'] else 'BAD'} {c['control']:<22} {c['verdict']}", flush=True)++ if not cases:+ return not_measured("discovery found zero (data, shapes) pairs", report, args.json)++ rows, skipped_pairs = [], []+ for case in cases:+ data = [f for f in case["data"] if os.path.exists(f)]+ size = sum(os.path.getsize(f) for f in data) + os.path.getsize(case["shapes"])+ if size > MAX_BYTES:+ reason = f"{size/1e6:.0f} MB over the {MAX_BYTES/1e6:.0f} MB cap"+ skipped_pairs.append({"name": case["name"], "reason": reason})+ print(f"{'NOT_RUN':<17} {case['name']}\n {reason}", flush=True)+ continue+ try:+ ours = run_open_ontologies(data, case["shapes"])+ theirs = run_pyshacl(data, case["shapes"], case.get("originals"))+ verdict, detail = classify(ours, theirs)+ if "expected" in case and verdict not in ("ERROR", "UNDETERMINED"):+ verdict, detail = arbitrate(verdict, detail, ours, theirs, case["expected"],+ case.get("baseline"))+ except subprocess.TimeoutExpired:+ reason = f"timed out after {TIMEOUT}s"+ skipped_pairs.append({"name": case["name"], "reason": reason})+ print(f"{'NOT_RUN':<17} {case['name']}\n {reason}", flush=True)+ continue+ except Exception as exc: # a parse failure in either engine is a result, not a crash+ verdict, detail = "ERROR", f"{type(exc).__name__}: {exc}"[:300]+ rows.append({"name": case["name"], "verdict": verdict, "detail": detail})+ print(f"{verdict:<17} {case['name']}\n {detail}", flush=True)counts = {}for r in rows:counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1+ measured = sum(n for k, n in counts.items() if k in MEASURED)+ report.update({"rows": rows, "not_run": skipped_pairs, "counts": counts, "measured": measured})print("\n" + "=" * 70)- print(f"{len(rows)} pair(s) run, {len(skipped_pairs)} not run")- for k in ("FALSE_CLEAN", "FALSE_ALARM", "PARTIAL", "UNDETERMINED", "AGREE", "ERROR"):+ print(f"{len(cases)} pair(s) discovered, {len(rows)} run, {len(skipped_pairs)} not run, "+ f"{measured} compared")+ for k in ("FALSE_CLEAN", "AGREE_AGAINST_W3C", "KNOWN_FALSE_CLEAN", "FALSE_ALARM", "PARTIAL",+ "ORACLE_WRONG",+ "UNDETERMINED", "AGREE", "ERROR"):if k in counts:- print(f" {k:<13} {counts[k]}")+ print(f" {k:<17} {counts[k]}")for s in skipped_pairs:print(f" not run: {s['name']} - {s['reason']}")- if args.json:- json.dump({"rows": rows, "not_run": skipped_pairs, "counts": counts}, open(args.json, "w"), indent=1)+ if control_failures:+ code, status = RED, "RED"+ for f in control_failures:+ print(f"\nRED: {f}")+ print("A comparator that misclassifies a planted difference cannot be trusted on the "+ "corpus, whatever the corpus said.")+ elif any(counts.get(k) for k in BLOCKING):+ code, status = RED, "RED"+ elif skipped_pairs:+ return not_measured(f"{len(skipped_pairs)} of {len(cases)} discovered pair(s) were not run",+ report, args.json)+ elif measured == 0:+ return not_measured(f"none of the {len(rows)} pair(s) run yielded a comparison "+ "(every one ERROR or UNDETERMINED)", report, args.json)+ else:+ code, status = GREEN, "GREEN"- # A false clean is the one result that must fail a pipeline.- return 1 if counts.get("FALSE_CLEAN") else 0+ report.update({"status": status, "exit": code})+ if args.json:+ with open(args.json, "w") as fh:+ json.dump(report, fh, indent=1, default=sorted)+ return codeif __name__ == "__main__":