Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

SHACL differential exited 0 when it measured nothing and its oracle was unpinned; exit 2 NOT MEASURED, plant controls, pin pyshacl, arbitrate on W3C (#320)

FABIOTESS2e8025ec6266
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index b49affa5..a6c22294 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -252,14 +252,32 @@ jobs:
# test that returns early: it reports assurance that did not happen.
#
# The oracle exits 1 on a FALSE_CLEAN, meaning we said conforms where pyshacl
- # said not, which is the one failure this validator must never have. That is
- # what gates here. Disagreements in the other direction and undetermined
- # verdicts are printed and uploaded, not failed on, because they are a work
- # list rather than a defect.
+ # said not, which is the one failure this validator must never have, and on
+ # a planted control it misclassifies. It exits 2, NOT MEASURED, when there is
+ # no oracle, the wrong oracle, nothing discovered, fewer pairs run than
+ # discovered, or no pair compared. Any non-zero exit fails this job.
+ # Disagreements in the other direction and undetermined verdicts are printed
+ # and uploaded, not failed on, because they are a work list rather than a
+ # defect.
+ #
+ # Before #320 the script exited 0 on everything but a FALSE_CLEAN, and this
+ # job was green on a run that compared nothing: measured 24 Oct 2026, the
+ # case-studies corpus yields 4 pairs and all 4 are UNDETERMINED. The W3C suite
+ # is therefore run in the same invocation, where each disagreement is
+ # arbitrated against the Working Group's expected report (73 of 117 pairs
+ # compared on that date).
+ #
+ # pyshacl and rdflib are pinned, and the script refuses (exit 2) any other
+ # pyshacl than PYSHACL_VERSION, because an unpinned oracle makes a pyshacl
+ # release look like a change in this engine. The versions and the sha256 of
+ # every input are in the uploaded JSON.
shacl-differential:
timeout-minutes: 30
name: SHACL differential vs pyshacl
runs-on: ubuntu-latest
+ env:
+ PYSHACL_VERSION: "0.40.1"
+ RDFLIB_VERSION: "7.6.0"
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
@@ -276,14 +294,16 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- - name: Install pyshacl
- run: pip install pyshacl
+ - name: Install the pinned oracle
+ run: pip install "pyshacl==${PYSHACL_VERSION}" "rdflib==${RDFLIB_VERSION}"
- name: Build the binary the oracle drives
run: cargo build --release
- - name: Run the differential over the case-study corpus
+ - name: Run the differential over the case-study corpus and the W3C suite
run: |
python3 tools/shacl_differential.py \
--discover case-studies \
+ --w3c tests/w3c-shacl \
+ --pyshacl-version "${PYSHACL_VERSION}" \
--json shacl-differential.json
- name: Upload the report
if: always()
@@ -417,6 +437,7 @@ jobs:
tests/test_horn_rule_table.py \
tests/test_shacl.py \
tests/test_shacl_vacuous.py \
+ tests/test_shacl_differential_harness.py \
tests/test_align.py \
tests/test_vocab_check.py
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8d3d900c..72d7c636 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -23,6 +23,23 @@ All notable changes to Open Ontologies are documented here.
the engine did not read `sh:severity`, had been false since #206 and is gone.
A planted control mutates one PASS case's expected report in each of the six
fields and requires each copy to be rejected by name.
+- **`tools/shacl_differential.py` can no longer go green on nothing**
+ (reported by @noahgift in #320). It exited 0 on everything but a FALSE_CLEAN,
+ including no cases, no pyshacl and no comparisons, and the CI job's
+ case-studies corpus yields 4 pairs that all come back UNDETERMINED. It now
+ exits 2 (NOT MEASURED) for a missing or unpinned oracle, a missing engine,
+ empty discovery, fewer pairs run than discovered, or no pair compared; plants
+ a must-agree control and four mutated copies of pyshacl's report on every run
+ and exits 1 if any is misclassified; keys results on (focus, path, severity,
+ source shape, component) instead of (focus, path); records pyshacl, rdflib,
+ the engine binary's sha256 and every input's sha256; and with `--w3c`
+ arbitrates each disagreement against the W3C expected report. CI pins
+ pyshacl 0.40.1 and rdflib 7.6.0 and runs the corpus and the W3C suite
+ together: 117 pairs, 73 compared, 0 blocking, with 12 W3C-confirmed
+ FALSE_CLEANs reported as KNOWN because the conformance ratchet already
+ holds each as FAIL. `tools/fol_differential.py`
+ (every goal undetermined) and `tools/horn_differential.py` (every case not
+ run) had the same exit-0-on-nothing and now exit 2.
## [2.1.0] - 2026-10-23
diff --git a/docs/ci-gates.md b/docs/ci-gates.md
index 173256fa..c5359b21 100644
--- a/docs/ci-gates.md
+++ b/docs/ci-gates.md
@@ -8,8 +8,8 @@ over the same documents — ran its preflight, found neither of the two tools it
needs, skipped every test in the file, and exited 0.
This page is the answer to "does a green tick mean this ran?", so that nobody has
-to work it out again from six workflow files and the thirty-four test files that
-can skip — twenty-five Rust, nine Python. It is a statement about CI
+to work it out again from six workflow files and the test files that can skip —
+twenty-five Rust, fourteen Python. It is a statement about CI
configuration, which changes. Its Rust half is now checked by a test rather than
by hand (`tests/ci_gate_coverage_test.rs`), because the by-hand version went
stale exactly where it mattered; the Python half still carries a date at the
@@ -40,7 +40,17 @@ Two tool scripts mirror the variable under their own names because they are CLIs
rather than test suites, and say so where they define it:
`HORN_DIFF_REQUIRE_TOOLS=1` in `tools/horn_differential.py` and
`FOL_DIFF_REQUIRE_ATP=1` in `tools/fol_differential.py`. Both exit 2 rather than
-panicking.
+panicking. Both also exit 2, flag or no flag, when the tools ARE present and the
+run compared nothing: every case NOT_RUN in the first, every goal UNDETERMINED
+or none asked in the second (#320).
+
+`tools/shacl_differential.py` has no opt-out flag, because its CI job always
+provides the oracle. It exits 2, NOT MEASURED, when pyshacl is missing or is not
+the pinned version, when the engine binary is missing, when discovery finds
+nothing, when fewer pairs ran than were discovered, and when no pair yielded a
+comparison; it exits 1 when one of its planted controls is misclassified. Before
+#320 all of those exited 0, and the `shacl-differential` job was green over a
+corpus whose 4 pairs were all UNDETERMINED.
## The table
@@ -118,6 +128,11 @@ path to make strict. `module_extract_test.rs` reads
| `test_horn_rule_table.py` | `tests/fixtures/horn/` + git | `python` job | **strict** |
| `test_shacl.py` | `pyshacl` (the `shacl` extra) | `python` job installs it | **strict** |
| `test_shacl_vacuous.py` | `pyshacl` | `python` job | **strict** |
+| `test_shacl_differential_harness.py` | `pyshacl` | `python` job | **strict** |
+| `test_shacl_falsy_terms.py` | `pyshacl` | `python` job installs it | **runs, not strict**: not in the strict leg, so a skip would pass |
+| `test_shacl_targeted_shapes.py` | `pyshacl` | `python` job installs it | **runs, not strict** |
+| `test_align_failed_insertion.py` | `hnswlib` | `python` job installs it | **runs, not strict** |
+| `test_horn_checker_relative.py` | a symlink to an executable in a temp directory | any POSIX runner | **runs, not strict** |
| `test_align.py` | `hnswlib` (the `align` extra) | `python` job installs it | **strict** |
| `test_vocab_check.py` | `pyoxigraph` (a base dependency) | always present | **strict** |
| `test_dataframe.py` | `fenic` | **nothing** — `fenic` is in no extra and no dependency list | **skips** |
@@ -154,7 +169,15 @@ The other seven files under `python/tests/` have no skip path.
is absent or the tree is not a checkout, which on a GitHub runner after
`actions/checkout` cannot happen. Measured on the current tree: the eight
strict Python files are 67 passed, 1 skipped, and the one skip is the
- `oo_opt_in` twelve-minute differential.
+ `oo_opt_in` twelve-minute differential. The ninth,
+ `test_shacl_differential_harness.py`, adds 9 passed (measured 24 October
+ 2026 against pyshacl 0.40.1).
+- **Four Python files can skip and are in no strict leg**:
+ `test_shacl_falsy_terms.py`, `test_shacl_targeted_shapes.py`,
+ `test_align_failed_insertion.py` and `test_horn_checker_relative.py`. The
+ `python` job provides what each needs, so they run there today, but a skip in
+ any of them would pass. They were missing from this page until 24 October
+ 2026, when the count below was re-taken and came to fourteen rather than nine.
- **The `features/breadth` job runs no tests** (`test: false`); it is
`cargo check` plus clippy across all features. That is deliberate and is not a
skip, but it does mean an all-features build is type-checked and never run.
@@ -207,13 +230,13 @@ property is that CI cannot skip in silence, not that a laptop cannot skip at all
The Python half is still by hand:
```bash
-# every Python file that can skip: 11 hits. conftest.py is the mechanism and
+# every Python file that can skip: 16 hits. conftest.py is the mechanism and
# not a test, and test_conftest_collection_skip.py tests it: it writes a
-# skipping module into a temp directory and cannot skip itself. So 9, and the
-# table has 9 rows
+# skipping module into a temp directory and cannot skip itself. So 14, and the
+# table has 14 rows
grep -rln 'pytest.skip\|importorskip\|skipif' python/tests/*.py
# every leg CI makes strict
grep -rn 'OO_REQUIRE_FIXTURES=1' .github/workflows/
```
-Last checked against `.github/workflows/` on 15 September 2026.
+Last checked against `.github/workflows/` on 24 October 2026.
diff --git a/python/tests/test_shacl_differential_harness.py b/python/tests/test_shacl_differential_harness.py
new file mode 100644
index 00000000..a7595e8c
--- /dev/null
+++ b/python/tests/test_shacl_differential_harness.py
@@ -0,0 +1,227 @@
+"""`tools/shacl_differential.py` cannot go green on nothing, and its planted
+controls bite (#320).
+
+Before #320 the script exited 0 whenever it found no FALSE_CLEAN, which it also
+did when it found no cases, when pyshacl was not installed (every pair came back
+ERROR from the import inside the run), and when every pair was capped out. The
+CI job "SHACL differential vs pyshacl" was therefore green on a run that
+compared nothing. Each test below fails against that version of the script.
+
+The engine here is a stand-in: a script that answers `load` and `shacl` the way
+the release binary does, by running pyshacl and printing the result in the
+engine's JSON shape. A faithful stand-in is the right fixture for testing the
+HARNESS, because every disagreement it shows is then the harness's own doing,
+and it keeps this file from needing a release build. The real binary is what
+the CI job drives.
+"""
+
+import importlib.util
+import json
+import os
+import pathlib
+import subprocess
+import sys
+import textwrap
+
+import pytest
+
+pytest.importorskip("pyshacl")
+
+REPO = pathlib.Path(__file__).resolve().parents[2]
+TOOL = REPO / "tools" / "shacl_differential.py"
+
+FAKE_ENGINE = textwrap.dedent(
+ '''\
+ import json, pathlib, sys
+ from pyshacl import validate
+ from rdflib import BNode, Graph, Namespace, URIRef
+
+ SH = Namespace("http://www.w3.org/ns/shacl#")
+ args = sys.argv[1:]
+ store = pathlib.Path(args[args.index("--data-dir") + 1])
+ cmd, target = args[args.index("--data-dir") + 2], args[args.index("--data-dir") + 3]
+ if cmd == "load":
+ with open(store / "loaded.txt", "a") as fh:
+ fh.write(target + "\\n")
+ sys.exit(0)
+ data = Graph()
+ for f in (store / "loaded.txt").read_text().split():
+ data.parse(f)
+ shapes = Graph().parse(target)
+ conforms, results, _ = validate(data, shacl_graph=shapes)
+
+ def term(t):
+ if t is None:
+ return None
+ if isinstance(t, BNode):
+ return "_:" + str(t)
+ return str(t) if isinstance(t, URIRef) else t.n3()
+
+ violations = []
+ for r in results.subjects(SH.resultSeverity, None):
+ shape = results.value(r, SH.sourceShape)
+ parent = next(iter(shapes.subjects(SH.property, shape)), None)
+ violations.append({
+ "focus_node": term(results.value(r, SH.focusNode)),
+ "result_path": term(results.value(r, SH.resultPath)),
+ "severity": str(results.value(r, SH.resultSeverity)).rsplit("#", 1)[-1],
+ "source_shape": term(shape),
+ "node_shape": term(parent) if parent is not None else term(shape),
+ "source_constraint_component": term(results.value(r, SH.sourceConstraintComponent)),
+ })
+ print(json.dumps({"conforms": conforms, "violations": violations}))
+ '''
+)
+
+DATA = """\
+@prefix ex: <http://example.org/t#> .
+ex:a a ex:Thing .
+"""
+SHAPES = """\
+@prefix ex: <http://example.org/t#> .
+@prefix sh: <http://www.w3.org/ns/shacl#> .
+ex:S a sh:NodeShape ; sh:targetClass ex:Thing ; sh:property [ sh:path ex:p ; sh:minCount 1 ] .
+"""
+
+
+def engine(tmp_path):
+ exe = tmp_path / "fake-engine"
+ exe.write_text(f"#!{sys.executable}\n" + FAKE_ENGINE)
+ exe.chmod(0o755)
+ return exe
+
+
+def corpus(tmp_path):
+ root = tmp_path / "corpus"
+ (root / "one" / "shacl").mkdir(parents=True)
+ (root / "one" / "data").mkdir()
+ (root / "one" / "shacl" / "s.ttl").write_text(SHAPES)
+ (root / "one" / "data" / "d.ttl").write_text(DATA)
+ return root
+
+
+def run(args, engine, extra_env=None):
+ env = dict(os.environ, OPEN_ONTOLOGIES_BIN=str(engine), **(extra_env or {}))
+ return subprocess.run([sys.executable, str(TOOL), *args], capture_output=True, text=True, env=env)
+
+
+def load_tool(monkeypatch, engine):
+ monkeypatch.setenv("OPEN_ONTOLOGIES_BIN", str(engine))
+ spec = importlib.util.spec_from_file_location("shacl_differential_under_test", TOOL)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+def test_a_faithful_engine_is_green_and_the_receipt_is_written(engine, corpus, tmp_path):
+ out = tmp_path / "r.json"
+ p = run(["--discover", str(corpus), "--json", str(out)], engine)
+ assert p.returncode == 0, p.stdout + p.stderr
+ r = json.loads(out.read_text())
+ assert r["status"] == "GREEN"
+ assert r["counts"] == {"AGREE": 1}
+ import pyshacl
+
+ assert r["receipt"]["pyshacl"] == pyshacl.__version__
+ assert r["receipt"]["engine"]["sha256"]
+ assert sorted(r["receipt"]["inputs"]) == sorted(
+ str(p) for p in [corpus / "one" / "shacl" / "s.ttl", corpus / "one" / "data" / "d.ttl"]
+ )
+ assert all(len(h) == 64 for h in r["receipt"]["inputs"].values())
+ assert [c["ok"] for c in r["controls"]] == [True] * 5, r["controls"]
+
+
+def test_empty_discovery_is_not_measured(engine, tmp_path):
+ empty = tmp_path / "empty"
+ empty.mkdir()
+ p = run(["--discover", str(empty)], engine)
+ assert p.returncode == 2, p.stdout + p.stderr
+ assert "NOT MEASURED" in p.stdout
+
+
+def test_missing_pyshacl_is_not_measured(engine, corpus, tmp_path):
+ # A package named pyshacl that refuses to import, first on the path: the
+ # interpreter sees pyshacl as absent, exactly as on a runner without it.
+ stub = tmp_path / "stub" / "pyshacl"
+ stub.mkdir(parents=True)
+ (stub / "__init__.py").write_text("raise ImportError('pyshacl withheld by the test')\n")
+ p = run(["--discover", str(corpus)], engine, {"PYTHONPATH": str(stub.parent)})
+ assert p.returncode == 2, p.stdout + p.stderr
+ assert "NOT MEASURED" in p.stdout and "pyshacl" in p.stdout
+
+
+def test_a_pin_mismatch_is_not_measured(engine, corpus):
+ p = run(["--discover", str(corpus), "--pyshacl-version", "0.0.0"], engine)
+ assert p.returncode == 2, p.stdout + p.stderr
+
+
+def test_fewer_run_than_discovered_is_not_measured(engine, corpus):
+ p = run(["--discover", str(corpus)], engine, {"SHACL_DIFF_MAX_BYTES": "1"})
+ assert p.returncode == 2, p.stdout + p.stderr
+ assert "were not run" in p.stdout
+
+
+def test_a_comparator_blind_to_results_fails_on_its_controls(engine, corpus, monkeypatch, capsys):
+ """A classifier that compares the verdict and nothing else would call every
+ corpus pair here AGREE. The planted controls are what catch it."""
+ mod = load_tool(monkeypatch, engine)
+ real = mod.classify
+
+ def verdict_only(ours, theirs):
+ verdict, detail = real(ours, theirs)
+ return ("AGREE", detail) if verdict == "PARTIAL" else (verdict, detail)
+
+ monkeypatch.setattr(mod, "classify", verdict_only)
+ assert mod.main(["--discover", str(corpus)]) == 1
+ out = capsys.readouterr().out
+ for name in ("one-result-dropped", "severity-changed", "source-shape-changed", "component-changed"):
+ assert f"control `{name}` came out AGREE" in out, out
+
+
+def test_a_comparator_that_never_agrees_fails_on_its_controls(engine, corpus, monkeypatch, capsys):
+ mod = load_tool(monkeypatch, engine)
+ monkeypatch.setattr(mod, "classify", lambda ours, theirs: ("PARTIAL", "always"))
+ assert mod.main(["--discover", str(corpus)]) == 1
+ assert "control `must-agree` came out PARTIAL" in capsys.readouterr().out
+
+
+def test_w3c_arbitration_names_the_wrong_side(engine, monkeypatch):
+ """`arbitrate` is what turns a disagreement on a W3C test into a verdict
+ about one engine. Driven directly, on hand-built reports, so each branch is
+ shown to be reachable and to say what it claims."""
+ mod = load_tool(monkeypatch, engine)
+ right = {"conforms": False, "keys": {("f", None, "sev", "shape", "comp")}}
+ wrong = {"conforms": True, "keys": set()}
+
+ # Both engines agree and the Working Group does not: the blind spot of a
+ # two-engine differential, so it blocks.
+ assert mod.arbitrate("AGREE", "", wrong, wrong, right, None)[0] == "AGREE_AGAINST_W3C"
+ assert "AGREE_AGAINST_W3C" in mod.BLOCKING
+ # We match the expected report and pyshacl does not.
+ assert mod.arbitrate("FALSE_ALARM", "", right, wrong, right, None)[0] == "ORACLE_WRONG"
+ # pyshacl is right and we are clean where we should not be: blocking unless
+ # the conformance ratchet already holds this test as failing.
+ assert mod.arbitrate("FALSE_CLEAN", "", wrong, right, right, "PASS")[0] == "FALSE_CLEAN"
+ assert mod.arbitrate("FALSE_CLEAN", "", wrong, right, right, None)[0] == "FALSE_CLEAN"
+ assert mod.arbitrate("FALSE_CLEAN", "", wrong, right, right, "FAIL")[0] == "KNOWN_FALSE_CLEAN"
+ assert "KNOWN_FALSE_CLEAN" not in mod.BLOCKING
+
+
+def test_horn_differential_is_not_measured_when_every_case_is_capped(tmp_path, monkeypatch, capsys):
+ """The same hole in tools/horn_differential.py: its zero-cases check ran
+ before the size cap, so a corpus capped out entirely compared nothing and
+ exited 0. The preflight is stubbed because the hole is in what happens
+ after it."""
+ (tmp_path / "a.ttl").write_text(DATA)
+ spec = importlib.util.spec_from_file_location(
+ "horn_differential_under_test", REPO / "tools" / "horn_differential.py"
+ )
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ monkeypatch.setattr(mod, "preflight", lambda: [])
+ argv = ["horn_differential.py", "--root", str(tmp_path), "--max-bytes", "0", "--no-coverage"]
+ monkeypatch.setattr(sys, "argv", argv)
+ assert mod.main() == 2
+ assert "compared nothing" in capsys.readouterr().out
diff --git a/tests/w3c_shacl_conformance_test.rs b/tests/w3c_shacl_conformance_test.rs
index 09b8ac60..cc3d3fef 100644
--- a/tests/w3c_shacl_conformance_test.rs
+++ b/tests/w3c_shacl_conformance_test.rs
@@ -107,6 +107,10 @@
//!
//! That figure is for the gated column as it stood before #319, with every
//! blank source shape collapsed to `_:`; the script still applies that lens.
+//! Per-case arbitration of this engine against pyshacl on the W3C expected
+//! reports, with severity and the re-keyed source shape in the key, is
+//! `tools/shacl_differential.py --w3c tests/w3c-shacl`, which the
+//! `shacl-differential` CI job runs.
//!
//! # Four buckets, because this engine has three answers
//!
diff --git a/tools/fol_differential.py b/tools/fol_differential.py
index 2adbf878..180ebb36 100644
--- a/tools/fol_differential.py
+++ b/tools/fol_differential.py
@@ -642,7 +642,17 @@ def main():
"counts": counts, "proof_counts": proof_counts}, fh, indent=1)
# A conclusion the engine claims and the ATP refutes is the one result that
- # must fail a pipeline. An undetermined run is not a pass and not a failure.
+ # must fail a pipeline. An undetermined row is not a pass and not a failure,
+ # but a run in which EVERY goal is undetermined, or no goal was asked, put
+ # nothing to the prover that came back with an answer, and used to exit 0
+ # exactly like a run that agreed on everything (#320). It exits 2, as a
+ # missing prover does under FOL_DIFF_REQUIRE_ATP=1.
+ if all(r["verdict"] == "UNDETERMINED" for r in rows):
+ print(
+ f"\nNOT MEASURED: {len(rows)} goal(s) asked and none answered. This is not "
+ "agreement; it exits 2."
+ )
+ return 2
rejected = sum(n for k, n in proof_counts.items() if k.startswith("PROOF_REJECTED"))
if rejected:
print(
diff --git a/tools/horn_differential.py b/tools/horn_differential.py
index a1069f20..d4dcd0ea 100644
--- a/tools/horn_differential.py
+++ b/tools/horn_differential.py
@@ -855,6 +855,13 @@ def main() -> int:
else:
print(f"{row['verdict']:<20} {row['name']} ({row['detail']})", flush=True)
shutil.rmtree(specdir, ignore_errors=True)
+ # The empty-corpus check above runs before the size cap and the workers, so
+ # a corpus whose every case was capped, timed out or crashed reached the
+ # exit at the bottom with zero rows and returned 0 (#320). Nothing compared
+ # is not agreement.
+ if not rows:
+ print(f"all {len(cases)} case(s) were NOT_RUN: the differential compared nothing", flush=True)
+ return 2
counts = {}
for r in rows:
diff --git a/tools/shacl_differential.py b/tools/shacl_differential.py
index b13594cd..12c38451 100644
--- a/tools/shacl_differential.py
+++ b/tools/shacl_differential.py
@@ -2,33 +2,86 @@
"""Differential oracle: open-ontologies SHACL against pyshacl, over real shapes.
A validator is trusted on evidence, not on its own test suite. This runs both
-engines over the same (data, shapes) pairs and compares three things in order of
+engines over the same (data, shapes) pairs and compares them in order of
severity:
1. FALSE CLEAN - we say conforms, pyshacl says not. The one failure mode the
validator must not have. Any of these is a stop-the-line bug.
2. FALSE ALARM - we say not conforms, pyshacl says conforms. Noise that makes
a gate untrustworthy in the other direction.
- 3. UNDETERMINED - we return null because something was skipped. Honest, but a
+ 3. PARTIAL - the verdicts agree and the result sets do not.
+ 4. UNDETERMINED - we return null because something was skipped. Honest, but a
gap: the shapes graph got no verdict. These are the work list.
Agreement on the verdict is not enough, so where both engines give a verdict the
-violation sets are compared as (focus_node, path) pairs.
+result sets are compared on (focus node, result path, severity, source shape,
+source constraint component). A blank-node source shape is keyed by the IRI of
+the node shape holding it under sh:property plus its sh:path, the same key
+tests/w3c_shacl_conformance_test.rs uses, because a blank label is not portable
+between two parses, let alone two engines. sh:value is NOT in the key: on the
+default path the engine does not emit it yet (#318), so including it would turn
+every case with a violation into PARTIAL and hide every other difference behind
+that one.
+
+Exit codes, after the CRUX harness rule (aprender, contracts/crux-shacl-jena-v1):
+
+ 0 GREEN every planted control classified as planted, and no FALSE_CLEAN
+ 1 RED a FALSE_CLEAN, an AGREE_AGAINST_W3C, or a misclassified control
+ 2 NOT MEASURED no pyshacl, the wrong pyshacl, no engine, nothing discovered,
+ fewer pairs run than discovered, or no pair yielding a
+ comparison. Never green: before #320 every one of these exited
+ 0, so a run that compared nothing looked like a run that found
+ nothing.
+
+Planted controls run on EVERY invocation, before the corpus, so a run is never
+trusted on a comparator nobody has seen disagree. One must-agree pair goes
+through both real engines and must come out AGREE. Copies of pyshacl's report on
+it are then mutated (one result dropped; one result's severity, source shape and
+component changed in turn) and each copy must come out PARTIAL or FALSE_CLEAN.
+
+W3C arbitration. With --w3c, the cases are the vendored W3C SHACL suite and each
+carries the Working Group's expected report. When the two engines disagree the
+expected report says which one is wrong: we = W3C != pyshacl is ORACLE_WRONG
+(GREEN, recorded), we = pyshacl != W3C is AGREE_AGAINST_W3C (RED, because two
+engines agreeing on a wrong answer is the case a two-engine differential is
+blind to). A FALSE_CLEAN the W3C report confirms, on a test the conformance
+ratchet already records as failing, is KNOWN_FALSE_CLEAN and does not block
+(see `arbitrate`). Entries expecting sht:Failure carry no report and are listed
+as EXCLUDED. The Working Group's own `sh:value` is dropped from the key for the
+same reason as above.
+
+Why the CI job passes --w3c as well as --discover case-studies: measured on 24
+Oct 2026, discovery over case-studies/ finds 4 pairs and all 4 come back
+UNDETERMINED (two skip sh:xone / sh:not / sh:or, two match no focus node), so
+on its own the corpus compares nothing, and before #320 that run exited 0 and
+was the whole job.
+
+Every run writes a receipt into the JSON: the pyshacl and rdflib versions, the
+engine binary's sha256, and the sha256 of every input file, so a drift in the
+oracle and a change in the engine are told apart by a diff.
Usage:
python3 tools/shacl_differential.py CORPUS.json [--json out.json]
+ python3 tools/shacl_differential.py --discover case-studies [--json out.json]
+ python3 tools/shacl_differential.py --w3c tests/w3c-shacl [--json out.json]
+ python3 tools/shacl_differential.py --discover case-studies --w3c tests/w3c-shacl
-CORPUS.json is a list of {"name", "data": [ttl...], "shapes": ttl} objects, or
-pass --discover ROOT to build one from a tree laid out as */shacl/*.ttl or
+The sources combine; the exit rules apply to the whole run.
+
+CORPUS.json is a list of {"name", "data": [ttl...], "shapes": ttl} objects.
+--discover ROOT builds one from a tree laid out as */shacl/*.ttl or
*/shapes/*.ttl with data in */graph/*.ttl, */data/*.ttl or */examples/*.ttl.
+--pyshacl-version V makes any other installed version NOT MEASURED.
Nothing is capped silently: every pair excluded for size or timeout is listed in
-the report with the reason.
+the report with the reason, and makes the run NOT MEASURED.
"""
import argparse
+import hashlib
import json
import os
import pathlib
+import platform
import subprocess
import sys
import tempfile
@@ -43,12 +96,177 @@ BIN = os.environ.get(
MAX_BYTES = int(os.environ.get("SHACL_DIFF_MAX_BYTES", 40 * 1024 * 1024))
TIMEOUT = int(os.environ.get("SHACL_DIFF_TIMEOUT", 300))
+GREEN, RED, NOT_MEASURED = 0, 1, 2
+SH = "http://www.w3.org/ns/shacl#"
+RDF_NS = "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+W3C_BASE = "https://w3c.github.io/data-shapes/data-shapes-test-suite/tests/"
+# Verdicts that are a comparison. UNDETERMINED and ERROR are not, and a run made
+# only of them measured nothing.
+MEASURED = {"AGREE", "PARTIAL", "FALSE_CLEAN", "KNOWN_FALSE_CLEAN", "FALSE_ALARM", "ORACLE_WRONG",
+ "AGREE_AGAINST_W3C"}
+BLOCKING = ("FALSE_CLEAN", "AGREE_AGAINST_W3C")
+
+CONTROL_DATA = """\
+@prefix ex: <http://example.org/oo-control#> .
+ex:alice a ex:Person ; ex:age "forty" .
+ex:bob a ex:Person ; ex:name "Bob" ; ex:age 40 .
+"""
+# Two results, chosen to exercise every widened key field: a BLANK property
+# shape (keyed by parent + path), an IRI property shape, and a non-default
+# severity.
+CONTROL_SHAPES = """\
+@prefix ex: <http://example.org/oo-control#> .
+@prefix sh: <http://www.w3.org/ns/shacl#> .
+@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+ex:PersonShape a sh:NodeShape ;
+ sh:targetClass ex:Person ;
+ sh:property [ sh:path ex:name ; sh:minCount 1 ] ;
+ sh:property ex:AgeShape .
+ex:AgeShape a sh:PropertyShape ;
+ sh:path ex:age ;
+ sh:datatype xsd:integer ;
+ sh:severity sh:Warning .
+"""
+
+
+def sha256(path):
+ h = hashlib.sha256()
+ with open(path, "rb") as fh:
+ for chunk in iter(lambda: fh.read(1 << 20), b""):
+ h.update(chunk)
+ return h.hexdigest()
+
+
+def oracle_versions():
+ """pyshacl and rdflib versions, or None when pyshacl cannot be imported."""
+ try:
+ import pyshacl
+ import rdflib
+ except ImportError:
+ return None
+ return {"pyshacl": pyshacl.__version__, "rdflib": rdflib.__version__}
+
+
+# ── keys ─────────────────────────────────────────────────────────────────────
+
+
+def shape_key(parent, path):
+ return f"{parent} sh:property [sh:path {path}]"
+
+
+def severity_iri(s):
+ """The engine reports the local name of sh:severity; read a bare name as
+ being in the SHACL namespace, as tests/w3c_shacl_conformance_test.rs does."""
+ if not s:
+ return None
+ return s if ":" in s else SH + s
+
+
+def our_term(s):
+ """The engine prints IRIs bare and literals in N-Triples form; blank nodes
+ are not portable, so they collapse."""
+ if s is None or s == "":
+ return None
+ return "_:" if s.startswith("_:") else s
+
+
+def ours_keys(report):
+ out = set()
+ for v in report.get("violations", []) or []:
+ # Every violation passes through `attribute()`, which mirrors `path`
+ # into `result_path`, and a sh:sparql result populates only
+ # `result_path`, so reading it first is what makes SPARQL-based
+ # results comparable at all.
+ path = our_term(v.get("result_path") or v.get("path"))
+ shape = v.get("source_shape")
+ if shape and shape.startswith("_:"):
+ parent = v.get("node_shape")
+ shape = shape_key(parent, path) if parent and not parent.startswith("_:") and path else "_:"
+ out.add((
+ our_term(v.get("focus_node")),
+ path,
+ severity_iri(v.get("severity")),
+ our_term(shape),
+ our_term(v.get("source_constraint_component")),
+ ))
+ return out
+
+
+def rdf_term(t):
+ from rdflib import BNode, URIRef
+
+ if t is None:
+ return None
+ if isinstance(t, BNode):
+ return "_:"
+ if isinstance(t, URIRef):
+ return str(t)
+ return t.n3()
+
+
+def path_form(graph, t, depth=0):
+ """Structural rendering of a SHACL path, the same canonical form the Rust
+ harness uses: bare IRI, ^IRI, (a/b), (a|b), (p)* (p)+ (p)?. A well-formed
+ path is an acyclic tree, so equal renderings mean isomorphic paths."""
+ from rdflib import BNode, Namespace
+ from rdflib.namespace import RDF
+
+ sh = Namespace(SH)
+ if not isinstance(t, BNode):
+ return rdf_term(t)
+ if depth > 32:
+ return "(path too deep)"
+ sub = lambda x: path_form(graph, x, depth + 1) # noqa: E731
+ inv = graph.value(t, sh.inversePath)
+ if inv is not None:
+ return "^" + sub(inv)
+ alt = graph.value(t, sh.alternativePath)
+ if alt is not None:
+ return "(" + "|".join(sub(x) for x in graph.items(alt)) + ")"
+ for pred, op in (("zeroOrMorePath", "*"), ("oneOrMorePath", "+"), ("zeroOrOnePath", "?")):
+ x = graph.value(t, sh[pred])
+ if x is not None:
+ return f"({sub(x)}){op}"
+ if graph.value(t, RDF.first) is not None:
+ return "(" + "/".join(sub(x) for x in graph.items(t)) + ")"
+ return "_:"
+
+
+def rdf_keys(results_graph, results, shapes_graph):
+ """Keys for rdflib sh:result nodes. `shapes_graph` resolves a blank source
+ shape to its parent; pyshacl reports the shapes graph's own blank node."""
+ from rdflib import BNode, Namespace, URIRef
+
+ sh = Namespace(SH)
+ out = set()
+ for r in results:
+ p = results_graph.value(r, sh.resultPath)
+ path = None
+ if p is not None:
+ path = path_form(results_graph if (p, None, None) in results_graph else shapes_graph, p)
+ shape = results_graph.value(r, sh.sourceShape)
+ if isinstance(shape, BNode):
+ parents = list(shapes_graph.subjects(sh.property, shape))
+ spath = shapes_graph.value(shape, sh.path)
+ if len(parents) == 1 and isinstance(parents[0], URIRef) and spath is not None:
+ shape_s = shape_key(str(parents[0]), path_form(shapes_graph, spath))
+ else:
+ shape_s = "_:"
+ else:
+ shape_s = rdf_term(shape)
+ out.add((
+ rdf_term(results_graph.value(r, sh.focusNode)),
+ path,
+ rdf_term(results_graph.value(r, sh.resultSeverity)),
+ shape_s,
+ rdf_term(results_graph.value(r, sh.sourceConstraintComponent)),
+ ))
+ return out
+
+
+# ── engines ──────────────────────────────────────────────────────────────────
+
-# Every violation passes through `attribute()`, which mirrors `path` into
-# `result_path` and is the only field a sh:sparql result populates, so reading
-# `result_path` first is what makes SPARQL-based results comparable at all.
-# Keyed on `path` alone, two engines that agreed exactly printed PARTIAL for
-# every constraint that binds ?path, which would have masked a real difference.
def run_open_ontologies(data_files, shapes_file):
with tempfile.TemporaryDirectory() as d:
env = dict(os.environ, OPEN_ONTOLOGIES_STORAGE_MODE="persistent")
@@ -64,40 +282,44 @@ def run_open_ontologies(data_files, shapes_file):
capture_output=True, text=True, env=env, timeout=TIMEOUT,
)
try:
- return json.loads(r.stdout)
+ report = json.loads(r.stdout)
except json.JSONDecodeError:
return {"error": f"unparseable report: {(r.stdout + r.stderr).strip()[:300]}"}
+ report["keys"] = ours_keys(report)
+ return report
-def run_pyshacl(data_files, shapes_file):
+def run_pyshacl(data_files, shapes_file, originals=None):
+ """`originals` maps a rewritten N-Triples file back to (Turtle source,
+ base). pyshacl is handed the source: measured on the W3C suite, the
+ N-Triples rewrite drops the prefix bindings pyshacl resolves `sh:prefixes`
+ through, and `sparql/node/prefixes-001` became a pyshacl ERROR that the
+ original file does not cause."""
from pyshacl import validate
- from rdflib import Graph
+ from rdflib import Graph, Namespace
+
+ sh = Namespace(SH)
+ originals = originals or {}
+
+ def parse(graph, f):
+ src, base = originals.get(str(f), (str(f), None))
+ return graph.parse(src, format="turtle", publicID=base) if base else graph.parse(src)
data = Graph()
for f in data_files:
- data.parse(str(f))
- shapes = Graph().parse(str(shapes_file))
+ parse(data, f)
+ shapes = parse(Graph(), shapes_file)
conforms, results_graph, _ = validate(data, shacl_graph=shapes)
- violations = set()
- from rdflib.namespace import Namespace
- SH = Namespace("http://www.w3.org/ns/shacl#")
# Every severity, not just sh:Violation. Collecting only Violations here
# compared pyshacl's violations against our results at ALL severities, and
# reported the difference as our defect: four sh:Warning results in the
# enterprise-knowledge vertical showed up as four extra violations we had
# invented. An oracle that manufactures findings is worse than no oracle.
- for result in set(results_graph.subjects(SH.resultSeverity, None)):
- focus = results_graph.value(result, SH.focusNode)
- path = results_graph.value(result, SH.resultPath)
- violations.add((str(focus), str(path) if path else None))
- return {"conforms": conforms, "violations": violations}
+ results = set(results_graph.subjects(sh.resultSeverity, None))
+ return {"conforms": conforms, "keys": rdf_keys(results_graph, results, shapes)}
-def ours_violation_set(report):
- out = set()
- for v in report.get("violations", []) or []:
- out.add((str(v.get("focus_node")), v.get("result_path") or v.get("path")))
- return out
+# ── classification ───────────────────────────────────────────────────────────
def classify(ours, theirs):
@@ -115,19 +337,91 @@ def classify(ours, theirs):
return "UNDETERMINED", ours.get("warning", "no verdict")
if our_conforms is True and their_conforms is False:
- return "FALSE_CLEAN", f"pyshacl found {len(theirs['violations'])} violation(s), we found none"
+ return "FALSE_CLEAN", f"pyshacl found {len(theirs['keys'])} result(s), we found none"
if our_conforms is False and their_conforms is True:
- return "FALSE_ALARM", f"we reported {len(ours_violation_set(ours))} violation(s), pyshacl found none"
+ return "FALSE_ALARM", f"we reported {len(ours['keys'])} result(s), pyshacl found none"
- ours_v, theirs_v = ours_violation_set(ours), theirs["violations"]
- missed, extra = theirs_v - ours_v, ours_v - theirs_v
+ missed, extra = theirs["keys"] - ours["keys"], ours["keys"] - theirs["keys"]
if missed or extra:
return "PARTIAL", f"{len(missed)} missed, {len(extra)} extra (verdict agrees)"
- return "AGREE", f"{len(ours_v)} violation(s), identical sets"
+ return "AGREE", f"{len(ours['keys'])} result(s), identical sets"
+
+
+def same(a, b):
+ return a["conforms"] == b["conforms"] and a["keys"] == b["keys"]
+
+
+def arbitrate(verdict, detail, ours, theirs, expected, baseline):
+ """Let the W3C expected report decide a disagreement, or flag an agreement
+ it contradicts. Only called where the engine gave a verdict.
+
+ A FALSE_CLEAN the W3C report confirms, on a test the conformance ratchet
+ (tests/w3c_shacl_baseline.json) does not record as PASS, is KNOWN_FALSE_CLEAN:
+ reported on every run, and not blocking here, because the `w3c-shacl` job
+ already fails if that FAIL count rises or any recorded PASS regresses.
+ Measured 24 Oct 2026: 12 of the 113 comparable W3C tests, every one a
+ `conforms: true` where the suite expects false, and every one already FAIL
+ in that baseline. On a test the baseline records as PASS it stays FALSE_CLEAN
+ and blocks, since that is a regression the ratchet has not seen."""
+ if verdict == "AGREE":
+ if not same(theirs, expected):
+ return "AGREE_AGAINST_W3C", "both engines agree and the W3C expected report does not"
+ return verdict, detail
+ if same(ours, expected):
+ return "ORACLE_WRONG", f"W3C sides with us against pyshacl ({verdict}: {detail})"
+ side = "W3C sides with pyshacl" if same(theirs, expected) else "W3C matches neither engine"
+ if verdict == "FALSE_CLEAN" and baseline not in (None, "PASS"):
+ return "KNOWN_FALSE_CLEAN", f"{detail}; {side}; held by the w3c-shacl ratchet as {baseline}"
+ return verdict, f"{detail}; {side}"
+
+
+# ── planted controls ─────────────────────────────────────────────────────────
+
+
+def run_controls(workdir):
+ """Plant the controls and return (rows, failures). A failure is a control
+ whose classification is not the one it was planted to have."""
+ data = pathlib.Path(workdir) / "control-data.ttl"
+ shapes = pathlib.Path(workdir) / "control-shapes.ttl"
+ data.write_text(CONTROL_DATA)
+ shapes.write_text(CONTROL_SHAPES)
+ ours = run_open_ontologies([data], shapes)
+ theirs = run_pyshacl([data], shapes)
+
+ rows, failures = [], []
+
+ def plant(name, planted_theirs, allowed):
+ verdict, detail = classify(ours, planted_theirs)
+ ok = verdict in allowed
+ rows.append({"control": name, "verdict": verdict, "expected": sorted(allowed),
+ "detail": detail, "ok": ok})
+ if not ok:
+ failures.append(f"control `{name}` came out {verdict}, planted to be "
+ f"{' or '.join(sorted(allowed))}: {detail}")
+
+ plant("must-agree", theirs, {"AGREE"})
+ keys = sorted(theirs["keys"], key=lambda k: tuple(x or "" for x in k))
+ if len(keys) < 2:
+ # Without two results the mutations below cannot be planted at all, and
+ # a control that is not planted is not a control.
+ failures.append(f"control pair yielded {len(keys)} pyshacl result(s), need 2 to plant mutations")
+ return rows, failures
+ victim = keys[0]
+ plant("one-result-dropped", {**theirs, "keys": set(keys[1:])}, {"PARTIAL", "FALSE_CLEAN"})
+ for field, i in (("severity", 2), ("source-shape", 3), ("component", 4)):
+ changed = list(victim)
+ changed[i] = f"{changed[i]}-planted"
+ plant(f"{field}-changed", {**theirs, "keys": set(keys[1:]) | {tuple(changed)}}, {"PARTIAL"})
+ return rows, failures
+
+
+# ── discovery ────────────────────────────────────────────────────────────────
def discover(root):
cases, root = [], pathlib.Path(root)
+ if not root.is_dir():
+ return cases
for repo in sorted(p for p in root.iterdir() if p.is_dir()):
shapes = sorted(list((repo / "shacl").glob("*.ttl")) + list((repo / "shapes").glob("*.ttl")))
if not shapes:
@@ -143,57 +437,216 @@ def discover(root):
return cases
-def main():
+def discover_w3c(suite, workdir):
+ """Walk the vendored W3C suite from its root manifest, the walk
+ tests/w3c_shacl_conformance_test.rs does. Each case's graphs are re-written
+ as N-Triples with the suite's published base, because every file uses
+ relative IRIs and `open-ontologies shacl` takes no base IRI. Entries whose
+ mf:result is sht:Failure carry no report to compare and are returned as
+ excluded, by name."""
+ from rdflib import Graph, Namespace
+ from rdflib.namespace import RDF
+
+ mf = Namespace("http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#")
+ sht = Namespace("http://www.w3.org/ns/shacl-test#")
+ sh = Namespace(SH)
+ root = os.path.realpath(suite)
+ if not os.path.isfile(os.path.join(root, "manifest.ttl")):
+ return [], []
+
+ def base_of(p):
+ return W3C_BASE + os.path.relpath(p, root)
+
+ def path_of(iri):
+ return os.path.join(root, str(iri)[len(W3C_BASE):])
+
+ def load(p):
+ return Graph().parse(p, format="turtle", publicID=base_of(p))
+
+ # The conformance ratchet's per-test outcomes. A W3C test it does not record
+ # as PASS is an engine defect that ratchet already holds (its FAIL count may
+ # not rise), so a FALSE_CLEAN there is KNOWN rather than new; see arbitrate.
+ baseline_file = pathlib.Path(root).parent / "w3c_shacl_baseline.json"
+ baseline = json.loads(baseline_file.read_text())["outcomes"] if baseline_file.is_file() else {}
+
+ cases, excluded, seen = [], [], set()
+ out = pathlib.Path(workdir) / "w3c"
+
+ def absolute(p):
+ dest = out / (os.path.relpath(p, root) + ".nt")
+ if not dest.exists():
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ load(p).serialize(destination=str(dest), format="nt")
+ return str(dest)
+
+ def walk(path):
+ path = os.path.realpath(path)
+ if path in seen:
+ return
+ seen.add(path)
+ g = load(path)
+ name = os.path.relpath(path, root)[: -len(".ttl")]
+ for manifest in g.subjects(RDF.type, mf.Manifest):
+ for entries in g.objects(manifest, mf.entries):
+ for entry in g.items(entries):
+ if (entry, RDF.type, sht.Validate) not in g:
+ continue
+ result = g.value(entry, mf.result)
+ if result == sht.Failure:
+ excluded.append({"name": "w3c/" + name, "reason": "expects sht:Failure, not a report"})
+ continue
+ action = g.value(entry, mf.action)
+ data = path_of(g.value(action, sht.dataGraph))
+ shapes = path_of(g.value(action, sht.shapesGraph))
+ expected = {
+ "conforms": bool(g.value(result, sh.conforms).toPython()),
+ "keys": rdf_keys(g, set(g.objects(result, sh.result)), g),
+ }
+ nt_data, nt_shapes = absolute(data), absolute(shapes)
+ cases.append({"name": "w3c/" + name, "data": [nt_data], "shapes": nt_shapes,
+ "originals": {nt_data: (data, base_of(data)),
+ nt_shapes: (shapes, base_of(shapes))},
+ "inputs": sorted({path, data, shapes}), "expected": expected,
+ "baseline": baseline.get(name)})
+ for inc in sorted(g.objects(None, mf.include)):
+ walk(path_of(inc))
+
+ walk(os.path.join(root, "manifest.ttl"))
+ cases.sort(key=lambda c: c["name"])
+ return cases, excluded
+
+
+# ── main ─────────────────────────────────────────────────────────────────────
+
+
+def not_measured(why, report, json_path):
+ print(f"\nNOT MEASURED: {why}\nThis is not agreement between the engines; it is the absence "
+ "of a comparison, and it exits 2.", flush=True)
+ report.update({"status": "NOT_MEASURED", "exit": NOT_MEASURED, "why": why})
+ if json_path:
+ with open(json_path, "w") as fh:
+ json.dump(report, fh, indent=1, default=sorted)
+ return NOT_MEASURED
+
+
+def main(argv=None):
ap = argparse.ArgumentParser()
ap.add_argument("corpus", nargs="?")
ap.add_argument("--discover")
+ ap.add_argument("--w3c")
ap.add_argument("--json")
ap.add_argument("--limit", type=int)
- args = ap.parse_args()
-
- cases = discover(args.discover) if args.discover else json.load(open(args.corpus))
- if args.limit:
- cases = cases[: args.limit]
-
- rows, skipped_pairs = [], []
- for case in cases:
- data = [f for f in case["data"] if os.path.exists(f)]
- size = sum(os.path.getsize(f) for f in data) + os.path.getsize(case["shapes"])
- if size > MAX_BYTES:
- reason = f"{size/1e6:.0f} MB over the {MAX_BYTES/1e6:.0f} MB cap"
- skipped_pairs.append({"name": case["name"], "reason": reason})
- print(f"{'NOT_RUN':<13} {case['name']}\n {reason}", flush=True)
- continue
- try:
- ours = run_open_ontologies(data, case["shapes"])
- theirs = run_pyshacl(data, case["shapes"])
- verdict, detail = classify(ours, theirs)
- except subprocess.TimeoutExpired:
- reason = f"timed out after {TIMEOUT}s"
- skipped_pairs.append({"name": case["name"], "reason": reason})
- print(f"{'NOT_RUN':<13} {case['name']}\n {reason}", flush=True)
- continue
- except Exception as exc: # a parse failure in either engine is a result, not a crash
- verdict, detail = "ERROR", f"{type(exc).__name__}: {exc}"[:300]
- rows.append({"name": case["name"], "verdict": verdict, "detail": detail})
- print(f"{verdict:<13} {case['name']}\n {detail}", flush=True)
+ ap.add_argument("--pyshacl-version")
+ args = ap.parse_args(argv)
+
+ report = {"receipt": {"python": platform.python_version(), "engine": {"path": BIN}}}
+ versions = oracle_versions()
+ if versions is None:
+ return not_measured("pyshacl is not importable; there is no oracle", report, args.json)
+ report["receipt"].update(versions)
+ print(f"oracle: pyshacl {versions['pyshacl']}, rdflib {versions['rdflib']}", flush=True)
+ if args.pyshacl_version and versions["pyshacl"] != args.pyshacl_version:
+ return not_measured(f"pyshacl {versions['pyshacl']} is installed and {args.pyshacl_version} "
+ "is pinned; a different oracle is a different measurement", report, args.json)
+ if not os.path.isfile(BIN):
+ return not_measured(f"no engine binary at {BIN}", report, args.json)
+ report["receipt"]["engine"]["sha256"] = sha256(BIN)
+
+ with tempfile.TemporaryDirectory() as workdir:
+ if not (args.w3c or args.discover or args.corpus):
+ ap.error("give a CORPUS.json, --discover ROOT and/or --w3c SUITE")
+ cases, excluded = [], []
+ if args.corpus:
+ cases += json.load(open(args.corpus))
+ if args.discover:
+ cases += discover(args.discover)
+ if args.w3c:
+ w3c_cases, excluded = discover_w3c(args.w3c, workdir)
+ cases += w3c_cases
+ if args.limit:
+ cases = cases[: args.limit]
+ report["excluded"] = excluded
+ for e in excluded:
+ print(f"{'EXCLUDED':<17} {e['name']}\n {e['reason']}", flush=True)
+
+ inputs = set()
+ for case in cases:
+ inputs.update(case.get("inputs") or (list(case["data"]) + [case["shapes"]]))
+ report["receipt"]["inputs"] = {f: sha256(f) for f in sorted(inputs) if os.path.isfile(f)}
+ report["discovered"] = len(cases)
+
+ controls, control_failures = run_controls(workdir)
+ report["controls"] = controls
+ for c in controls:
+ print(f"CONTROL {'ok ' if c['ok'] else 'BAD'} {c['control']:<22} {c['verdict']}", flush=True)
+
+ if not cases:
+ return not_measured("discovery found zero (data, shapes) pairs", report, args.json)
+
+ rows, skipped_pairs = [], []
+ for case in cases:
+ data = [f for f in case["data"] if os.path.exists(f)]
+ size = sum(os.path.getsize(f) for f in data) + os.path.getsize(case["shapes"])
+ if size > MAX_BYTES:
+ reason = f"{size/1e6:.0f} MB over the {MAX_BYTES/1e6:.0f} MB cap"
+ skipped_pairs.append({"name": case["name"], "reason": reason})
+ print(f"{'NOT_RUN':<17} {case['name']}\n {reason}", flush=True)
+ continue
+ try:
+ ours = run_open_ontologies(data, case["shapes"])
+ theirs = run_pyshacl(data, case["shapes"], case.get("originals"))
+ verdict, detail = classify(ours, theirs)
+ if "expected" in case and verdict not in ("ERROR", "UNDETERMINED"):
+ verdict, detail = arbitrate(verdict, detail, ours, theirs, case["expected"],
+ case.get("baseline"))
+ except subprocess.TimeoutExpired:
+ reason = f"timed out after {TIMEOUT}s"
+ skipped_pairs.append({"name": case["name"], "reason": reason})
+ print(f"{'NOT_RUN':<17} {case['name']}\n {reason}", flush=True)
+ continue
+ except Exception as exc: # a parse failure in either engine is a result, not a crash
+ verdict, detail = "ERROR", f"{type(exc).__name__}: {exc}"[:300]
+ rows.append({"name": case["name"], "verdict": verdict, "detail": detail})
+ print(f"{verdict:<17} {case['name']}\n {detail}", flush=True)
counts = {}
for r in rows:
counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1
+ measured = sum(n for k, n in counts.items() if k in MEASURED)
+ report.update({"rows": rows, "not_run": skipped_pairs, "counts": counts, "measured": measured})
print("\n" + "=" * 70)
- print(f"{len(rows)} pair(s) run, {len(skipped_pairs)} not run")
- for k in ("FALSE_CLEAN", "FALSE_ALARM", "PARTIAL", "UNDETERMINED", "AGREE", "ERROR"):
+ print(f"{len(cases)} pair(s) discovered, {len(rows)} run, {len(skipped_pairs)} not run, "
+ f"{measured} compared")
+ for k in ("FALSE_CLEAN", "AGREE_AGAINST_W3C", "KNOWN_FALSE_CLEAN", "FALSE_ALARM", "PARTIAL",
+ "ORACLE_WRONG",
+ "UNDETERMINED", "AGREE", "ERROR"):
if k in counts:
- print(f" {k:<13} {counts[k]}")
+ print(f" {k:<17} {counts[k]}")
for s in skipped_pairs:
print(f" not run: {s['name']} - {s['reason']}")
- if args.json:
- json.dump({"rows": rows, "not_run": skipped_pairs, "counts": counts}, open(args.json, "w"), indent=1)
+ if control_failures:
+ code, status = RED, "RED"
+ for f in control_failures:
+ print(f"\nRED: {f}")
+ print("A comparator that misclassifies a planted difference cannot be trusted on the "
+ "corpus, whatever the corpus said.")
+ elif any(counts.get(k) for k in BLOCKING):
+ code, status = RED, "RED"
+ elif skipped_pairs:
+ return not_measured(f"{len(skipped_pairs)} of {len(cases)} discovered pair(s) were not run",
+ report, args.json)
+ elif measured == 0:
+ return not_measured(f"none of the {len(rows)} pair(s) run yielded a comparison "
+ "(every one ERROR or UNDETERMINED)", report, args.json)
+ else:
+ code, status = GREEN, "GREEN"
- # A false clean is the one result that must fail a pipeline.
- return 1 if counts.get("FALSE_CLEAN") else 0
+ report.update({"status": status, "exit": code})
+ if args.json:
+ with open(args.json, "w") as fh:
+ json.dump(report, fh, indent=1, default=sorted)
+ return code
if __name__ == "__main__":