Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

Make the gate that checks the proofs able to run, and the claims about it true

FABIOTESS0a6d9a0bbef4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 49181ae4..696eb94d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -20,6 +20,77 @@ All notable changes to Open Ontologies are documented here.
trace. See docs/lean-certificates.md and decision 0002.
### Fixed
+- **`sh:sparql` ignored `sh:prefixes` and merged every `sh:declare` in the
+ shapes graph into one prologue.** A prefix bound to two namespaces emitted
+ two `PREFIX` lines, SPARQL took the last, and which one won was decided by
+ store row order: the constraint pointing at the loser matched nothing and the
+ run reported `conforms: true` with nothing in `skipped_constraints`. Two
+ shapes files that were the same RDF graph could give opposite verdicts. A
+ false clean, in the function the `#132` fix sits next to. Declarations are
+ now scoped per constraint through `sh:prefixes`, `owl:imports*` and
+ `sh:declare` as SHACL 5.2.1 says; a constraint that names no set keeps the
+ permissive whole-graph merge, but an ambiguous merge is refused with a
+ recorded reason rather than guessed.
+- **`sh:deactivated true` was ignored on a `sh:sparql` constraint node.** The
+ constraint ran, its rows were reported, and the run returned a confident
+ `conforms: false` where SHACL 5.3 says there are no results. The predicate
+ was already honoured on node shapes and property shapes. There was also no
+ unimplemented-predicate complement over constraint nodes at all, so any other
+ `sh:` predicate written there was invisible; there is one now.
+- **Any `sh:select`, `sh:pattern` or `sh:message` containing a typed or
+ language-tagged literal was truncated.** The helper that strips a literal's
+ quoting searched the whole string for `^^` and `"@` and cut there, so
+ `FILTER(?d < "2025-01-01"^^xsd:date)` was chopped mid-query. The constraint
+ then failed to parse and was recorded as unrunnable with an error blaming the
+ author, leaving `conforms: null`, zero violations and exit 0. Date and
+ numeric comparisons are the most common SPARQL constraints there are. The
+ value is now read with the RDF parser instead of by string surgery.
+- **A blank-node node shape applied every property shape in the file to its own
+ targets.** The shape was spliced into the query text, and `_:label` in a
+ SPARQL body is a non-distinguished variable, so `[] a sh:NodeShape` meant
+ "anything that has a property shape". Two such shapes with disjoint targets
+ reported conforming data as non-conforming. Shape terms are now bound by
+ substitution, the same mechanism `$this` pre-binding uses.
+- **`source_shape` named the enclosing node shape rather than the shape
+ carrying the constraint**, for all fifteen property-borne constraint sites.
+ Two property shapes on one path under one node shape therefore stayed
+ indistinguishable, which is the case #131 was filed about. pyshacl returns
+ the property shape and this now agrees. The node shape is still reported,
+ under `node_shape`.
+- **Duplicate results.** One shape carrying two target declarations that select
+ the same node reported every violation on that node twice. A validation
+ report is a set.
+- **The reasoner was not a fixpoint of its own rule set.** Only the `rdf:type`,
+ `rdfs:subClassOf` and `rdfs:subPropertyOf` indices were rebuilt each
+ iteration; every schema index was filtered once out of the run-start
+ snapshot, so a schema triple the reasoner itself derived was never used and
+ running `reason` twice derived more than running it once. That matters most
+ for certificates: materialising turns one run's conclusions into the next
+ run's premises, so `asserted.tsv` could list the reasoner's own output as an
+ axiom.
+- **Four rules could conclude a triple with a literal subject.** `prp-symp`,
+ `prp-inv1`, `prp-inv2` and `eq-sym` take their conclusion's subject from an
+ object position, so a literal object produced a triple no serialisation can
+ express. Materialisation then failed on the whole batch, losing every
+ inference including the sound ones, at a line number that moved between runs
+ because it depends on hash iteration order.
+- **`cls-hv1` was not W3C's `cls-hv1`.** It was the composite of `cax-sco` and
+ `cls-hv1`: it required an explicit `rdfs:subClassOf` hop, so an individual
+ typed directly with the restriction derived nothing, while the certificate
+ put a W3C rule name in front of the reader. The W3C form is used now and
+ loses nothing, since the composite case is `rdfs9` followed by it.
+- **`rdfs3` dropped every range inference onto a blank node.** The guard
+ required an IRI when the invariant it needs is "not a literal", so a
+ blank-node value never got typed and `rdfs9` starved behind it.
+- **`oo-cert` exited 1 rather than the documented 2 when a certificate file
+ could not be read**, so a harness written to the contract reported an
+ unreadable file as a rejected certificate.
+- **The `lean` CI job could not pass.** The `lake --version` probe ran from the
+ crate root, the one directory with no `lean-toolchain` in its ancestry, and
+ `leanprover/lean-action` installs elan with no default toolchain. The probe
+ reported lake as missing and `OO_REQUIRE_FIXTURES=1` turned that into five
+ panics. Every local run was green because a developer machine has a default
+ toolchain.
- **`sh:sparql` reported `conforms: true` when any single focus node
conformed (#132).** The author's SELECT was wrapped as a subquery under a
`VALUES ?this` clause. A subquery is evaluated bottom-up with no outer
diff --git a/README.md b/README.md
index a7c52f2e..eb4d21ca 100644
--- a/README.md
+++ b/README.md
@@ -1003,7 +1003,8 @@ read. `lean/` holds a checker for that certificate whose soundness is a
machine-checked theorem (`OOCert.certificate_sound`, core Lean, no Mathlib), so
a run whose certificate checks contains only triples entailed by the asserted
graph, whatever this engine did to find them. CI builds the proofs and
-certifies every RDF file in this repository. The first thing the checker caught
+certifies every RDF file in this repository, naming with a reason any it cannot
+read or that exceeds the size cap. The first thing the checker caught
was in the engine: `cls-svf1` derived the converse of a subclass axiom. Details
in [docs/lean-certificates.md](docs/lean-certificates.md) and
[decision 0002](docs/decisions/0002-an-inference-carries-a-certificate.md).
diff --git a/docs/lean-certificates.md b/docs/lean-certificates.md
index 6313a99b..cd06b4fa 100644
--- a/docs/lean-certificates.md
+++ b/docs/lean-certificates.md
@@ -114,6 +114,13 @@ interpretation of the premises plus the single consequence the semantics does fo
## What it caught on day one
+An adversarial audit of the whole layer on 13 September 2026 found nine more defects, seven of them
+predating the certificate work: a false clean from unscoped prefix declarations, an ignored
+`sh:deactivated`, a truncation that silently disabled any constraint mentioning a typed literal, a
+blank-node shape acting as a wildcard, a reasoner that was not a fixpoint, four rules that could
+emit an unserialisable triple, and a CI job that could not pass. All are fixed and pinned; the
+CHANGELOG lists them.
+
`cls-svf1` in the `owl-rl-ext` profile derived `x rdf:type C` from `C rdfs:subClassOf ∃p.D`,
`x p y` and `y rdf:type D`. That is the converse of the axiom. It also treated `x p D`, with `D` the
filler class IRI itself, as a witness. Neither has a sound rule, so neither could be given one in
@@ -123,8 +130,29 @@ kept in `tests/lean_certificate_test.rs` as a forged certificate the checker mus
## In CI
The `lean` job builds `lean/` (which is the proof check), then runs
-`tests/lean_certificate_test.rs` with `OO_REQUIRE_FIXTURES=1`: every RDF file under
-`case-studies/`, `demo/`, `tests/fixtures/`, `data/` and `examples/` is loaded, reasoned under
-`owl-rl-ext` with a certificate, and the certificate checked. Files over 4 MB and files that do not
-parse are listed with the reason, never dropped silently. The same test appends three forgeries
-and requires each to be rejected.
+`tests/lean_certificate_test.rs` with `OO_REQUIRE_FIXTURES=1`. Every RDF file the repository
+tracks, enumerated from `git ls-files`, is loaded, reasoned under `owl-rl-ext` with a certificate,
+and the certificate checked: 122 files and 37,133 derivations at the time of writing. Files over
+4 MB and files that do not parse are listed with the reason, never dropped silently. The same test
+appends three forgeries and requires each to be rejected.
+
+The corpus used to be five hand-named directories, three of which hold no RDF, so 46% of the
+repository's RDF was never walked and was excluded without being named. Widening it is what
+surfaced the literal-subject defect in `prp-symp`, `prp-inv1`, `prp-inv2` and `eq-sym`, which lived
+in `benchmark/`.
+
+## Known limitations
+
+Stated rather than discovered later.
+
+- **The rule table is this engine's, not W3C's.** `by_rule` describes what this engine did. It is
+ not an OWL 2 RL conformance claim, and the engine does not implement every rule in the profile.
+- **`asserted.tsv` is the store, not your file.** Quads are flattened, so a triple present in two
+ named graphs appears on two lines. Literals are in the store's post-parse canonical spelling, so
+ `"01"^^xsd:integer` is written `"1"^^xsd:integer`. The guarantee is relative to that file.
+- **Reasoning twice into one store.** The run now reaches a fixpoint, so a second run adds nothing,
+ but if you materialise into a store that already held inferences they appear in `asserted.tsv` as
+ assumptions with nothing marking them derived. Use `inference_graph: true` (decision 0001) when
+ that distinction matters.
+- **`focus_nodes` in a SHACL report is a sum over target declarations**, so a node selected by two
+ declarations of one shape is counted twice. The violations themselves are deduplicated.
diff --git a/lean/Main.lean b/lean/Main.lean
index f3fba0ec..22b5d7f4 100644
--- a/lean/Main.lean
+++ b/lean/Main.lean
@@ -35,11 +35,27 @@ def jsonStr (s : String) : String :=
def showTriple (t : Triple) : String := s!"{t.s} {t.p} {t.o}"
+/-- Read a file, or report why not. A read failure is exit 2, the code reserved
+for "a file could not be read or parsed"; letting the exception escape `main`
+exits 1, which is the code that means "the checker rejected a step". A harness
+written to the documented contract would then report an unreadable file as a
+failed certificate, which is the wrong alarm in the wrong direction. -/
+def readOrFail (path : String) : IO (Except UInt32 String) := do
+ try
+ return .ok (← IO.FS.readFile path)
+ catch e =>
+ IO.eprintln s!"cannot read {path}: {e}"
+ return .error 2
+
def main (args : List String) : IO UInt32 := do
match args with
| [gPath, dPath] =>
- let g ← IO.FS.readFile gPath
- let d ← IO.FS.readFile dPath
+ let g ← match ← readOrFail gPath with
+ | .ok s => pure s
+ | .error c => return c
+ let d ← match ← readOrFail dPath with
+ | .ok s => pure s
+ | .error c => return c
match Parse.parseTriples g, Parse.parseSteps d with
| .ok G, .ok steps =>
if checkCert G steps then
diff --git a/tests/lean_certificate_test.rs b/tests/lean_certificate_test.rs
index 4a984ea4..7276a70e 100644
--- a/tests/lean_certificate_test.rs
+++ b/tests/lean_certificate_test.rs
@@ -36,8 +36,20 @@ fn lean_dir() -> PathBuf {
}
fn lake_available() -> bool {
+ // `.current_dir(lean_dir())` is not cosmetic. elan resolves the toolchain
+ // from the working directory's `lean-toolchain`, and the crate root is the
+ // one directory in this repository with none in its ancestry. Run from
+ // there against an elan that has no default toolchain, which is exactly
+ // what `leanprover/lean-action` installs, `lake --version` exits non-zero
+ // with "no default toolchain configured". The probe then reported lake as
+ // missing, `skip()` returned false, and `OO_REQUIRE_FIXTURES=1` turned
+ // that into five panics: the CI job that gates the proofs could not pass,
+ // while every local run was green because a developer machine has a
+ // default toolchain. `checker()` below already did this correctly, which
+ // is why the real `lake build` would have worked.
Command::new("lake")
.arg("--version")
+ .current_dir(lean_dir())
.output()
.map(|o| o.status.success())
.unwrap_or(false)
@@ -239,33 +251,49 @@ fn owl_dl_refuses_to_pretend_it_has_a_certificate() {
assert!(err.to_string().contains("no derivation certificate"), "{err}");
}
-/// Everything under these roots that parses as RDF, up to the size cap.
-const ROOTS: [&str; 5] = ["case-studies", "demo", "tests/fixtures", "data", "examples"];
+/// Every RDF file the repository tracks, rather than a hand-picked list of
+/// directories.
+///
+/// The list used to be five named directories, three of which hold no RDF at
+/// all, so 59 of the 128 tracked RDF files were never walked, and they were
+/// excluded SILENTLY. That contradicted this file's own promise that anything
+/// excluded is named with the reason, and the README's claim that CI certifies
+/// every RDF file in the repository. Walking from the root makes both true, and
+/// it is what surfaced the literal-subject defect in `prp-symp` and friends
+/// (`tests/reason_literal_subject_test.rs`), which lived in `benchmark/`, a
+/// directory the old list did not name.
const SIZE_CAP: u64 = 4 * 1024 * 1024;
-fn walk(dir: &Path, out: &mut Vec<PathBuf>) {
- let Ok(entries) = std::fs::read_dir(dir) else { return };
- for entry in entries.flatten() {
- let path = entry.path();
- let name = entry.file_name().to_string_lossy().to_string();
- if path.is_dir() {
- // Tooling trees that live beside the corpus locally but are not
- // part of it: a Python venv drops pyshacl's own shacl.ttl under
- // site-packages, and it certified happily, but it is not ours.
- if matches!(
- name.as_str(),
- "node_modules" | "target" | ".lake" | ".git" | ".venv" | "venv" | "site-packages" | "__pycache__"
- ) {
- continue;
- }
- walk(&path, out);
- } else if matches!(
- path.extension().and_then(|e| e.to_str()),
- Some("ttl" | "owl" | "rdf" | "nt")
- ) {
- out.push(path);
- }
- }
+/// Build output, dependency trees and virtual environments. Each holds RDF that
+/// is not this repository's to certify: a Python venv drops pyshacl's own
+/// `shacl.ttl` under site-packages, and it certified happily, but it is theirs.
+const SKIP_DIRS: [&str; 8] = [
+ "node_modules", "target", ".lake", ".git", ".venv", "venv", "site-packages", "__pycache__",
+];
+
+/// The RDF files this repository actually contains, from `git ls-files`.
+///
+/// Enumerating from git rather than from the filesystem is what makes "every
+/// RDF file in this repository" a statement with one meaning. A filesystem walk
+/// sees whatever a developer has generated locally as well, so it takes 190
+/// LUBM files nobody committed on one machine and none on a fresh clone, and
+/// the run stops being comparable between them. It is also minutes slower for
+/// no extra coverage.
+fn corpus() -> Vec<PathBuf> {
+ let out = Command::new("git")
+ .args(["ls-files", "-z", "*.ttl", "*.owl", "*.rdf", "*.nt"])
+ .current_dir(repo())
+ .output()
+ .expect("run git ls-files");
+ assert!(out.status.success(), "git ls-files failed: {}", String::from_utf8_lossy(&out.stderr));
+ let mut files: Vec<PathBuf> = String::from_utf8_lossy(&out.stdout)
+ .split('\0')
+ .filter(|p| !p.is_empty())
+ .filter(|p| !SKIP_DIRS.iter().any(|d| p.split('/').any(|seg| seg == *d)))
+ .map(|p| repo().join(p))
+ .collect();
+ files.sort();
+ files
}
#[test]
@@ -273,12 +301,8 @@ fn every_shipped_ontology_certifies() {
if skip() {
return;
}
- let mut files = Vec::new();
- for root in ROOTS {
- walk(&repo().join(root), &mut files);
- }
- files.sort();
- assert!(files.len() >= 40, "expected the shipped corpus, found {} files", files.len());
+ let files = corpus();
+ assert!(files.len() >= 120, "expected the whole corpus, found {} files", files.len());
let mut excluded: Vec<(String, String)> = Vec::new();
let mut failures: Vec<(String, String)> = Vec::new();
@@ -324,7 +348,7 @@ fn every_shipped_ontology_certifies() {
for (f, why) in &excluded {
eprintln!("EXCLUDED {f}: {why}");
}
- assert!(certified >= 30, "too few files certified ({certified}); excluded: {excluded:?}");
+ assert!(certified >= 100, "too few files certified ({certified}); excluded: {excluded:?}");
assert!(total_derivations > 0, "the corpus produced no inferences at all, so nothing was checked");
assert!(
failures.is_empty(),
diff --git a/tools/shacl_differential.py b/tools/shacl_differential.py
index a9abf7a2..b13594cd 100644
--- a/tools/shacl_differential.py
+++ b/tools/shacl_differential.py
@@ -44,6 +44,11 @@ MAX_BYTES = int(os.environ.get("SHACL_DIFF_MAX_BYTES", 40 * 1024 * 1024))
TIMEOUT = int(os.environ.get("SHACL_DIFF_TIMEOUT", 300))
+# Every violation passes through `attribute()`, which mirrors `path` into
+# `result_path` and is the only field a sh:sparql result populates, so reading
+# `result_path` first is what makes SPARQL-based results comparable at all.
+# Keyed on `path` alone, two engines that agreed exactly printed PARTIAL for
+# every constraint that binds ?path, which would have masked a real difference.
def run_open_ontologies(data_files, shapes_file):
with tempfile.TemporaryDirectory() as d:
env = dict(os.environ, OPEN_ONTOLOGIES_STORAGE_MODE="persistent")
@@ -91,7 +96,7 @@ def run_pyshacl(data_files, shapes_file):
def ours_violation_set(report):
out = set()
for v in report.get("violations", []) or []:
- out.add((str(v.get("focus_node")), v.get("path")))
+ out.add((str(v.get("focus_node")), v.get("result_path") or v.get("path")))
return out