Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

Derive the comparison number instead of typing it, and refresh two stale claims

FABIOTESS0d63e9abf31e
diff --git a/docs/decisions/0003-a-rule-is-data-and-an-assumption-is-not-a-fact.md b/docs/decisions/0003-a-rule-is-data-and-an-assumption-is-not-a-fact.md
index 2fa9b7f5..8383580c 100644
--- a/docs/decisions/0003-a-rule-is-data-and-an-assumption-is-not-a-fact.md
+++ b/docs/decisions/0003-a-rule-is-data-and-an-assumption-is-not-a-fact.md
@@ -1,10 +1,9 @@
# 0003 · A rule is data, and an assumption is not a fact
- **Status**: implemented · `lean/OOCert/Horn.lean` with `horn_certificate_sound` machine-checked ·
- `lean/OOCert/HornBuiltin.lean` discharges 19 of the engine's rules against the semantics, giving
+ `lean/OOCert/HornBuiltin.lean` discharges the engine's Horn rules against the semantics (27 table rows covering 25 W3C rules, since `scm-eqc1` and `scm-eqp1` each license two conclusions), giving
`entails_of_builtin_horn` · checker `oo-horn`, gated by `tests/lean_horn_certificate_test.rs` and
- a CI leg · **the Rust reasoner does not yet emit Horn certificates**, and
- `lean/OOCert/Mixed.lean` is deferred (see Not done)
+ a CI leg · `reason --rules` emits Horn certificates, and `lean/OOCert/Mixed.lean` has landed
- **Written**: 2026-09-14
- **Related**: decision 0002 (an inference carries a certificate); the adversarial-SHACL assurance
laundering work, which named the failure this decision is built to avoid
@@ -26,7 +25,7 @@ Extending the checker rule by rule would mean a Lean edit per customer.
over rules, because there is no per-rule structure left in the checker. Axioms are
`propext`, `Classical.choice`, `Quot.sound`, identical to the existing theorem. The trust surface
does not grow.
-3. **The built-ins are a rule table like any other.** 19 of the engine's rules are Horn rules and are
+3. **The built-ins are a rule table like any other.** Most of the engine's rules are Horn rules and are
written out as data in `HornBuiltin.lean`, each with a one-line lemma deriving it from the
semantic conditions. `cls-int1` and `cls-uni` are not Horn rules and stay as arms: their premise
count is the length of an RDF list, which is data rather than fixed by the rule. That is not a gap
@@ -52,11 +51,16 @@ whether the rule is true, whether the rule set is consistent, or whether the ass
accurate. Under a user table the guarantee is conditional on assumptions the user wrote, and the
report says so in the verdict rather than in a footnote.
-## Not done
+## Since written
-- The Rust reasoner cannot yet evaluate a supplied rule table or emit a Horn certificate, so today
- the layer checks certificates rather than producing them. That is the next piece.
-- `Mixed.lean`, which lets one certificate carry both built-in arms and Horn steps, needs
- `Soundness.lean` generalised from `Entails G` to `EntailsIn P`. The change is mechanical across
- about twenty cases and was deferred rather than rushed. Until it lands, a user-rule certificate
- cannot also cite `cls-int1` or `cls-uni`.
+Both items listed here as not done have landed. `reason --rules` evaluates a supplied table and
+emits a certificate, and `Mixed.lean` lets one certificate carry both built-in arms and Horn steps,
+which needed `Soundness.lean` generalised from `Entails G` to `EntailsIn P` across about twenty
+cases. The engine's rule count has since grown; the checker covers 29 of the OWL 2 RL profile's 78
+rules, 30 counting `cax-dw` in the refutation layer, and four of them stay as hand-written arms
+because their premise count is the length of an RDF list.
+
+## Still not done
+
+- Nothing in `src/` writes the refutation format, so inconsistency can be checked and not yet
+ produced by this engine.
diff --git a/tests/shacl_core_verified_test.rs b/tests/shacl_core_verified_test.rs
index 370fc4a8..6d105eae 100644
--- a/tests/shacl_core_verified_test.rs
+++ b/tests/shacl_core_verified_test.rs
@@ -45,11 +45,19 @@
//! development decodes an N-Triples escape, so it cannot tell a literal backslash
//! from an escape sequence.
//!
-//! The Rust engine's own number, under its own harness at the same suite commit, is
-//! PASS 36 / FAIL 24 / UNDETERMINED 60 of 120. That is the comparison to hold this
-//! against, and it is not an apples-to-apples one: the two answer different SETS of
-//! tests. What is comparable is that the SPARQL-backed engine answers 60 of the 120
-//! and is wrong on 24 of them, and this one answers 91 and is wrong on none.
+//! The Rust engine's own number is printed alongside at run time, read from its
+//! committed baseline rather than repeated here, because a number typed into
+//! prose goes stale silently and this one did within a day. The comparison is
+//! not apples to apples either way: the two answer different SETS of tests.
+//! What is comparable is how many of the 120 each ANSWERS and how many of those
+//! it gets wrong.
+//!
+//! And FAIL at 0 claims less than it looks. It is 0 on the tests this evaluator
+//! answers, under a lens that ignores sh:resultPath, sh:value, sh:resultSeverity,
+//! sh:resultMessage, result multiplicity and blank-node identity. It is 0 on
+//! this suite, which is not the same as 0 against the Recommendation: the suite
+//! is the only gate on the compiler, so a constraint the suite never exercises
+//! is not checked by anything here.
//!
//! # The comparison, which is the same one the other harness uses
//!
@@ -1007,6 +1015,19 @@ fn the_vendored_suite_is_whole() {
}
/// The headline: what the verified evaluator scores, and where the rest went.
+/// The other harness's counts, from the baseline it commits. Returns None rather
+/// than a guess if the file is absent or its shape changes.
+fn rust_engine_baseline() -> Option<(u64, u64, u64)> {
+ let raw = std::fs::read_to_string(repo().join("tests").join("w3c_shacl_baseline.json")).ok()?;
+ let v: serde_json::Value = serde_json::from_str(&raw).ok()?;
+ let counts = v.get("counts").unwrap_or(&v);
+ Some((
+ counts.get("pass")?.as_u64()?,
+ counts.get("fail")?.as_u64()?,
+ counts.get("undetermined")?.as_u64()?,
+ ))
+}
+
#[test]
fn the_verified_conformance_report() {
if skip() {
@@ -1055,13 +1076,26 @@ fn the_verified_conformance_report() {
refusal is NOT credited as a reported failure.",
r.failure_expected, r.failure_undetermined
);
- eprintln!(
- "\n for comparison, the SPARQL-backed Rust engine under its own harness at the same\n \
- suite commit: PASS 36 / FAIL 24 / UNDETERMINED 60. It ANSWERS {} of the 120 and gets\n \
- 24 of those wrong; this one answers {} and gets none wrong.",
- 36 + 24,
- c.pass + c.fail
- );
+ // Read from the other harness's committed baseline rather than typed here.
+ // These constants were typed once and went stale within a day: they said the
+ // Rust engine was at 36/24/60 when it was at 35/17/68, inside a report whose
+ // entire value is that its numbers are measured. A number standing next to
+ // the screen that shows it has to be derived from the same source, or it
+ // becomes a claim nothing checks.
+ match rust_engine_baseline() {
+ Some((rp, rf, ru)) => eprintln!(
+ "\n for comparison, the SPARQL-backed Rust engine under its own harness at the same\n \
+ suite commit, read from tests/w3c_shacl_baseline.json: PASS {rp} / FAIL {rf} /\n \
+ UNDETERMINED {ru}. It ANSWERS {} of the 120 and is wrong on {rf}; this one answers\n \
+ {} and is wrong on none of those it answers.",
+ rp + rf,
+ c.pass + c.fail
+ ),
+ None => eprintln!(
+ "\n the Rust engine's baseline could not be read, so no comparison is printed.\n \
+ A missing number is better than a stale one."
+ ),
+ }
// Where the undetermined tests went, clustered, with every test named under
// its cluster so the next piece of work can be picked by evidence rather than