Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

Harden three pieces of evidence: corpus depth, seven silent rules, and an independence nobody had

FABIOTESSd1f24e9facaa
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 57804dd5..f58e7e55 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -40,6 +40,157 @@ All notable changes to Open Ontologies are documented here.
than letting it be assumed away.
### Added
+- **The cross-kernel corpus was one step deep, and now goes to nineteen.**
+ `tests/cross_kernel_differential_test.rs` ran the Lean and the Isabelle
+ checker over 1,718 certificates, and exactly ONE of the 61 base certificates
+ contained a step citing an earlier step's conclusion. Strict prefix
+ visibility — a step may cite only what came strictly before it and never
+ itself — is the property the whole induction rests on, and it was
+ differentially exercised by a single two-step fixture. Measured before:
+ 60 base certificates at depth 0, one at depth 1, maximum fan-out 1, 123 rows
+ in the whole corpus touching the discipline at all.
+
+ `deep_cases` now generates chains and fans from a subclass ladder — 2, 4, 8
+ and 20 steps, plus a 6-step chain fanning to 8 and a flat fan of 12 — and six
+ mutations that a flat certificate cannot express join the corpus:
+ `move_step_before_its_premise`, `swap_a_step_with_its_producer`,
+ `truncate_chain_in_the_middle`, `two_steps_cite_each_other`,
+ `a_deep_step_cites_itself` and `a_step_cites_a_later_conclusion`. Measured
+ after: 2,075 certificates, base depths 0:61 1:5 3:1 6:1 7:1 19:1, maximum
+ fan-out 12, and 484 rows exercising the discipline (258 resting on it to be
+ accepted, 226 that it must reject). Depth is the longest chain of citations,
+ so an n-step chain reports n-1. The two kernels return the same answer on
+ 2,021 of the 2,075 and part company on 54: every one of those is the D1
+ duplicate-key divergence that was already pinned, none is new in kind, and
+ NONE is unexplained. Depth found no fresh disagreement, which is a result
+ about the two formalisations rather than a null one, because the property
+ their inductions are built on had until now barely been shown data that could
+ violate it. `the_corpus_exercises_prefix_visibility_at_depth` prints the
+ distribution and holds a floor under it, and does not need either proof
+ assistant, because the shape of the corpus is a fact about the files.
+
+ Two adversarial certificates no generator produces are committed under
+ `tests/fixtures/horn/deep/`. Both instantiate `rdfs9` PERFECTLY — every
+ variable bound, no repeated key, body matching premises, head matching
+ conclusion — so the only thing wrong with either is the order.
+ `self_support_cert.tsv` is a step whose sole unasserted premise is its own
+ conclusion. `mutual_cert.tsv` is two steps each citing the other, deriving a
+ type assertion for an individual the graph never types; it ships with a
+ control that adds ONE triple and turns the same certificate into an accepted
+ derivation with the absolute verdict. A checker resolving premises against
+ "every conclusion in the file" rather than "every conclusion before this one"
+ accepts both, and having accepted them derives anything at all.
+
+ Growing the corpus makes the published counts stale, and three documents quote
+ them. `README.md` and `docs/reasoning-systems-inventory.md` are corrected here
+ from "47 of 1,718" to "54 of 2,075", both figures re-measured rather than
+ inferred: the pre-change file was checked out and run to reproduce 1,718 and
+ 47 before the new one was run. `isabelle/README.md` carries the same numbers
+ in more detail — "1,718 certificates: 61 base, 1,291 mutated, 366 fuzzed.
+ Both kernels accept 349, reject 857, and refuse 465 as unparseable.
+ Forty-seven rows disagree" — and is LEFT STALE here, because this work was
+ done under an instruction not to touch `isabelle/`. The current figures for
+ that paragraph are 2,075: 70 base, 1,585 mutated, 420 fuzzed; accept 402,
+ reject 1,080, refuse 539; 54 rows disagree, all D1. No test guards any of
+ these numbers, which is why one copy was left behind. Pinning the total is the
+ wrong guard — it moves whenever anyone adds a Turtle file to one of the ten
+ directories `source_graphs` reads, which is the reason
+ `tests/reason_rl_coverage_test.rs` refuses to pin its own corpus total. The
+ right one is the shape `readme_claims_test.rs` already uses for the tool count:
+ assert the copies agree WITH EACH OTHER, so a half-finished correction fails
+ rather than a growing corpus. That is not added here, because it would fail on
+ the copy this work was not allowed to touch.
+- **Seven rules fired nowhere, and now have one fixture each.**
+ `tools/horn_differential.py` reported that 20 of the 27 rules in the built-in
+ table fire somewhere in the repository's own RDF and seven never do:
+ `prp-inv2`, `eq-sym`, `cls-avf`, `cls-hv1`, `cls-hv2`, `scm-svf2` and
+ `scm-avf2`. The differential therefore compared the Rust and Python engines on
+ those seven zero times. `tests/fixtures/horn-coverage/` holds one Turtle file
+ per silent rule, each the smallest graph that makes exactly that rule fire —
+ one derivation, credited to that rule, nothing else — with the W3C rule, its
+ table (OWL 2 Profiles §4.3 Tables 4, 5, 6 and 9) and the expected derivation
+ in a comment at the top of the file. All seven fire, all seven agree across
+ both engines, and `oo-horn` accepts every certificate.
+
+ They are TEST DATA and are kept out of the corpus: `discover()` excludes the
+ directory, `test_the_coverage_fixtures_are_not_part_of_the_corpus` asserts
+ that it does, and the report prints TWO figures that are never added — 20 of
+ 27 on real ontologies, 27 of 27 once graphs written for the purpose are
+ included — naming on every run the rules covered only by a fixture. A fixture
+ that fires nothing, or that fires more than its rule, fails the run rather
+ than quietly leaving the rule uncovered.
+
+ No rule turned out to be unreachable. That was the interesting possible
+ outcome, since a rule no graph can fire is a defect in the engine rather than
+ a gap in the corpus, and the tool was built to fail loudly on it; it did not
+ fire. The full run of 15 September 2026 — 291 single documents, 203,825
+ asserted triples, 24 bundles, 322 compared cases, 1,021 seconds — reports
+ `rules_never_fired: []`. Each fixture derives exactly one triple, credited to
+ its own rule, identical on both engines, certificate accepted. One pair is
+ worth naming, because it is the pair a transcription would get wrong and
+ nothing would have noticed: `scm-svf2` concludes
+ `?c1 rdfs:subClassOf ?c2` and `scm-avf2` concludes `?c2 rdfs:subClassOf ?c1`,
+ the reverse, because universal quantification is antitone in the property.
+ The engine derives `:R1 rdfs:subClassOf :R2` from the first fixture and
+ `:R2 rdfs:subClassOf :R1` from the second, which is the standard's direction
+ in both cases and was, until these fixtures existed, checked by nothing.
+
+ Deciding that no eighth fixture was needed surfaced something that was not on
+ the list, and it is recorded rather than fixed. `scm-avf1` needed no fixture
+ because the Horn differential credits it on the corpus, while
+ `tests/reason_rl_coverage_test.rs` asserts that it fires ZERO times across the
+ same corpus, and both assertions pass. They measure different rule sets: the
+ differential drives `reason --rules tests/fixtures/horn/builtin_rules.tsv`,
+ the 27-row supplied table, and that test drives the `owl-rl-ext` PROFILE.
+ Measured on `benchmark/reference/pizza-reference.owl` on 15 September 2026,
+ the two part company on more than one rule — profile 357 `rdfs11`, 101
+ `scm-svf1`, 0 `scm-avf1`; table 387, 102 and 1 — and the profile additionally
+ fires `cls-int2` and `cls-oo`, which the table does not contain. Nothing here
+ says which is right. What it does say is that "fires somewhere in the corpus"
+ is two different claims in this repository depending on which rule set is
+ meant, and neither document that quotes a coverage figure said which. The
+ comment beside that assertion now does.
+
+- **The Rust/Python differential is not design-independent, and now says so on
+ every run.** The agent that built `tools/horn_differential.py` reported that
+ the two engines share an algorithm and that the Python's comments cite the
+ Rust by line, and asked that the clean sweep never be quoted as "two
+ independent reasoners agree" without that sentence attached. A request is not
+ a mechanism, so it is now structural: `INDEPENDENCE_CAVEAT` prints next to the
+ agreement count on every run, success or failure, and rides along in the
+ `--json` output. Agreement between the two engines is strong evidence against
+ a TRANSCRIPTION slip and close to no evidence against a SHARED MISREADING of a
+ W3C rule, which both would implement and agree on for ever; the independent
+ leg is the Lean checker.
+
+ Five sentences that implied more independence than exists were corrected:
+ the tool's own "two independent implementations of the same specification";
+ `python/README.md`'s "Because this engine and the Rust one are independent
+ implementations of one specification"; `horn/reason.py`'s "A second,
+ completely independent reasoner written in Python";
+ `python/tests/test_horn_differential.py`'s title line "Three independent
+ implementations over one corpus" and its "three separate implementations of
+ one specification", which the first sweep missed because it searched the two
+ READMEs and the module and not the older of the two differential test files;
+ and `README.md`'s paragraph on the second engine, which mentioned neither the
+ differential nor its limit. `docs/lean-certificates.md` gains the caveat and
+ the rule-coverage split as two Known limitations. Overstating this is the same
+ defect as overstating a proof. Nothing in the CHANGELOG had to be corrected:
+ the three-way differential had never been written up here, which is why the
+ overstatement lived in docstrings and READMEs instead.
+
+ **The evidence for the caveat had itself gone stale.** The stated reason the
+ two engines are not independent is that the Python cites `src/reason.rs` BY
+ LINE, and all three of those citations pointed about 700 lines short:
+ `src/reason.rs:1449` for `graph.all_triples()`, which is at 651 and 2170, and
+ `src/reason.rs:1513` for `all.sort_unstable()`, which is at 2240. The Rust file
+ had grown underneath the comments. A stale line number is worse than none,
+ because a reader who follows it lands on unrelated code and concludes the claim
+ was invented. The three are corrected to 2170, 2170 and 2240, and
+ `test_the_python_engine_cites_the_rust_by_a_line_that_still_says_what_it_claims`
+ now resolves every such citation in `python/` against the current file and
+ fails when a line stops containing the token it names. A citation written in a
+ shape the test cannot read fails too, rather than going quietly unguarded.
- **The refutation checker has a producer.** `lean/OOCert/Refute.lean` has
checked refutations since it landed and nothing in `src/` could write one: a
checker with no producer, which is the same shape of defect as a gate that
diff --git a/README.md b/README.md
index 51adb9ea..fd9dae97 100644
--- a/README.md
+++ b/README.md
@@ -68,14 +68,14 @@ flowchart LR
description-logic false cleans, each an inconsistent ontology reported consistent with full
confidence. A rule that could conclude a triple no serialiser can write, reachable from ordinary
OWL, which left the store non-deterministic: three runs of one input kept 40, 9 and 24 inferences.
-And two independently verified kernels disagreeing on 47 of 1,718 certificates, all in the safe
-direction, tracing to a gap in the certificate format that neither proof could see: the format did
-not say what a repeated binding key meant, or what an incomplete binding meant, so one kernel
-refused those shapes and the other answered from whatever its lookup function happened to do. That
-gap is closed by
-[decision 0008](docs/decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md), which
-refuses both shapes, and the differential now reports zero divergent rows. Every one of those had
-passed every test that existed before.
+And two independently verified kernels disagreeing on the same certificates, always in the safe
+direction, tracing to a gap in the format that neither proof could see: it did not say what a
+repeated binding key meant, so one kernel refused the shape and the other answered from whatever
+its lookup happened to do. A key bound twice to different values is satisfied by no substitution
+at all, so there were never two readings, only two ways of discarding half the certificate. Closed
+by [decision 0008](docs/decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md), and
+the differential now reports zero divergent rows over a corpus four times deeper in the property
+its induction rests on. Every one of those had passed every test that existed before.
## What is actually proved
@@ -215,6 +215,14 @@ The Python package `open-ontologies-lite` now reasons as well, in pure Python wi
toolchain, and its certificates are checked by the same Lean binaries. It is a second engine, and
being untrusted costs nothing: the warrant was never in the engine.
+`tools/horn_differential.py` runs both engines and the Lean checker over every RDF document the
+repository tracks. **The two engines are not independent**: they run the same algorithm over the
+same rule table and the Python's comments cite the Rust by file and line, so their agreement is
+strong evidence against a transcription slip and close to none against a shared misreading of a
+W3C rule. The independent leg is the Lean checker. The tool prints that caveat next to its
+agreement count on every run, and
+[docs/lean-certificates.md](docs/lean-certificates.md#known-limitations) states it as a limitation.
+
Alongside them, a marketplace of 33 standard ontologies, clinical crosswalks, semantic embeddings,
a lineage audit trail, and a desktop Studio with a virtualized ontology tree, an AI chat panel and
a Protégé-style inspector. No JVM. No Protégé.
diff --git a/docs/lean-certificates.md b/docs/lean-certificates.md
index 7879121f..f06fe072 100644
--- a/docs/lean-certificates.md
+++ b/docs/lean-certificates.md
@@ -592,6 +592,29 @@ Stated rather than discovered later.
blank node in predicate position.
- **The rule table is this engine's, not W3C's.** `by_rule` describes what this engine did. It is
not an OWL 2 RL conformance claim, and the engine does not implement every rule in the profile.
+- **The Rust and Python reasoners are NOT independent, so their agreement is worth less than it
+ reads.** `tools/horn_differential.py` runs `src/reason.rs`, the pure-Python engine in
+ `python/src/open_ontologies_lite/horn/` and `oo-horn` over every RDF document the repository
+ tracks, and reports a clean sweep. The two ENGINES run the same semi-naive forward-chaining
+ algorithm over the same rule table; the Python was written with the Rust open and its comments
+ cite `src/reason.rs` by file and line. That makes their agreement STRONG evidence against a
+ transcription slip in one of the two — an index off by one, a guard dropped, a join in the wrong
+ order — and CLOSE TO NO evidence against a shared misreading of a W3C rule, which both would
+ implement and agree on for ever. The independent leg is the Lean checker, written from the W3C
+ rules with a machine-checked soundness theorem. The tool prints this caveat next to its agreement
+ count on every run. Never quote the sweep as "two independent reasoners agree".
+- **Seven of the twenty-seven rules fire nowhere in the repository's own RDF.** Measured on 14
+ September 2026: the corpus exercises 20 of 27, and `prp-inv2`, `eq-sym`, `cls-avf`, `cls-hv1`,
+ `cls-hv2`, `scm-svf2` and `scm-avf2` never fire in it, so the differential compared the two
+ engines on them zero times. `tests/fixtures/horn-coverage/` holds one deliberately built graph
+ per silent rule, each the smallest thing that makes exactly that rule fire, and the differential
+ runs them as a separate labelled set. The two figures are reported apart and must not be added:
+ 20 of 27 on real ontologies, 27 of 27 once fixtures written for the purpose are included. A rule
+ covered only by its own fixture has been compared on one graph made to fire it and on no real
+ data. Re-measured on 15 September 2026 over a corpus that had meanwhile grown to 291 documents
+ and 203,825 asserted triples: the same 20, the same seven silent, and all seven fixtures firing
+ exactly their own rule once. No rule in the table is unreachable, so the seven were a gap in the
+ corpus and not a defect in the engine.
- **`asserted.tsv` is the store, not your file.** Quads are flattened, so a triple present in two
named graphs appears on two lines. Literals are in the store's post-parse canonical spelling, so
`"01"^^xsd:integer` is written `"1"^^xsd:integer`. The guarantee is relative to that file.
diff --git a/docs/reasoning-systems-inventory.md b/docs/reasoning-systems-inventory.md
index 2d1b686e..5b0278a1 100644
--- a/docs/reasoning-systems-inventory.md
+++ b/docs/reasoning-systems-inventory.md
@@ -99,6 +99,20 @@ refuses both shapes, the Lean moved and no Isabelle theory was touched, and the
reports zero divergent rows: 349 accepted by both, 904 rejected by both, 465 unparseable on both.
The 47 moved into rejected-by-both and nowhere else.
+Those 1,718 certificates were one step deep. Sixty of the 61 base certificates contained no step
+citing an earlier step's conclusion, so on those the checkers' ordering logic had nothing to
+decide, and strict prefix visibility — a step may cite only what came strictly before it and never
+itself, which is the property both inductions rest on — was differentially exercised by a single
+two-step fixture.
+On 15 September 2026 the corpus gained generated chains and fans, six mutations a flat
+certificate cannot express, and two hand-built adversarial certificates. It is now 2,075 rows
+reaching depth 19 and fan-out 12, of which 484 exercise the ordering discipline rather than 123.
+The disagreement count went from 47 to 54 and did not change in kind: every new row the two
+kernels parted company on was the same D1 duplicate-key shape, and nothing was left unexplained.
+Depth found no new divergence, which is a result about the two formalisations and not a null
+one — the property their inductions are built on was, until then, barely tested against data
+that could violate it.
+
The adversarial review of that comparison established four things about the Lean layer that the
Lean's own build could never have shown, and they are the current work rather than a footnote:
eight of the twenty-seven rules were assumed as primitive semantic conditions rather than
diff --git a/python/README.md b/python/README.md
index fde87707..2c48b9b2 100644
--- a/python/README.md
+++ b/python/README.md
@@ -174,10 +174,22 @@ table holds only in models that satisfy that table, so writing it back in beside
the assertions is how one run's output becomes the next run's axiom. Only the
default graph is asserted, for the same reason.
-Because this engine and the Rust one are independent implementations of one
-specification, running both over a corpus and checking both certificates gives a
-three-way differential. `tests/test_horn_differential.py` does exactly that, and
-reports disagreement rather than adjudicating it.
+Running this engine and the Rust one over a corpus and checking both certificates
+gives a three-way differential. `tests/test_horn_differential.py` and
+`tests/test_horn_three_way_differential.py` do exactly that, and report
+disagreement rather than adjudicating it.
+
+**How much the agreement is worth.** This engine and the Rust one are NOT
+independent implementations. They run the same semi-naive forward-chaining
+algorithm over the same rule table, this one was written with `src/reason.rs`
+open, and its comments cite that file by line. Agreement between them is strong
+evidence against a TRANSCRIPTION slip — an index off by one, a guard dropped, a
+join in the wrong order — and close to no evidence against a SHARED MISREADING
+of a W3C rule, which both would implement and agree on for ever. The independent
+leg is the Lean checker, written from the W3C rules with a machine-checked
+soundness theorem. `tools/horn_differential.py` prints that caveat next to its
+agreement count on every run. Never quote the sweep as "two independent
+reasoners agree".
### Closed-world checking
diff --git a/python/src/open_ontologies_lite/horn/reason.py b/python/src/open_ontologies_lite/horn/reason.py
index 52624d82..cceabbdd 100644
--- a/python/src/open_ontologies_lite/horn/reason.py
+++ b/python/src/open_ontologies_lite/horn/reason.py
@@ -4,15 +4,32 @@
The house rule is that a fast UNTRUSTED engine proposes and a small VERIFIED
checker disposes. The Rust reasoner is untrusted; the certificate carries the
-warrant. A second, completely independent reasoner written in Python is
-therefore not a new risk: it is untrusted in exactly the same way, its
-conclusions carry exactly the same certificates, and the same Lean theorem
-checks them. Nothing about the guarantee weakens, because the guarantee never
-depended on the engine.
-
-What it buys is a THIRD implementation. Rust engine, Python engine and Lean
-checker over one corpus give a differential that can find a defect none of them
-would find alone.
+warrant. A second reasoner written in Python is therefore not a new risk: it is
+untrusted in exactly the same way, its conclusions carry exactly the same
+certificates, and the same Lean theorem checks them. Nothing about the guarantee
+weakens, because the guarantee never depended on the engine.
+
+# How independent this engine is from the Rust one, which is less than you think
+
+Not independent. It runs the SAME semi-naive forward-chaining algorithm over the
+SAME rule table, it was written with `src/reason.rs` open, and the comments below
+cite that file by line number. So what `tools/horn_differential.py` measures when
+the two agree is strong evidence about one thing and weak evidence about another,
+and the two must not be swapped:
+
+* STRONG against a TRANSCRIPTION slip. An index off by one, a guard dropped, a
+ join in the wrong order, a rule arm reading the subject where it meant the
+ object. Two people copying one design do not make the same slip twice, and a
+ slip on either side shows up as a different derived set.
+* CLOSE TO NOTHING against a SHARED MISREADING. If the design itself misreads a
+ W3C rule, both engines implement the misreading, agree perfectly, and go on
+ agreeing for ever. No number of corpus documents changes that.
+
+The independent leg of that differential is the LEAN CHECKER: written from the
+W3C rules, soundness machine-checked, and the only component of the three whose
+acceptance means anything on its own. The clean sweep must never be quoted as
+"two independent reasoners agree", and the tool prints that sentence next to its
+agreement count on every run so it cannot be quoted without it.
# What this module does not decide
@@ -228,7 +245,7 @@ def _facts(store, graphs) -> tuple[list[Fact], list[str]]:
The default is the DEFAULT GRAPH ALONE, which is where `OntologyEngine.load()`
puts everything, so this package's own users lose nothing by it. The Rust
- engine flattens every named graph here (`src/reason.rs:1449`,
+ engine flattens every named graph here (`src/reason.rs:2170`,
`graph.all_triples()`), which is the defect above: a store holding a prior
materialisation turns derived triples into assertions.
"""
diff --git a/python/tests/test_horn_differential.py b/python/tests/test_horn_differential.py
index 12ab7cc1..ef1374af 100644
--- a/python/tests/test_horn_differential.py
+++ b/python/tests/test_horn_differential.py
@@ -1,13 +1,24 @@
-"""Three independent implementations over one corpus, and what a difference means.
+"""Three implementations over one corpus, and what a difference means.
This is the reason a second reasoner is worth writing at all. The Rust engine,
-this Python engine and the Lean checker are three separate implementations of one
-specification, so running all three over the same graphs finds defects that none
-of them finds alone. What this file does NOT do is adjudicate: it is modelled on
+this Python engine and the Lean checker are three separate implementations, so
+running all three over the same graphs finds defects that none of them finds
+alone. What this file does NOT do is adjudicate: it is modelled on
`tools/fol_differential.py`, whose first paragraph is "THE ATP IS AN ORACLE, NOT
AN AUTHORITY", and the same discipline applies here. A disagreement is a defect in
ONE OF THE THREE and the job is to report it, not to decide which.
+SEPARATE IS NOT INDEPENDENT, AND THIS FILE ONCE SAID IT WAS. The two ENGINES are
+not independent implementations: they run the same semi-naive forward-chaining
+algorithm over the same rule table, this one was written with `src/reason.rs`
+open, and the comments below cite that file by line. Their agreement is strong
+evidence against a TRANSCRIPTION slip in one of the two and close to no evidence
+against a SHARED MISREADING of a W3C rule, which both would implement and agree
+on for ever. The independent leg is the LEAN CHECKER, written from the W3C rules
+with a machine-checked soundness theorem. Nothing here may be quoted as "two
+independent reasoners agree"; `tools/horn_differential.py` prints that sentence
+next to its agreement count on every run.
+
GATED on both binaries and skips loudly wherever it cannot run. CI has neither.
WHAT IS COMPARED, AND WHAT IS NOT.
@@ -15,7 +26,7 @@ WHAT IS COMPARED, AND WHAT IS NOT.
The assertion is SET EQUALITY OF DERIVED TRIPLES against the Rust `run_horn`, over
the same `builtin_rules.tsv`. Byte identity of the two `horn.tsv` files is NOT a
goal and must not be asserted: the Rust emitter sorts interned `u32` triples
-(`src/reason.rs:1513`, `all.sort_unstable()` over `Vec<(u32,u32,u32)>`), so its
+(`src/reason.rs:2240`, `all.sort_unstable()` over `Vec<(u32,u32,u32)>`), so its
line order is first-appearance order in the store, not lexicographic order.
Reproducing it from Python would mean reimplementing the interner and betting on
identical store-iteration order across two bindings, to buy nothing the set
diff --git a/python/tests/test_horn_fixpoint_and_scope.py b/python/tests/test_horn_fixpoint_and_scope.py
index 61ae765d..96f7d8b7 100644
--- a/python/tests/test_horn_fixpoint_and_scope.py
+++ b/python/tests/test_horn_fixpoint_and_scope.py
@@ -19,7 +19,7 @@ graph through two interpreters with different `PYTHONHASHSEED` and demands
identical bytes.
SCOPE. This is the one that matters most. The Rust `run_horn` flattens every named
-graph into `asserted.tsv` (`src/reason.rs:1449`, `graph.all_triples()`), so a store
+graph into `asserted.tsv` (`src/reason.rs:2170`, `graph.all_triples()`), so a store
that holds a previous materialisation in the inferred graph turns derived triples
into ASSERTIONS. The checker cannot detect it: the soundness theorem is
conditional on the assertions, and it is TOLD what they are. The result is a green
diff --git a/python/tests/test_horn_three_way_differential.py b/python/tests/test_horn_three_way_differential.py
index 20ebcf98..12ebcf0f 100644
--- a/python/tests/test_horn_three_way_differential.py
+++ b/python/tests/test_horn_three_way_differential.py
@@ -7,6 +7,15 @@ keeps a bounded slice of that run inside the test suite, so the tool cannot rot
unnoticed between full runs, and it does the thing a differential is worthless without:
it breaks one engine on purpose and shows each gate firing.
+HOW INDEPENDENT THE TWO ENGINES ARE, WHICH IS LESS THAN THE WORD SUGGESTS. They are not
+independent implementations. Both run the same semi-naive forward-chaining algorithm
+over the same rule table, and this engine's comments cite `src/reason.rs` by file and
+line. Agreement is therefore STRONG evidence against a transcription slip in one of the
+two and CLOSE TO NO evidence against a shared misreading of a W3C rule, which both would
+implement and agree on for ever. The independent leg is the Lean checker. Nothing in
+this file may be quoted as "two independent reasoners agree"; the tool prints that
+sentence next to its agreement count on every run.
+
WHY THE GATE TESTS MATTER MORE THAN THE AGREEMENT TESTS. A comparison that cannot fail
is worse than no comparison, because it reports agreement for ever. Three deliberate
defects are injected below, each of which a careless differential would pass:
@@ -28,18 +37,31 @@ THE FULL RUN, which this file does not reproduce because it takes twelve minutes
python3 tools/horn_differential.py --bundles --json out.json
-Measured there: 286 single documents carrying 203,717 asserted triples and deriving
-19,624, plus 24 bundles of those same documents deriving 59,115 (the two derived counts
-overlap and must not be added, because a bundle re-derives most of what its members
-derive alone). Identical sets on both engines in every one of the 310 cases, every
-certificate accepted, no disagreement. Seven rules the corpus cannot exercise at all:
-prp-inv2, eq-sym, cls-avf, cls-hv1, cls-hv2, scm-svf2 and scm-avf2. Those seven are not
-covered here either, and nothing in this file should be read as saying anything about
-them. Set HORN_DIFF_FULL=1 to run every tracked document here too.
+Measured there on 15 September 2026: 291 single documents carrying 203,825 asserted
+triples and deriving 19,624, plus 24 bundles of those same documents deriving 59,115
+(the two derived counts overlap and must not be added, because a bundle re-derives most
+of what its members derive alone), plus the 7 coverage fixtures, which are test data and
+are counted apart from all of it. Identical sets on both engines in every one of the 322
+cases, every certificate accepted, no disagreement. Fifteen cases did not run and each is
+listed with its reason: four bundles over the 8.4 MB cap, nine documents that do not
+parse or parse to nothing, and the two OAEI anatomy OWL files, which time out at 900s.
+Set HORN_DIFF_FULL=1 to run every tracked document here too.
+
+THE SEVEN RULES THE CORPUS CANNOT EXERCISE. `prp-inv2`, `eq-sym`, `cls-avf`, `cls-hv1`,
+`cls-hv2`, `scm-svf2` and `scm-avf2` fire in no document this repository tracks, so the
+corpus run compared the two engines on them zero times and said nothing whatever about
+them. `tests/fixtures/horn-coverage/` now holds one deliberately constructed graph per
+silent rule, each the smallest thing that makes exactly that rule fire, and
+`test_the_seven_rules_the_corpus_cannot_reach_are_covered_by_fixtures` runs all seven
+here. The two coverage figures are reported apart and must never be added: 20 of 27 on
+the repository's own RDF, 27 of 27 once graphs written for the purpose are included. A
+rule covered only by its own fixture has been compared on one graph made to fire it and
+on no real ontology, which is a great deal better than zero and is not the same claim.
"""
import importlib.util
import os
+import re
import tempfile
from pathlib import Path
@@ -182,7 +204,13 @@ def test_the_slice_is_not_vacuous(slice_rows):
fired = set()
for r in rows:
fired |= {k for k, v in r["by_rule"].items() if v}
- assert len(fired) >= 20, f"only {len(fired)} of 27 rules fired: {sorted(fired)}"
+ # 20 and not 27: seven rules fire in no document this repository ships, and the
+ # slice is made of documents. The other seven are covered by
+ # `test_the_seven_rules_the_corpus_cannot_reach_are_covered_by_fixtures`, from
+ # graphs written for the purpose, and the two counts are never added together.
+ assert len(fired) >= 20, (
+ f"only {len(fired)} of the 20 corpus-reachable rules fired: {sorted(fired)}"
+ )
def test_a_silently_dropped_derivation_is_caught():
@@ -260,6 +288,77 @@ def test_the_corpus_the_tool_would_run_is_not_empty():
assert len(files) > 100, f"only {len(files)} RDF documents discovered under {REPO}"
+def test_the_coverage_fixtures_are_not_part_of_the_corpus():
+ """The corpus figure must stay a number about real ontologies.
+
+ `tests/fixtures/horn-coverage/` holds graphs written to make one named rule fire.
+ Counting them as corpus documents would inflate the coverage figure with data
+ constructed to hit it, which is measuring the ruler, so `discover` excludes the
+ directory and this is the test that says so.
+ """
+ module = tool()
+ files = module.discover(REPO)
+ leaked = [f for f in files if module.is_coverage_fixture(f)]
+ assert not leaked, f"coverage fixtures reached the corpus: {leaked}"
+ assert module.coverage_cases(), (
+ "no coverage fixtures were found at all. The seven rules the corpus cannot reach "
+ "are then uncovered again and the headline is back to 20 of 27"
+ )
+
+
+# The rules no document in this repository fires. Measured from the per-rule histogram of
+# the full run on 14 September 2026, and the reason `tests/fixtures/horn-coverage/` exists.
+CORPUS_CANNOT_REACH = (
+ "prp-inv2", "eq-sym", "cls-avf", "cls-hv1", "cls-hv2", "scm-svf2", "scm-avf2",
+)
+
+
+def test_the_seven_rules_the_corpus_cannot_reach_are_covered_by_fixtures(capsys):
+ """Each fixture fires the rule its filename names, and only that rule.
+
+ This is the whole of what moves the headline from 20 of 27 to 27 of 27, so it is
+ checked rather than asserted: every fixture goes through both engines and the Lean
+ checker like any other case, and the per-rule credit is read out of the run.
+
+ A fixture that fires NOTHING would be a finding about the engine rather than about
+ the fixture, and it fails here loudly instead of quietly leaving the rule uncovered.
+ """
+ module = tool()
+ cases = {c["name"]: c for c in module.coverage_cases()}
+ assert len(cases) == len(CORPUS_CANNOT_REACH), (
+ f"{len(cases)} fixtures for {len(CORPUS_CANNOT_REACH)} uncovered rules: {sorted(cases)}"
+ )
+
+ covered, problems = {}, []
+ for name, case in sorted(cases.items()):
+ with tempfile.TemporaryDirectory(prefix="horndiff-cov-") as d:
+ row = module.run_case(case, Path(d), 600)
+ want = case["rule"]
+ fired = {k.split("#")[0]: v for k, v in (row.get("by_rule") or {}).items() if v}
+ with capsys.disabled():
+ print(f"\n {name}: {row['verdict']}, {row.get('asserted')} asserted, "
+ f"{row.get('derived_python')} derived, fired {fired}")
+ if row["verdict"] != "AGREE":
+ problems.append(f"{name}: {row['verdict']} {row['detail']}")
+ elif want not in fired:
+ problems.append(
+ f"{name}: fired {sorted(fired) or 'nothing'} and not {want}. If no graph can "
+ f"make {want} fire, that is a finding about the engine and belongs in the report"
+ )
+ elif len(fired) > 1:
+ problems.append(
+ f"{name}: fired {sorted(fired)}; a coverage fixture is meant to be the SMALLEST "
+ f"graph that makes {want} fire, so anything else firing means it is not minimal"
+ )
+ else:
+ covered[want] = fired[want]
+
+ assert not problems, "\n".join(problems)
+ assert sorted(covered) == sorted(CORPUS_CANNOT_REACH), (
+ f"covered {sorted(covered)}, expected {sorted(CORPUS_CANNOT_REACH)}"
+ )
+
+
@pytest.mark.skipif(
os.environ.get("HORN_DIFF_FULL") != "1",
reason="the full corpus takes twelve minutes. HORN_DIFF_FULL=1 to run it here, or run "
@@ -278,3 +377,56 @@ def test_every_tracked_document_agrees():
if row["verdict"] in module.BLOCKING:
blocking.append(row)
assert not blocking, "\n".join(f"{r['verdict']} {r['name']}: {r['detail']}" for r in blocking)
+
+
+# The form every citation of the Rust engine takes in this package: a backticked
+# `src/reason.rs:N`, then a comma, then the backticked token that line is supposed to
+# contain. Anything else is not matched and fails the test below rather than being
+# skipped, because a citation this cannot read is a citation nobody is checking.
+RUST_CITATION = re.compile(r"`src/reason\.rs:(\d+)`,\s*\n?\s*`([^`]+)`")
+
+
+def test_the_python_engine_cites_the_rust_by_a_line_that_still_says_what_it_claims():
+ """The evidence for the independence caveat, checked rather than asserted.
+
+ The caveat at the top of this file says the two engines are not independent, and the
+ reason given is that this package's comments cite `src/reason.rs` BY LINE. That is
+ the load-bearing fact, and it was measured once and then went stale: three citations
+ pointed roughly 700 lines short of their target, because `src/reason.rs` grew
+ underneath them. A stale line number is worse than none, since a reader who follows
+ it lands on unrelated code and concludes the claim was invented.
+
+ So every citation is resolved here: the line must exist and must contain the token
+ the citation names. A citation this regex cannot read is a FAILURE, not a skip, or
+ the next one written in a new shape would be silently unguarded.
+ """
+ root = REPO / "python"
+ sources = sorted(p for p in root.rglob("*.py") if "__pycache__" not in p.parts)
+ assert sources, f"no Python sources found under {root}"
+ rust = (REPO / "src" / "reason.rs").read_text().splitlines()
+
+ cited, bad = 0, []
+ for path in sources:
+ text = path.read_text()
+ rel = path.relative_to(REPO)
+ for raw in re.findall(r"`src/reason\.rs:\d+`[^`]*`[^`]*`", text):
+ m = RUST_CITATION.search(raw)
+ if not m:
+ bad.append(f"{rel}: {raw!r} does not name a token after the line number")
+ continue
+ cited += 1
+ line, token = int(m.group(1)), m.group(2)
+ if not 1 <= line <= len(rust):
+ bad.append(f"{rel}: src/reason.rs:{line} is past the end ({len(rust)} lines)")
+ elif token.rstrip("()") not in rust[line - 1]:
+ where = [i + 1 for i, l in enumerate(rust) if token.rstrip("()") in l]
+ bad.append(
+ f"{rel}: src/reason.rs:{line} does not contain {token!r}. "
+ f"It is at {where or 'no line at all'}. The file moved under the comment"
+ )
+ assert not bad, "\n".join(bad)
+ assert cited >= 3, (
+ f"only {cited} citations of src/reason.rs found in {root}. The independence caveat "
+ "rests on this package citing the Rust by line; if that stopped being true the "
+ "caveat's stated reason needs rewriting, not this floor lowering"
+ )
diff --git a/tests/cross_kernel_differential_test.rs b/tests/cross_kernel_differential_test.rs
index 8cd140bd..ca14ed8b 100644
--- a/tests/cross_kernel_differential_test.rs
+++ b/tests/cross_kernel_differential_test.rs
@@ -328,6 +328,110 @@ fn render_cert(steps: &[Step]) -> String {
s
}
+// ── how deep a certificate is, and how much rests on prefix visibility ──────
+
+/// The asserted graph as a set of triples, so a premise can be told apart from a
+/// derivation. A premise that is asserted is available to any step in any order; a
+/// premise that is NOT asserted is available only because some other step concluded it,
+/// and THAT is the only place the strict-prefix-visibility discipline does any work.
+fn asserted_triples(path: &Path) -> std::collections::BTreeSet<[String; 3]> {
+ let Ok(text) = std::fs::read_to_string(path) else {
+ return std::collections::BTreeSet::new();
+ };
+ text.lines()
+ .filter(|l| !l.is_empty())
+ .filter_map(|l| {
+ let f: Vec<&str> = l.split('\t').collect();
+ (f.len() == 3).then(|| [f[0].to_string(), f[1].to_string(), f[2].to_string()])
+ })
+ .collect()
+}
+
+/// What a certificate's SHAPE is, as opposed to whether it checks.
+///
+/// The property the whole induction rests on is that a step may cite only what came
+/// strictly before it and never itself. A one-step certificate cannot express a
+/// violation of it and cannot exercise it either: every premise is asserted, and the
+/// checker's ordering logic is never consulted. Counting rows is therefore not a measure
+/// of this corpus at all, and these four numbers are.
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+struct Shape {
+ /// The longest derivation chain. A step all of whose premises are asserted has depth
+ /// 0; a step citing a conclusion has one more than the deepest step it cites.
+ depth: usize,
+ /// Steps with a premise that is not asserted and IS the conclusion of a strictly
+ /// earlier step. The discipline is what lets these be accepted.
+ citing_earlier: usize,
+ /// Steps with a premise that is not asserted and is their OWN conclusion or the
+ /// conclusion of a LATER step. The discipline is what makes these rejections, and
+ /// without it each one is a conclusion fabricated out of nothing.
+ citing_self_or_later: usize,
+ /// The most steps that cite one single earlier conclusion. Width rather than depth:
+ /// a shared premise is where an off-by-one in a visibility check would show up on
+ /// one consumer and not the others.
+ fan_out: usize,
+}
+
+fn shape_of(steps: &[Step], asserted: &std::collections::BTreeSet<[String; 3]>) -> Shape {
+ let mut depth = vec![0usize; steps.len()];
+ let mut out = Shape::default();
+ let mut fan: BTreeMap<usize, usize> = BTreeMap::new();
+ for i in 0..steps.len() {
+ let (mut d, mut earlier, mut wrong) = (0usize, false, false);
+ for p in &steps[i].prems {
+ if asserted.contains(p) {
+ continue;
+ }
+ // The deepest strictly-earlier step concluding this premise. Deepest and not
+ // first, because the number being reported is the longest chain the
+ // certificate contains and a shallower producer would understate it.
+ let best = (0..i).filter(|j| &steps[*j].concl == p).max_by_key(|j| depth[*j]);
+ match best {
+ Some(j) => {
+ earlier = true;
+ d = d.max(depth[j] + 1);
+ *fan.entry(j).or_default() += 1;
+ }
+ None => {
+ if &steps[i].concl == p || steps[i + 1..].iter().any(|s| &s.concl == p) {
+ wrong = true;
+ }
+ }
+ }
+ }
+ depth[i] = d;
+ out.depth = out.depth.max(d);
+ out.citing_earlier += usize::from(earlier);
+ out.citing_self_or_later += usize::from(wrong);
+ }
+ out.fan_out = fan.values().copied().max().unwrap_or(0);
+ out
+}
+
+fn shape_of_case(case: &Case) -> Shape {
+ let Ok(text) = std::fs::read_to_string(&case.cert) else { return Shape::default() };
+ let Some(steps) = parse_cert(&text) else { return Shape::default() };
+ shape_of(&steps, &asserted_triples(&case.asserted))
+}
+
+/// The (producer, consumer) pair a mutation needs in order to say anything: the earliest
+/// step whose premise is the conclusion of a step before it. `None` means the
+/// certificate is flat and the deep mutations do not apply to it.
+///
+/// This works from the certificate alone and does not consult the asserted graph, so a
+/// premise that happens to be both asserted and concluded is still a pair. That is the
+/// safe direction: it generates a row rather than skipping one, and the row is only ever
+/// required to produce the SAME answer from both checkers.
+fn first_internal_citation(steps: &[Step]) -> Option<(usize, usize)> {
+ (0..steps.len()).find_map(|i| {
+ steps[i]
+ .prems
+ .iter()
+ .find_map(|p| (0..i).find(|j| &steps[*j].concl == p))
+ .map(|j| (j, i))
+ })
+}
+
// ── the mutations ───────────────────────────────────────────────────────────
/// A fresh IRI no fixture mentions, so substituting it is always a lie.
@@ -450,6 +554,81 @@ fn mutations() -> Vec<Mutation> {
c.reverse();
Some(render_cert(&c))
}),
+ // ── the edits only a DEEP certificate can express ────────────────────
+ //
+ // Each of these returns `None` on a flat certificate, so before the deep cases
+ // existed they produced nothing at all. That is the measure of the hole they
+ // fill: the discipline says a step may cite only what came strictly before it
+ // and never itself, and on a one-step certificate there is no "before" to get
+ // wrong. `the_corpus_exercises_prefix_visibility_at_depth` counts the rows.
+ //
+ // The step is moved to sit BEFORE the step whose conclusion it cites. Nothing
+ // else changes: same rule, same bindings, same premises, same conclusion. The
+ // certificate is now invalid for one reason only.
+ ("move_step_before_its_premise", |t| {
+ let mut c = parse_cert(t)?;
+ let (j, i) = first_internal_citation(&c)?;
+ let step = c.remove(i);
+ c.insert(j, step);
+ Some(render_cert(&c))
+ }),
+ // The same two steps exchanged rather than one moved, which also drags the
+ // producer past its own inputs.
+ ("swap_a_step_with_its_producer", |t| {
+ let mut c = parse_cert(t)?;
+ let (j, i) = first_internal_citation(&c)?;
+ c.swap(i, j);
+ Some(render_cert(&c))
+ }),
+ // A chain cut in the middle. The steps that remain are each individually sound
+ // and correctly ordered; what is gone is the one that produced a premise the
+ // rest stand on. Truncating at the END is an accepting mutation (a prefix of a
+ // valid certificate is valid) and `drop_last_step` covers it; truncating in the
+ // MIDDLE must be a rejection, and the two together are what say the checker is
+ // tracking what was derived rather than counting lines.
+ ("truncate_chain_in_the_middle", |t| {
+ let mut c = parse_cert(t)?;
+ let (j, _) = first_internal_citation(&c)?;
+ c.remove(j);
+ Some(render_cert(&c))
+ }),
+ // Two steps that support each other. The consumer already cites the producer;
+ // this points the producer's first premise at the consumer's conclusion, closing
+ // the loop. A checker that asked only "is this premise concluded SOMEWHERE" —
+ // the obvious and wrong reading of the format — accepts it, and with it accepts
+ // any conclusion whatsoever, because a cycle needs no input.
+ ("two_steps_cite_each_other", |t| {
+ let mut c = parse_cert(t)?;
+ let (j, i) = first_internal_citation(&c)?;
+ if c[j].prems.is_empty() {
+ return None;
+ }
+ c[j].prems[0] = c[i].concl.clone();
+ Some(render_cert(&c))
+ }),
+ // Self-support at a step that is NOT the first, which `step_cites_own_conclusion`
+ // cannot reach: it edits step 0, where a checker that only compared against the
+ // asserted set would reject for the ordinary reason and never consult the
+ // visibility rule at all.
+ ("a_deep_step_cites_itself", |t| {
+ let mut c = parse_cert(t)?;
+ let (_, i) = first_internal_citation(&c)?;
+ let earlier: Vec<[String; 3]> = c[..i].iter().map(|s| s.concl.clone()).collect();
+ let at = c[i].prems.iter().position(|p| earlier.contains(p))?;
+ c[i].prems[at] = c[i].concl.clone();
+ Some(render_cert(&c))
+ }),
+ // The first step reaches FORWARD to the last step's conclusion. Applies to any
+ // certificate with two steps, deep or flat, and is the plainest statement of the
+ // rule: what is cited must already exist.
+ ("a_step_cites_a_later_conclusion", |t| {
+ let mut c = parse_cert(t)?;
+ if c.len() < 2 || c[0].prems.is_empty() {
+ return None;
+ }
+ c[0].prems[0] = c.last()?.concl.clone();
+ Some(render_cert(&c))
+ }),
("swap_first_two_steps", |t| {
let mut c = parse_cert(t)?;
if c.len() < 2 {
@@ -715,6 +894,139 @@ fn probe_cases() -> Vec<Case> {
.collect()
}
+// ── deep certificates, because the shipped ones are one step tall ───────────
+
+const RDF_TYPE: &str = "<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>";
+const SUBCLASS: &str = "<http://www.w3.org/2000/01/rdf-schema#subClassOf>";
+/// `rdfs9` in `tests/fixtures/horn/builtin_rules.tsv`, zero-based. Every certificate
+/// below cites this one rule, so `deep_cases` checks the row still is what it thinks
+/// before writing anything: a table edit that shifted the indices would otherwise turn
+/// this whole family into certificates about a different rule, which both checkers would
+/// reject in agreement and which would look exactly like a passing test.
+const RDFS9: usize = 4;
+
+fn class(i: usize) -> String {
+ format!("<http://ex.org/deep/C{i}>")
+}
+
+/// One `rdfs9` step: `?x type ?a`, `?a subClassOf ?b` ⊢ `?x type ?b`. The premise ORDER
+/// is the rule's body order and is part of the contract on both sides.
+fn rdfs9_step(x: &str, a: &str, b: &str) -> Step {
+ Step {
+ idx: RDFS9.to_string(),
+ binds: vec![
+ ("x".to_string(), x.to_string()),
+ ("a".to_string(), a.to_string()),
+ ("b".to_string(), b.to_string()),
+ ],
+ concl: [x.to_string(), RDF_TYPE.to_string(), b.to_string()],
+ prems: vec![
+ [x.to_string(), RDF_TYPE.to_string(), a.to_string()],
+ [a.to_string(), SUBCLASS.to_string(), b.to_string()],
+ ],
+ }
+}
+
+/// A chain `d` steps tall over a subclass ladder, then `w` steps fanning out from its
+/// tip. Step 0's premises are all asserted; step k cites step k-1's conclusion and
+/// nothing else that is not asserted; each of the final `w` steps cites the SAME
+/// conclusion, which is where a visibility check that is right for one consumer and
+/// wrong for the rest would show.
+///
+/// Every one of these is a genuine `rdfs9` derivation and both checkers must accept it.
+/// The value is not the acceptance, which is easy; it is that the mutations below then
+/// have something to break.
+fn ladder(d: usize, w: usize) -> (String, String) {
+ let ind = "<http://ex.org/deep/a>";
+ let mut asserted = format!("{ind}\t{RDF_TYPE}\t{}\n", class(0));
+ for k in 0..d {
+ asserted.push_str(&format!("{}\t{SUBCLASS}\t{}\n", class(k), class(k + 1)));
+ }
+ for j in 0..w {
+ asserted.push_str(&format!("{}\t{SUBCLASS}\t<http://ex.org/deep/B{j}>\n", class(d)));
+ }
+ let mut steps: Vec<Step> = (0..d).map(|k| rdfs9_step(ind, &class(k), &class(k + 1))).collect();
+ steps.extend(
+ (0..w).map(|j| rdfs9_step(ind, &class(d), &format!("<http://ex.org/deep/B{j}>"))),
+ );
+ (asserted, render_cert(&steps))
+}
+
+/// The deep half of the corpus: chains, a wide fan, and two of each combined. Written to
+/// scratch rather than committed because they are PARAMETERISED — the point is the
+/// distribution of depths, and a fixture per depth would be twelve files saying one
+/// thing. The hand-built adversarial pair that cannot be generated is committed instead,
+/// under `tests/fixtures/horn/deep/`; see `deep_fixture_cases`.
+fn deep_cases(scratch: &Path) -> Vec<Case> {
+ let rules = horn_fixture("builtin_rules.tsv");
+ let Ok(table) = std::fs::read_to_string(&rules) else { return Vec::new() };
+ let row = table.lines().nth(RDFS9).unwrap_or_default();
+ assert!(
+ row.starts_with("rdfs9\t2\t"),
+ "row {RDFS9} of builtin_rules.tsv is {row:?}, not rdfs9 with a two-atom body. The \
+ deep certificates cite that index by number and would now be about a different \
+ rule, which both kernels would reject together and which would read as a pass"
+ );
+
+ // (name, chain height, fan-out width). Heights chosen so the distribution is not one
+ // number: a two-step cert, a handful, and one long enough that an off-by-one in a
+ // visibility bound has somewhere to hide.
+ let shapes = [
+ ("chain-02", 2, 0),
+ ("chain-04", 4, 0),
+ ("chain-08", 8, 0),
+ ("chain-20", 20, 0),
+ ("wide-12", 1, 12),
+ ("ladder-06x08", 6, 8),
+ ];
+ let mut cases = Vec::new();
+ for (name, d, w) in shapes {
+ let dir = scratch.join("deep").join(name);
+ if std::fs::create_dir_all(&dir).is_err() {
+ continue;
+ }
+ let (asserted, cert) = ladder(d, w);
+ let ap = dir.join("asserted.tsv");
+ let cp = dir.join("cert.tsv");
+ if std::fs::write(&ap, asserted).is_err() || std::fs::write(&cp, cert).is_err() {
+ continue;
+ }
+ cases.push(Case {
+ name: format!("deep-{name}"),
+ kind: "base".to_string(),
+ rules: rules.clone(),
+ asserted: ap,
+ cert: cp,
+ });
+ }
+ cases
+}
+
+/// The two adversarial certificates that no generator produces, because a generator
+/// emits derivations and these are not derivations. Both instantiate their rule
+/// PERFECTLY: every binding covers the rule, every premise matches the body, the
+/// conclusion matches the head. The only thing wrong with either is the order, which is
+/// exactly the property under test, and both are pinned by their own tests below.
+fn deep_fixture_cases() -> Vec<Case> {
+ let d = |n: &str| repo().join("tests").join("fixtures").join("horn").join("deep").join(n);
+ let rules = horn_fixture("builtin_rules.tsv");
+ let rows: Vec<(&str, PathBuf, PathBuf)> = vec![
+ ("deep-self-support", d("self_support_asserted.tsv"), d("self_support_cert.tsv")),
+ ("deep-mutual-support", d("mutual_asserted.tsv"), d("mutual_cert.tsv")),
+ ("deep-mutual-seeded", d("mutual_seeded_asserted.tsv"), d("mutual_cert.tsv")),
+ ];
+ rows.into_iter()
+ .filter(|(_, a, c)| a.exists() && c.exists())
+ .map(|(n, a, c)| Case {
+ name: n.to_string(),
+ kind: "base".to_string(),
+ rules: rules.clone(),
+ asserted: a,
+ cert: c,
+ })
+ .collect()
+}
+
/// Expand each base case into its mutants.
fn mutate(bases: &[Case], scratch: &Path) -> Vec<Case> {
let mut out = Vec::new();
@@ -763,8 +1075,16 @@ fn mutate(bases: &[Case], scratch: &Path) -> Vec<Case> {
out
}
+/// A directory of this run's own, and a DIFFERENT one on every call.
+///
+/// The process id alone was enough while one test generated a corpus. Three do now, and
+/// `cargo test` runs them in parallel by default, so a shared path would have one test
+/// deleting another's certificates mid-run — a flake that looks like a disagreement,
+/// which is the one failure this file must never report falsely.
fn scratch_dir() -> PathBuf {
- let d = std::env::temp_dir().join(format!("oo-cross-kernel-{}", std::process::id()));
+ static NEXT: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0);
+ let n = NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ let d = std::env::temp_dir().join(format!("oo-cross-kernel-{}-{n}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).expect("create scratch dir");
d
@@ -1075,6 +1395,8 @@ fn the_two_kernels_agree_on_the_whole_corpus() {
let scratch = scratch_dir();
let mut bases = fixture_cases();
bases.extend(probe_cases());
+ bases.extend(deep_fixture_cases());
+ bases.extend(deep_cases(&scratch));
bases.extend(generate_base_certificates(&scratch));
let mutants = mutate(&bases, &scratch);
let fuzzed = fuzz(&bases, &scratch);
@@ -1147,9 +1469,23 @@ fn the_two_kernels_agree_on_the_whole_corpus() {
}
let refused_total: usize = refused.values().sum();
+ // The corpus measured by SHAPE and not only by row count, printed next to the
+ // agreement number so the two are read together. A row count says how many files
+ // were compared; these say how many of them could express the ordering property the
+ // checker's induction rests on. See
+ // `the_corpus_exercises_prefix_visibility_at_depth`, which holds the floor.
+ let shapes: Vec<Shape> =
+ bases.iter().chain(mutants.iter()).chain(fuzzed.iter()).map(shape_of_case).collect();
+ let deepest = shapes.iter().map(|s| s.depth).max().unwrap_or(0);
+ let widest = shapes.iter().map(|s| s.fan_out).max().unwrap_or(0);
+ let uses = shapes.iter().filter(|s| s.citing_earlier > 0).count();
+ let violates = shapes.iter().filter(|s| s.citing_self_or_later > 0).count();
+
println!(
"cross-kernel differential\n \
certificates: {total} ({} base, {} mutated, {} fuzzed, seed {FUZZ_SEED:#x})\n \
+ prefix visibility: {} rows exercise it ({uses} rest on it to be accepted, \
+ {violates} must be rejected by it); deepest chain {deepest}, widest fan-out {widest}\n \
both accept: {agree_accept}\n \
both reject: {agree_reject}\n \
both exit 2: {agree_unreadable}\n \
@@ -1158,6 +1494,7 @@ fn the_two_kernels_agree_on_the_whole_corpus() {
bases.len(),
mutants.len(),
fuzzed.len(),
+ uses + violates,
divergent.len()
);
for (kind, n) in &refused {
@@ -1757,3 +2094,231 @@ fn agreement_is_about_the_definitions_and_is_not_itself_a_proof() {
);
}
}
+
+/// **The measure, not the claim.** How deep the corpus actually is.
+///
+/// A differential over 1,718 certificates sounds like a lot until you ask how many of
+/// them can express the property the checker's induction rests on. A step may cite only
+/// what came strictly before it and never itself; a ONE-STEP certificate cannot violate
+/// that and cannot exercise it either, because every premise it has is asserted and the
+/// ordering logic is never reached. Before the deep cases were added, exactly one base
+/// certificate in the corpus contained a step citing an earlier step's conclusion, so
+/// the discipline was differentially tested by a single two-step fixture.
+///
+/// This test prints the distribution and holds a floor under it. It deliberately does
+/// NOT require either proof assistant: the shape of the corpus is a fact about the
+/// files, and it should still be measurable on a machine that cannot run the kernels.
+#[test]
+fn the_corpus_exercises_prefix_visibility_at_depth() {
+ let scratch = scratch_dir();
+ let mut bases = fixture_cases();
+ bases.extend(probe_cases());
+ bases.extend(deep_fixture_cases());
+ bases.extend(deep_cases(&scratch));
+ bases.extend(generate_base_certificates(&scratch));
+ let mutants = mutate(&bases, &scratch);
+ let fuzzed = fuzz(&bases, &scratch);
+
+ let report = |label: &str, cases: &[Case]| -> (usize, usize) {
+ let shapes: Vec<Shape> = cases.iter().map(shape_of_case).collect();
+ let mut hist: BTreeMap<usize, usize> = BTreeMap::new();
+ for s in &shapes {
+ *hist.entry(s.depth).or_default() += 1;
+ }
+ let uses = shapes.iter().filter(|s| s.citing_earlier > 0).count();
+ let violates = shapes.iter().filter(|s| s.citing_self_or_later > 0).count();
+ println!(
+ " {label:<10} {:>5} certificates depth {} max fan-out {}\n \
+ {uses:>5} cite an earlier conclusion, {violates:>5} cite themselves or a later step",
+ cases.len(),
+ hist.iter().map(|(d, n)| format!("{d}:{n}")).collect::<Vec<_>>().join(" "),
+ shapes.iter().map(|s| s.fan_out).max().unwrap_or(0),
+ );
+ (uses, violates)
+ };
+
+ println!("prefix-visibility coverage (depth d:count, d = longest derivation chain)");
+ let (base_uses, base_violates) = report("base", &bases);
+ let (mut_uses, mut_violates) = report("mutated", &mutants);
+ let (fuzz_uses, fuzz_violates) = report("fuzzed", &fuzzed);
+ let uses = base_uses + mut_uses + fuzz_uses;
+ let violates = base_violates + mut_violates + fuzz_violates;
+ println!(
+ " TOTAL {} rows exercise the discipline: {uses} rest on it to be accepted, \
+ {violates} must be rejected by it",
+ uses + violates
+ );
+
+ let deepest = bases.iter().map(|c| shape_of_case(c).depth).max().unwrap_or(0);
+ let widest = bases.iter().map(|c| shape_of_case(c).fan_out).max().unwrap_or(0);
+ assert!(
+ deepest >= 15,
+ "the deepest base certificate is {deepest} steps of chaining. A shallow corpus \
+ tests the checker's ordering logic on one or two links, which is where an \
+ off-by-one hides"
+ );
+ assert!(widest >= 10, "no base certificate has {widest} or more steps citing one conclusion");
+ assert!(
+ base_uses >= 8,
+ "only {base_uses} base certificates cite an earlier conclusion at all; the corpus \
+ has gone flat again and the discipline is back to being tested by one fixture"
+ );
+ assert!(
+ violates >= 40,
+ "only {violates} rows in the whole corpus cite themselves or a later step, so the \
+ REJECTING half of the discipline is barely exercised"
+ );
+
+ let _ = std::fs::remove_dir_all(&scratch);
+}
+
+/// **Self-support, in the one form that isolates it.** A step whose only non-asserted
+/// premise is its OWN conclusion, and which is otherwise perfect.
+///
+/// `tests/fixtures/horn/deep/self_support_cert.tsv` cites `rdfs9` with `a` and `b` bound
+/// to the SAME class, so the rule's body is `<a> type <A>`, `<A> subClassOf <A>` and its
+/// head is `<a> type <A>`. The head and the first premise are then the same triple. The
+/// binding covers every variable of the rule, has no repeated key, and instantiates the
+/// body into exactly the premises the step lists: nothing a duplicate-key or
+/// coverage check would catch is wrong with it.
+///
+/// `self_support_asserted.tsv` supplies `<A> subClassOf <A>` and NOTHING else, so the
+/// remaining premise is unavailable unless a step may cite itself. Both checkers reject,
+/// and Isabelle names the reason: `premise_unknown:0`.
+///
+/// Why it is worth committing rather than mutating into existence: this is the case the
+/// induction in both formalisations is FOR. A checker that resolved premises against
+/// "the asserted graph plus every conclusion in the file" rather than "plus every
+/// conclusion strictly before this one" accepts it, and having accepted it accepts the
+/// derivation of anything at all, since a self-supporting step needs no input. The
+/// shipped fixture set could not express it: `bad_self.tsv` swaps a premise for the
+/// conclusion under a binding that no longer instantiates the body, so it is rejected
+/// for two reasons at once and says nothing about which.
+#[test]
+fn a_step_may_not_cite_its_own_conclusion() {
+ if skip() {
+ return;
+ }
+ let dir = repo().join("tests").join("fixtures").join("horn").join("deep");
+ let rules = horn_fixture("builtin_rules.tsv");
+ let asserted = dir.join("self_support_asserted.tsv");
+ let cert = dir.join("self_support_cert.tsv");
+ if common::skip_unless(
+ asserted.exists() && cert.exists(),
+ "tests/fixtures/horn/deep/self_support_*.tsv",
+ "they are committed",
+ ) {
+ return;
+ }
+
+ // The shape is asserted from the file, not assumed, or a later edit could turn this
+ // into a test of an ordinary rejection without anyone noticing.
+ let steps = parse_cert(&std::fs::read_to_string(&cert).unwrap()).expect("it must parse");
+ assert_eq!(steps.len(), 1);
+ assert_eq!(steps[0].prems[0], steps[0].concl, "premise 0 must BE the conclusion");
+ let known = asserted_triples(&asserted);
+ assert!(!known.contains(&steps[0].concl), "and it must not be asserted, or nothing is proved");
+ assert!(known.contains(&steps[0].prems[1]), "while the other premise must be asserted");
+
+ let lean = run_lean(&rules, &asserted, &cert);
+ let isa = run_isabelle(&rules, &asserted, &cert);
+ assert_eq!(lean, isa, "the two kernels disagree about self-support: {lean:?} vs {isa:?}");
+ assert_eq!(lean.exit, 1, "a self-supporting step must be REJECTED, not accepted and not a parse error");
+}
+
+/// **Mutual support.** Two steps, each citing the other's conclusion, and a control that
+/// shows the rejection is about the ordering and about nothing else in the three files.
+///
+/// `mutual_cert.tsv` is two `rdfs9` steps over `<A> subClassOf <B>` and
+/// `<B> subClassOf <A>`: step 0 concludes `<a> type <B>` from `<a> type <A>`, and step 1
+/// concludes `<a> type <A>` from `<a> type <B>`. Each premise the other step supplies.
+/// `mutual_asserted.tsv` asserts only the two subclass axioms, so `<a>` is never said to
+/// be of any type at all, and the pair manufactures both conclusions out of each other.
+///
+/// The control is the SAME certificate against `mutual_seeded_asserted.tsv`, which adds
+/// the single triple `<a> type <A>`. Now step 0's first premise is asserted, step 1's is
+/// step 0's conclusion, the order is legal, and both checkers accept with the ABSOLUTE
+/// verdict. One triple is the whole difference between a cycle and a derivation, and the
+/// discipline is the only thing that tells them apart.
+#[test]
+fn two_steps_may_not_support_each_other() {
+ if skip() {
+ return;
+ }
+ let dir = repo().join("tests").join("fixtures").join("horn").join("deep");
+ let rules = horn_fixture("builtin_rules.tsv");
+ let cert = dir.join("mutual_cert.tsv");
+ let cycle = dir.join("mutual_asserted.tsv");
+ let seeded = dir.join("mutual_seeded_asserted.tsv");
+ if common::skip_unless(
+ cert.exists() && cycle.exists() && seeded.exists(),
+ "tests/fixtures/horn/deep/mutual_*.tsv",
+ "they are committed",
+ ) {
+ return;
+ }
+
+ let steps = parse_cert(&std::fs::read_to_string(&cert).unwrap()).expect("it must parse");
+ assert_eq!(steps.len(), 2);
+ assert_eq!(steps[0].prems[0], steps[1].concl, "step 0 must cite step 1's conclusion");
+ assert_eq!(steps[1].prems[0], steps[0].concl, "and step 1 must cite step 0's");
+ let unseeded = asserted_triples(&cycle);
+ assert!(
+ !unseeded.contains(&steps[0].concl) && !unseeded.contains(&steps[1].concl),
+ "neither conclusion may be asserted, or the cycle is not load-bearing"
+ );
+ assert_eq!(
+ asserted_triples(&seeded).difference(&unseeded).count(),
+ 1,
+ "the control must differ by exactly ONE triple"
+ );
+
+ let lean = run_lean(&rules, &cycle, &cert);
+ let isa = run_isabelle(&rules, &cycle, &cert);
+ assert_eq!(lean, isa, "the two kernels disagree about mutual support: {lean:?} vs {isa:?}");
+ assert_eq!(lean.exit, 1, "a pair of steps supporting each other must be REJECTED");
+
+ let lean = run_lean(&rules, &seeded, &cert);
+ let isa = run_isabelle(&rules, &seeded, &cert);
+ assert_eq!(lean, isa, "the two kernels disagree on the control: {lean:?} vs {isa:?}");
+ assert_eq!(
+ lean.exit, 0,
+ "the SAME certificate, with one triple asserted, must be accepted; if it is not, \
+ the rejection above was about something other than the ordering and this pair \
+ proves nothing"
+ );
+ assert_eq!(lean.verdict.as_deref(), Some("entailed"));
+}
+
+/// The deep certificates are accepted, and are deep. Two separate claims: a generator
+/// that quietly emitted flat certificates would still pass the differential, because
+/// both kernels agree on flat certificates too.
+#[test]
+fn the_generated_deep_certificates_check_and_are_deep() {
+ if skip() {
+ return;
+ }
+ let scratch = scratch_dir();
+ let cases = deep_cases(&scratch);
+ assert!(!cases.is_empty(), "the deep generator produced nothing");
+ for case in &cases {
+ let shape = shape_of_case(case);
+ assert!(
+ shape.depth >= 1 && shape.citing_earlier >= 1,
+ "{} is flat: {shape:?}. A base case that cites nothing exercises no ordering",
+ case.name
+ );
+ let lean = run_lean(&case.rules, &case.asserted, &case.cert);
+ let isa = run_isabelle(&case.rules, &case.asserted, &case.cert);
+ assert_eq!(lean, isa, "the two kernels disagree on {}: {lean:?} vs {isa:?}", case.name);
+ assert_eq!(
+ lean.exit, 0,
+ "{} is a real rdfs9 derivation in a legal order and must be ACCEPTED (depth \
+ {}, fan-out {}); if it is rejected the generator is wrong and every deep \
+ mutation below is mutating rubbish",
+ case.name, shape.depth, shape.fan_out
+ );
+ assert_eq!(lean.verdict.as_deref(), Some("entailed"), "{} over the built-in table", case.name);
+ }
+ let _ = std::fs::remove_dir_all(&scratch);
+}
diff --git a/tests/fixtures/horn-coverage/README.md b/tests/fixtures/horn-coverage/README.md
new file mode 100644
index 00000000..07e5e054
--- /dev/null
+++ b/tests/fixtures/horn-coverage/README.md
@@ -0,0 +1,38 @@
+# One graph per rule the repository's own RDF never fires
+
+`tools/horn_differential.py` compares the Rust reasoner against the Python one over every
+RDF document this repository tracks. On 14 September 2026 that corpus fired **20 of the
+27 rules** in the built-in Horn table. The differential therefore compared the two
+engines zero times on the other seven, and its clean sweep said nothing whatever about
+them:
+
+ prp-inv2 eq-sym cls-avf cls-hv1 cls-hv2 scm-svf2 scm-avf2
+
+This directory holds one Turtle file per silent rule. Each is the smallest graph that
+makes exactly that rule fire — one derivation, credited to that rule, nothing else — and
+each carries the W3C rule, its table, and its expected derivation in a comment at the top
+of the file.
+
+## These are TEST DATA and are not part of the corpus
+
+They are deliberately constructed to make one rule fire. Mixing them into the corpus of
+real ontologies would inflate the coverage number with graphs that were written to hit
+it, which is measuring the ruler. So:
+
+* `discover()` in `tools/horn_differential.py` EXCLUDES this directory, and a test in
+ `python/tests/test_horn_three_way_differential.py` asserts that it does.
+* The tool runs them as a separate, labelled set and reports two coverage numbers: how
+ many rules fired on the repository's own RDF, and how many fired once these are added.
+ Rules that fire ONLY here are named on every run.
+
+A rule covered only by a fixture in this directory has been differentially compared on a
+graph written for the purpose and on nothing else. That is much better than zero and it
+is not the same as evidence from a real ontology, which is why the report keeps the two
+apart rather than adding them up.
+
+## Adding one
+
+Name the file after the rule exactly as the rule table names it (`<rule>.ttl`); the tool
+reads the target rule off the stem and checks that this is the rule that fires. Keep it
+minimal, and put the W3C rule text in the header comment so a reader can check the
+fixture against the standard without leaving the file.
diff --git a/tests/fixtures/horn-coverage/cls-avf.ttl b/tests/fixtures/horn-coverage/cls-avf.ttl
new file mode 100644
index 00000000..b966b442
--- /dev/null
+++ b/tests/fixtures/horn-coverage/cls-avf.ttl
@@ -0,0 +1,34 @@
+# COVERAGE FIXTURE. Deliberate test data, not an ontology about anything.
+#
+# RULE: cls-avf
+# W3C OWL 2 Web Ontology Language Profiles (Second Edition), section 4.3,
+# Table 6: The Semantics of Classes.
+# https://www.w3.org/TR/owl2-profiles/#Reasoning_in_OWL_2_RL_and_RDF_Graphs_using_Rules
+#
+# T(?x, owl:allValuesFrom, ?y)
+# T(?x, owl:onProperty, ?p)
+# T(?u, rdf:type, ?x)
+# T(?u, ?p, ?v)
+# -> T(?v, rdf:type, ?y)
+#
+# WHY IT EXISTS. The repository ships allValuesFrom restrictions in its TBoxes
+# and instance data that uses the restricted properties, but never both in one
+# graph, so tools/horn_differential.py compared the Rust and Python engines on
+# cls-avf zero times and its clean sweep said nothing about it.
+#
+# EXPECTED: exactly one derivation, credited to cls-avf:
+# :y rdf:type :C
+#
+# The restriction is a NAMED node rather than the blank node OWL would normally
+# use. The rule is a triple pattern and does not care, and a named node keeps
+# the expected derivation writable here without a blank-node label to track.
+
+@prefix : <http://ex.org/cover/cls-avf#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+
+:R a owl:Restriction ;
+ owl:onProperty :p ;
+ owl:allValuesFrom :C .
+
+:x a :R ;
+ :p :y .
diff --git a/tests/fixtures/horn-coverage/cls-hv1.ttl b/tests/fixtures/horn-coverage/cls-hv1.ttl
new file mode 100644
index 00000000..3dd868c6
--- /dev/null
+++ b/tests/fixtures/horn-coverage/cls-hv1.ttl
@@ -0,0 +1,35 @@
+# COVERAGE FIXTURE. Deliberate test data, not an ontology about anything.
+#
+# RULE: cls-hv1
+# W3C OWL 2 Web Ontology Language Profiles (Second Edition), section 4.3,
+# Table 6: The Semantics of Classes.
+# https://www.w3.org/TR/owl2-profiles/#Reasoning_in_OWL_2_RL_and_RDF_Graphs_using_Rules
+#
+# T(?x, owl:hasValue, ?y)
+# T(?x, owl:onProperty, ?p)
+# T(?u, rdf:type, ?x)
+# -> T(?u, ?p, ?y)
+#
+# WHY IT EXISTS. No document in this repository asserts owl:hasValue, so
+# tools/horn_differential.py compared the Rust and Python engines on cls-hv1
+# zero times and its clean sweep said nothing about it. That is worth closing
+# for this rule in particular: src/reason.rs records that the arm emitted under
+# this name was once the COMPOSITE of cax-sco and W3C cls-hv1, demanding a
+# subclass step the standard rule does not ask for. A rule that has already been
+# wrong once, and that no corpus document exercises, is the weakest row in the
+# table.
+#
+# EXPECTED: exactly one derivation, credited to cls-hv1:
+# :x :p :v
+#
+# cls-hv2 then fires on that conclusion and re-derives ":x rdf:type :R", which
+# is asserted, so it adds no step and takes no credit.
+
+@prefix : <http://ex.org/cover/cls-hv1#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+
+:R a owl:Restriction ;
+ owl:onProperty :p ;
+ owl:hasValue :v .
+
+:x a :R .
diff --git a/tests/fixtures/horn-coverage/cls-hv2.ttl b/tests/fixtures/horn-coverage/cls-hv2.ttl
new file mode 100644
index 00000000..abf8a096
--- /dev/null
+++ b/tests/fixtures/horn-coverage/cls-hv2.ttl
@@ -0,0 +1,35 @@
+# COVERAGE FIXTURE. Deliberate test data, not an ontology about anything.
+#
+# RULE: cls-hv2
+# W3C OWL 2 Web Ontology Language Profiles (Second Edition), section 4.3,
+# Table 6: The Semantics of Classes.
+# https://www.w3.org/TR/owl2-profiles/#Reasoning_in_OWL_2_RL_and_RDF_Graphs_using_Rules
+#
+# T(?x, owl:hasValue, ?y)
+# T(?x, owl:onProperty, ?p)
+# T(?u, ?p, ?y)
+# -> T(?u, rdf:type, ?x)
+#
+# WHY IT EXISTS. No document in this repository asserts owl:hasValue, so
+# tools/horn_differential.py compared the Rust and Python engines on cls-hv2
+# zero times and its clean sweep said nothing about it.
+#
+# EXPECTED: exactly one derivation, credited to cls-hv2:
+# :x rdf:type :R
+#
+# The converse of cls-hv1.ttl, and deliberately a separate file: this one starts
+# from the PROPERTY assertion and concludes the type, the other starts from the
+# type and concludes the property assertion. Putting both in one graph would
+# leave each rule's conclusion asserted for the other and neither would fire.
+#
+# cls-hv1 then fires on the conclusion and re-derives ":x :p :v", which is
+# asserted, so it adds no step and takes no credit.
+
+@prefix : <http://ex.org/cover/cls-hv2#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+
+:R a owl:Restriction ;
+ owl:onProperty :p ;
+ owl:hasValue :v .
+
+:x :p :v .
diff --git a/tests/fixtures/horn-coverage/eq-sym.ttl b/tests/fixtures/horn-coverage/eq-sym.ttl
new file mode 100644
index 00000000..87f53a7c
--- /dev/null
+++ b/tests/fixtures/horn-coverage/eq-sym.ttl
@@ -0,0 +1,28 @@
+# COVERAGE FIXTURE. Deliberate test data, not an ontology about anything.
+#
+# RULE: eq-sym
+# W3C OWL 2 Web Ontology Language Profiles (Second Edition), section 4.3,
+# Table 4: The Semantics of Equality.
+# https://www.w3.org/TR/owl2-profiles/#Reasoning_in_OWL_2_RL_and_RDF_Graphs_using_Rules
+#
+# T(?x, owl:sameAs, ?y)
+# -> T(?y, owl:sameAs, ?x)
+#
+# WHY IT EXISTS. No document in this repository asserts owl:sameAs, so
+# tools/horn_differential.py compared the Rust and Python engines on eq-sym zero
+# times and its clean sweep said nothing about it.
+#
+# EXPECTED: exactly one derivation, credited to eq-sym:
+# :b owl:sameAs :a
+#
+# eq-sym then fires on its own conclusion and re-derives ":a owl:sameAs :b",
+# which is asserted, so the closure is reached in one round.
+#
+# The table holds eq-sym alone out of the eq-* family: none of eq-ref, eq-trans
+# or the eq-rep-* replacement rules is a built-in here, so nothing in this file
+# propagates the identity into any other triple.
+
+@prefix : <http://ex.org/cover/eq-sym#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+
+:a owl:sameAs :b .
diff --git a/tests/fixtures/horn-coverage/prp-inv2.ttl b/tests/fixtures/horn-coverage/prp-inv2.ttl
new file mode 100644
index 00000000..672eac6a
--- /dev/null
+++ b/tests/fixtures/horn-coverage/prp-inv2.ttl
@@ -0,0 +1,26 @@
+# COVERAGE FIXTURE. Deliberate test data, not an ontology about anything.
+#
+# RULE: prp-inv2
+# W3C OWL 2 Web Ontology Language Profiles (Second Edition), section 4.3,
+# Table 5: The Semantics of Axioms about Properties.
+# https://www.w3.org/TR/owl2-profiles/#Reasoning_in_OWL_2_RL_and_RDF_Graphs_using_Rules
+#
+# T(?p1, owl:inverseOf, ?p2)
+# T(?x, ?p2, ?y)
+# -> T(?y, ?p1, ?x)
+#
+# WHY IT EXISTS. The repository's own RDF never fires prp-inv2, so
+# tools/horn_differential.py compared the Rust and Python engines on it zero
+# times and its clean sweep said nothing about it.
+#
+# EXPECTED: exactly one derivation, credited to prp-inv2:
+# :y :p :x
+#
+# The mirror rule prp-inv1 then fires on that conclusion and re-derives
+# ":x :q :y", which is already asserted, so it adds no step and takes no credit.
+
+@prefix : <http://ex.org/cover/prp-inv2#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+
+:p owl:inverseOf :q .
+:x :q :y .
diff --git a/tests/fixtures/horn-coverage/scm-avf2.ttl b/tests/fixtures/horn-coverage/scm-avf2.ttl
new file mode 100644
index 00000000..6bb47b65
--- /dev/null
+++ b/tests/fixtures/horn-coverage/scm-avf2.ttl
@@ -0,0 +1,46 @@
+# COVERAGE FIXTURE. Deliberate test data, not an ontology about anything.
+#
+# RULE: scm-avf2
+# W3C OWL 2 Web Ontology Language Profiles (Second Edition), section 4.3,
+# Table 9: The Semantics of Schema Vocabulary.
+# https://www.w3.org/TR/owl2-profiles/#Reasoning_in_OWL_2_RL_and_RDF_Graphs_using_Rules
+#
+# T(?c1, owl:allValuesFrom, ?y)
+# T(?c1, owl:onProperty, ?p1)
+# T(?c2, owl:allValuesFrom, ?y)
+# T(?c2, owl:onProperty, ?p2)
+# T(?p1, rdfs:subPropertyOf, ?p2)
+# -> T(?c2, rdfs:subClassOf, ?c1)
+#
+# WHY IT EXISTS. Same shape as scm-svf2.ttl with allValuesFrom in place of
+# someValuesFrom, and nothing in this repository has it, so
+# tools/horn_differential.py compared the Rust and Python engines on scm-avf2
+# zero times and its clean sweep said nothing about it.
+#
+# NOTE THE DIRECTION. The consequent is T(?c2, rdfs:subClassOf, ?c1), which is
+# the REVERSE of scm-svf2's. Universal quantification is antitone in the
+# property: widening the property narrows the class. A transcription that
+# copied the someValuesFrom row and changed the vocabulary would get the
+# direction wrong, and with no corpus document firing the rule nothing would
+# have noticed. This fixture is what would notice, because it pins which of
+# :R1 and :R2 is the subclass.
+#
+# EXPECTED: exactly one derivation, credited to scm-avf2:
+# :R2 rdfs:subClassOf :R1
+#
+# scm-avf1 does not also fire: it would need :C rdfs:subClassOf :C, which this
+# file does not assert, and the built-in table has no reflexivity rule.
+
+@prefix : <http://ex.org/cover/scm-avf2#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+
+:R1 a owl:Restriction ;
+ owl:onProperty :p1 ;
+ owl:allValuesFrom :C .
+
+:R2 a owl:Restriction ;
+ owl:onProperty :p2 ;
+ owl:allValuesFrom :C .
+
+:p1 rdfs:subPropertyOf :p2 .
diff --git a/tests/fixtures/horn-coverage/scm-svf2.ttl b/tests/fixtures/horn-coverage/scm-svf2.ttl
new file mode 100644
index 00000000..25012a0a
--- /dev/null
+++ b/tests/fixtures/horn-coverage/scm-svf2.ttl
@@ -0,0 +1,42 @@
+# COVERAGE FIXTURE. Deliberate test data, not an ontology about anything.
+#
+# RULE: scm-svf2
+# W3C OWL 2 Web Ontology Language Profiles (Second Edition), section 4.3,
+# Table 9: The Semantics of Schema Vocabulary.
+# https://www.w3.org/TR/owl2-profiles/#Reasoning_in_OWL_2_RL_and_RDF_Graphs_using_Rules
+#
+# T(?c1, owl:someValuesFrom, ?y)
+# T(?c1, owl:onProperty, ?p1)
+# T(?c2, owl:someValuesFrom, ?y)
+# T(?c2, owl:onProperty, ?p2)
+# T(?p1, rdfs:subPropertyOf, ?p2)
+# -> T(?c1, rdfs:subClassOf, ?c2)
+#
+# WHY IT EXISTS. scm-svf2 needs two someValuesFrom restrictions over the SAME
+# filler whose properties are related by rdfs:subPropertyOf. Nothing in this
+# repository has that shape, so tools/horn_differential.py compared the Rust and
+# Python engines on scm-svf2 zero times and its clean sweep said nothing about
+# it. Its sibling scm-svf1, which needs one shared property and two fillers
+# related by rdfs:subClassOf, does fire in the corpus; the two differ only in
+# WHICH of the two positions carries the ordering, which is exactly the kind of
+# difference a reader assumes is covered because the neighbour is.
+#
+# EXPECTED: exactly one derivation, credited to scm-svf2:
+# :R1 rdfs:subClassOf :R2
+#
+# scm-svf1 does not also fire: it would need :C rdfs:subClassOf :C, which this
+# file does not assert, and the built-in table has no reflexivity rule.
+
+@prefix : <http://ex.org/cover/scm-svf2#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+
+:R1 a owl:Restriction ;
+ owl:onProperty :p1 ;
+ owl:someValuesFrom :C .
+
+:R2 a owl:Restriction ;
+ owl:onProperty :p2 ;
+ owl:someValuesFrom :C .
+
+:p1 rdfs:subPropertyOf :p2 .
diff --git a/tests/fixtures/horn/deep/mutual_asserted.tsv b/tests/fixtures/horn/deep/mutual_asserted.tsv
new file mode 100644
index 00000000..7b347002
--- /dev/null
+++ b/tests/fixtures/horn/deep/mutual_asserted.tsv
@@ -0,0 +1,2 @@
+<http://ex.org/A> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://ex.org/B>
+<http://ex.org/B> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://ex.org/A>
diff --git a/tests/fixtures/horn/deep/mutual_cert.tsv b/tests/fixtures/horn/deep/mutual_cert.tsv
new file mode 100644
index 00000000..8cc5af0c
--- /dev/null
+++ b/tests/fixtures/horn/deep/mutual_cert.tsv
@@ -0,0 +1,2 @@
+4 3 x <http://ex.org/a> a <http://ex.org/A> b <http://ex.org/B> <http://ex.org/a> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ex.org/B> <http://ex.org/a> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ex.org/A> <http://ex.org/A> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://ex.org/B>
+4 3 x <http://ex.org/a> a <http://ex.org/B> b <http://ex.org/A> <http://ex.org/a> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ex.org/A> <http://ex.org/a> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ex.org/B> <http://ex.org/B> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://ex.org/A>
diff --git a/tests/fixtures/horn/deep/mutual_seeded_asserted.tsv b/tests/fixtures/horn/deep/mutual_seeded_asserted.tsv
new file mode 100644
index 00000000..1f0b84c9
--- /dev/null
+++ b/tests/fixtures/horn/deep/mutual_seeded_asserted.tsv
@@ -0,0 +1,3 @@
+<http://ex.org/A> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://ex.org/B>
+<http://ex.org/B> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://ex.org/A>
+<http://ex.org/a> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ex.org/A>
diff --git a/tests/fixtures/horn/deep/self_support_asserted.tsv b/tests/fixtures/horn/deep/self_support_asserted.tsv
new file mode 100644
index 00000000..4bddfd46
--- /dev/null
+++ b/tests/fixtures/horn/deep/self_support_asserted.tsv
@@ -0,0 +1 @@
+<http://ex.org/A> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://ex.org/A>
diff --git a/tests/fixtures/horn/deep/self_support_cert.tsv b/tests/fixtures/horn/deep/self_support_cert.tsv
new file mode 100644
index 00000000..0718c08f
--- /dev/null
+++ b/tests/fixtures/horn/deep/self_support_cert.tsv
@@ -0,0 +1 @@
+4 3 x <http://ex.org/a> a <http://ex.org/A> b <http://ex.org/A> <http://ex.org/a> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ex.org/A> <http://ex.org/a> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ex.org/A> <http://ex.org/A> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://ex.org/A>
diff --git a/tests/reason_rl_coverage_test.rs b/tests/reason_rl_coverage_test.rs
index c3b855d0..44bdc8f9 100644
--- a/tests/reason_rl_coverage_test.rs
+++ b/tests/reason_rl_coverage_test.rs
@@ -473,14 +473,44 @@ fn the_new_rules_earn_their_place_on_the_shipped_corpus() {
const SKIP_DIRS: [&str; 8] = [
"node_modules", "target", ".lake", ".git", ".venv", "venv", "site-packages", "__pycache__",
];
+ // `tests/fixtures/horn-coverage/` holds one graph per rule the shipped corpus
+ // never fires, each written to make exactly that rule fire, so the Rust/Python
+ // differential can compare the two engines on rules no real ontology reaches. They
+ // are EXCLUDED here, and the exclusion is the whole point of the directory: this
+ // test asks whether the ten new rules earn their place on the corpus this
+ // repository SHIPS, and a rule that earns its place on a file written to make it
+ // fire has earned nothing. Without this line the silent list at the bottom would
+ // empty itself out of its own test data. `tools/horn_differential.py` excludes the
+ // same directory for the same reason and reports it as a separate figure.
+ const COVERAGE_FIXTURES: &str = "tests/fixtures/horn-coverage/";
+ let coverage_dir = repo().join("tests").join("fixtures").join("horn-coverage");
+ assert!(
+ coverage_dir.is_dir(),
+ "{} is missing. If the directory moved, the exclusion below is a no-op and this \
+ test is now measuring data written to make its own assertions pass",
+ coverage_dir.display()
+ );
+
let out = Command::new("git")
.args(["ls-files", "-z", "*.ttl", "*.owl", "*.rdf", "*.nt"])
.current_dir(repo())
.output()
.expect("run git ls-files");
- let mut files: Vec<PathBuf> = String::from_utf8_lossy(&out.stdout)
+ let tracked: Vec<String> = String::from_utf8_lossy(&out.stdout)
.split('\0')
.filter(|p| !p.is_empty())
+ .map(|p| p.to_string())
+ .collect();
+ let excluded = tracked.iter().filter(|p| p.starts_with(COVERAGE_FIXTURES)).count();
+ assert!(
+ excluded > 0,
+ "no tracked file under {COVERAGE_FIXTURES} was excluded. Either the fixtures are \
+ untracked, in which case `git ls-files` never saw them and this guard is fine to \
+ relax, or the prefix has drifted and they are being counted as corpus"
+ );
+ let mut files: Vec<PathBuf> = tracked
+ .iter()
+ .filter(|p| !p.starts_with(COVERAGE_FIXTURES))
.filter(|p| !SKIP_DIRS.iter().any(|d| p.split('/').any(|seg| seg == *d)))
.map(|p| repo().join(p))
.collect();
@@ -567,6 +597,25 @@ fn the_new_rules_earn_their_place_on_the_shipped_corpus() {
// biggest contributor below, not because a file here needs them, and the
// honest way to say so is an assertion that fails the day it stops being
// true.
+ //
+ // `tests/fixtures/horn-coverage/scm-svf2.ttl` and `scm-avf2.ttl` DO fire two of
+ // these three, and are excluded from the sweep above precisely so that they cannot
+ // shorten this list. A graph written to make a rule fire is evidence that the rule
+ // is implemented, which is what that directory is for; it is not evidence that any
+ // ontology needs the rule, which is what this list is about.
+ //
+ // `scm-avf1` has NO fixture, and the reason is a discrepancy rather than an
+ // oversight. This sweep runs the `owl-rl-ext` PROFILE and measures zero.
+ // `tools/horn_differential.py` runs the 27-row SUPPLIED TABLE in
+ // `tests/fixtures/horn/builtin_rules.tsv` and credits `scm-avf1` once, on
+ // `benchmark/reference/pizza-reference.owl`, so it was never on that tool's list of
+ // rules the corpus cannot reach and no coverage fixture was written for it. The two
+ // rule sets are not the same rule set and the same file separates them elsewhere
+ // too: on pizza, measured 15 September 2026, the profile derives 357 `rdfs11` and
+ // 101 `scm-svf1` where the table derives 387 and 102, and the profile additionally
+ // fires `cls-int2` and `cls-oo`, which the table does not contain. Neither number is
+ // known to be wrong. What is wrong is quoting either as "the corpus", and this
+ // comment is here so the zero below is not read as a claim about the other one.
for silent in ["scm-svf2", "scm-avf1", "scm-avf2"] {
assert_eq!(
totals.get(silent).copied().unwrap_or(0),
diff --git a/tools/horn_differential.py b/tools/horn_differential.py
index 0b630ad3..a1069f20 100644
--- a/tools/horn_differential.py
+++ b/tools/horn_differential.py
@@ -3,11 +3,33 @@
over one Horn rule table and every ontology this repository ships.
WHY THIS EXISTS. `src/reason.rs` and `python/src/open_ontologies_lite/horn/` are two
-independent implementations of the same specification, and `lean/` holds a checker
-whose soundness is a machine-checked theorem. Running all three over one corpus finds
-defects none of them finds alone. Agreement on a handful of hand-made files is weak
-evidence, so the corpus is every RDF document tracked in the repository, and the report
-says how many ontologies and how many derived triples the claim rests on.
+implementations of the same specification, and `lean/` holds a checker whose soundness
+is a machine-checked theorem. Running all three over one corpus finds defects none of
+them finds alone. Agreement on a handful of hand-made files is weak evidence, so the
+corpus is every RDF document tracked in the repository, and the report says how many
+ontologies and how many derived triples the claim rests on.
+
+HOW INDEPENDENT THE TWO ENGINES ARE, WHICH IS LESS THAN "INDEPENDENT". This is the
+first thing to know about every number below and it is printed on every run, next to
+the agreement count, so it cannot be quoted without it.
+
+The Rust and Python reasoners are NOT independent implementations. They run the same
+semi-naive forward-chaining algorithm over the same rule table, the Python was written
+with the Rust open, and the Python's comments cite `src/reason.rs` by file and line.
+What their agreement is therefore strong evidence about is a TRANSCRIPTION SLIP: an
+index off by one, a guard dropped, a join in the wrong order, a rule arm that reads the
+subject where it meant the object. Two people copying one design do not make the same
+slip twice.
+
+What it is close to NO evidence about is a SHARED MISREADING. If the design itself
+misreads a W3C rule, both engines implement the misreading and agree perfectly for
+ever. The clean sweep says nothing about that, and this tool must never be quoted as
+"two independent reasoners agree".
+
+The leg of this differential that IS independent is the LEAN CHECKER. It was written
+from the W3C rules, its soundness theorem is machine-checked, and it is the only
+component here whose acceptance means anything on its own. Two engines and a checker,
+and the checker is the one carrying the warrant.
THIS TOOL DOES NOT ADJUDICATE. It is modelled on `tools/shacl_differential.py` and on
`tools/fol_differential.py`, whose first paragraph is "THE ATP IS AN ORACLE, NOT AN
@@ -42,6 +64,29 @@ WHAT IS COMPARED.
says nothing about. The per-rule histogram is printed for exactly that reason, and
a reader who skips it will overstate what agreement was shown.
+ Measured on 14 September 2026, the repository's own RDF fired 20 of the 27 rules
+ and left seven silent: prp-inv2, eq-sym, cls-avf, cls-hv1, cls-hv2, scm-svf2 and
+ scm-avf2. `tests/fixtures/horn-coverage/` now holds one deliberately constructed
+ graph per silent rule, each the smallest thing that makes exactly that rule fire.
+ They are run as a SEPARATE, LABELLED set and are excluded from `discover()`, so
+ the corpus number stays a number about real ontologies. Two coverage figures are
+ reported and never added together: how many rules fired on the repository's own
+ RDF, and how many fired once the fixtures are included. A rule covered only by a
+ fixture written to cover it has been compared on a graph made for the purpose and
+ on nothing else, which is better than zero and is not the same thing. Re-measured
+ on 15 September 2026 over a corpus that had grown to 291 documents: the same 20,
+ the same seven, and every fixture firing exactly its own rule once. No rule in the
+ table is unreachable.
+
+ WHICH RULE SET THIS COUNTS. The 27 rows of `tests/fixtures/horn/builtin_rules.tsv`,
+ which is what this tool passes to `reason --rules`. It is NOT the `owl-rl-ext`
+ profile, and the two differ on real data: on `benchmark/reference/pizza-reference.owl`
+ the table derives 387 `rdfs11`, 102 `scm-svf1` and 1 `scm-avf1` where the profile
+ derives 357, 101 and 0, and the profile also fires `cls-int2` and `cls-oo`, which
+ this table does not contain. `tests/reason_rl_coverage_test.rs` measures the profile
+ and asserts `scm-avf1` fires nowhere; this tool credits it once. Both are true of
+ their own rule set, and neither may be quoted as "the corpus" without saying which.
+
BUNDLES, AND WHY THEY ARE NOT OPTIONAL. Run one file at a time and most of the OWL
rules never fire, because the repository ships a TBox in one directory and the ABox
that instantiates it in another: `rdfs9` needs a type assertion and a subclass axiom in
@@ -86,7 +131,7 @@ Usage:
python3 tools/horn_differential.py [--bundles] [--json out.json] [--limit N]
[--jobs N] [--timeout S] [--max-bytes N]
[--include SUBSTRING] [--exclude SUBSTRING]
- [--root DIR] [--verbose]
+ [--root DIR] [--no-coverage] [--verbose]
Requires the Rust binary and the Lean checker. If either is missing the run SKIPS
LOUDLY and exits 0, the way the Lean-dependent tests skip on a missing `lake`; set
@@ -135,6 +180,28 @@ CROSS_BUNDLES = {
"benchmark/reference + benchmark/data": ("benchmark/reference", "benchmark/data"),
}
+# Deliberate test data: one graph per rule the corpus never fires, each written to make
+# exactly that rule fire. Kept OUT of the corpus by `discover`, reported separately, and
+# never added to the corpus coverage figure. See the directory's README.md.
+COVERAGE_DIR = REPO / "tests" / "fixtures" / "horn-coverage"
+COVERAGE_PREFIX = "coverage "
+
+# The caveat that travels with every agreement number this tool prints. It is here as a
+# constant rather than in the docstring alone because a docstring is not printed, and the
+# agent that built this tool asked that the sweep never be quoted without it.
+INDEPENDENCE_CAVEAT = """\
+ WHAT THE AGREEMENT ABOVE IS WORTH
+ The Rust and Python reasoners are NOT independent implementations. They run the
+ same semi-naive forward-chaining algorithm over the same rule table, and the
+ Python's comments cite src/reason.rs by file and line. Their agreement is STRONG
+ evidence against a transcription slip in one of the two, and it is close to NO
+ evidence against a shared misreading of a W3C rule, because a misreading in the
+ design is copied into both and they then agree for ever.
+ Do not quote this run as "two independent reasoners agree".
+ The independent leg is the LEAN CHECKER: written from the W3C rules, soundness
+ machine-checked, and the only component here whose acceptance means anything on
+ its own."""
+
BLOCKING = ("DISAGREE", "CHECKER_REJECTED", "VERDICT_DIFFERS", "INPUT_DIFFERS")
ORDER = (
"DISAGREE",
@@ -512,9 +579,23 @@ def worker(spec_path: str, timeout: int) -> dict:
return row
+def is_coverage_fixture(path: pathlib.Path) -> bool:
+ """Is this one of the graphs written to make a silent rule fire?
+
+ The exclusion is by DIRECTORY and is applied in `discover`, so a fixture can never
+ be counted as a corpus document by accident. Mixing the two would inflate the
+ coverage figure with graphs written to hit it, which is measuring the ruler.
+ """
+ try:
+ return COVERAGE_DIR.resolve() in path.resolve().parents
+ except OSError:
+ return False
+
+
def discover(root: pathlib.Path) -> list:
- """Every RDF document the repository tracks. `git ls-files` rather than a walk, so
- build output and scratch files cannot inflate the corpus count."""
+ """Every RDF document the repository tracks, EXCEPT the coverage fixtures.
+ `git ls-files` rather than a walk, so build output and scratch files cannot inflate
+ the corpus count."""
try:
out = subprocess.run(
["git", "-C", str(root), "ls-files", *(f"*{e}" for e in EXTENSIONS)],
@@ -524,10 +605,63 @@ def discover(root: pathlib.Path) -> list:
).stdout.split("\n")
files = [root / f for f in out if f.strip()]
if files:
- return sorted(files)
+ return sorted(f for f in files if not is_coverage_fixture(f))
except (subprocess.CalledProcessError, FileNotFoundError):
pass
- return sorted(p for e in EXTENSIONS for p in root.rglob(f"*{e}"))
+ return sorted(
+ p for e in EXTENSIONS for p in root.rglob(f"*{e}") if not is_coverage_fixture(p)
+ )
+
+
+def coverage_cases() -> list:
+ """One case per file in `tests/fixtures/horn-coverage/`.
+
+ The target rule is read off the FILENAME, so the check below is against what the
+ fixture was named for rather than against whatever it turned out to do. A fixture
+ that fires nothing, or that fires a rule other than its name, is reported and fails
+ the run: in the first case the rule cannot be made to fire and that is a finding
+ about the engine, in the second the fixture is not the minimal graph it claims.
+ """
+ if not COVERAGE_DIR.is_dir():
+ return []
+ return [
+ {"name": COVERAGE_PREFIX + p.name, "files": [str(p)], "rule": p.stem}
+ for p in sorted(COVERAGE_DIR.iterdir())
+ if p.suffix in EXTENSIONS
+ ]
+
+
+def audit_coverage(rows: list) -> tuple:
+ """Did each coverage fixture fire the rule it is named for, and only that rule?
+
+ Returns `(lines, failures)`: one report line per fixture, and the fixtures that did
+ not do what their name says. `by_rule` keys are `name#index`, and two rows of the
+ built-in table share the name `scm-eqc1`, so the comparison is on the NAME half.
+ """
+ lines, failures = [], []
+ for row in sorted(rows, key=lambda r: r["name"]):
+ want = row["name"][len(COVERAGE_PREFIX):].rsplit(".", 1)[0]
+ fired = {k.split("#")[0]: v for k, v in (row.get("by_rule") or {}).items() if v}
+ note = f"{row['verdict']}, derived {row.get('derived_python')}"
+ if row["verdict"] not in COMPARED:
+ state = "NOT COMPARED"
+ failures.append((row["name"], f"{row['verdict']}: {row['detail']}"))
+ elif want not in fired:
+ state = "DID NOT FIRE"
+ failures.append(
+ (row["name"], f"fired {sorted(fired) or 'nothing'}, not {want}. Either the "
+ f"fixture does not instantiate the rule or the engine cannot reach it")
+ )
+ elif len(fired) > 1:
+ state = "FIRED MORE THAN ITS RULE"
+ failures.append(
+ (row["name"], f"fired {sorted(fired)}; a coverage fixture is supposed to be "
+ f"the SMALLEST graph that makes {want} fire")
+ )
+ else:
+ state = f"fires {want} x{fired[want]}"
+ lines.append(f" {row['name']:<28} {state:<26} ({note})")
+ return lines, failures
def bundle_cases(files, root: pathlib.Path) -> list:
@@ -624,6 +758,11 @@ def main() -> int:
ap.add_argument("--exclude", action="append", default=[])
ap.add_argument("--bundles", action="store_true")
ap.add_argument("--no-singles", action="store_true")
+ ap.add_argument(
+ "--no-coverage",
+ action="store_true",
+ help="skip tests/fixtures/horn-coverage/, reproducing the corpus-only figure",
+ )
ap.add_argument("--verbose", action="store_true")
ap.add_argument("--one", help=argparse.SUPPRESS)
args = ap.parse_args()
@@ -648,6 +787,11 @@ def main() -> int:
cases += bundle_cases(files, root)
if args.limit:
cases = cases[: args.limit]
+ # After the limit, deliberately: a `--limit` that silently dropped the coverage
+ # fixtures would print a coverage figure that is about a truncated corpus while
+ # reading like the real one.
+ if not args.no_coverage:
+ cases += coverage_cases()
if not cases:
print("no RDF documents found: the differential ran over nothing", flush=True)
return 2
@@ -715,17 +859,30 @@ def main() -> int:
counts = {}
for r in rows:
counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1
- compared = [r for r in rows if r["verdict"] in COMPARED]
+ coverage_rows = [r for r in rows if r["name"].startswith(COVERAGE_PREFIX)]
+ compared = [
+ r
+ for r in rows
+ if r["verdict"] in COMPARED and not r["name"].startswith(COVERAGE_PREFIX)
+ ]
singles = [r for r in compared if not r["name"].startswith("bundle ")]
bundles = [r for r in compared if r["name"].startswith("bundle ")]
def total(group, field):
return sum(r.get(field, 0) for r in group)
- by_rule = {}
- for r in compared:
- for k, v in (r.get("by_rule") or {}).items():
- by_rule[k] = by_rule.get(k, 0) + v
+ def histogram(group):
+ out = {}
+ for r in group:
+ for k, v in (r.get("by_rule") or {}).items():
+ out[k] = out.get(k, 0) + v
+ return out
+
+ # Two histograms, never one. `by_rule` is the corpus of real ontologies; the
+ # fixtures are graphs written to make a named rule fire and are counted apart.
+ compared_fixtures = [r for r in coverage_rows if r["verdict"] in COMPARED]
+ by_rule = histogram(compared)
+ by_rule_fixture = histogram(compared_fixtures)
print("\n" + "=" * 74)
print("THREE-WAY DIFFERENTIAL: rust engine, python engine, lean checker")
@@ -742,22 +899,55 @@ def main() -> int:
print(" (a bundle re-derives much of what its members "
"derive alone:")
print(" the two rows overlap and must not be summed)")
+ print(f" coverage fixtures {len(coverage_rows):>6} "
+ f"(tests/fixtures/horn-coverage/, deliberate test data, NOT corpus)")
print(f" derived sets identical on both engines in every case: "
- f"{all(r.get('derived_python') == r.get('derived_rust') for r in compared)}")
+ f"{all(r.get('derived_python') == r.get('derived_rust') for r in compared + compared_fixtures)}")
print(f" cases not run {len(not_run):>6}")
print(f" wall clock {time.time() - start:.0f}s")
+ print()
+ print(INDEPENDENCE_CAVEAT)
print("\n verdicts")
for k in ORDER:
if k in counts:
print(f" {k:<22} {counts[k]}")
- print("\n coverage by rule (python's credit, summed over every compared case)")
- silent = [k for k, v in by_rule.items() if not v]
- for k in sorted(by_rule, key=lambda k: int(k.split("#")[1])):
- print(f" {k:<18} {by_rule[k]:>9}{'' if by_rule[k] else ' NEVER FIRED'}")
- print(f"\n {len(by_rule) - len(silent)} of {len(by_rule)} rules fired somewhere in the corpus.")
- if silent:
- print(" THIS RUN SAYS NOTHING ABOUT: "
- + ", ".join(sorted(silent, key=lambda k: int(k.split("#")[1]))))
+
+ print("\n coverage by rule (python's credit; corpus | fixture, counted apart)")
+ every = sorted(set(by_rule) | set(by_rule_fixture), key=lambda k: int(k.split("#")[1]))
+ silent = [k for k in every if not by_rule.get(k)]
+ fixture_only = [k for k in silent if by_rule_fixture.get(k)]
+ still_silent = [k for k in silent if not by_rule_fixture.get(k)]
+ for k in every:
+ corpus, fixture = by_rule.get(k, 0), by_rule_fixture.get(k, 0)
+ if corpus:
+ note = ""
+ elif fixture:
+ note = " FIXTURE ONLY"
+ else:
+ note = " NEVER FIRED"
+ print(f" {k:<18} {corpus:>9} | {fixture:<4}{note}")
+ print(f"\n {len(every) - len(silent)} of {len(every)} rules fired on the "
+ f"repository's own RDF.")
+ if coverage_rows:
+ print(f" {len(every) - len(still_silent)} of {len(every)} once "
+ f"tests/fixtures/horn-coverage/ is included.")
+ else:
+ print(" The coverage fixtures did not run (--no-coverage), so there is no second "
+ "figure.")
+ if fixture_only:
+ print(" COVERED ONLY BY A FIXTURE WRITTEN FOR THE PURPOSE, so these rules have "
+ "been compared\n on a graph made to fire them and on no real ontology: "
+ + ", ".join(fixture_only))
+ if still_silent:
+ print(" THIS RUN SAYS NOTHING WHATEVER ABOUT: " + ", ".join(still_silent))
+
+ coverage_failures = []
+ if coverage_rows:
+ lines, coverage_failures = audit_coverage(coverage_rows)
+ print("\n coverage fixtures, each against the rule its filename names")
+ for line in lines:
+ print(line)
+
if not_run:
print("\n not run")
for s in not_run:
@@ -780,6 +970,12 @@ def main() -> int:
"bundle_refused": total(bundles, "refused_python"),
},
"by_rule": by_rule,
+ "by_rule_coverage_fixtures": by_rule_fixture,
+ "rules_fired_on_corpus": len(every) - len(silent),
+ "rules_fired_including_fixtures": len(every) - len(still_silent),
+ "rules_covered_only_by_a_fixture": fixture_only,
+ "rules_never_fired": still_silent,
+ "independence_caveat": INDEPENDENCE_CAVEAT,
},
fh,
indent=1,
@@ -789,6 +985,13 @@ def main() -> int:
if blocking:
print(f"\n {blocking} STOP-THE-LINE result(s). One of the three implementations is wrong.")
return 1
+ if coverage_failures:
+ print(f"\n {len(coverage_failures)} coverage fixture(s) did not do what their name says.")
+ for name, why in coverage_failures:
+ print(f" {name}\n {why}")
+ print(" A fixture that cannot make its rule fire is a finding about the ENGINE and "
+ "belongs\n in the report, not in a quiet skip.")
+ return 1
return 0