Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

Re-measure the differential, derive the figures, and fix two gates that could not fail

FABIOTESSf12d025bc4e5
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 800b74ea..c720d104 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -38,7 +38,25 @@ jobs:
- name: Build
run: cargo build
- name: Test
- run: cargo test -- --nocapture 2>&1 | tee cargo-test.log
+ # `shell: bash` and `set -o pipefail`, and neither is decoration.
+ #
+ # This step was `cargo test ... | tee cargo-test.log` with no shell named.
+ # On Linux a `run:` block is `bash -e`, which does NOT set pipefail, so the
+ # step's exit status was TEE's and never cargo's, and the whole suite could
+ # report `test result: FAILED` under a green tick. Measured, not reasoned:
+ # on the first run of this branch the Linux leg passed with a failing test
+ # in its own log while the Windows leg, whose default shell is PowerShell
+ # and which therefore keeps cargo's exit code, failed on the same test.
+ # A job that runs every test and cannot fail on one is the same defect this
+ # repository keeps finding elsewhere, in the job that runs the most tests.
+ #
+ # Naming bash makes both legs of the matrix behave the same way, which is
+ # the point of running a matrix, and the skip-counter step below already
+ # names it for the same reason.
+ shell: bash
+ run: |
+ set -o pipefail
+ cargo test -- --nocapture 2>&1 | tee cargo-test.log
- name: Report tests that skipped for a missing fixture
# Some tests need data this repository does not carry, so they skip
# rather than fail. A skip reports `ok`, which means a green run can
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4a0e39c9..68ddb149 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,65 @@ All notable changes to Open Ontologies are documented here.
## [Unreleased]
+### Fixed
+- **A front-page claim was gated by a test that ran nowhere, and now runs in CI.** `README.md`
+ reported that the cross-kernel differential "reports zero divergent rows", and
+ `tests/cross_kernel_differential_test.rs` does require exactly that and fails on any row at
+ all. No workflow ran it. Its `skip()` wants lake AND Poly/ML: the `build` job has neither, so
+ it skipped there, and the `lean` job, which has lake, invoked the file from no leg. A skipped
+ test reports `ok`. `docs/ci-gates.md` listed this under "What is still open" and decision 0008
+ carried a "Not run in CI" bullet, so it was known, written down, and unable to fail.
+
+ What CI installs is NOT Isabelle, and the old reasoning that it would have to be is what kept
+ this open. `isabelle/driver/oo_horn_generated.ML` is committed, so the missing piece was a
+ compiler for it: Isabelle publishes Poly/ML as a 39 MB component, pinned here by SHA-256,
+ cached, and the same 5.9.2-2 build that generated that file, so CI and the developer machine
+ run one compiler and not two. Not `apt-get install polyml`, which was checked rather than
+ assumed because `prover9` having no apt candidate on noble broke this job once already: Ubuntu
+ noble carries 5.7.1 and ships `/usr/bin/poly` and `libpolymain.a` with no `libpolyml.a`, so
+ there is nothing to link against, and both `poly_dir()` and `build_native.sh` now say so by
+ name rather than failing in the linker. `isabelle/build_native.sh` was macOS-only, looking for
+ `libgmp.dylib` and an app bundle under `/Applications`; it now takes `POLYDIR`, finds gmp in
+ either shape or as the bundled versioned copy, adds `-lpthread -lm -ldl` on Linux only because
+ macOS has no libdl to pass, and prints the linker's stderr on failure instead of discarding it.
+
+- **The `build` job could not fail on a failing test, and this branch proved it by accident.**
+ `cargo test -- --nocapture 2>&1 | tee cargo-test.log` in a `run:` block is `bash -e` without
+ `pipefail`, so the step's exit status was `tee`'s. The first CI run of this work carried a test
+ that failed: the Linux leg printed `test result: FAILED. 2 passed; 1 failed` in its own log and
+ reported SUCCESS, while the Windows leg, whose default shell keeps cargo's exit code, failed on
+ the same test. The job that runs the whole suite on two platforms was not a gate on one of them.
+ `shell: bash` and `set -o pipefail`, which is what the skip-counter step below it already does
+ and for the same reason.
+
+- **A skip in CI is now a failure even in a file no job invokes.** `common::skip_unless` already
+ panicked under `OO_REQUIRE_FIXTURES=1` and twenty legs already set it, so no second variable is
+ added: a second name for an existing mechanism is a second thing to forget. What that mechanism
+ cannot see is a test file that runs nowhere, which is what this defect was.
+ `tests/ci_gate_coverage_test.rs` requires every `tests/*.rs` with a skip path to be strict in
+ some workflow or on a written exception list with its reason, refuses an exception for a file
+ that no longer skips or that is strict after all, holds `docs/ci-gates.md`'s table against what
+ the workflows actually say, and fails on any test printing the `SKIPPED_FIXTURE:` marker by hand
+ rather than through the helper. Five files are excepted, each because CI deliberately does not
+ carry what they need. It does NOT make every file strict: a contributor without Poly/ML must
+ still be able to run `cargo test` and get a pass, and can.
+
+- **The corpus figures in the documents are derived rather than typed.** Three documents quoted
+ this corpus and none of them had measured the tree they were committed to. The 47-of-1,718 and
+ the zero that replaced it were measured on the shallow corpus; the 54-of-2,075 was measured on a
+ branch cut before the binding fix. The two were authored eleven minutes apart and merged
+ separately, so the combination nobody ran was the deep corpus under the fixed checker, which is
+ the combination the README asserted a result for. Measured now, in CI: **2,075 certificates, 402
+ accepted by both, 1,134 rejected by both, 539 unparseable on both, and ZERO divergent**, with 332
+ rows carrying a binding decision 0008 refuses and both kernels refusing every one. The shape of
+ the shallow result repeats exactly: 1,080 plus 54 is 1,134, and the accepted and unparseable
+ counts did not move. `the_corpus_exercises_prefix_visibility_at_depth` now formats the corpus
+ size, its depth, its fan-out and its prefix-visibility count out of the measurement and fails if
+ `README.md` or `docs/reasoning-systems-inventory.md` says anything else, so a growing corpus
+ fails until the prose is corrected and a correction that reaches one document out of two fails
+ as well. `isabelle/README.md`'s per-bucket counts are deleted rather than corrected: that file
+ already said the numbers belong in the test output, three sections before writing them out.
+
### Changed
- **A binding is data, and a certificate that admits two readings is refused.** Running the
Lean checker in `lean/OOCert/Horn.lean` and the independent Isabelle/HOL one in `isabelle/`
@@ -59,10 +118,14 @@ All notable changes to Open Ontologies are documented here.
after: 2,075 certificates, base depths 0:61 1:5 3:1 6:1 7:1 19:1, maximum
fan-out 12, and 484 rows exercising the discipline (258 resting on it to be
accepted, 226 that it must reject). Depth is the longest chain of citations,
- so an n-step chain reports n-1. The two kernels return the same answer on
- 2,021 of the 2,075 and part company on 54: every one of those is the D1
+ so an n-step chain reports n-1. The two kernels returned the same answer on
+ 2,021 of the 2,075 and parted company on 54: every one of those is the D1
duplicate-key divergence that was already pinned, none is new in kind, and
- NONE is unexplained. Depth found no fresh disagreement, which is a result
+ NONE is unexplained. **That count was measured on this branch, which was cut
+ before the binding fix above landed, and it is history rather than a
+ description: on the merged tree the figure is ZERO, and the 54 moved into
+ rejected-by-both exactly as the 47 did. See the Fixed section at the top.**
+ Depth found no fresh disagreement, which is a result
about the two formalisations rather than a null one, because the property
their inductions are built on had until now barely been shown data that could
violate it. `the_corpus_exercises_prefix_visibility_at_depth` prints the
diff --git a/README.md b/README.md
index 6445eae8..fbd30484 100644
--- a/README.md
+++ b/README.md
@@ -74,8 +74,15 @@ repeated binding key meant, so one kernel refused the shape and the other answer
its lookup happened to do. A key bound twice to different values is satisfied by no substitution
at all, so there were never two readings, only two ways of discarding half the certificate. Closed
by [decision 0008](docs/decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md), and
-the differential now reports zero divergent rows over a corpus four times deeper in the property
-its induction rests on. Every one of those had passed every test that existed before.
+the two kernels now return the same answer on every row of a corpus of 2,075 certificates, 484 of
+which exercise the ordering property both inductions rest on, against 123 before that corpus was
+deepened. Every one of those had passed every test that existed before.
+
+That last sentence was, until 15 September 2026, checked by nothing. No workflow installed the
+second kernel, so the test that requires zero divergence skipped in the one job that ran it and was
+invoked by no job that could have made it strict, and a skipped test reports `ok`. CI runs both
+kernels over the whole corpus on every pull request now, and
+[docs/ci-gates.md](docs/ci-gates.md) is the table of which other gates do and do not fire.
## What is actually proved
diff --git a/docs/decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md b/docs/decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md
index 14022aa0..ef49c015 100644
--- a/docs/decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md
+++ b/docs/decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md
@@ -18,10 +18,15 @@
- **Written**: 2026-09-14
- **Related**: decision 0003 (a rule is data, and an assumption is not a fact), whose format this
amends; decision 0002 (an inference carries a certificate)
-- **Not run in CI.** The differential needs Isabelle2025-2 and Poly/ML, which the workflow does not
- install, so `cargo test --test cross_kernel_differential_test` is a LOCAL gate and the 47-to-0
- number is reproduced by running it, not by a green badge. `lean_horn_certificate_test` and
- `reason_horn_emit_test` do run in CI and carry the Lean-side and producer-side halves.
+- **Run in CI since 15 September 2026, and not before.** This bullet used to say the opposite, and
+ it was right: the differential was a LOCAL gate, because no workflow installed the second kernel,
+ so the file skipped in the one job that ran it and was invoked by no job that could have made it
+ strict. It needed Poly/ML rather than Isabelle2025-2, which is the thing that was wrong with the
+ old reasoning: `isabelle/driver/oo_horn_generated.ML` is committed, so a 39 MB component installs
+ it. The `lean` job now runs the file under `OO_REQUIRE_FIXTURES=1`, and
+ `tests/ci_gate_coverage_test.rs` fails if any test file that can skip ever again runs nowhere.
+ `lean_horn_certificate_test` and `reason_horn_emit_test` carry the Lean-side and producer-side
+ halves and always did.
## The problem
@@ -224,6 +229,32 @@ binding were already rejected by both kernels for some other reason; the gate ru
rather than only on the ones that disagree, because a check that ran only on divergences could be
satisfied by silence. It splits 89 D1 and 197 D2.
+### Re-measured on the deep corpus, 15 September 2026
+
+The pair above is the SHALLOW corpus, which is what existed on the day this decision was written.
+The corpus was deepened hours later, on a branch cut before this decision landed, and the two
+pieces of work were merged separately: the pair that nobody had run was the deep corpus under the
+fixed checker, and that is the pair the README stated a result for. Run in CI, which now installs
+the second kernel:
+
+Before, over 2,075 certificates (70 base, 1,585 mutated, 420 fuzzed, same seed):
+
+ both accept: 402
+ both reject: 1,080
+ both exit 2: 539
+ known divergence: 54
+
+After:
+
+ both accept: 402
+ both reject: 1,134
+ both exit 2: 539
+ divergent: 0
+ malformed binding, refused by both (decision 0008): 332
+
+The shape repeats exactly: 1,080 plus 54 is 1,134, and the accepted and unparseable counts did not
+move. Depth changed the size of every column and changed nothing about the decision.
+
One detail worth stating rather than rounding off: **all 47 corpus divergences were D1.** D2's
three certificates are committed probes with their own tests and are not members of the generated
corpus, so the corpus never counted them, and the before-run's classification prints only D1 rows.
diff --git a/docs/reasoning-systems-inventory.md b/docs/reasoning-systems-inventory.md
index 5b0278a1..a865134f 100644
--- a/docs/reasoning-systems-inventory.md
+++ b/docs/reasoning-systems-inventory.md
@@ -95,9 +95,9 @@ refused. The defect was in the FORMAT, which never said what a repeated key mean
incomplete binding means, and only a second implementation could have surfaced it.
That gap is now CLOSED. `docs/decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md`
-refuses both shapes, the Lean moved and no Isabelle theory was touched, and the same corpus now
-reports zero divergent rows: 349 accepted by both, 904 rejected by both, 465 unparseable on both.
-The 47 moved into rejected-by-both and nowhere else.
+refuses both shapes, the Lean moved and no Isabelle theory was touched, and the 47 moved into
+rejected-by-both and nowhere else. The per-bucket counts that decision was measured against are
+in the decision record, dated, and describe the corpus as it was that day.
Those 1,718 certificates were one step deep. Sixty of the 61 base certificates contained no step
citing an earlier step's conclusion, so on those the checkers' ordering logic had nothing to
@@ -107,12 +107,28 @@ two-step fixture.
On 15 September 2026 the corpus gained generated chains and fans, six mutations a flat
certificate cannot express, and two hand-built adversarial certificates. It is now 2,075 rows
reaching depth 19 and fan-out 12, of which 484 exercise the ordering discipline rather than 123.
-The disagreement count went from 47 to 54 and did not change in kind: every new row the two
-kernels parted company on was the same D1 duplicate-key shape, and nothing was left unexplained.
Depth found no new divergence, which is a result about the two formalisations and not a null
one — the property their inductions are built on was, until then, barely tested against data
that could violate it.
+**Read the numbers in the two paragraphs above as history, because for a day they were read as a
+description.** The 47-of-1,718 and the zero that replaced it were both measured on the shallow
+corpus. A later count of 54 disagreements over the deep one was measured on a branch cut before
+the binding fix, and is not repeated here for that reason. The two pieces of work were authored
+eleven minutes apart and merged separately, so the combination nobody had run was the deep corpus
+under the FIXED checker, which is exactly the combination the README asserted a result for. It
+has now been run, on this tree and in CI, and the two kernels return the same answer on every
+one of the 2,075 rows. The 54 moved into rejected-by-both and nowhere else, the accepted and
+unparseable counts did not move at all, and the per-bucket figures are in
+[decision 0008](decisions/0008-a-binding-is-data-and-evidence-admits-one-reading.md), dated, next
+to the shallow pair they repeat the shape of.
+
+The reason none of this was caught is worth more than the numbers. The test that requires zero
+divergence needs Poly/ML as well as lake, no workflow installed Poly/ML, so it skipped in the one
+job that ran it and was invoked by no job that could have made it strict. A skipped test reports
+`ok`. [docs/ci-gates.md](ci-gates.md) now states which gates fire and is itself held to the
+workflows by a test.
+
The adversarial review of that comparison established four things about the Lean layer that the
Lean's own build could never have shown, and they are the current work rather than a footnote:
eight of the twenty-seven rules were assumed as primitive semantic conditions rather than
diff --git a/tests/ci_gate_coverage_test.rs b/tests/ci_gate_coverage_test.rs
index de626e3c..24ba6034 100644
--- a/tests/ci_gate_coverage_test.rs
+++ b/tests/ci_gate_coverage_test.rs
@@ -38,6 +38,20 @@ fn repo() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
}
+/// This file, by name, so the scans below can leave it out.
+///
+/// Both of them look for a fragment of source text, and this file has to CONTAIN those
+/// fragments in order to look for them. Without this it matches itself: the first run in
+/// CI reported that `ci_gate_coverage_test.rs` can skip and has no row in the table, on
+/// the strength of the string literal `"skip_unless("` three functions down. `file!()`
+/// rather than a written-out name, so a rename cannot reintroduce it.
+fn this_file() -> &'static str {
+ std::path::Path::new(file!())
+ .file_name()
+ .and_then(|n| n.to_str())
+ .expect("file!() always has a file name")
+}
+
/// Files that can skip and are not run strictly anywhere, each with the reason. A file
/// belongs here when the thing it needs is one CI deliberately does not provide; it does
/// NOT belong here because wiring it up looked like work.
@@ -78,12 +92,16 @@ fn files_that_can_skip() -> BTreeSet<String> {
if path.extension().and_then(|e| e.to_str()) != Some("rs") {
continue;
}
+ let name = path.file_name().expect("file name").to_string_lossy().into_owned();
+ if name == this_file() {
+ continue;
+ }
let text = std::fs::read_to_string(&path).expect("test file must be readable");
// The CALL and not the identifier. This file names `skip_unless` a dozen times in
// prose and calls it never, so a bare substring match would put the gate on the
// gates into its own list of things to worry about.
if text.contains("skip_unless(") {
- out.insert(path.file_name().expect("file name").to_string_lossy().into_owned());
+ out.insert(name);
}
}
out
@@ -213,6 +231,9 @@ fn only_the_helper_emits_the_skip_marker() {
if path.extension().and_then(|e| e.to_str()) != Some("rs") {
continue;
}
+ if path.file_name().map(|n| n.to_string_lossy() == this_file()) == Some(true) {
+ continue;
+ }
let text = std::fs::read_to_string(&path).expect("test file must be readable");
for line in text.lines() {
let t = line.trim_start();
diff --git a/tests/cross_kernel_differential_test.rs b/tests/cross_kernel_differential_test.rs
index fae897d8..abb4e959 100644
--- a/tests/cross_kernel_differential_test.rs
+++ b/tests/cross_kernel_differential_test.rs
@@ -2145,6 +2145,12 @@ fn agreement_is_about_the_definitions_and_is_not_itself_a_proof() {
/// This test prints the distribution and holds a floor under it. It deliberately does
/// NOT require either proof assistant: the shape of the corpus is a fact about the
/// files, and it should still be measurable on a machine that cannot run the kernels.
+///
+/// It also CHECKS THE DOCUMENTS that quote these figures against the figures, at the
+/// bottom, rather than in a test of its own. Generating this corpus means running the
+/// engine over every ontology the repository ships, three tests in this file already pay
+/// for that, and a fourth copy of the cost to compare two strings is not worth it. The
+/// reason the check exists at all is in the comment where it is made.
#[test]
fn the_corpus_exercises_prefix_visibility_at_depth() {
let scratch = scratch_dir();
@@ -2206,9 +2212,94 @@ fn the_corpus_exercises_prefix_visibility_at_depth() {
REJECTING half of the discipline is barely exercised"
);
+ // ── the documents, against the corpus that was just counted ──────────────
+ //
+ // The house rule is that a figure next to the thing it describes must be DERIVED and
+ // never typed, and this corpus is the case that made the rule. Three documents quoted
+ // it, two different versions of it were measured on two branches eleven minutes apart
+ // and merged separately, and what landed on main was a README sentence reporting zero
+ // divergent rows next to an inventory reporting fifty-four, neither of them measured
+ // against the tree they were committed to. `isabelle/README.md` said so about itself
+ // and was left stale on purpose, because the work that found it was not allowed to
+ // touch that directory.
+ //
+ // The corpus TOTAL is the wrong thing to pin on its own: it moves whenever a Turtle
+ // file lands in one of the ten directories `source_graphs` reads, and a growing
+ // corpus is not a regression. What is checked instead is that every document quoting
+ // it quotes THIS measurement, so a corpus that grows fails until the prose is
+ // corrected, and a correction that reaches one document out of two fails as well.
+ // That is the shape `tests/readme_claims_test.rs` uses for the tool count.
+ //
+ // The divergence count is deliberately NOT here. It cannot be measured without both
+ // kernels, and `the_two_kernels_agree_on_the_whole_corpus` asserts it directly rather
+ // than in prose: the documents say the two agree, and that sentence is true exactly
+ // when that test passes.
+ let total = bases.len() + mutants.len() + fuzzed.len();
+ let exercising = uses + violates;
+ let claims: [(&str, &str, String); 2] = [
+ (
+ "README.md",
+ "the paragraph on what the discipline has caught",
+ format!(
+ "a corpus of {} certificates, {exercising} of which exercise the ordering \
+ property",
+ commas(total)
+ ),
+ ),
+ (
+ "docs/reasoning-systems-inventory.md",
+ "the paragraph on corpus depth",
+ format!(
+ "{} rows reaching depth {deepest} and fan-out {widest}, of which \
+ {exercising} exercise the ordering discipline",
+ commas(total)
+ ),
+ ),
+ ];
+
+ // Whitespace is collapsed on both sides before comparing. These documents are hard
+ // wrapped at about a hundred columns, so any phrase long enough to be worth checking
+ // straddles a line break, and a literal `contains` would fail on prose that is
+ // correct and pass on nothing. It is the claim being checked, not its typesetting.
+ let flatten = |s: &str| s.split_whitespace().collect::<Vec<_>>().join(" ");
+
+ let mut wrong = Vec::new();
+ for (file, where_, claim) in &claims {
+ let text = std::fs::read_to_string(repo().join(file))
+ .unwrap_or_else(|_| panic!("{file} must exist: a claim is checked against it"));
+ if !flatten(&text).contains(&flatten(claim)) {
+ wrong.push(format!("{file}, {where_}:\n expected to find {claim:?}"));
+ }
+ }
+ assert!(
+ wrong.is_empty(),
+ "The corpus this file generates is {} certificates, {exercising} of them \
+ exercising prefix visibility, deepest base chain {deepest} and widest fan-out \
+ {widest}, and a document says otherwise.\n\n{}\n\nCorrect the document rather \
+ than this test: the corpus is the measurement and the prose is the claim. If a \
+ sentence was deliberately reworded, change the expected phrase here in the same \
+ commit and say why.",
+ commas(total),
+ wrong.join("\n\n")
+ );
+
let _ = std::fs::remove_dir_all(&scratch);
}
+/// Thousands separators, because the documents are written for people and a claim checked
+/// against `2075` would pass over prose that says `2,075`.
+fn commas(n: usize) -> String {
+ let s = n.to_string();
+ let mut out = String::with_capacity(s.len() + s.len() / 3);
+ for (i, c) in s.char_indices() {
+ if i > 0 && (s.len() - i) % 3 == 0 {
+ out.push(',');
+ }
+ out.push(c);
+ }
+ out
+}
+
/// **Self-support, in the one form that isolates it.** A step whose only non-asserted
/// premise is its OWN conclusion, and which is otherwise perfect.
///