Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
docs: record the withdrawn HermiT claim and both determinism defects
FABIOTESS54ee20e9cdca
diff --git a/README.md b/README.mdindex cb2008df..e4f4fab8 100644--- a/README.md+++ b/README.md@@ -960,6 +960,7 @@ flowchart TD| IES SPARQL Examples | [docs/ies-examples.md](docs/ies-examples.md) || IES:Building Alignment | [docs/ies-alignment.md](docs/ies-alignment.md) || Benchmarks | [docs/benchmarks.md](docs/benchmarks.md) |+| Determinism & corrected results | [docs/determinism.md](docs/determinism.md) || Contributing | [CONTRIBUTING.md](CONTRIBUTING.md) || Changelog | [CHANGELOG.md](CHANGELOG.md) |diff --git a/benchmark/reasoner/README.md b/benchmark/reasoner/README.mdindex 13d40f0e..56461d5b 100644--- a/benchmark/reasoner/README.md+++ b/benchmark/reasoner/README.md@@ -39,3 +39,46 @@ bash benchmark/reasoner/run_lubm_performance.sh```Results are saved to `benchmark/reasoner/results/`.++## Findings, and one withdrawn claim++### The LUBM speed claim is withdrawn++Earlier versions of this repository, and `arXiv:2605.09184v1`, reported OWL-RL+forward chaining at 14-15 ms against HermiT's 112-24,490 ms on LUBM, a ratio of+**1,633x** at 50,000 axioms. **That claim is withdrawn. It was measured on an+empty store.**++Three faults stacked in `run_lubm_performance.sh`:++1. `generate_lubm.py` wrote Turtle into a file with an `.owl` extension, and+ format detection keyed on the extension and chose an RDF/XML parser, so every+ load failed.+2. `load` and `reason` ran as separate processes. Process setup builds a fresh+ empty store, so `reason` always executed over zero triples.+3. Both failures were routed to `/dev/null` and replaced with a default value.++The published 14-15 ms was process start-up time, and its flatness across a 50x+scale-up was the tell. Fixed by content sniffing in `detect_format_sniffed`, by+running through `batch -` so a single store is shared, and by asserting+`triples > 0` before any timing is trusted.++### Corrected comparison++Measured properly on the canonical `pizza.owl` (1,944 triples, 99 classes, 218+individuals), same task, same machine, HermiT 1.4.3.456 on JDK 17:++| | HermiT | Open Ontologies |+| --- | --- | --- |+| Wall time | **170 ms** | 10.9 s (budget expiry, not completion) |+| Inferred subsumptions | **311** | **0** |+| Outcome | complete | incomplete, 143 undetermined |++The claim was not merely unsupported. It was inverted. **No speed claim against+any Java reasoner should be made from this repository.** The reason our reasoner+does not complete is documented in [`regressions/`](regressions/): it does not+terminate on nominals, and LUBM emits one `owl:hasValue` per department.++For a workload this engine *is* suited to, see the compiled claim-verification+benchmark in [`docs/benchmarks.md`](../../docs/benchmarks.md), which is+task-matched and audited against HermiT for agreement rather than speed alone.diff --git a/docs/benchmarks.md b/docs/benchmarks.mdindex 989dad4e..0e812e56 100644--- a/docs/benchmarks.md+++ b/docs/benchmarks.md@@ -156,3 +156,16 @@ make bench-pizza # Just Pizzamake bench-ontoaxiom # Just OntoAxiommake bench-reasoner # Just reasoner comparison```++## Determinism and corrected results++Two subsystems in this repository were nondeterministic: tableaux classification+and alignment both let `HashMap` iteration order decide their output. Both are+fixed, and the Anatomy figure recorded before the fix (F1 0.832) was one draw+from a distribution whose deterministic value is 0.829. Full record, including+the five divergent alignment hashes and the reproduction commands:+[`docs/determinism.md`](determinism.md).++A speed claim against HermiT previously carried here and in `arXiv:2605.09184v1`+is **withdrawn**: it was measured on an empty store, and inverts when measured+correctly. See [`benchmark/reasoner/README.md`](../benchmark/reasoner/README.md).diff --git a/docs/determinism.md b/docs/determinism.mdnew file mode 100644index 00000000..deee9e1a--- /dev/null+++ b/docs/determinism.md@@ -0,0 +1,114 @@+# Determinism++Two subsystems in this repository returned different answers on identical input.+Both are fixed. This page records what happened, because a flaky test is the+symptom and the disease is worth naming: **if output depends on hash iteration+order, the system is not a function of its input.**++Rust seeds `HashMap` and `HashSet` iteration order per process. Any traversal+derived from one, collected into a `Vec` and then walked, is a per-run coin+flip. That is harmless if the consumer is order-insensitive. Neither of these+was.++## 1. Tableaux classification++**Symptom.** `w3c_pizza_vegetarian_classification` in+`tests/owl2_conformance_test.rs` failed intermittently. On a fixed commit and a+fixed machine it failed on **2 runs in 6**. The assertion is that+`Margherita ⊑ VegetarianPizza` is inferred; on failing runs the classification+returned only `Pizza`.++**Cause.** `named_classes` is a `HashSet`, and `src/tableaux.rs` collected+`self.nodes.keys()` into a `Vec` at five expansion sites: the merge redirect,+the expansion loop, the max-cardinality rule, the disjunction rule and the+blocking update. Class order fixed candidate-pair order, which fixed which+subsumption checks were reached before a finite node or depth budget was spent.+Within a single check it fixed the order nodes were visited in.++The conformance tests do **not** disable the node and depth budgets. On an+unlucky ordering a budget was exhausted, `Tableau::decide` correctly returned+`Verdict::Unknown`, `is_subsumed` correctly declined to assert, and the+entailment was simply absent from the output. Nothing false was ever claimed;+something true was silently omitted, and omission is indistinguishable from a+correct negative without a complete oracle.++**Fix.** Sort both traversals by identifier. Node ids are assigned sequentially,+so sorting by id is creation order, the natural deterministic traversal.++| | Before | After |+| --- | --- | --- |+| Consecutive full runs of the conformance binary passing | 4 / 6 | **12 / 12** |+| Wall time, that binary | 1.17-2.57 s | **0.01 s** |++The two-order-of-magnitude speed-up says the previous flakiness was budget+exhaustion under bad orderings rather than genuine hardness. This is a+determinism fix, not a completeness fix; the nominal blowup in+[`benchmark/reasoner/regressions/`](../benchmark/reasoner/regressions/) is+untouched and still open.++## 2. Alignment++**Symptom.** The OAEI Anatomy candidate count moved between runs of the same+binary on the same input. Five runs, hashing the sorted correspondence set:++| Run | Correspondences | Set hash |+| --- | ---: | --- |+| 1 | 1151 | `c23fd913a17a` |+| 2 | 1153 | `77db7f7c5f79` |+| 3 | 1151 | `05db643ab8dc` |+| 4 | 1152 | `56f1e1cb5774` |+| 5 | 1154 | `b4f525ec6056` |++No two output sets matched.++**Cause.** Two links. `extract_classes` in `src/align.rs` ended in+`class_map.into_values().collect()`, so the class list arrived in hash order and+set the order candidate pairs were generated in. The candidate comparator then+sorted on confidence alone, which is **not a total order** here: the+zero-structural-signal branch assigns very many pairs the identical+`label_sim * 0.85`. `sort_by` is stable, so tied candidates kept generation+order, and the greedy 1-to-1 `retain` that follows is first-come-first-served.++**Fix.** Sort `extract_classes` output by IRI, and break confidence ties on+`(source_iri, target_iri)` so the comparator is total. Either alone would suffice+today; both together mean a future upstream change cannot reintroduce it.++| | Correspondences | Set hash |+| --- | ---: | --- |+| 5 consecutive runs after the fix | **1152** | **`469e0e04cb58`** (all five) |++**Consequence for the recorded result.** The Anatomy figure previously recorded+in this repository, P 0.963 / R 0.733 / F1 0.832, was one draw from the+distribution above. The deterministic value is:++| | P | R | F1 | Correspondences |+| --- | ---: | ---: | ---: | ---: |+| Deterministic | 0.960 | 0.730 | **0.829** | 1,152 |++Rank in the OAEI 2025 field is unchanged at 9th of 13. Three decimal places had+been quoted for a quantity whose third decimal was not a property of the system.++## Reproducing++```bash+# Tableaux: run the conformance binary repeatedly; all runs must agree.+for i in $(seq 1 12); do cargo test --test owl2_conformance_test 2>&1 | grep "^test result"; done++# Alignment: five runs must produce one hash.+for i in 1 2 3 4 5; do+ ./target/release/open-ontologies align \+ benchmark/oaei/data/anatomy/mouse.owl \+ benchmark/oaei/data/anatomy/human.owl \+ --min-confidence 0.80 --dry-run \+ | python3 -c "import json,sys,hashlib; c=json.load(sys.stdin)['candidates']; \+p=[(x['source_iri'],x['target_iri']) for x in c]; \+print(len(c), hashlib.sha256(repr(sorted(p)).encode()).hexdigest()[:12])"+done+```++## The general rule++Sort every hash-derived traversal, and make every comparator a total order.+Determinism is not a performance concern or a testing nicety. Both defects here+were found only because something downstream wobbled by one, and in both cases+the wobble was hiding a result that was not reproducible.