Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
fix(align): make alignment deterministic; five runs gave five different alignments
FABIOTESS36a93a3efd25
diff --git a/README.md b/README.mdindex 37edd009..cb2008df 100644--- a/README.md+++ b/README.md@@ -281,8 +281,8 @@ The benchmarks below are not a feature tour. Each one exists to answer a specifi| --- | --- | --- | --- || **RQ1** | Does structured tool access give an LLM a materially different mode of access to an ontology than handing it the raw OWL file, or than giving it nothing but class and property names? | [OntoAxiom](#ontoaxiom--llm-axiom-identification), 3 conditions, 2 models | **Partly.** Both beat name lists decisively. Tool extraction and raw-file reading are at **parity** with each other: raw OWL wins the macro average, extraction wins the micro. The tools' remaining advantage is that every pair traces to a query against real triples. || **RQ2** | When two evaluation conditions disagree, how much of the gap is the method and how much is the scorer? | [OntoAxiom](#ontoaxiom--llm-axiom-identification), legacy vs unified evaluator | **Enough to invert the finding.** Three scorer asymmetries, all pointing one way, produced a reported result whose sign reverses under a shared evaluator, on both models and under both averages. |-| **RQ3** | In ontology alignment, which carries the result: how the similarity signals are weighted, or the constraint that the matching be 1-to-1? | [OAEI Anatomy](#oaei-ontology-alignment--anatomy-track), 5 weight configurations vs stable-matching ablation | **The constraint, overwhelmingly.** F1 moves 0.0033 across five weight configurations and collapses from 0.832 to 0.182 when stable matching is removed. |-| **RQ4** | How far does an alignment system get on a biomedical track with **no** domain background knowledge (no UMLS, no BioPortal, no LLM oracle)? | [OAEI Anatomy](#oaei-ontology-alignment--anatomy-track) and [Conference](#oaei-ontology-alignment--conference-track), full 2025 field | **Not far enough.** 9th of 13 on Anatomy, level with the lightweight lexical matcher and +0.066 F1 over a string-equality baseline. Below every system and both baselines on Conference. Precision is competitive; recall is the failure. |+| **RQ3** | In ontology alignment, which carries the result: how the similarity signals are weighted, or the constraint that the matching be 1-to-1? | [OAEI Anatomy](#oaei-ontology-alignment--anatomy-track), 5 weight configurations vs stable-matching ablation | **The constraint, overwhelmingly.** Removing stable matching as the only variable drops F1 from 0.829 to 0.728; the five-weight-configuration spread is 0.0033, and even that overstates it (the zero-structural-signal branch bypasses the weights entirely). |+| **RQ4** | How far does an alignment system get on a biomedical track with **no** domain background knowledge (no UMLS, no BioPortal, no LLM oracle)? | [OAEI Anatomy](#oaei-ontology-alignment--anatomy-track) and [Conference](#oaei-ontology-alignment--conference-track), full 2025 field | **Not far enough.** 9th of 13 on Anatomy, level with the lightweight lexical matcher and +0.063 F1 over a string-equality baseline. Below every system and both baselines on Conference. Precision is competitive; recall is the failure. || **RQ5** | Does a closed-world vocabulary check catch generated terms that open-world SHACL validation silently accepts? | [`onto-correctness-bench`](case-studies/onto-correctness-bench/): 3 vocabularies, 418 fabricated terms, 300 graphs | **Yes, completely.** SHACL returned `conforms=true` on **300/300** graphs containing a fabricated term. The closed-world gate flagged **300/300**, with zero false positives on clean graphs. Open-world semantics treat an undeclared predicate as merely unknown, so SHACL is structurally unable to see it. |RQ2 and RQ4 are the ones worth reading if you are deciding whether to trust this repo. Both are negative results about work done here.@@ -448,13 +448,13 @@ The comparison below is the **complete** OAEI 2025 Anatomy field, reproduced fro| MDMapper | 0.899 | 0.879 | **0.889** || LogMap | 0.917 | 0.848 | **0.881** || LogMapKG | 0.917 | 0.848 | **0.881** |-| **Open Ontologies** | **0.963** | **0.733** | **0.832** |+| **Open Ontologies** | **0.960** | **0.730** | **0.829** || DRAL-OA | 0.830 | 0.827 | **0.828** || LogMapLt | 0.962 | 0.728 | **0.828** || *StringEquiv (baseline)* | 0.997 | 0.622 | **0.766** || LSMatch | 0.952 | 0.634 | **0.761** |-**Read this honestly.** Open Ontologies ranks **9th of 13** on F1. Its precision (0.963) is third in the field, but its recall (0.733) is second-from-bottom among non-baseline systems, and the resulting F1 sits **level with LogMapLt**, the deliberately lightweight lexical matcher, and 0.109 below the leader. The `StringEquiv` baseline reaches 0.766 by normalised string equality alone, so the margin this system earns over pure string matching is **+0.066 F1**. That is the number to beat, and it is not yet a competitive result.+**Read this honestly.** Open Ontologies ranks **9th of 13** on F1. Its precision (0.960) is third in the field, but its recall (0.730) is second-from-bottom among non-baseline systems, and the resulting F1 sits **level with LogMapLt**, the deliberately lightweight lexical matcher, and 0.112 below the leader. The `StringEquiv` baseline reaches 0.766 by normalised string equality alone, so the margin this system earns over pure string matching is **+0.063 F1**. That is the number to beat, and it is not yet a competitive result.The gap is recall, and its cause is identifiable: this system carries **no domain background knowledge**. Anatomy is a track where UMLS and BioPortal lookups are what separate the 0.88+ band from the rest, and every system above Open Ontologies in the table either uses biomedical background knowledge, an LLM oracle, or both. Alignment here uses 7 weighted signals (label similarity, property/parent/instance/restriction/neighbourhood overlap, embedding similarity), stable 1-to-1 matching, and a label penalty when no structural evidence is available.@@ -462,7 +462,7 @@ The defensible finding from this track is therefore **not** the headline F1. It| Configuration | Precision | Recall | F1 || --- | ---: | ---: | ---: |-| With stable matching (5 weight configurations) | 0.963 | 0.733 | 0.832 ± 0.004 |+| With stable matching | 0.960 | 0.730 | 0.829 || Without stable matching | 0.102 | 0.846 | 0.182 |Removing stable matching produces 12,557 candidates against a 1,516-mapping reference. See issues [#8](https://github.com/fabio-rovai/open-ontologies/issues/8), [#9](https://github.com/fabio-rovai/open-ontologies/issues/9), [#10](https://github.com/fabio-rovai/open-ontologies/issues/10); background-knowledge integration is the open work.diff --git a/benchmark/oaei/README.md b/benchmark/oaei/README.mdindex 88416aba..7dd312fd 100644--- a/benchmark/oaei/README.md+++ b/benchmark/oaei/README.md@@ -32,7 +32,7 @@ Measured with `onto_align` at `min_confidence = 0.8`, stable 1-to-1 matching, la| Track | Precision | Recall | F1 | Predicted | Reference | Runtime ||---|---:|---:|---:|---:|---:|---:|-| Anatomy | 0.963 | 0.733 | **0.832** | 1,154 | 1,516 | 47s |+| Anatomy | 0.960 | 0.730 | **0.829** | 1,152 | 1,516 | 47s || Conference (15 of 21 pairs, micro) | 0.693 | 0.320 | **0.438** | 101 | 219 | — |## Comparison Systems@@ -51,13 +51,13 @@ The full OAEI 2025 field, reproduced from Tables 9 and 10 of the official result| MDMapper | 0.899 | **0.889** | 0.879 | 0.707 || LogMap | 0.917 | **0.881** | 0.848 | 0.602 || LogMapKG | 0.917 | **0.881** | 0.848 | 0.602 |-| **Open Ontologies** | **0.963** | **0.832** | **0.733** | — |+| **Open Ontologies** | **0.960** | **0.829** | **0.730** | — || DRAL-OA | 0.830 | **0.828** | 0.827 | 0.560 || LogMapLt | 0.962 | **0.828** | 0.728 | 0.288 || *StringEquiv (baseline)* | 0.997 | **0.766** | 0.622 | 0.000 || LSMatch | 0.952 | **0.761** | 0.634 | 0.037 |-Open Ontologies ranks **9th of 13**. Precision is third in the field; recall is second-from-bottom among non-baseline systems. F1 is level with LogMapLt, the lightweight lexical matcher, and **+0.066** over the `StringEquiv` string-equality baseline. Every system above it uses biomedical background knowledge, an LLM oracle, or both.+Open Ontologies ranks **9th of 13**. Precision is third in the field; recall is second-from-bottom among non-baseline systems. F1 is level with LogMapLt, the lightweight lexical matcher, and **+0.063** over the `StringEquiv` string-equality baseline. Every system above it uses biomedical background knowledge, an LLM oracle, or both.`Recall+` (recall excluding trivial same-normalised-label correspondences) is the metric that most exposes the gap and is not yet computed here; LogMapLt scores 0.288 on it against Matcha's 0.820.@@ -82,11 +82,36 @@ Per-pair spread (see `results/conference_results.json`): best `ekaw-iasted` 0.58## Ablation: stable matching dominates-Five weight configurations were run over the six structural signal weights (`full`, `label_only`, `no_label`, `equal`, `label_parent`). Measured F1 spans 0.8303 to 0.8336, a spread of **0.0033**. Removing stable 1-to-1 matching collapses the result:+Removing stable 1-to-1 matching as the **only** variable, threshold held at 0.80+(`run_ablation_no_stable.py`, raw data in `results/ablation_no_stable.json`):| Configuration | Precision | Recall | F1 | Candidates ||---|---:|---:|---:|---:|-| Stable matching (any weight configuration) | 0.963 | 0.733 | **0.832** | 1,154 |-| No stable matching | 0.102 | 0.846 | **0.182** | 12,557 |+| With stable matching | 0.960 | 0.730 | **0.829** | 1,152 |+| Without stable matching | 0.711 | 0.746 | **0.728** | 1,590 |-This, rather than the headline F1, is the transferable finding from the track: the assignment constraint carries the result and the signal weights are close to irrelevant once it is applied. Raw ablation data in `results/ablation_signals.json`.+Distinguish this from a second figure quoted elsewhere: the system *before* stable+matching, the label penalty and a raised pre-filter were introduced together scored+P 0.102 / R 0.846 / **F1 0.182** with 12,557 candidates. That is three simultaneous+changes and must not be attributed to the constraint alone.++### The weight ablation does not test what it appears to test++Five weight configurations with stable matching span F1 0.8303 to 0.8336, a spread of+0.0033; **without** stable matching all three configurations tested produce byte-identical+output. The obvious reading is "weights are irrelevant". The real reason is narrower.+`src/align.rs` branches:++```rust+if structural_sum == 0.0 { best_signal * 0.85 } else { signals·weights }+```++On Anatomy most class pairs share no properties, instances or restrictions, so+`structural_sum` is zero, the fallback fires, and **the weight vector is never read**. A+weight ablation on this track leaves most confidences untouched by construction. The+honest statement is that on this track weights are largely *not exercised*, and any+ablation reporting otherwise is measuring the fallback.++The conclusion about the constraint is unaffected, because it rests on the controlled+comparison above (0.101) rather than on the weight sweep (0.003). Raw data in+`results/ablation_signals.json` and `results/ablation_no_stable.json`.diff --git a/benchmark/oaei/results/oaei_results.json b/benchmark/oaei/results/oaei_results.jsonindex c9f142b4..a6a45883 100644--- a/benchmark/oaei/results/oaei_results.json+++ b/benchmark/oaei/results/oaei_results.json@@ -10,24 +10,27 @@],"date": "2026-04-21","notes": "v2: stable matching + label penalty (fixes #8). Previous results (v1, 2026-03-31) in oaei_results_v1.json",- "comparators": "OAEI 2025 (CEUR Vol-4144, om2025-oaei-paper0)"+ "comparators": "OAEI 2025 (CEUR Vol-4144, om2025-oaei-paper0)",+ "deterministic_since": "align.rs: sort extract_classes by IRI; total-order candidate sort"},"anatomy": {- "precision": 0.963,- "recall": 0.733,- "f1": 0.832,- "tp": 1112,- "fp": 42,- "fn": 404,+ "precision": 0.9601,+ "recall": 0.7296,+ "f1": 0.8291,+ "tp": 1106,+ "fp": 46,+ "fn": 410,"reference_size": 1516,- "predicted_size": 1154,+ "predicted_size": 1152,"time_s": 47,"min_confidence": 0.8,"changes": ["stable 1-to-1 matching (keep top candidate per source and target class)","label penalty (label_sim * 0.85) when structural signals are zero","label pre-filter raised from 0.70 to 0.75"- ]+ ],+ "deterministic": true,+ "notes": "Deterministic as of the align.rs ordering fix. Before it, extract_classes returned classes in HashMap iteration order and the confidence sort was not a total order, so the greedy 1-to-1 selection varied per process: five runs of the same binary gave 1151/1153/1151/1152/1154 candidates and five distinct alignments. The previously recorded P=0.963 R=0.733 F1=0.832 was one draw from that distribution."},"anatomy_v1": {"precision": 0.1021,diff --git a/docs/benchmarks.md b/docs/benchmarks.mdindex 77b112c3..989dad4e 100644--- a/docs/benchmarks.md+++ b/docs/benchmarks.md@@ -64,7 +64,7 @@ Given the Manchester Pizza OWL and a 13-row restaurant CSV, map the data into th## OntoAxiom Benchmark — Tool Access vs Raw File vs Name Lists-[OntoAxiom](https://arxiv.org/abs/2512.05594) tests LLM axiom identification across 9 ontologies and 3,042 ground truth axioms.+[OntoAxiom](https://arxiv.org/abs/2512.05594) tests LLM axiom identification across five axiom types. The paper states 2,771 axioms; the released ground truth contains 3,042 ordered pairs over those types, or 2,514 if symmetric pairs are collapsed. We score the 3,042 and say so.All conditions below are scored by **one** evaluator (`score_all_conditions.py`): shared normalizer, `domain`/`range`-only pair flipping, empty ground-truth cells skipped, both averages reported. Numbers are rescored from stored predictions, with no new inference.@@ -90,7 +90,7 @@ Full writeup, including the three-bug analysis and reproduction commands: [`benc## OAEI Ontology Alignment-Anatomy 0.832 F1 (9th of 13 in the OAEI 2025 field), Conference 0.438 (below every participating system and both baselines). Full field, both baselines, and the stable-matching ablation: [`benchmark/oaei/README.md`](../benchmark/oaei/README.md)+Anatomy 0.829 F1 (9th of 13 in the OAEI 2025 field), Conference 0.438 (below every participating system and both baselines). Full field, both baselines, and the stable-matching ablation: [`benchmark/oaei/README.md`](../benchmark/oaei/README.md)## Claim Verification — Compiled Reasoning vs HermiTdiff --git a/src/align.rs b/src/align.rsindex 5b51840a..3bda5083 100644--- a/src/align.rs+++ b/src/align.rs@@ -112,7 +112,16 @@ impl AlignmentEngine {}}- class_map.into_values().collect()+ // Sorted by IRI. `into_values()` yields HashMap iteration order, which Rust+ // seeds per process. That order propagates into the candidate-generation+ // loop, and because the confidence sort below is stable, equal-confidence+ // candidates keep it; the greedy 1-to-1 retain then picks a different+ // winner among ties on every run. Measured before this fix: five runs of+ // the same binary on OAEI Anatomy produced five different alignments+ // (1151/1153/1151/1152/1154 candidates, all five sets distinct).+ let mut out: Vec<ClassInfo> = class_map.into_values().collect();+ out.sort_by(|a, b| a.iri.cmp(&b.iri));+ out}/// Extract property IRIs whose domain is the given class.@@ -584,11 +593,21 @@ impl AlignmentEngine {}}- // Sort by confidence descending+ // Sort by confidence descending, ties broken by (source IRI, target IRI).+ //+ // Confidence alone is not a total order: on Anatomy many pairs tie exactly,+ // because the zero-structural-signal branch gives them all label_sim * 0.85.+ // `sort_by` is stable, so a confidence-only comparator leaves tied pairs in+ // input order, which makes the 1-to-1 selection below depend on how the+ // candidate list happened to be built. Breaking ties on the IRIs makes the+ // comparator total, so the result no longer depends on upstream ordering+ // even if that ordering changes again.candidates.sort_by(|a, b| {b["confidence"].as_f64().unwrap_or(0.0).partial_cmp(&a["confidence"].as_f64().unwrap_or(0.0)).unwrap_or(std::cmp::Ordering::Equal)+ .then_with(|| a["source_iri"].as_str().unwrap_or("").cmp(b["source_iri"].as_str().unwrap_or("")))+ .then_with(|| a["target_iri"].as_str().unwrap_or("").cmp(b["target_iri"].as_str().unwrap_or("")))});// Stable matching: for each source class, keep only the top-scoring target.