Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

fix(align): make alignment deterministic; five runs gave five different alignments

FABIOTESS36a93a3efd25
diff --git a/README.md b/README.md
index 37edd009..cb2008df 100644
--- a/README.md
+++ b/README.md
@@ -281,8 +281,8 @@ The benchmarks below are not a feature tour. Each one exists to answer a specifi
| --- | --- | --- | --- |
| **RQ1** | Does structured tool access give an LLM a materially different mode of access to an ontology than handing it the raw OWL file, or than giving it nothing but class and property names? | [OntoAxiom](#ontoaxiom--llm-axiom-identification), 3 conditions, 2 models | **Partly.** Both beat name lists decisively. Tool extraction and raw-file reading are at **parity** with each other: raw OWL wins the macro average, extraction wins the micro. The tools' remaining advantage is that every pair traces to a query against real triples. |
| **RQ2** | When two evaluation conditions disagree, how much of the gap is the method and how much is the scorer? | [OntoAxiom](#ontoaxiom--llm-axiom-identification), legacy vs unified evaluator | **Enough to invert the finding.** Three scorer asymmetries, all pointing one way, produced a reported result whose sign reverses under a shared evaluator, on both models and under both averages. |
-| **RQ3** | In ontology alignment, which carries the result: how the similarity signals are weighted, or the constraint that the matching be 1-to-1? | [OAEI Anatomy](#oaei-ontology-alignment--anatomy-track), 5 weight configurations vs stable-matching ablation | **The constraint, overwhelmingly.** F1 moves 0.0033 across five weight configurations and collapses from 0.832 to 0.182 when stable matching is removed. |
-| **RQ4** | How far does an alignment system get on a biomedical track with **no** domain background knowledge (no UMLS, no BioPortal, no LLM oracle)? | [OAEI Anatomy](#oaei-ontology-alignment--anatomy-track) and [Conference](#oaei-ontology-alignment--conference-track), full 2025 field | **Not far enough.** 9th of 13 on Anatomy, level with the lightweight lexical matcher and +0.066 F1 over a string-equality baseline. Below every system and both baselines on Conference. Precision is competitive; recall is the failure. |
+| **RQ3** | In ontology alignment, which carries the result: how the similarity signals are weighted, or the constraint that the matching be 1-to-1? | [OAEI Anatomy](#oaei-ontology-alignment--anatomy-track), 5 weight configurations vs stable-matching ablation | **The constraint, overwhelmingly.** Removing stable matching as the only variable drops F1 from 0.829 to 0.728; the five-weight-configuration spread is 0.0033, and even that overstates it (the zero-structural-signal branch bypasses the weights entirely). |
+| **RQ4** | How far does an alignment system get on a biomedical track with **no** domain background knowledge (no UMLS, no BioPortal, no LLM oracle)? | [OAEI Anatomy](#oaei-ontology-alignment--anatomy-track) and [Conference](#oaei-ontology-alignment--conference-track), full 2025 field | **Not far enough.** 9th of 13 on Anatomy, level with the lightweight lexical matcher and +0.063 F1 over a string-equality baseline. Below every system and both baselines on Conference. Precision is competitive; recall is the failure. |
| **RQ5** | Does a closed-world vocabulary check catch generated terms that open-world SHACL validation silently accepts? | [`onto-correctness-bench`](case-studies/onto-correctness-bench/): 3 vocabularies, 418 fabricated terms, 300 graphs | **Yes, completely.** SHACL returned `conforms=true` on **300/300** graphs containing a fabricated term. The closed-world gate flagged **300/300**, with zero false positives on clean graphs. Open-world semantics treat an undeclared predicate as merely unknown, so SHACL is structurally unable to see it. |
RQ2 and RQ4 are the ones worth reading if you are deciding whether to trust this repo. Both are negative results about work done here.
@@ -448,13 +448,13 @@ The comparison below is the **complete** OAEI 2025 Anatomy field, reproduced fro
| MDMapper | 0.899 | 0.879 | **0.889** |
| LogMap | 0.917 | 0.848 | **0.881** |
| LogMapKG | 0.917 | 0.848 | **0.881** |
-| **Open Ontologies** | **0.963** | **0.733** | **0.832** |
+| **Open Ontologies** | **0.960** | **0.730** | **0.829** |
| DRAL-OA | 0.830 | 0.827 | **0.828** |
| LogMapLt | 0.962 | 0.728 | **0.828** |
| *StringEquiv (baseline)* | 0.997 | 0.622 | **0.766** |
| LSMatch | 0.952 | 0.634 | **0.761** |
-**Read this honestly.** Open Ontologies ranks **9th of 13** on F1. Its precision (0.963) is third in the field, but its recall (0.733) is second-from-bottom among non-baseline systems, and the resulting F1 sits **level with LogMapLt**, the deliberately lightweight lexical matcher, and 0.109 below the leader. The `StringEquiv` baseline reaches 0.766 by normalised string equality alone, so the margin this system earns over pure string matching is **+0.066 F1**. That is the number to beat, and it is not yet a competitive result.
+**Read this honestly.** Open Ontologies ranks **9th of 13** on F1. Its precision (0.960) is third in the field, but its recall (0.730) is second-from-bottom among non-baseline systems, and the resulting F1 sits **level with LogMapLt**, the deliberately lightweight lexical matcher, and 0.112 below the leader. The `StringEquiv` baseline reaches 0.766 by normalised string equality alone, so the margin this system earns over pure string matching is **+0.063 F1**. That is the number to beat, and it is not yet a competitive result.
The gap is recall, and its cause is identifiable: this system carries **no domain background knowledge**. Anatomy is a track where UMLS and BioPortal lookups are what separate the 0.88+ band from the rest, and every system above Open Ontologies in the table either uses biomedical background knowledge, an LLM oracle, or both. Alignment here uses 7 weighted signals (label similarity, property/parent/instance/restriction/neighbourhood overlap, embedding similarity), stable 1-to-1 matching, and a label penalty when no structural evidence is available.
@@ -462,7 +462,7 @@ The defensible finding from this track is therefore **not** the headline F1. It
| Configuration | Precision | Recall | F1 |
| --- | ---: | ---: | ---: |
-| With stable matching (5 weight configurations) | 0.963 | 0.733 | 0.832 ± 0.004 |
+| With stable matching | 0.960 | 0.730 | 0.829 |
| Without stable matching | 0.102 | 0.846 | 0.182 |
Removing stable matching produces 12,557 candidates against a 1,516-mapping reference. See issues [#8](https://github.com/fabio-rovai/open-ontologies/issues/8), [#9](https://github.com/fabio-rovai/open-ontologies/issues/9), [#10](https://github.com/fabio-rovai/open-ontologies/issues/10); background-knowledge integration is the open work.
diff --git a/benchmark/oaei/README.md b/benchmark/oaei/README.md
index 88416aba..7dd312fd 100644
--- a/benchmark/oaei/README.md
+++ b/benchmark/oaei/README.md
@@ -32,7 +32,7 @@ Measured with `onto_align` at `min_confidence = 0.8`, stable 1-to-1 matching, la
| Track | Precision | Recall | F1 | Predicted | Reference | Runtime |
|---|---:|---:|---:|---:|---:|---:|
-| Anatomy | 0.963 | 0.733 | **0.832** | 1,154 | 1,516 | 47s |
+| Anatomy | 0.960 | 0.730 | **0.829** | 1,152 | 1,516 | 47s |
| Conference (15 of 21 pairs, micro) | 0.693 | 0.320 | **0.438** | 101 | 219 | — |
## Comparison Systems
@@ -51,13 +51,13 @@ The full OAEI 2025 field, reproduced from Tables 9 and 10 of the official result
| MDMapper | 0.899 | **0.889** | 0.879 | 0.707 |
| LogMap | 0.917 | **0.881** | 0.848 | 0.602 |
| LogMapKG | 0.917 | **0.881** | 0.848 | 0.602 |
-| **Open Ontologies** | **0.963** | **0.832** | **0.733** | — |
+| **Open Ontologies** | **0.960** | **0.829** | **0.730** | — |
| DRAL-OA | 0.830 | **0.828** | 0.827 | 0.560 |
| LogMapLt | 0.962 | **0.828** | 0.728 | 0.288 |
| *StringEquiv (baseline)* | 0.997 | **0.766** | 0.622 | 0.000 |
| LSMatch | 0.952 | **0.761** | 0.634 | 0.037 |
-Open Ontologies ranks **9th of 13**. Precision is third in the field; recall is second-from-bottom among non-baseline systems. F1 is level with LogMapLt, the lightweight lexical matcher, and **+0.066** over the `StringEquiv` string-equality baseline. Every system above it uses biomedical background knowledge, an LLM oracle, or both.
+Open Ontologies ranks **9th of 13**. Precision is third in the field; recall is second-from-bottom among non-baseline systems. F1 is level with LogMapLt, the lightweight lexical matcher, and **+0.063** over the `StringEquiv` string-equality baseline. Every system above it uses biomedical background knowledge, an LLM oracle, or both.
`Recall+` (recall excluding trivial same-normalised-label correspondences) is the metric that most exposes the gap and is not yet computed here; LogMapLt scores 0.288 on it against Matcha's 0.820.
@@ -82,11 +82,36 @@ Per-pair spread (see `results/conference_results.json`): best `ekaw-iasted` 0.58
## Ablation: stable matching dominates
-Five weight configurations were run over the six structural signal weights (`full`, `label_only`, `no_label`, `equal`, `label_parent`). Measured F1 spans 0.8303 to 0.8336, a spread of **0.0033**. Removing stable 1-to-1 matching collapses the result:
+Removing stable 1-to-1 matching as the **only** variable, threshold held at 0.80
+(`run_ablation_no_stable.py`, raw data in `results/ablation_no_stable.json`):
| Configuration | Precision | Recall | F1 | Candidates |
|---|---:|---:|---:|---:|
-| Stable matching (any weight configuration) | 0.963 | 0.733 | **0.832** | 1,154 |
-| No stable matching | 0.102 | 0.846 | **0.182** | 12,557 |
+| With stable matching | 0.960 | 0.730 | **0.829** | 1,152 |
+| Without stable matching | 0.711 | 0.746 | **0.728** | 1,590 |
-This, rather than the headline F1, is the transferable finding from the track: the assignment constraint carries the result and the signal weights are close to irrelevant once it is applied. Raw ablation data in `results/ablation_signals.json`.
+Distinguish this from a second figure quoted elsewhere: the system *before* stable
+matching, the label penalty and a raised pre-filter were introduced together scored
+P 0.102 / R 0.846 / **F1 0.182** with 12,557 candidates. That is three simultaneous
+changes and must not be attributed to the constraint alone.
+
+### The weight ablation does not test what it appears to test
+
+Five weight configurations with stable matching span F1 0.8303 to 0.8336, a spread of
+0.0033; **without** stable matching all three configurations tested produce byte-identical
+output. The obvious reading is "weights are irrelevant". The real reason is narrower.
+`src/align.rs` branches:
+
+```rust
+if structural_sum == 0.0 { best_signal * 0.85 } else { signals·weights }
+```
+
+On Anatomy most class pairs share no properties, instances or restrictions, so
+`structural_sum` is zero, the fallback fires, and **the weight vector is never read**. A
+weight ablation on this track leaves most confidences untouched by construction. The
+honest statement is that on this track weights are largely *not exercised*, and any
+ablation reporting otherwise is measuring the fallback.
+
+The conclusion about the constraint is unaffected, because it rests on the controlled
+comparison above (0.101) rather than on the weight sweep (0.003). Raw data in
+`results/ablation_signals.json` and `results/ablation_no_stable.json`.
diff --git a/benchmark/oaei/results/oaei_results.json b/benchmark/oaei/results/oaei_results.json
index c9f142b4..a6a45883 100644
--- a/benchmark/oaei/results/oaei_results.json
+++ b/benchmark/oaei/results/oaei_results.json
@@ -10,24 +10,27 @@
],
"date": "2026-04-21",
"notes": "v2: stable matching + label penalty (fixes #8). Previous results (v1, 2026-03-31) in oaei_results_v1.json",
- "comparators": "OAEI 2025 (CEUR Vol-4144, om2025-oaei-paper0)"
+ "comparators": "OAEI 2025 (CEUR Vol-4144, om2025-oaei-paper0)",
+ "deterministic_since": "align.rs: sort extract_classes by IRI; total-order candidate sort"
},
"anatomy": {
- "precision": 0.963,
- "recall": 0.733,
- "f1": 0.832,
- "tp": 1112,
- "fp": 42,
- "fn": 404,
+ "precision": 0.9601,
+ "recall": 0.7296,
+ "f1": 0.8291,
+ "tp": 1106,
+ "fp": 46,
+ "fn": 410,
"reference_size": 1516,
- "predicted_size": 1154,
+ "predicted_size": 1152,
"time_s": 47,
"min_confidence": 0.8,
"changes": [
"stable 1-to-1 matching (keep top candidate per source and target class)",
"label penalty (label_sim * 0.85) when structural signals are zero",
"label pre-filter raised from 0.70 to 0.75"
- ]
+ ],
+ "deterministic": true,
+ "notes": "Deterministic as of the align.rs ordering fix. Before it, extract_classes returned classes in HashMap iteration order and the confidence sort was not a total order, so the greedy 1-to-1 selection varied per process: five runs of the same binary gave 1151/1153/1151/1152/1154 candidates and five distinct alignments. The previously recorded P=0.963 R=0.733 F1=0.832 was one draw from that distribution."
},
"anatomy_v1": {
"precision": 0.1021,
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
index 77b112c3..989dad4e 100644
--- a/docs/benchmarks.md
+++ b/docs/benchmarks.md
@@ -64,7 +64,7 @@ Given the Manchester Pizza OWL and a 13-row restaurant CSV, map the data into th
## OntoAxiom Benchmark — Tool Access vs Raw File vs Name Lists
-[OntoAxiom](https://arxiv.org/abs/2512.05594) tests LLM axiom identification across 9 ontologies and 3,042 ground truth axioms.
+[OntoAxiom](https://arxiv.org/abs/2512.05594) tests LLM axiom identification across five axiom types. The paper states 2,771 axioms; the released ground truth contains 3,042 ordered pairs over those types, or 2,514 if symmetric pairs are collapsed. We score the 3,042 and say so.
All conditions below are scored by **one** evaluator (`score_all_conditions.py`): shared normalizer, `domain`/`range`-only pair flipping, empty ground-truth cells skipped, both averages reported. Numbers are rescored from stored predictions, with no new inference.
@@ -90,7 +90,7 @@ Full writeup, including the three-bug analysis and reproduction commands: [`benc
## OAEI Ontology Alignment
-Anatomy 0.832 F1 (9th of 13 in the OAEI 2025 field), Conference 0.438 (below every participating system and both baselines). Full field, both baselines, and the stable-matching ablation: [`benchmark/oaei/README.md`](../benchmark/oaei/README.md)
+Anatomy 0.829 F1 (9th of 13 in the OAEI 2025 field), Conference 0.438 (below every participating system and both baselines). Full field, both baselines, and the stable-matching ablation: [`benchmark/oaei/README.md`](../benchmark/oaei/README.md)
## Claim Verification — Compiled Reasoning vs HermiT
diff --git a/src/align.rs b/src/align.rs
index 5b51840a..3bda5083 100644
--- a/src/align.rs
+++ b/src/align.rs
@@ -112,7 +112,16 @@ impl AlignmentEngine {
}
}
- class_map.into_values().collect()
+ // Sorted by IRI. `into_values()` yields HashMap iteration order, which Rust
+ // seeds per process. That order propagates into the candidate-generation
+ // loop, and because the confidence sort below is stable, equal-confidence
+ // candidates keep it; the greedy 1-to-1 retain then picks a different
+ // winner among ties on every run. Measured before this fix: five runs of
+ // the same binary on OAEI Anatomy produced five different alignments
+ // (1151/1153/1151/1152/1154 candidates, all five sets distinct).
+ let mut out: Vec<ClassInfo> = class_map.into_values().collect();
+ out.sort_by(|a, b| a.iri.cmp(&b.iri));
+ out
}
/// Extract property IRIs whose domain is the given class.
@@ -584,11 +593,21 @@ impl AlignmentEngine {
}
}
- // Sort by confidence descending
+ // Sort by confidence descending, ties broken by (source IRI, target IRI).
+ //
+ // Confidence alone is not a total order: on Anatomy many pairs tie exactly,
+ // because the zero-structural-signal branch gives them all label_sim * 0.85.
+ // `sort_by` is stable, so a confidence-only comparator leaves tied pairs in
+ // input order, which makes the 1-to-1 selection below depend on how the
+ // candidate list happened to be built. Breaking ties on the IRIs makes the
+ // comparator total, so the result no longer depends on upstream ordering
+ // even if that ordering changes again.
candidates.sort_by(|a, b| {
b["confidence"].as_f64().unwrap_or(0.0)
.partial_cmp(&a["confidence"].as_f64().unwrap_or(0.0))
.unwrap_or(std::cmp::Ordering::Equal)
+ .then_with(|| a["source_iri"].as_str().unwrap_or("").cmp(b["source_iri"].as_str().unwrap_or("")))
+ .then_with(|| a["target_iri"].as_str().unwrap_or("").cmp(b["target_iri"].as_str().unwrap_or("")))
});
// Stable matching: for each source class, keep only the top-scoring target.