Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

feat(align): borderline-candidate surfacing for LLM-orchestrated review (#20)

Fabio Rovaia7d3990069cf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 606aefee..785c1b24 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,7 @@ All notable changes to Open Ontologies are documented here.
### Added
- **KGCL output format for `onto_drift`** (#17). The drift detector can now emit results in the [Knowledge Graph Change Language](https://github.com/INCATools/kgcl) (Mungall et al., Database 2025, doi:10.1093/database/baae133) alongside the existing JSON. Two new format options on the MCP tool: `format = "kgcl"` produces line-oriented CNL (`create node <iri>`, `obsolete node <iri>`, `obsolete node <iri> with replacement <iri>`) consumed by ROBOT and BioPortal; `format = "kgcl_json"` produces structured JSON-LD. High-confidence likely_renames (above `rename_threshold`, default 0.7) collapse into a `NodeObsoletion` with `has_direct_replacement` instead of plain add+remove pairs. New module `src/kgcl.rs` with 8 unit tests plus `tests/kgcl_drift_test.rs` integration suite.
+- **LLM-orchestrated borderline-candidate review for `onto_align`** (#16). The alignment engine now splits its output into three buckets driven by two thresholds rather than a single `min_confidence` cliff: candidates with confidence above `high_threshold` (default 0.85) auto-apply as today, those in `[low_threshold, high_threshold)` (default low 0.4) surface in a new `borderline` array enriched with `context` (source/target labels and parent IRIs), and those below `low_threshold` are dropped. The MCP tool returns a `summary_for_review` instructing the connected LLM to inspect each borderline pair and call `onto_align_feedback` to record verdicts — those verdicts flow into the existing self-calibrating-weights loop. This is the MCP-native form of the LogMap-LLM "LLM-as-oracle" pattern (Jiménez-Ruiz et al., EACL 2026 main, top-2 in OAEI 2025 Bio-ML): no extra LLM client, no API key, no provider abstraction — the connected orchestrator does the judging via the conversation that already exists. New public method `AlignmentEngine::align_with_thresholds(source, target, high, low, dry_run)`; the legacy `align(source, target, min_confidence, dry_run)` remains and delegates with a degenerate range (empty borderline bucket) for back-compat. New `OntoAlignInput` fields `high_threshold` + `low_threshold` (both optional); `min_confidence` retained as the back-compat alias for `high_threshold`. New `tests/align_borderline_test.rs` (5 tests) covering bucket boundaries, context enrichment, summary text, and back-compat.
- **DuckDB SQL data backbone**. New optional `duckdb` Cargo feature (and `sql` umbrella combining `postgres` + `duckdb`) wires DuckDB in alongside PostgreSQL as a *data integration* backbone — not as a SPARQL parser. DuckDB's extensions (`httpfs`, `parquet`, `csv`, `json`, `postgres_scanner`, `iceberg`, `delta`, …) let one SQL query federate over remote files, object stores, and other databases; rows then flow into the existing mapping/SHACL/reason pipeline.
- **New MCP tool `onto_sql_ingest`** — runs a SQL `SELECT` against PostgreSQL or DuckDB and ingests result rows into the triple store using the same `MappingConfig` shape as `onto_ingest`. Connection-string scheme is auto-detected (`postgres://`, `postgresql://`, `duckdb://`, `:memory:`, or a `*.duckdb` / `*.ddb` file path).
- **New CLI command `sql-ingest`** mirroring the MCP tool, with `--mapping`, `--inline-mapping`, `--base-iri`, and `-` (stdin) for the SQL.
diff --git a/src/align.rs b/src/align.rs
index 41db2ccf..d25181db 100644
--- a/src/align.rs
+++ b/src/align.rs
@@ -317,6 +317,30 @@ impl AlignmentEngine {
min_confidence: f64,
dry_run: bool,
) -> anyhow::Result<String> {
+ // Back-compat wrapper: degenerate borderline range (low == high) reproduces old behaviour.
+ self.align_with_thresholds(source, target, min_confidence, min_confidence, dry_run)
+ }
+
+ /// Two-threshold alignment with borderline-candidate surfacing for LLM-orchestrated review.
+ ///
+ /// - candidates with confidence ≥ `high_threshold` land in `auto_applied`
+ /// (and are persisted as triples unless `dry_run`)
+ /// - candidates with confidence in `[low_threshold, high_threshold)` land in `borderline`
+ /// with enriched context (labels, parents) so the calling LLM can judge them and record
+ /// verdicts via `onto_align_feedback`
+ /// - candidates below `low_threshold` are dropped
+ ///
+ /// If `low_threshold >= high_threshold`, no borderline bucket is produced (back-compat mode).
+ pub fn align_with_thresholds(
+ &self,
+ source: &str,
+ target: Option<&str>,
+ high_threshold: f64,
+ low_threshold: f64,
+ dry_run: bool,
+ ) -> anyhow::Result<String> {
+ // Clamp: degenerate range means "everything above high_threshold; nothing borderline".
+ let low_threshold = low_threshold.min(high_threshold);
// Load source into a temporary graph (detect format from content)
let source_store = GraphStore::new();
if std::path::Path::new(source).exists() {
@@ -403,12 +427,13 @@ impl AlignmentEngine {
signals.iter().zip(weights.iter()).map(|(s, w)| s * w).sum()
};
- // Skip low-confidence pairs
- if confidence < min_confidence {
+ // Drop pairs below the low threshold entirely.
+ if confidence < low_threshold {
continue;
}
let relation = Self::classify_relation(label_sim, prop_overlap, parent_ovlp);
+ let requires_review = confidence < high_threshold;
#[allow(unused_mut)]
let mut signals_json = serde_json::json!({
@@ -431,6 +456,7 @@ impl AlignmentEngine {
"confidence": (confidence * 1000.0).round() / 1000.0,
"signals": signals_json,
"applied": false,
+ "requires_review": requires_review,
}));
}
}
@@ -462,12 +488,12 @@ impl AlignmentEngine {
});
}
- // Auto-apply above threshold
+ // Auto-apply above high_threshold.
let mut applied_count = 0;
if !dry_run {
for candidate in &mut candidates {
let conf = candidate["confidence"].as_f64().unwrap_or(0.0);
- if conf >= min_confidence {
+ if conf >= high_threshold {
let source_iri = candidate["source_iri"].as_str().unwrap();
let target_iri = candidate["target_iri"].as_str().unwrap();
let relation = candidate["relation"].as_str().unwrap();
@@ -481,13 +507,77 @@ impl AlignmentEngine {
}
}
+ // Enrich borderline candidates with context for LLM-orchestrated review.
+ // Cheap: at most one extract_parents call per borderline candidate per side.
+ let source_class_map: std::collections::HashMap<&str, &ClassInfo> =
+ source_classes.iter().map(|c| (c.iri.as_str(), c)).collect();
+ let target_class_map: std::collections::HashMap<&str, &ClassInfo> =
+ target_classes.iter().map(|c| (c.iri.as_str(), c)).collect();
+ for candidate in &mut candidates {
+ if !candidate["requires_review"].as_bool().unwrap_or(false) {
+ continue;
+ }
+ let s_iri = candidate["source_iri"].as_str().unwrap_or("").to_string();
+ let t_iri = candidate["target_iri"].as_str().unwrap_or("").to_string();
+ let s_labels: Vec<String> = source_class_map
+ .get(s_iri.as_str())
+ .map(|c| c.labels.clone())
+ .unwrap_or_default();
+ let t_labels: Vec<String> = target_class_map
+ .get(t_iri.as_str())
+ .map(|c| c.labels.clone())
+ .unwrap_or_default();
+ let s_parents = Self::extract_parents(&source_store, &s_iri);
+ let t_parents = Self::extract_parents(target_ref, &t_iri);
+ candidate["context"] = serde_json::json!({
+ "source_labels": s_labels,
+ "target_labels": t_labels,
+ "source_parents": s_parents,
+ "target_parents": t_parents,
+ });
+ }
+
+ // Partition into auto-applied vs borderline buckets (back-compat: `candidates`
+ // keeps the unified sorted list; new `auto_applied` / `borderline` are views).
+ let auto_applied: Vec<serde_json::Value> = candidates
+ .iter()
+ .filter(|c| !c["requires_review"].as_bool().unwrap_or(false))
+ .cloned()
+ .collect();
+ let borderline: Vec<serde_json::Value> = candidates
+ .iter()
+ .filter(|c| c["requires_review"].as_bool().unwrap_or(false))
+ .cloned()
+ .collect();
+
let total = candidates.len();
+ let borderline_count = borderline.len();
+
+ let summary_for_review = if borderline_count > 0 {
+ format!(
+ "Found {} auto-applied matches and {} borderline pair(s) needing review. \
+ For each borderline pair, inspect the `context` field (labels, parents) \
+ and call `onto_align_feedback` with accepted=true|false to record your \
+ verdict — the self-calibrating weight model will learn from it.",
+ auto_applied.len(),
+ borderline_count
+ )
+ } else {
+ String::new()
+ };
Ok(serde_json::json!({
"candidates": candidates,
+ "auto_applied": auto_applied,
+ "borderline": borderline,
"applied_count": applied_count,
"total_candidates": total,
- "threshold": min_confidence,
+ "borderline_count": borderline_count,
+ "high_threshold": high_threshold,
+ "low_threshold": low_threshold,
+ // Back-compat alias — older callers read "threshold".
+ "threshold": high_threshold,
+ "summary_for_review": summary_for_review,
}).to_string())
}
diff --git a/src/inputs.rs b/src/inputs.rs
index 1a29a6f9..e63e5946 100644
--- a/src/inputs.rs
+++ b/src/inputs.rs
@@ -363,8 +363,18 @@ pub struct OntoAlignInput {
pub source: String,
/// Target ontology: inline Turtle content or file path. If omitted, aligns against loaded store
pub target: Option<String>,
- /// Minimum confidence threshold for auto-apply (default 0.85)
+ /// Minimum confidence threshold for auto-apply (default 0.85). Back-compat alias for
+ /// `high_threshold` — if both are set, `high_threshold` wins.
pub min_confidence: Option<f64>,
+ /// Confidence threshold above which a candidate is auto-applied (default 0.85, or
+ /// `min_confidence` if provided for back-compat). Candidates above this land in
+ /// `auto_applied`.
+ pub high_threshold: Option<f64>,
+ /// Confidence threshold below which a candidate is dropped entirely (default 0.4).
+ /// Candidates in [low_threshold, high_threshold] are surfaced in `borderline` with
+ /// enriched context (parents, siblings, labels) so the calling LLM can judge them
+ /// and record verdicts via `onto_align_feedback`.
+ pub low_threshold: Option<f64>,
/// If true, return candidates only without inserting triples (default false)
pub dry_run: Option<bool>,
}
diff --git a/src/server.rs b/src/server.rs
index 70f3d803..906c3092 100644
--- a/src/server.rs
+++ b/src/server.rs
@@ -1358,12 +1358,20 @@ impl OpenOntologiesServer {
None => None,
};
- let min_conf = input.min_confidence.unwrap_or(0.85);
+ let high = input.high_threshold.or(input.min_confidence).unwrap_or(0.85);
+ // Default low_threshold = 0.4 surfaces a borderline bucket for LLM-orchestrated review.
+ // Callers wanting the old strict behaviour pass low_threshold == high_threshold.
+ let low = input.low_threshold.unwrap_or(0.4).min(high);
let dry_run = input.dry_run.unwrap_or(false);
- match engine.align(&source, target.as_deref(), min_conf, dry_run) {
+ match engine.align_with_thresholds(&source, target.as_deref(), high, low, dry_run) {
Ok(result) => {
- self.lineage().record(&self.session_id, "AL", "align", &format!("threshold={}", min_conf));
+ self.lineage().record(
+ &self.session_id,
+ "AL",
+ "align",
+ &format!("high={},low={}", high, low),
+ );
result
}
Err(e) => format!(r#"{{"error":"{}"}}"#, e),
diff --git a/tests/align_borderline_test.rs b/tests/align_borderline_test.rs
new file mode 100644
index 00000000..ce4b97cf
--- /dev/null
+++ b/tests/align_borderline_test.rs
@@ -0,0 +1,176 @@
+//! Integration tests for the MCP-native LLM-as-oracle pattern (#16).
+//!
+//! Verifies that `AlignmentEngine::align_with_thresholds` correctly partitions candidates
+//! into `auto_applied` (≥ high_threshold) and `borderline` (in [low_threshold, high_threshold))
+//! buckets, and that borderline candidates carry the context fields (labels, parents) the
+//! calling LLM needs to judge them.
+
+use open_ontologies::align::AlignmentEngine;
+use open_ontologies::graph::GraphStore;
+use open_ontologies::state::StateDb;
+use std::sync::Arc;
+use tempfile::NamedTempFile;
+
+fn setup() -> (StateDb, Arc<GraphStore>) {
+ let tmp = NamedTempFile::new().unwrap();
+ let path = tmp.path().to_path_buf();
+ std::mem::forget(tmp);
+ let db = StateDb::open(&path).unwrap();
+ let graph = Arc::new(GraphStore::new());
+ (db, graph)
+}
+
+const SOURCE: &str = r#"
+ @prefix owl: <http://www.w3.org/2002/07/owl#> .
+ @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+ @prefix ex: <http://source.org/> .
+
+ ex:Animal a owl:Class ; rdfs:label "Animal" .
+ ex:Dog a owl:Class ; rdfs:label "Dog" ; rdfs:subClassOf ex:Animal .
+ ex:Cat a owl:Class ; rdfs:label "Cat" ; rdfs:subClassOf ex:Animal .
+"#;
+
+const TARGET: &str = r#"
+ @prefix owl: <http://www.w3.org/2002/07/owl#> .
+ @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+ @prefix tg: <http://target.org/> .
+
+ tg:Animal a owl:Class ; rdfs:label "Animal" .
+ tg:Dog a owl:Class ; rdfs:label "Dog" ; rdfs:subClassOf tg:Animal .
+ tg:Cat a owl:Class ; rdfs:label "Cat" ; rdfs:subClassOf tg:Animal .
+"#;
+
+#[test]
+fn degenerate_thresholds_preserve_legacy_behaviour() {
+ // low == high → no borderline bucket. Output shape matches the old align() semantics.
+ let (db, graph) = setup();
+ let engine = AlignmentEngine::new(db, graph);
+
+ let result_str = engine
+ .align_with_thresholds(SOURCE, Some(TARGET), 0.5, 0.5, true)
+ .expect("align_with_thresholds");
+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");
+
+ assert_eq!(
+ result["borderline_count"].as_u64().unwrap_or(99),
+ 0,
+ "degenerate range must produce empty borderline bucket"
+ );
+ assert!(
+ result["borderline"].as_array().unwrap().is_empty(),
+ "borderline array must be empty when low == high"
+ );
+ // All candidates that survived stable matching should be in auto_applied.
+ let candidates = result["candidates"].as_array().unwrap();
+ let auto_applied = result["auto_applied"].as_array().unwrap();
+ assert_eq!(auto_applied.len(), candidates.len());
+}
+
+#[test]
+fn borderline_bucket_collects_mid_confidence_candidates() {
+ // With a wide range, mid-confidence matches land in borderline.
+ let (db, graph) = setup();
+ let engine = AlignmentEngine::new(db, graph);
+
+ // high=0.95 is above what label-only matching produces here; low=0.4 catches
+ // the label-similarity-only matches.
+ let result_str = engine
+ .align_with_thresholds(SOURCE, Some(TARGET), 0.95, 0.4, true)
+ .expect("align_with_thresholds");
+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");
+
+ let borderline = result["borderline"].as_array().unwrap();
+ assert!(
+ !borderline.is_empty(),
+ "expected at least one borderline candidate with high=0.95, low=0.4; got result:\n{}",
+ result_str
+ );
+
+ // Every borderline candidate must have the LLM-review marker and the context block.
+ for cand in borderline {
+ assert_eq!(
+ cand["requires_review"].as_bool(),
+ Some(true),
+ "borderline candidate missing requires_review=true: {}",
+ cand
+ );
+ let ctx = &cand["context"];
+ assert!(
+ ctx.is_object(),
+ "borderline candidate missing `context` field: {}",
+ cand
+ );
+ assert!(ctx["source_labels"].is_array(), "context.source_labels missing");
+ assert!(ctx["target_labels"].is_array(), "context.target_labels missing");
+ assert!(ctx["source_parents"].is_array(), "context.source_parents missing");
+ assert!(ctx["target_parents"].is_array(), "context.target_parents missing");
+ }
+}
+
+#[test]
+fn auto_applied_candidates_lack_context_field() {
+ // Context enrichment is for borderline only — auto-applied candidates skip the parent
+ // lookups, keeping the hot path cheap on large ontologies.
+ let (db, graph) = setup();
+ let engine = AlignmentEngine::new(db, graph);
+
+ let result_str = engine
+ .align_with_thresholds(SOURCE, Some(TARGET), 0.5, 0.5, true)
+ .expect("align_with_thresholds");
+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");
+
+ for cand in result["auto_applied"].as_array().unwrap() {
+ assert_eq!(cand["requires_review"].as_bool(), Some(false));
+ assert!(
+ cand["context"].is_null(),
+ "auto_applied candidates must not carry context to keep enrichment cost off the hot path; got {}",
+ cand
+ );
+ }
+}
+
+#[test]
+fn summary_for_review_present_when_borderline_nonempty() {
+ let (db, graph) = setup();
+ let engine = AlignmentEngine::new(db, graph);
+
+ let result_str = engine
+ .align_with_thresholds(SOURCE, Some(TARGET), 0.95, 0.4, true)
+ .expect("align_with_thresholds");
+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");
+
+ let borderline_count = result["borderline_count"].as_u64().unwrap_or(0);
+ let summary = result["summary_for_review"].as_str().unwrap_or("");
+ if borderline_count > 0 {
+ assert!(
+ summary.contains("onto_align_feedback"),
+ "summary must instruct the LLM to call onto_align_feedback; got: {}",
+ summary
+ );
+ assert!(
+ summary.contains("borderline"),
+ "summary must mention borderline; got: {}",
+ summary
+ );
+ }
+}
+
+#[test]
+fn back_compat_align_method_still_works() {
+ // The old align(source, target, min_confidence, dry_run) signature is still callable
+ // and produces a sensible (degenerate-bucket) result.
+ let (db, graph) = setup();
+ let engine = AlignmentEngine::new(db, graph);
+
+ let result_str = engine
+ .align(SOURCE, Some(TARGET), 0.5, true)
+ .expect("legacy align");
+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");
+
+ // No borderline produced under back-compat (low_threshold = min_confidence).
+ assert_eq!(result["borderline_count"].as_u64().unwrap_or(99), 0);
+ // Existing keys still present.
+ assert!(result["candidates"].is_array());
+ assert!(result["total_candidates"].is_number());
+ assert!(result["threshold"].is_number());
+}