Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
feat(align): borderline-candidate surfacing for LLM-orchestrated review (#20)
Fabio Rovaia7d3990069cf
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 606aefee..785c1b24 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,7 @@ All notable changes to Open Ontologies are documented here.### Added- **KGCL output format for `onto_drift`** (#17). The drift detector can now emit results in the [Knowledge Graph Change Language](https://github.com/INCATools/kgcl) (Mungall et al., Database 2025, doi:10.1093/database/baae133) alongside the existing JSON. Two new format options on the MCP tool: `format = "kgcl"` produces line-oriented CNL (`create node <iri>`, `obsolete node <iri>`, `obsolete node <iri> with replacement <iri>`) consumed by ROBOT and BioPortal; `format = "kgcl_json"` produces structured JSON-LD. High-confidence likely_renames (above `rename_threshold`, default 0.7) collapse into a `NodeObsoletion` with `has_direct_replacement` instead of plain add+remove pairs. New module `src/kgcl.rs` with 8 unit tests plus `tests/kgcl_drift_test.rs` integration suite.+- **LLM-orchestrated borderline-candidate review for `onto_align`** (#16). The alignment engine now splits its output into three buckets driven by two thresholds rather than a single `min_confidence` cliff: candidates with confidence above `high_threshold` (default 0.85) auto-apply as today, those in `[low_threshold, high_threshold)` (default low 0.4) surface in a new `borderline` array enriched with `context` (source/target labels and parent IRIs), and those below `low_threshold` are dropped. The MCP tool returns a `summary_for_review` instructing the connected LLM to inspect each borderline pair and call `onto_align_feedback` to record verdicts — those verdicts flow into the existing self-calibrating-weights loop. This is the MCP-native form of the LogMap-LLM "LLM-as-oracle" pattern (Jiménez-Ruiz et al., EACL 2026 main, top-2 in OAEI 2025 Bio-ML): no extra LLM client, no API key, no provider abstraction — the connected orchestrator does the judging via the conversation that already exists. New public method `AlignmentEngine::align_with_thresholds(source, target, high, low, dry_run)`; the legacy `align(source, target, min_confidence, dry_run)` remains and delegates with a degenerate range (empty borderline bucket) for back-compat. New `OntoAlignInput` fields `high_threshold` + `low_threshold` (both optional); `min_confidence` retained as the back-compat alias for `high_threshold`. New `tests/align_borderline_test.rs` (5 tests) covering bucket boundaries, context enrichment, summary text, and back-compat.- **DuckDB SQL data backbone**. New optional `duckdb` Cargo feature (and `sql` umbrella combining `postgres` + `duckdb`) wires DuckDB in alongside PostgreSQL as a *data integration* backbone — not as a SPARQL parser. DuckDB's extensions (`httpfs`, `parquet`, `csv`, `json`, `postgres_scanner`, `iceberg`, `delta`, …) let one SQL query federate over remote files, object stores, and other databases; rows then flow into the existing mapping/SHACL/reason pipeline.- **New MCP tool `onto_sql_ingest`** — runs a SQL `SELECT` against PostgreSQL or DuckDB and ingests result rows into the triple store using the same `MappingConfig` shape as `onto_ingest`. Connection-string scheme is auto-detected (`postgres://`, `postgresql://`, `duckdb://`, `:memory:`, or a `*.duckdb` / `*.ddb` file path).- **New CLI command `sql-ingest`** mirroring the MCP tool, with `--mapping`, `--inline-mapping`, `--base-iri`, and `-` (stdin) for the SQL.diff --git a/src/align.rs b/src/align.rsindex 41db2ccf..d25181db 100644--- a/src/align.rs+++ b/src/align.rs@@ -317,6 +317,30 @@ impl AlignmentEngine {min_confidence: f64,dry_run: bool,) -> anyhow::Result<String> {+ // Back-compat wrapper: degenerate borderline range (low == high) reproduces old behaviour.+ self.align_with_thresholds(source, target, min_confidence, min_confidence, dry_run)+ }++ /// Two-threshold alignment with borderline-candidate surfacing for LLM-orchestrated review.+ ///+ /// - candidates with confidence ≥ `high_threshold` land in `auto_applied`+ /// (and are persisted as triples unless `dry_run`)+ /// - candidates with confidence in `[low_threshold, high_threshold)` land in `borderline`+ /// with enriched context (labels, parents) so the calling LLM can judge them and record+ /// verdicts via `onto_align_feedback`+ /// - candidates below `low_threshold` are dropped+ ///+ /// If `low_threshold >= high_threshold`, no borderline bucket is produced (back-compat mode).+ pub fn align_with_thresholds(+ &self,+ source: &str,+ target: Option<&str>,+ high_threshold: f64,+ low_threshold: f64,+ dry_run: bool,+ ) -> anyhow::Result<String> {+ // Clamp: degenerate range means "everything above high_threshold; nothing borderline".+ let low_threshold = low_threshold.min(high_threshold);// Load source into a temporary graph (detect format from content)let source_store = GraphStore::new();if std::path::Path::new(source).exists() {@@ -403,12 +427,13 @@ impl AlignmentEngine {signals.iter().zip(weights.iter()).map(|(s, w)| s * w).sum()};- // Skip low-confidence pairs- if confidence < min_confidence {+ // Drop pairs below the low threshold entirely.+ if confidence < low_threshold {continue;}let relation = Self::classify_relation(label_sim, prop_overlap, parent_ovlp);+ let requires_review = confidence < high_threshold;#[allow(unused_mut)]let mut signals_json = serde_json::json!({@@ -431,6 +456,7 @@ impl AlignmentEngine {"confidence": (confidence * 1000.0).round() / 1000.0,"signals": signals_json,"applied": false,+ "requires_review": requires_review,}));}}@@ -462,12 +488,12 @@ impl AlignmentEngine {});}- // Auto-apply above threshold+ // Auto-apply above high_threshold.let mut applied_count = 0;if !dry_run {for candidate in &mut candidates {let conf = candidate["confidence"].as_f64().unwrap_or(0.0);- if conf >= min_confidence {+ if conf >= high_threshold {let source_iri = candidate["source_iri"].as_str().unwrap();let target_iri = candidate["target_iri"].as_str().unwrap();let relation = candidate["relation"].as_str().unwrap();@@ -481,13 +507,77 @@ impl AlignmentEngine {}}+ // Enrich borderline candidates with context for LLM-orchestrated review.+ // Cheap: at most one extract_parents call per borderline candidate per side.+ let source_class_map: std::collections::HashMap<&str, &ClassInfo> =+ source_classes.iter().map(|c| (c.iri.as_str(), c)).collect();+ let target_class_map: std::collections::HashMap<&str, &ClassInfo> =+ target_classes.iter().map(|c| (c.iri.as_str(), c)).collect();+ for candidate in &mut candidates {+ if !candidate["requires_review"].as_bool().unwrap_or(false) {+ continue;+ }+ let s_iri = candidate["source_iri"].as_str().unwrap_or("").to_string();+ let t_iri = candidate["target_iri"].as_str().unwrap_or("").to_string();+ let s_labels: Vec<String> = source_class_map+ .get(s_iri.as_str())+ .map(|c| c.labels.clone())+ .unwrap_or_default();+ let t_labels: Vec<String> = target_class_map+ .get(t_iri.as_str())+ .map(|c| c.labels.clone())+ .unwrap_or_default();+ let s_parents = Self::extract_parents(&source_store, &s_iri);+ let t_parents = Self::extract_parents(target_ref, &t_iri);+ candidate["context"] = serde_json::json!({+ "source_labels": s_labels,+ "target_labels": t_labels,+ "source_parents": s_parents,+ "target_parents": t_parents,+ });+ }++ // Partition into auto-applied vs borderline buckets (back-compat: `candidates`+ // keeps the unified sorted list; new `auto_applied` / `borderline` are views).+ let auto_applied: Vec<serde_json::Value> = candidates+ .iter()+ .filter(|c| !c["requires_review"].as_bool().unwrap_or(false))+ .cloned()+ .collect();+ let borderline: Vec<serde_json::Value> = candidates+ .iter()+ .filter(|c| c["requires_review"].as_bool().unwrap_or(false))+ .cloned()+ .collect();+let total = candidates.len();+ let borderline_count = borderline.len();++ let summary_for_review = if borderline_count > 0 {+ format!(+ "Found {} auto-applied matches and {} borderline pair(s) needing review. \+ For each borderline pair, inspect the `context` field (labels, parents) \+ and call `onto_align_feedback` with accepted=true|false to record your \+ verdict — the self-calibrating weight model will learn from it.",+ auto_applied.len(),+ borderline_count+ )+ } else {+ String::new()+ };Ok(serde_json::json!({"candidates": candidates,+ "auto_applied": auto_applied,+ "borderline": borderline,"applied_count": applied_count,"total_candidates": total,- "threshold": min_confidence,+ "borderline_count": borderline_count,+ "high_threshold": high_threshold,+ "low_threshold": low_threshold,+ // Back-compat alias — older callers read "threshold".+ "threshold": high_threshold,+ "summary_for_review": summary_for_review,}).to_string())}diff --git a/src/inputs.rs b/src/inputs.rsindex 1a29a6f9..e63e5946 100644--- a/src/inputs.rs+++ b/src/inputs.rs@@ -363,8 +363,18 @@ pub struct OntoAlignInput {pub source: String,/// Target ontology: inline Turtle content or file path. If omitted, aligns against loaded storepub target: Option<String>,- /// Minimum confidence threshold for auto-apply (default 0.85)+ /// Minimum confidence threshold for auto-apply (default 0.85). Back-compat alias for+ /// `high_threshold` — if both are set, `high_threshold` wins.pub min_confidence: Option<f64>,+ /// Confidence threshold above which a candidate is auto-applied (default 0.85, or+ /// `min_confidence` if provided for back-compat). Candidates above this land in+ /// `auto_applied`.+ pub high_threshold: Option<f64>,+ /// Confidence threshold below which a candidate is dropped entirely (default 0.4).+ /// Candidates in [low_threshold, high_threshold] are surfaced in `borderline` with+ /// enriched context (parents, siblings, labels) so the calling LLM can judge them+ /// and record verdicts via `onto_align_feedback`.+ pub low_threshold: Option<f64>,/// If true, return candidates only without inserting triples (default false)pub dry_run: Option<bool>,}diff --git a/src/server.rs b/src/server.rsindex 70f3d803..906c3092 100644--- a/src/server.rs+++ b/src/server.rs@@ -1358,12 +1358,20 @@ impl OpenOntologiesServer {None => None,};- let min_conf = input.min_confidence.unwrap_or(0.85);+ let high = input.high_threshold.or(input.min_confidence).unwrap_or(0.85);+ // Default low_threshold = 0.4 surfaces a borderline bucket for LLM-orchestrated review.+ // Callers wanting the old strict behaviour pass low_threshold == high_threshold.+ let low = input.low_threshold.unwrap_or(0.4).min(high);let dry_run = input.dry_run.unwrap_or(false);- match engine.align(&source, target.as_deref(), min_conf, dry_run) {+ match engine.align_with_thresholds(&source, target.as_deref(), high, low, dry_run) {Ok(result) => {- self.lineage().record(&self.session_id, "AL", "align", &format!("threshold={}", min_conf));+ self.lineage().record(+ &self.session_id,+ "AL",+ "align",+ &format!("high={},low={}", high, low),+ );result}Err(e) => format!(r#"{{"error":"{}"}}"#, e),diff --git a/tests/align_borderline_test.rs b/tests/align_borderline_test.rsnew file mode 100644index 00000000..ce4b97cf--- /dev/null+++ b/tests/align_borderline_test.rs@@ -0,0 +1,176 @@+//! Integration tests for the MCP-native LLM-as-oracle pattern (#16).+//!+//! Verifies that `AlignmentEngine::align_with_thresholds` correctly partitions candidates+//! into `auto_applied` (≥ high_threshold) and `borderline` (in [low_threshold, high_threshold))+//! buckets, and that borderline candidates carry the context fields (labels, parents) the+//! calling LLM needs to judge them.++use open_ontologies::align::AlignmentEngine;+use open_ontologies::graph::GraphStore;+use open_ontologies::state::StateDb;+use std::sync::Arc;+use tempfile::NamedTempFile;++fn setup() -> (StateDb, Arc<GraphStore>) {+ let tmp = NamedTempFile::new().unwrap();+ let path = tmp.path().to_path_buf();+ std::mem::forget(tmp);+ let db = StateDb::open(&path).unwrap();+ let graph = Arc::new(GraphStore::new());+ (db, graph)+}++const SOURCE: &str = r#"+ @prefix owl: <http://www.w3.org/2002/07/owl#> .+ @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .+ @prefix ex: <http://source.org/> .++ ex:Animal a owl:Class ; rdfs:label "Animal" .+ ex:Dog a owl:Class ; rdfs:label "Dog" ; rdfs:subClassOf ex:Animal .+ ex:Cat a owl:Class ; rdfs:label "Cat" ; rdfs:subClassOf ex:Animal .+"#;++const TARGET: &str = r#"+ @prefix owl: <http://www.w3.org/2002/07/owl#> .+ @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .+ @prefix tg: <http://target.org/> .++ tg:Animal a owl:Class ; rdfs:label "Animal" .+ tg:Dog a owl:Class ; rdfs:label "Dog" ; rdfs:subClassOf tg:Animal .+ tg:Cat a owl:Class ; rdfs:label "Cat" ; rdfs:subClassOf tg:Animal .+"#;++#[test]+fn degenerate_thresholds_preserve_legacy_behaviour() {+ // low == high → no borderline bucket. Output shape matches the old align() semantics.+ let (db, graph) = setup();+ let engine = AlignmentEngine::new(db, graph);++ let result_str = engine+ .align_with_thresholds(SOURCE, Some(TARGET), 0.5, 0.5, true)+ .expect("align_with_thresholds");+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");++ assert_eq!(+ result["borderline_count"].as_u64().unwrap_or(99),+ 0,+ "degenerate range must produce empty borderline bucket"+ );+ assert!(+ result["borderline"].as_array().unwrap().is_empty(),+ "borderline array must be empty when low == high"+ );+ // All candidates that survived stable matching should be in auto_applied.+ let candidates = result["candidates"].as_array().unwrap();+ let auto_applied = result["auto_applied"].as_array().unwrap();+ assert_eq!(auto_applied.len(), candidates.len());+}++#[test]+fn borderline_bucket_collects_mid_confidence_candidates() {+ // With a wide range, mid-confidence matches land in borderline.+ let (db, graph) = setup();+ let engine = AlignmentEngine::new(db, graph);++ // high=0.95 is above what label-only matching produces here; low=0.4 catches+ // the label-similarity-only matches.+ let result_str = engine+ .align_with_thresholds(SOURCE, Some(TARGET), 0.95, 0.4, true)+ .expect("align_with_thresholds");+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");++ let borderline = result["borderline"].as_array().unwrap();+ assert!(+ !borderline.is_empty(),+ "expected at least one borderline candidate with high=0.95, low=0.4; got result:\n{}",+ result_str+ );++ // Every borderline candidate must have the LLM-review marker and the context block.+ for cand in borderline {+ assert_eq!(+ cand["requires_review"].as_bool(),+ Some(true),+ "borderline candidate missing requires_review=true: {}",+ cand+ );+ let ctx = &cand["context"];+ assert!(+ ctx.is_object(),+ "borderline candidate missing `context` field: {}",+ cand+ );+ assert!(ctx["source_labels"].is_array(), "context.source_labels missing");+ assert!(ctx["target_labels"].is_array(), "context.target_labels missing");+ assert!(ctx["source_parents"].is_array(), "context.source_parents missing");+ assert!(ctx["target_parents"].is_array(), "context.target_parents missing");+ }+}++#[test]+fn auto_applied_candidates_lack_context_field() {+ // Context enrichment is for borderline only — auto-applied candidates skip the parent+ // lookups, keeping the hot path cheap on large ontologies.+ let (db, graph) = setup();+ let engine = AlignmentEngine::new(db, graph);++ let result_str = engine+ .align_with_thresholds(SOURCE, Some(TARGET), 0.5, 0.5, true)+ .expect("align_with_thresholds");+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");++ for cand in result["auto_applied"].as_array().unwrap() {+ assert_eq!(cand["requires_review"].as_bool(), Some(false));+ assert!(+ cand["context"].is_null(),+ "auto_applied candidates must not carry context to keep enrichment cost off the hot path; got {}",+ cand+ );+ }+}++#[test]+fn summary_for_review_present_when_borderline_nonempty() {+ let (db, graph) = setup();+ let engine = AlignmentEngine::new(db, graph);++ let result_str = engine+ .align_with_thresholds(SOURCE, Some(TARGET), 0.95, 0.4, true)+ .expect("align_with_thresholds");+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");++ let borderline_count = result["borderline_count"].as_u64().unwrap_or(0);+ let summary = result["summary_for_review"].as_str().unwrap_or("");+ if borderline_count > 0 {+ assert!(+ summary.contains("onto_align_feedback"),+ "summary must instruct the LLM to call onto_align_feedback; got: {}",+ summary+ );+ assert!(+ summary.contains("borderline"),+ "summary must mention borderline; got: {}",+ summary+ );+ }+}++#[test]+fn back_compat_align_method_still_works() {+ // The old align(source, target, min_confidence, dry_run) signature is still callable+ // and produces a sensible (degenerate-bucket) result.+ let (db, graph) = setup();+ let engine = AlignmentEngine::new(db, graph);++ let result_str = engine+ .align(SOURCE, Some(TARGET), 0.5, true)+ .expect("legacy align");+ let result: serde_json::Value = serde_json::from_str(&result_str).expect("json");++ // No borderline produced under back-compat (low_threshold = min_confidence).+ assert_eq!(result["borderline_count"].as_u64().unwrap_or(99), 0);+ // Existing keys still present.+ assert!(result["candidates"].is_array());+ assert!(result["total_candidates"].is_number());+ assert!(result["threshold"].is_number());+}