Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
feat(support): claim support checking, the second axis beside conformance
FABIOTESS8a3aacf9d844
diff --git a/CLAUDE.md b/CLAUDE.mdindex 8e800b6c..950db190 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -87,6 +87,9 @@ Claude dynamically decides the next tool call based on what the previous tool re| `onto_align` | To detect alignment candidates (equivalentClass, exactMatch, subClassOf) between two ontologies using 7 weighted signals (6 structural + embedding similarity when embeddings are loaded). Labels are matched with their parsed BCP-47 language tag; with the multilingual embedder loaded, cross-lingual pairs that share no surface tokens (e.g. `Dog`↔`Chien`) are admitted via the embedding signal. Restrict the languages consulted with `[language] preferred = [...]` / `OPEN_ONTOLOGIES_LANGUAGES` (empty = all) || `onto_align_feedback` | To accept/reject alignment candidates for self-calibrating confidence weights || `onto_communities` | To cluster the entity graph and get a SKELETON per community (size, top members by degree, internal relations, bridges to other communities). Deterministic modularity optimisation, no model involved. Write one report per skeleton, then answer corpus-wide questions ("what are the themes", "summarise this corpus") from the reports instead of traversing from an anchor entity, which such questions do not have. The GraphRAG global-search pattern with the expensive half left to the orchestrator |+| `onto_support_check` | To check whether claims are backed by the sources they cite. Returns which claims cite NO source (computed) and, for the rest, verification TASKS: the claim as a sentence, the source, what to decide. Complements `onto_vocab_check`: conformance asks whether a claim is expressible, support asks whether it is true to its source, and a claim can fail either independently |+| `onto_support_verdict` | To record supported / refuted / unrelated for a claim after reading its source. Verdicts persist and are skipped on the next check |+| `onto_support_report` | To summarise provenance quality: unsourced-claim rate over the graph, support rate across the claims judged so far || `onto_pack` | To write the loaded graph and its verification evidence to a portable versioned artifact: sorted N-Triples, a manifest (name, version, counts, timestamp, tool version, sha256) and the lint/enforce results recorded at pack time. What you promote between environments is then a graph that has already passed its checks, with the evidence attached || `onto_unpack` | To load a pack, refusing it if the checksum does not match. `verify_only` inspects the manifest and evidence without loading || `onto_lineage` | To view the session's lineage trail (plan → enforce → apply → monitor → drift) |diff --git a/src/inputs.rs b/src/inputs.rsindex ae8f6fb1..0e3c0232 100644--- a/src/inputs.rs+++ b/src/inputs.rs@@ -1116,3 +1116,28 @@ pub struct OntoUnpackInput {/// Verify the checksum and report the manifest without loading (default false)pub verify_only: Option<bool>,}++#[derive(Deserialize, JsonSchema)]+pub struct OntoSupportCheckInput {+ /// Provenance predicate linking a claim to its source+ /// (default: prov:wasDerivedFrom)+ pub prov_predicate: Option<String>,+ /// Maximum tasks and unsourced examples to return (default 25)+ pub limit: Option<usize>,+}++#[derive(Deserialize, JsonSchema)]+pub struct OntoSupportVerdictInput {+ /// claim_id from onto_support_check+ pub claim_id: String,+ /// supported | refuted | unrelated+ pub verdict: String,+ /// Optional note: which passage decided it+ pub note: Option<String>,+}++#[derive(Deserialize, JsonSchema)]+pub struct OntoSupportReportInput {+ /// Provenance predicate (default: prov:wasDerivedFrom)+ pub prov_predicate: Option<String>,+}diff --git a/src/lib.rs b/src/lib.rsindex 2ae4c9ca..8f5dfc2c 100644--- a/src/lib.rs+++ b/src/lib.rs@@ -24,6 +24,7 @@ pub mod eval_rag;pub mod extract_scaffold;pub mod flora_pipeline;pub mod pack;+pub mod support;pub mod policy;pub mod projection_check;pub mod shape_combinatorics;diff --git a/src/server.rs b/src/server.rsindex 0c0bef09..19908321 100644--- a/src/server.rs+++ b/src/server.rs@@ -883,6 +883,36 @@ impl OpenOntologiesServer {}}+ #[tool(name = "onto_support_check", description = "Check whether the graph's claims are backed by the sources they cite. Returns which claims cite NO source at all (computed, no model needed) and, for those that do, verification TASKS: the claim as a sentence, the source, and what to decide. Read each source and record supported/refuted/unrelated with onto_support_verdict. Complements onto_vocab_check: conformance asks whether a claim is expressible, support asks whether it is true to its source, and a claim can fail either independently.")]+ async fn onto_support_check(&self, Parameters(input): Parameters<OntoSupportCheckInput>) -> String {+ use crate::support::SupportChecker;+ let checker = SupportChecker::new(self.graph.clone(), self.db.clone());+ match checker.check(input.prov_predicate.as_deref(), input.limit.unwrap_or(25)) {+ Ok(json) => json,+ Err(e) => format!(r#"{{"error":"{}"}}"#, e.to_string().replace('"', "'")),+ }+ }++ #[tool(name = "onto_support_verdict", description = "Record whether a cited source supports a claim: supported, refuted, or unrelated. Verdicts persist, so onto_support_check skips what has already been judged and onto_support_report can summarise.")]+ async fn onto_support_verdict(&self, Parameters(input): Parameters<OntoSupportVerdictInput>) -> String {+ use crate::support::SupportChecker;+ let checker = SupportChecker::new(self.graph.clone(), self.db.clone());+ match checker.record_verdict(&input.claim_id, &input.verdict, input.note.as_deref()) {+ Ok(json) => json,+ Err(e) => format!(r#"{{"error":"{}"}}"#, e.to_string().replace('"', "'")),+ }+ }++ #[tool(name = "onto_support_report", description = "Summarise provenance quality: the unsourced-claim rate over the whole graph, and the support rate across the claims judged so far.")]+ async fn onto_support_report(&self, Parameters(input): Parameters<OntoSupportReportInput>) -> String {+ use crate::support::SupportChecker;+ let checker = SupportChecker::new(self.graph.clone(), self.db.clone());+ match checker.report(input.prov_predicate.as_deref()) {+ Ok(json) => json,+ Err(e) => format!(r#"{{"error":"{}"}}"#, e.to_string().replace('"', "'")),+ }+ }+#[tool(name = "onto_pack", description = "Write the loaded graph and its verification evidence to a portable, versioned pack: sorted N-Triples plus a manifest (name, version, counts, timestamp, sha256, and the lint/enforce results recorded at pack time). Use to promote a verified graph between environments as one auditable artifact.")]async fn onto_pack(&self, Parameters(input): Parameters<OntoPackInput>) -> String {use crate::pack::Packer;diff --git a/src/state.rs b/src/state.rsindex ed5b4b65..e17e19b6 100644--- a/src/state.rs+++ b/src/state.rs@@ -86,6 +86,13 @@ CREATE TABLE IF NOT EXISTS align_feedback (CREATE INDEX IF NOT EXISTS idx_align_feedback_iris ON align_feedback(source_iri, target_iri);+CREATE TABLE IF NOT EXISTS support_verdicts (+ claim_id TEXT PRIMARY KEY,+ verdict TEXT NOT NULL,+ note TEXT,+ timestamp TEXT NOT NULL DEFAULT (datetime('now'))+);+CREATE TABLE IF NOT EXISTS tool_feedback (id INTEGER PRIMARY KEY AUTOINCREMENT,tool TEXT NOT NULL,diff --git a/src/support.rs b/src/support.rsnew file mode 100644index 00000000..08a45c36--- /dev/null+++ b/src/support.rs@@ -0,0 +1,320 @@+//! Claim support: is a fact in the graph actually backed by the source it+//! cites, and does it cite one at all.+//!+//! Conformance and support are independent failures. `onto_vocab_check`+//! answers "is this expressible in the declared vocabulary", which a+//! perfectly-sourced fact can fail and a completely fabricated one can pass.+//! Support answers the other half: the graph says X and points at document D,+//! but does D say X? A graph can be flawless on both counts only if both are+//! checked, and most pipelines check neither.+//!+//! Two things happen here, split along the line the project always draws:+//!+//! - what is computable is computed. Which claims cite no source at all is+//! a query, not a judgement, and an unsourced-claim rate is a number you+//! can put in a report today;+//! - what needs reading is not guessed. For claims that do cite a source,+//! the engine emits a verification TASK (the claim in words, the source,+//! what to decide) and the connected model returns a verdict through+//! `record_verdict`. No model client lives in here.+//!+//! Inspired by the ProVe line of work on reference verification for+//! knowledge graphs (Amaral et al., KCL), generalised: any provenance+//! predicate, any source, verdicts supplied by whoever is connected.++use crate::graph::GraphStore;+use crate::state::StateDb;+use std::sync::Arc;++pub struct SupportChecker {+ graph: Arc<GraphStore>,+ db: StateDb,+}++#[derive(serde::Serialize)]+pub struct SupportTask {+ /// Stable id for the claim, so a verdict can be recorded against it.+ pub claim_id: String,+ pub subject: String,+ pub predicate: String,+ pub object: String,+ /// The claim as a sentence, so the judgement does not require reading RDF.+ pub claim: String,+ pub source: String,+ #[serde(skip_serializing_if = "Option::is_none")]+ pub source_label: Option<String>,+}++const INSTRUCTION: &str = concat!(+ "For each task, decide whether the cited source supports the claim: ",+ "'supported' if the source states it, 'refuted' if the source states ",+ "something incompatible with it, 'unrelated' if the source is silent on ",+ "it. Read the source; do not decide from the claim's plausibility, which ",+ "is exactly the failure this check exists to catch. Record each decision ",+ "with onto_support_verdict. Claims listed under unsourced cite nothing at ",+ "all and need a source before they can be judged."+);++const PROV_DERIVED: &str = "http://www.w3.org/ns/prov#wasDerivedFrom";++fn strip(value: &str) -> String {+ let v = value.trim();+ if v.starts_with('<') && v.ends_with('>') {+ return v[1..v.len() - 1].to_string();+ }+ if v.starts_with('"') {+ let body = &v[1..];+ for cut in ["\"^^", "\"@", "\""] {+ if let Some(i) = body.find(cut) {+ return body[..i].to_string();+ }+ }+ }+ v.to_string()+}++fn local(iri: &str) -> &str {+ iri.rsplit(['#', '/']).next().unwrap_or(iri)+}++/// `hasGrowthCondition` reads as "has growth condition" in a sentence a+/// person or a model can judge without knowing the vocabulary.+fn humanise(predicate: &str) -> String {+ let name = local(predicate);+ let mut out = String::new();+ for (i, ch) in name.chars().enumerate() {+ if ch.is_uppercase() && i > 0 {+ out.push(' ');+ out.extend(ch.to_lowercase());+ } else {+ out.push(ch);+ }+ }+ out.replace('_', " ").to_lowercase()+}++impl SupportChecker {+ pub fn new(graph: Arc<GraphStore>, db: StateDb) -> Self {+ Self { graph, db }+ }++ fn rows(&self, query: &str) -> anyhow::Result<Vec<serde_json::Value>> {+ let raw = self.graph.sparql_select(query)?;+ let parsed: serde_json::Value = serde_json::from_str(&raw)?;+ Ok(parsed+ .get("results")+ .and_then(|r| r.as_array())+ .cloned()+ .unwrap_or_default())+ }++ /// Build verification tasks, and report what cites nothing.+ pub fn check(&self, prov_predicate: Option<&str>, limit: usize) -> anyhow::Result<String> {+ let prov = prov_predicate.unwrap_or(PROV_DERIVED);++ // Subjects that cite a source, and what they cite.+ let sourced = self.rows(&format!(+ "SELECT ?s ?src ?srcLabel WHERE {{ ?s <{prov}> ?src . \+ OPTIONAL {{ ?src <http://www.w3.org/2000/01/rdf-schema#label> ?srcLabel }} }} LIMIT 5000"+ ))?;+ let mut source_of: std::collections::BTreeMap<String, (String, Option<String>)> =+ std::collections::BTreeMap::new();+ for row in &sourced {+ if let (Some(s), Some(src)) = (+ row.get("s").and_then(|v| v.as_str()),+ row.get("src").and_then(|v| v.as_str()),+ ) {+ source_of.entry(strip(s)).or_insert((+ strip(src),+ row.get("srcLabel").and_then(|v| v.as_str()).map(strip),+ ));+ }+ }++ // The claims themselves: assertions about named things, excluding+ // schema triples and the provenance links.+ let claims = self.rows(&format!(+ "SELECT ?s ?p ?o WHERE {{ ?s ?p ?o . \+ FILTER(isIRI(?s)) \+ FILTER(?p != <{prov}>) \+ FILTER(?p != <http://www.w3.org/2000/01/rdf-schema#label>) \+ FILTER(?p != <http://www.w3.org/2000/01/rdf-schema#subClassOf>) \+ FILTER(?p != <http://www.w3.org/2000/01/rdf-schema#domain>) \+ FILTER(?p != <http://www.w3.org/2000/01/rdf-schema#range>) \+ FILTER(?p != <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> \+ || (isIRI(?o) && ?o != <http://www.w3.org/2002/07/owl#Class> \+ && ?o != <http://www.w3.org/2002/07/owl#ObjectProperty> \+ && ?o != <http://www.w3.org/2002/07/owl#DatatypeProperty>)) \+ }} LIMIT 5000"+ ))?;++ let labels = self.labels()?;+ let show = |iri: &str| -> String {+ labels.get(iri).cloned().unwrap_or_else(|| local(iri).to_string())+ };++ let judged = self.judged_ids()?;+ let mut tasks = Vec::new();+ let mut unsourced = Vec::new();+ let mut total = 0usize;++ for row in &claims {+ let (s, p, o) = match (+ row.get("s").and_then(|v| v.as_str()),+ row.get("p").and_then(|v| v.as_str()),+ row.get("o").and_then(|v| v.as_str()),+ ) {+ (Some(s), Some(p), Some(o)) => (strip(s), strip(p), strip(o)),+ _ => continue,+ };+ total += 1;++ let claim_id = claim_id(&s, &p, &o);+ let object_text = if o.starts_with("http") { show(&o) } else { o.clone() };+ let sentence = format!("{} {} {}.", show(&s), humanise(&p), object_text);++ match source_of.get(&s) {+ None => {+ if unsourced.len() < limit {+ unsourced.push(serde_json::json!({+ "claim_id": claim_id,+ "claim": sentence,+ }));+ }+ }+ Some((src, src_label)) => {+ if judged.contains(&claim_id) || tasks.len() >= limit {+ continue;+ }+ tasks.push(SupportTask {+ claim_id,+ subject: s,+ predicate: p,+ object: o,+ claim: sentence,+ source: src.clone(),+ source_label: src_label.clone(),+ });+ }+ }+ }++ let unsourced_total = total.saturating_sub(+ claims+ .iter()+ .filter(|row| {+ row.get("s")+ .and_then(|v| v.as_str())+ .map(|s| source_of.contains_key(&strip(s)))+ .unwrap_or(false)+ })+ .count(),+ );++ Ok(serde_json::json!({+ "ok": true,+ "claims_total": total,+ "unsourced_total": unsourced_total,+ "unsourced_rate": if total > 0 {+ (unsourced_total as f64 / total as f64 * 1000.0).round() / 1000.0+ } else { 0.0 },+ "already_judged": judged.len(),+ "tasks": tasks,+ "unsourced": unsourced,+ "instruction": INSTRUCTION,+ })+ .to_string())+ }++ /// Record a verdict for one claim.+ pub fn record_verdict(+ &self,+ claim_id: &str,+ verdict: &str,+ note: Option<&str>,+ ) -> anyhow::Result<String> {+ let verdict = verdict.trim().to_lowercase();+ if !["supported", "refuted", "unrelated"].contains(&verdict.as_str()) {+ return Ok(serde_json::json!({+ "error": "verdict must be one of: supported, refuted, unrelated"+ })+ .to_string());+ }+ let conn = self.db.conn();+ conn.execute(+ "INSERT OR REPLACE INTO support_verdicts (claim_id, verdict, note, timestamp) \+ VALUES (?1, ?2, ?3, datetime('now'))",+ rusqlite::params![claim_id, verdict, note],+ )?;+ Ok(serde_json::json!({"ok": true, "claim_id": claim_id, "verdict": verdict}).to_string())+ }++ /// The two-axis picture: how much of the graph is sourced, and of the+ /// claims judged so far, how many the sources actually bear out.+ pub fn report(&self, prov_predicate: Option<&str>) -> anyhow::Result<String> {+ let check: serde_json::Value = serde_json::from_str(&self.check(prov_predicate, 0)?)?;+ let conn = self.db.conn();+ let mut stmt =+ conn.prepare("SELECT verdict, COUNT(*) FROM support_verdicts GROUP BY verdict")?;+ let mut counts = std::collections::BTreeMap::new();+ let rows = stmt.query_map([], |r| {+ Ok((r.get::<_, String>(0)?, r.get::<_, i64>(1)?))+ })?;+ for row in rows {+ let (verdict, n) = row?;+ counts.insert(verdict, n);+ }+ let judged: i64 = counts.values().sum();+ let supported = *counts.get("supported").unwrap_or(&0);++ Ok(serde_json::json!({+ "ok": true,+ "claims_total": check.get("claims_total"),+ "unsourced_total": check.get("unsourced_total"),+ "unsourced_rate": check.get("unsourced_rate"),+ "verdicts": counts,+ "judged": judged,+ "support_rate": if judged > 0 {+ (supported as f64 / judged as f64 * 1000.0).round() / 1000.0+ } else { 0.0 },+ "note": "unsourced_rate is computed from the graph. support_rate covers only \+ the claims judged so far, so read it alongside judged, not alone.",+ })+ .to_string())+ }++ fn judged_ids(&self) -> anyhow::Result<std::collections::BTreeSet<String>> {+ let conn = self.db.conn();+ let mut stmt = conn.prepare("SELECT claim_id FROM support_verdicts")?;+ let rows = stmt.query_map([], |r| r.get::<_, String>(0))?;+ Ok(rows.filter_map(|r| r.ok()).collect())+ }++ fn labels(&self) -> anyhow::Result<std::collections::HashMap<String, String>> {+ let rows = self.rows(+ "SELECT ?s ?l WHERE { ?s <http://www.w3.org/2000/01/rdf-schema#label> ?l } LIMIT 20000",+ )?;+ let mut out = std::collections::HashMap::new();+ for row in rows {+ if let (Some(s), Some(l)) = (+ row.get("s").and_then(|v| v.as_str()),+ row.get("l").and_then(|v| v.as_str()),+ ) {+ out.entry(strip(s)).or_insert_with(|| strip(l));+ }+ }+ Ok(out)+ }+}++/// A short, stable id for a triple, so verdicts survive reloads and reorders.+fn claim_id(s: &str, p: &str, o: &str) -> String {+ use sha2::{Digest, Sha256};+ let mut h = Sha256::new();+ h.update(s.as_bytes());+ h.update(b"\x1f");+ h.update(p.as_bytes());+ h.update(b"\x1f");+ h.update(o.as_bytes());+ format!("{:x}", h.finalize())[..16].to_string()+}