Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.
Rust ai-nativedescription-logicsformal-verificationinfrastructure-as-codeisabelle-languageknowledge-graphleanlean4mcpmcp-serverontologyowlowl2reasoningrustshaclsparqltheorem-proving
feat(align): experimental multilingual / cross-lingual alignment logic
FABIOTESS46fe0a63e93d
diff --git a/CLAUDE.md b/CLAUDE.mdindex e1047a70..8dc7294f 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -82,14 +82,14 @@ Claude dynamically decides the next tool call based on what the previous tool re| `onto_crosswalk` | To look up clinical terminology mappings (ICD-10 ↔ SNOMED ↔ MeSH) || `onto_enrich` | To add skos:exactMatch triples linking classes to clinical codes || `onto_validate_clinical` | To check class labels against clinical crosswalk terminology |-| `onto_align` | To detect alignment candidates (equivalentClass, exactMatch, subClassOf) between two ontologies using 7 weighted signals (6 structural + embedding similarity when embeddings are loaded) |+| `onto_align` | To detect alignment candidates (equivalentClass, exactMatch, subClassOf) between two ontologies using 7 weighted signals (6 structural + embedding similarity when embeddings are loaded). Labels are matched with their parsed BCP-47 language tag; with the multilingual embedder loaded, cross-lingual pairs that share no surface tokens (e.g. `Dog`↔`Chien`) are admitted via the embedding signal. Restrict the languages consulted with `[language] preferred = [...]` / `OPEN_ONTOLOGIES_LANGUAGES` (empty = all) || `onto_align_feedback` | To accept/reject alignment candidates for self-calibrating confidence weights || `onto_lineage` | To view the session's lineage trail (plan → enforce → apply → monitor → drift) || `onto_lint_feedback` | To accept/dismiss a lint issue — teaches lint to suppress repeatedly dismissed warnings || `onto_enforce_feedback` | To accept/dismiss an enforce violation — teaches enforce to suppress repeatedly dismissed violations || `onto_dl_explain` | To explain why a class is unsatisfiable using DL tableaux reasoning — returns clash trace || `onto_dl_check` | To check if one class is subsumed by another using DL tableaux reasoning |-| `onto_embed` | After loading an ontology — generates text + Poincaré structural embeddings for all classes. Honours `[embeddings] provider = "local" \| "openai"` in `config.toml`; OpenAI-compatible gateways (Azure, Ollama, vLLM, LocalAI, …) are supported via `OPEN_ONTOLOGIES_EMBEDDINGS_*` env vars |+| `onto_embed` | After loading an ontology — generates text + Poincaré structural embeddings for all classes. The default local model is **multilingual** (`paraphrase-multilingual-MiniLM-L12-v2`), so labels in different natural languages embed into a shared space. Honours `[embeddings] provider = "local" \| "openai"` in `config.toml`; OpenAI-compatible gateways (Azure, Ollama, vLLM, LocalAI, …) are supported via `OPEN_ONTOLOGIES_EMBEDDINGS_*` env vars || `onto_search` | To find classes by natural language description — requires onto_embed first || `onto_similarity` | To compute embedding similarity between two specific IRIs || `onto_unload` | To unload the active ontology from memory. Optional `name` targets a specific cached entry; `delete_cache=true` also removes the on-disk N-Triples cache file |@@ -183,6 +183,26 @@ When exploring or aligning ontologies using semantic embeddings:7. When running `onto_align`, embedding similarity is automatically used as signal #7 if embeddings are loaded8. This catches semantically equivalent classes that have different labels (e.g., Vehicle ↔ Automobile)+### Cross-Lingual Alignment++The default `onto_embed` model is multilingual, so labels in different natural+languages embed into a shared vector space. `onto_align` parses the BCP-47+language tag off each label and, when label similarity is near zero (as it is+across languages — `Dog` vs `Chien` share no tokens), lets a strong embedding+match bypass the label pre-filter so the pair is still scored. Such pairs+typically surface as `borderline` candidates with their language-tagged labels+in the `context` block, for review via `onto_align_feedback`.++Control which languages are consulted with the `[language]` config section:++```toml+[language]+# Empty (default) = keep ALL languages — multilingual matching.+# Restrict to a set to pin matching to specific languages (untagged labels are+# always kept). Override with OPEN_ONTOLOGIES_LANGUAGES=en,fr+preferred = []+```+### Borderline-Candidate Review (LLM-as-Oracle Pattern)`onto_align` partitions candidates into three buckets by confidence:diff --git a/src/align.rs b/src/align.rsindex e419a526..5b51840a 100644--- a/src/align.rs+++ b/src/align.rs@@ -42,7 +42,11 @@ impl AlignmentEngine {}}- fn extract_classes(store: &GraphStore) -> Vec<ClassInfo> {+ /// Extract class IRIs and their labels, parsing the BCP-47 language tag off+ /// each literal and retaining only labels permitted by `preferred` (empty =+ /// keep all languages — multilingual mode). Untagged literals are always+ /// kept.+ fn extract_classes(store: &GraphStore, preferred: &[String]) -> Vec<ClassInfo> {let query = r#"SELECT ?class ?label ?altLabel WHERE {?class a <http://www.w3.org/2002/07/owl#Class> .@@ -85,25 +89,26 @@ impl AlignmentEngine {labels: Vec::new(),});- if let Some(label) = row["label"].as_str() {- let l = label.trim_matches('"').to_string();- if !entry.labels.contains(&l) {- entry.labels.push(l);- }- }- if let Some(alt) = row["altLabel"].as_str() {- let a = alt.trim_matches('"').to_string();- if !entry.labels.contains(&a) {- entry.labels.push(a);+ for key in ["label", "altLabel"] {+ if let Some(raw) = row[key].as_str() {+ let label = crate::language::parse_literal(raw);+ if !label.text.is_empty()+ && crate::language::label_matches_policy(&label, preferred)+ && !entry.labels.contains(&label)+ {+ entry.labels.push(label);+ }}}}}- // If no label found, use IRI local name+ // If no policy-matching label was found, fall back to the IRI local+ // name as a language-neutral label.for info in class_map.values_mut() {if info.labels.is_empty() {- info.labels.push(local_name(&info.iri));+ info.labels+ .push(crate::language::Label::new(local_name(&info.iri), None));}}@@ -271,8 +276,8 @@ impl AlignmentEngine {let mut best = 0.0f64;for la in &a.labels {for lb in &b.labels {- let na = normalize_label(la);- let nb = normalize_label(lb);+ let na = normalize_label(&la.text);+ let nb = normalize_label(&lb.text);// Jaro-Winkler on full normalized stringslet jw = jaro_winkler(&na, &nb);@@ -301,6 +306,13 @@ impl AlignmentEngine {#[cfg(feature = "embeddings")]const DEFAULT_WEIGHTS: [f64; 7] = [0.20, 0.15, 0.12, 0.12, 0.12, 0.09, 0.20];+ /// Minimum embedding cosine for a pair to bypass the label-similarity+ /// pre-filter. This is the gate that admits cross-lingual matches+ /// (e.g. `Dog`↔`Chien`), which share no surface tokens and so score ~0 on+ /// label similarity but high on a multilingual embedding model.+ #[cfg(feature = "embeddings")]+ const CROSS_LINGUAL_EMBED_MIN: f64 = 0.80;+#[cfg(feature = "embeddings")]/// Compute embedding similarity score using cosine similarity on text vectors.pub fn embedding_similarity_score(vec_a: &[f32], vec_b: &[f32]) -> f64 {@@ -360,6 +372,8 @@ impl AlignmentEngine {) -> anyhow::Result<String> {// Clamp: degenerate range means "everything above high_threshold; nothing borderline".let low_threshold = low_threshold.min(high_threshold);+ // Active language policy (empty = keep all languages / multilingual).+ let preferred = crate::runtime::preferred_languages();// For RRF the inline weighted-sum confidence is throwaway (recomputed after// collection). We collect all label-prefiltered candidates and only apply// `low_threshold` to the FINAL fused score post-rerank.@@ -372,7 +386,7 @@ impl AlignmentEngine {let format = Self::detect_content_format(source);source_store.load_content(source, format)?;}- let source_classes = Self::extract_classes(&source_store);+ let source_classes = Self::extract_classes(&source_store, &preferred);// Load target into a temporary graph (or use the main store)let target_store_owned;@@ -389,7 +403,7 @@ impl AlignmentEngine {} else {target_ref = &*self.graph;}- let target_classes = Self::extract_classes(target_ref);+ let target_classes = Self::extract_classes(target_ref, &preferred);// Get learned weights (or defaults)let weights = self.get_learned_weights();@@ -461,19 +475,9 @@ impl AlignmentEngine {let label_sim = Self::label_similarity(sc, tc);- // Pre-filter: skip pairs where label similarity is too low.- // Raised from 0.7 to 0.75 to reduce false positives on anatomy-style- // ontologies where many terms share token overlap (e.g., "bone" variants).- if label_sim < 0.75 {- continue;- }-- let prop_overlap = Self::property_overlap(&source_store, &sc.iri, target_ref, &tc.iri);- let parent_ovlp = Self::parent_overlap(&source_store, &sc.iri, target_ref, &tc.iri);- let inst_overlap = Self::instance_overlap(&source_store, &sc.iri, target_ref, &tc.iri);- let restr_sim = Self::restriction_similarity(&source_store, &sc.iri, target_ref, &tc.iri);- let neigh_sim = Self::neighborhood_similarity(&source_store, &sc.iri, target_ref, &tc.iri);-+ // Embedding similarity is computed BEFORE the label pre-filter so+ // it can vouch for cross-lingual pairs that share no surface+ // tokens (label_sim ~ 0) but embed close in a multilingual model.#[cfg(feature = "embeddings")]let embedding_sim = {if let Some(ref vs) = self.vecstore {@@ -487,6 +491,26 @@ impl AlignmentEngine {}};+ // Pre-filter: skip pairs where label similarity is too low.+ // Raised from 0.7 to 0.75 to reduce false positives on anatomy-style+ // ontologies where many terms share token overlap (e.g., "bone" variants).+ // With embeddings, a strong embedding match bypasses the gate so+ // cross-lingual equivalences are not discarded before scoring.+ #[cfg(feature = "embeddings")]+ if label_sim < 0.75 && embedding_sim < Self::CROSS_LINGUAL_EMBED_MIN {+ continue;+ }+ #[cfg(not(feature = "embeddings"))]+ if label_sim < 0.75 {+ continue;+ }++ let prop_overlap = Self::property_overlap(&source_store, &sc.iri, target_ref, &tc.iri);+ let parent_ovlp = Self::parent_overlap(&source_store, &sc.iri, target_ref, &tc.iri);+ let inst_overlap = Self::instance_overlap(&source_store, &sc.iri, target_ref, &tc.iri);+ let restr_sim = Self::restriction_similarity(&source_store, &sc.iri, target_ref, &tc.iri);+ let neigh_sim = Self::neighborhood_similarity(&source_store, &sc.iri, target_ref, &tc.iri);+#[cfg(feature = "embeddings")]let signals = [label_sim, prop_overlap, parent_ovlp, inst_overlap, restr_sim, neigh_sim, embedding_sim];@@ -494,13 +518,21 @@ impl AlignmentEngine {let signals = [label_sim, prop_overlap, parent_ovlp, inst_overlap, restr_sim, neigh_sim];// Compute confidence. When structural signals are all zero- // (common in lightweight OWL files), use label similarity with a- // penalty rather than the weighted sum (which would be ~0.25 * label_sim- // and too low to pass any threshold).+ // (common in lightweight OWL files), fall back to the strongest+ // lexical/semantic signal with a penalty rather than the weighted+ // sum (which would be ~0.25 * label_sim and too low to pass any+ // threshold).let structural_sum: f64 = signals[1..6].iter().sum();let confidence: f64 = if structural_sum == 0.0 {- // No structural evidence: use label_sim but apply 15% penalty- label_sim * 0.85+ // No structural evidence: apply a 15% penalty to the best+ // available signal. With embeddings this lets a strong+ // cross-lingual embedding match still surface (typically as a+ // borderline candidate for review).+ #[cfg(feature = "embeddings")]+ let best_signal = label_sim.max(embedding_sim);+ #[cfg(not(feature = "embeddings"))]+ let best_signal = label_sim;+ best_signal * 0.85} else {signals.iter().zip(weights.iter()).map(|(s, w)| s * w).sum()};@@ -612,11 +644,11 @@ impl AlignmentEngine {let t_iri = candidate["target_iri"].as_str().unwrap_or("").to_string();let s_labels: Vec<String> = source_class_map.get(s_iri.as_str())- .map(|c| c.labels.clone())+ .map(|c| c.labels.iter().map(|l| l.tagged()).collect()).unwrap_or_default();let t_labels: Vec<String> = target_class_map.get(t_iri.as_str())- .map(|c| c.labels.clone())+ .map(|c| c.labels.iter().map(|l| l.tagged()).collect()).unwrap_or_default();let s_parents = Self::extract_parents(&source_store, &s_iri);let t_parents = Self::extract_parents(target_ref, &t_iri);@@ -925,7 +957,11 @@ fn jaccard_similarity(a: &[String], b: &[String]) -> f64 {#[derive(Debug, Clone)]pub struct ClassInfo {pub iri: String,- pub labels: Vec<String>,+ /// Labels with their parsed language tags. Populated from rdfs:label /+ /// skos:prefLabel / skos:altLabel (and OBO synonyms), filtered by the+ /// active language policy. Falls back to a single language-neutral label+ /// derived from the IRI local name when no policy-matching label exists.+ pub labels: Vec<crate::language::Label>,}/// Extract local name from an IRI (after last # or /).@@ -975,11 +1011,14 @@ mod tests {fn test_label_similarity() {let a = ClassInfo {iri: "http://ex.org/Dog".into(),- labels: vec!["Dog".into()],+ labels: vec![crate::language::Label::new("Dog", None)],};let b = ClassInfo {iri: "http://other.org/Canine".into(),- labels: vec!["Dog".into(), "Canine".into()],+ labels: vec![+ crate::language::Label::new("Dog", None),+ crate::language::Label::new("Canine", None),+ ],};// Exact label match should give 1.0let sim = AlignmentEngine::label_similarity(&a, &b);@@ -1117,16 +1156,49 @@ mod tests {fn test_label_similarity_camelcase() {let a = ClassInfo {iri: "http://ex.org/DomesticCat".into(),- labels: vec!["DomesticCat".into()],+ labels: vec![crate::language::Label::new("DomesticCat", None)],};let b = ClassInfo {iri: "http://other.org/HouseCat".into(),- labels: vec!["Domestic Cat".into()],+ labels: vec![crate::language::Label::new("Domestic Cat", None)],};let sim = AlignmentEngine::label_similarity(&a, &b);assert!(sim > 0.95, "CamelCase split should match: {}", sim);}+ #[test]+ fn extract_classes_parses_language_tags_and_policy() {+ let gs = GraphStore::new();+ gs.load_content(+ r#"+ @prefix owl: <http://www.w3.org/2002/07/owl#> .+ @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .+ @prefix ex: <http://example.org/> .+ ex:Dog a owl:Class ; rdfs:label "Dog"@en, "Chien"@fr, "Perro"@es .+ "#,+ RdfFormat::Turtle,+ )+ .unwrap();++ // Multilingual policy (empty) keeps all three, each with its tag parsed+ // off — no more `Dog"@en` mangling.+ let all = AlignmentEngine::extract_classes(&gs, &[]);+ let dog = all.iter().find(|c| c.iri.ends_with("Dog")).unwrap();+ let texts: std::collections::HashSet<&str> =+ dog.labels.iter().map(|l| l.text.as_str()).collect();+ assert!(texts.contains("Dog"));+ assert!(texts.contains("Chien"));+ assert!(texts.contains("Perro"));+ let chien = dog.labels.iter().find(|l| l.text == "Chien").unwrap();+ assert_eq!(chien.lang.as_deref(), Some("fr"));++ // English-only policy drops the fr/es labels.+ let en = AlignmentEngine::extract_classes(&gs, &["en".to_string()]);+ let dog_en = en.iter().find(|c| c.iri.ends_with("Dog")).unwrap();+ let texts_en: Vec<&str> = dog_en.labels.iter().map(|l| l.text.as_str()).collect();+ assert_eq!(texts_en, vec!["Dog"]);+ }+#[test]fn test_align_feedback() {let tmp = tempfile::NamedTempFile::new().unwrap();diff --git a/src/config.rs b/src/config.rsindex 74ada7f7..0296ec5c 100644--- a/src/config.rs+++ b/src/config.rs@@ -8,6 +8,7 @@ use std::path::Path;pub struct Config {pub general: GeneralConfig,pub embeddings: EmbeddingsConfig,+ pub language: LanguageConfig,pub cache: CacheConfig,pub tools: ToolsConfig,pub webhook: WebhookConfig,@@ -104,16 +105,17 @@ pub struct EmbeddingsConfig {/// `OPEN_ONTOLOGIES_EMBEDDINGS_PROVIDER`.pub provider: Option<String>,/// Path to the ONNX model file (provider = "local" only).- /// Default: ~/.open-ontologies/models/bge-small-en-v1.5.onnx+ /// Default: ~/.open-ontologies/models/paraphrase-multilingual-MiniLM-L12-v2.onnx+ /// (falls back to a legacy bge-small-en-v1.5.onnx if that is what is on disk).pub model_path: Option<String>,/// Path to the tokenizer.json file (provider = "local" only)./// Default: ~/.open-ontologies/models/tokenizer.jsonpub tokenizer_path: Option<String>,- /// URL to download the ONNX model from. Default: BGE-small-en-v1.5 from Hugging Face+ /// URL to download the ONNX model from. Default: multilingual MiniLM-L12-v2 from Hugging Facepub model_url: Option<String>,- /// URL to download the tokenizer from. Default: BGE-small-en-v1.5 tokenizer from Hugging Face+ /// URL to download the tokenizer from. Default: multilingual MiniLM-L12-v2 tokenizer from Hugging Facepub tokenizer_url: Option<String>,- /// Filename for the downloaded model. Default: bge-small-en-v1.5.onnx+ /// Filename for the downloaded model. Default: paraphrase-multilingual-MiniLM-L12-v2.onnxpub model_name: Option<String>,// ─── OpenAI-compatible provider (provider = "openai") ───────────────@@ -198,6 +200,50 @@ pub fn resolve_embeddings_model(cfg: &EmbeddingsConfig) -> String {.unwrap_or_else(|| "text-embedding-3-small".to_string())}+/// `[language]` — natural-language policy for ontology labels used in+/// alignment / matching.+#[derive(Debug, Default, Deserialize, Clone)]+#[serde(default)]+pub struct LanguageConfig {+ /// Preferred natural-language tags (BCP-47, e.g. `["en", "cy"]`) for the+ /// `rdfs:label` / `skos:prefLabel` / `skos:altLabel` values consulted by+ /// `onto_align`. Untagged (plain or datatyped) literals are always kept.+ ///+ /// An empty list (the default) keeps **all** languages — i.e. fully+ /// multilingual matching. This only pays off when paired with a+ /// multilingual embedding model (the default `onto_embed` model is+ /// multilingual), because cross-lingual matches (`Dog`↔`Chien`) are carried+ /// by the embedding signal, not by string similarity.+ ///+ /// Override at runtime with `OPEN_ONTOLOGIES_LANGUAGES` (comma / colon /+ /// semicolon separated, e.g. `OPEN_ONTOLOGIES_LANGUAGES=en,fr`).+ #[serde(alias = "preferred_languages")]+ pub preferred: Vec<String>,+}++/// Resolve the preferred-language policy.+///+/// Precedence: `OPEN_ONTOLOGIES_LANGUAGES` env var > config field. Entries are+/// trimmed, lowercased, and emptied entries dropped. An empty result means+/// "keep all languages" (multilingual mode).+pub fn resolve_languages(cfg: &LanguageConfig) -> Vec<String> {+ let from_env = std::env::var("OPEN_ONTOLOGIES_LANGUAGES").ok();+ let raw: Vec<String> = match from_env {+ Some(v) if !v.trim().is_empty() => v+ .split([',', ':', ';'])+ .map(|s| s.trim().to_string())+ .filter(|s| !s.is_empty())+ .collect(),+ _ => cfg+ .preferred+ .iter()+ .map(|s| s.trim().to_string())+ .filter(|s| !s.is_empty())+ .collect(),+ };+ raw.into_iter().map(|s| s.to_lowercase()).collect()+}+#[derive(Debug, Deserialize, Clone)]#[serde(default)]pub struct CacheConfig {@@ -576,6 +622,31 @@ mod tests {assert_eq!(resolved, "local");}+ #[test]+ fn language_section_parses_and_defaults_to_multilingual() {+ // Empty / absent => keep all languages.+ let cfg: Config = toml::from_str("").expect("parse empty");+ assert!(cfg.language.preferred.is_empty());++ let toml_src = r#"+ [language]+ preferred = ["en", "CY"]+ "#;+ let cfg: Config = toml::from_str(toml_src).expect("parse");+ assert_eq!(cfg.language.preferred, vec!["en", "CY"]);+ // Resolver lowercases (env var unset in this path).+ if std::env::var("OPEN_ONTOLOGIES_LANGUAGES").is_err() {+ assert_eq!(resolve_languages(&cfg.language), vec!["en", "cy"]);+ }++ // Alias `preferred_languages` also populates the field.+ let aliased: Config = toml::from_str(+ "[language]\npreferred_languages = [\"fr\"]\n",+ )+ .expect("parse alias");+ assert_eq!(aliased.language.preferred, vec!["fr"]);+ }+#[test]fn base_url_alias_accepted() {// The legacy/alternative `base_url` key should also populatediff --git a/src/embed.rs b/src/embed.rsindex 7c47a207..660939b9 100644--- a/src/embed.rs+++ b/src/embed.rs@@ -8,11 +8,31 @@ use tract_onnx::prelude::*;use crate::poincare::l2_normalize;-/// Model download URLs for bge-small-en-v1.5-pub const BGE_SMALL_ONNX_URL: &str =- "https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx";-pub const BGE_SMALL_TOKENIZER_URL: &str =- "https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json";+/// Default embedding model: multilingual MiniLM (384-dim, BERT-style with+/// `token_type_ids`, mean pooling — a drop-in match for this loader's input+/// signature and pooling). Replaces the previous English-only+/// `bge-small-en-v1.5` so that labels in different natural languages embed into+/// a *shared* vector space, which is what makes cross-lingual alignment+/// (`Dog` ↔ `Chien` ↔ `Perro`) possible via the embedding signal in+/// `onto_align`. Override with `[embeddings] model_url/tokenizer_url/model_name`+/// or by switching to the OpenAI provider. The `Xenova/*` repo ships the+/// `onnx/model.onnx` + `tokenizer.json` layout this loader expects.+pub const DEFAULT_MODEL_ONNX_URL: &str =+ "https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/onnx/model.onnx";+pub const DEFAULT_MODEL_TOKENIZER_URL: &str =+ "https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/tokenizer.json";+/// On-disk filename for the default downloaded model.+pub const DEFAULT_MODEL_FILENAME: &str = "paraphrase-multilingual-MiniLM-L12-v2.onnx";+/// Legacy English-only model filename. Still loaded as a fallback when present+/// so existing installs that downloaded it before the multilingual switch keep+/// working (with English-only embeddings) until they re-run `init`.+pub const LEGACY_EN_MODEL_FILENAME: &str = "bge-small-en-v1.5.onnx";++// Back-compat aliases for the previous public constant names.+#[deprecated(note = "use DEFAULT_MODEL_ONNX_URL")]+pub const BGE_SMALL_ONNX_URL: &str = DEFAULT_MODEL_ONNX_URL;+#[deprecated(note = "use DEFAULT_MODEL_TOKENIZER_URL")]+pub const BGE_SMALL_TOKENIZER_URL: &str = DEFAULT_MODEL_TOKENIZER_URL;pub struct TextEmbedder {#[allow(clippy::type_complexity)]@@ -154,9 +174,17 @@ impl TextEmbedderProvider {.clone().map(|p| std::path::PathBuf::from(crate::config::expand_tilde(&p))).or_else(|| {- default_model_dir- .as_ref()- .map(|d| d.join("bge-small-en-v1.5.onnx"))+ default_model_dir.as_ref().map(|d| {+ // Prefer the multilingual default; fall back to a+ // legacy English model only if it is the one present+ // on disk (older installs).+ let preferred = d.join(DEFAULT_MODEL_FILENAME);+ if preferred.exists() {+ return preferred;+ }+ let legacy = d.join(LEGACY_EN_MODEL_FILENAME);+ if legacy.exists() { legacy } else { preferred }+ })});let tokenizer_path = cfgdiff --git a/src/language.rs b/src/language.rsnew file mode 100644index 00000000..0a228efc--- /dev/null+++ b/src/language.rs@@ -0,0 +1,203 @@+//! Natural-language handling for ontology labels.+//!+//! RDF labels carry an optional BCP-47 language tag (`"Dog"@en`). Before this+//! module, the alignment engine matched the raw N-Triples string returned by+//! `Term::to_string()`, which (a) silently mangled tagged literals+//! (`"Dog"@en` survived `trim_matches('"')` as `Dog"@en`) and (b) gave callers+//! no way to either pin matching to a preferred language or opt into fully+//! multilingual matching.+//!+//! This module parses the language tag off the literal and provides a small,+//! configurable policy so callers can keep all languages (default — pairs with+//! a multilingual embedding model for cross-lingual alignment) or restrict to a+//! preferred set. Untagged / datatyped literals are always treated as+//! language-neutral and retained regardless of policy.++/// A label with its optional (lowercased) BCP-47 language tag.+///+/// `lang == None` means a plain or datatyped literal — language-neutral, always+/// retained under any policy.+#[derive(Debug, Clone, PartialEq, Eq, Hash)]+pub struct Label {+ pub text: String,+ pub lang: Option<String>,+}++impl Label {+ pub fn new(text: impl Into<String>, lang: Option<String>) -> Self {+ Self {+ text: text.into(),+ lang: lang.map(|l| l.trim().to_lowercase()).filter(|l| !l.is_empty()),+ }+ }++ /// Human/LLM-readable form: `"text@lang"` when tagged, else just `"text"`.+ /// Used to surface the language of borderline alignment candidates so the+ /// reviewing orchestrator can judge cross-lingual pairs.+ pub fn tagged(&self) -> String {+ match &self.lang {+ Some(lang) => format!("{}@{}", self.text, lang),+ None => self.text.clone(),+ }+ }+}++/// Parse an oxigraph `Term::to_string()` literal into its lexical value and+/// language tag.+///+/// Handles the N-Triples literal forms:+/// - `"value"@en` → `(value, Some("en"))`+/// - `"value"^^<datatype>` → `(value, None)`+/// - `"value"` → `(value, None)`+/// - bare (IRI / local name) → `(raw, None)`+///+/// The closing quote is located as the last `"` in the string, which is correct+/// for well-formed literals because neither the `@lang` suffix nor the+/// `^^<datatype>` suffix contains a double quote. Common escapes inside the+/// lexical value (`\"`, `\\`, `\n`, `\t`, `\r`) are unescaped.+pub fn parse_literal(raw: &str) -> Label {+ let s = raw.trim();+ if !s.starts_with('"') {+ // Bare IRI or unquoted local name — no language tag.+ return Label::new(s.to_string(), None);+ }+ if let Some(close) = s.rfind('"') {+ if close > 0 {+ let value = unescape(&s[1..close]);+ let suffix = &s[close + 1..];+ let lang = suffix.strip_prefix('@').map(|l| l.to_string());+ return Label::new(value, lang);+ }+ }+ // Degenerate input (e.g. a lone `"`). Fall back to a best-effort strip.+ Label::new(s.trim_matches('"').to_string(), None)+}++/// Unescape the common N-Triples/Turtle string escapes that appear inside a+/// literal's lexical form. Anything else is passed through verbatim.+fn unescape(s: &str) -> String {+ if !s.contains('\\') {+ return s.to_string();+ }+ let mut out = String::with_capacity(s.len());+ let mut chars = s.chars();+ while let Some(c) = chars.next() {+ if c == '\\' {+ match chars.next() {+ Some('"') => out.push('"'),+ Some('\\') => out.push('\\'),+ Some('n') => out.push('\n'),+ Some('t') => out.push('\t'),+ Some('r') => out.push('\r'),+ Some(other) => {+ out.push('\\');+ out.push(other);+ }+ None => out.push('\\'),+ }+ } else {+ out.push(c);+ }+ }+ out+}++/// Should this label be retained under the given preferred-language policy?+///+/// An empty `preferred` list means "keep all languages" (multilingual mode).+/// Untagged labels (`lang == None`) are always kept. Otherwise the label's tag+/// must be present in `preferred`. Comparison is case-insensitive on the prefix+/// so that `"en"` matches `"en-GB"` / `"en-us"` etc.+pub fn label_matches_policy(label: &Label, preferred: &[String]) -> bool {+ if preferred.is_empty() {+ return true;+ }+ match &label.lang {+ None => true,+ Some(lang) => preferred.iter().any(|p| {+ let p = p.to_lowercase();+ lang == &p || lang.starts_with(&format!("{p}-"))+ }),+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn parse_tagged_literal() {+ let l = parse_literal("\"Dog\"@en");+ assert_eq!(l.text, "Dog");+ assert_eq!(l.lang.as_deref(), Some("en"));+ }++ #[test]+ fn parse_region_tag_is_lowercased() {+ let l = parse_literal("\"Colour\"@en-GB");+ assert_eq!(l.text, "Colour");+ assert_eq!(l.lang.as_deref(), Some("en-gb"));+ }++ #[test]+ fn parse_plain_literal() {+ let l = parse_literal("\"Dog\"");+ assert_eq!(l.text, "Dog");+ assert_eq!(l.lang, None);+ }++ #[test]+ fn parse_typed_literal_has_no_lang() {+ let l = parse_literal("\"42\"^^<http://www.w3.org/2001/XMLSchema#integer>");+ assert_eq!(l.text, "42");+ assert_eq!(l.lang, None);+ }++ #[test]+ fn parse_non_english_value() {+ assert_eq!(parse_literal("\"Chien\"@fr").text, "Chien");+ assert_eq!(parse_literal("\"Hund\"@de").lang.as_deref(), Some("de"));+ // Non-Latin scripts survive intact.+ assert_eq!(parse_literal("\"犬\"@ja").text, "犬");+ }++ #[test]+ fn parse_bare_value() {+ let l = parse_literal("Dog");+ assert_eq!(l.text, "Dog");+ assert_eq!(l.lang, None);+ }++ #[test]+ fn unescape_quotes_and_backslashes() {+ let l = parse_literal("\"say \\\"hi\\\"\"@en");+ assert_eq!(l.text, "say \"hi\"");+ assert_eq!(l.lang.as_deref(), Some("en"));+ }++ #[test]+ fn policy_empty_keeps_all() {+ assert!(label_matches_policy(&Label::new("Chien", Some("fr".into())), &[]));+ assert!(label_matches_policy(&Label::new("Dog", None), &[]));+ }++ #[test]+ fn policy_untagged_always_kept() {+ let pref = vec!["en".to_string()];+ assert!(label_matches_policy(&Label::new("Dog", None), &pref));+ }++ #[test]+ fn policy_filters_by_preferred() {+ let pref = vec!["en".to_string(), "cy".to_string()];+ assert!(label_matches_policy(&Label::new("Dog", Some("en".into())), &pref));+ assert!(label_matches_policy(&Label::new("Ci", Some("cy".into())), &pref));+ assert!(!label_matches_policy(&Label::new("Chien", Some("fr".into())), &pref));+ }++ #[test]+ fn policy_region_subtag_matches_base() {+ let pref = vec!["en".to_string()];+ assert!(label_matches_policy(&Label::new("Colour", Some("en-gb".into())), &pref));+ }+}diff --git a/src/lib.rs b/src/lib.rsindex 7668a484..6ee09426 100644--- a/src/lib.rs+++ b/src/lib.rs@@ -35,6 +35,7 @@ pub mod hnsw_index;pub mod ingest;pub mod inputs;pub mod kgcl;+pub mod language;pub mod lineage;pub mod mapping;pub mod marketplace;diff --git a/src/main.rs b/src/main.rsindex 02763b35..cb6ac222 100644--- a/src/main.rs+++ b/src/main.rs@@ -52,15 +52,26 @@ data_dir = "~/.open-ontologies"## ── Local provider (provider = "local", default) ────────────────────────# Paths to a local ONNX model and tokenizer (loaded at runtime).-# model_path = "~/.open-ontologies/models/bge-small-en-v1.5.onnx"+# Default is the multilingual MiniLM model (see model_url below); labels in+# different natural languages embed into a shared space for cross-lingual+# alignment.+# model_path = "~/.open-ontologies/models/paraphrase-multilingual-MiniLM-L12-v2.onnx"# tokenizer_path = "~/.open-ontologies/models/tokenizer.json"## URLs used by `open-ontologies init` to download the model.-# Override these to use a different sentence-transformer model (e.g. for non-English text).+# Override these to use a different sentence-transformer model. To go back to+# the smaller English-only model set these to BAAI/bge-small-en-v1.5.# The model must be exported to ONNX and use a Hugging Face tokenizer.json.-# model_url = "https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx"-# tokenizer_url = "https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json"-# model_name = "bge-small-en-v1.5.onnx"+# model_url = "https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/onnx/model.onnx"+# tokenizer_url = "https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/tokenizer.json"+# model_name = "paraphrase-multilingual-MiniLM-L12-v2.onnx"+#+# [language]+# Preferred natural-language tags for the labels onto_align consults. Untagged+# literals are always kept. Empty (default) = keep ALL languages (multilingual+# matching, paired with the multilingual embedding model above). Override with+# OPEN_ONTOLOGIES_LANGUAGES=en,fr+# preferred = []## ── OpenAI-compatible provider (provider = "openai") ────────────────────# Works with the official OpenAI API, Azure OpenAI, Ollama, vLLM, LocalAI,@@ -157,13 +168,13 @@ enum Commands {Init {#[arg(long, default_value = "~/.open-ontologies")]data_dir: String,- /// Custom ONNX model URL (default: BGE-small-en-v1.5 from Hugging Face)+ /// Custom ONNX model URL (default: multilingual MiniLM-L12-v2 from Hugging Face)#[arg(long)]model_url: Option<String>,- /// Custom tokenizer URL (default: BGE-small-en-v1.5 tokenizer from Hugging Face)+ /// Custom tokenizer URL (default: multilingual MiniLM-L12-v2 tokenizer from Hugging Face)#[arg(long)]tokenizer_url: Option<String>,- /// Filename for the downloaded ONNX model (default: bge-small-en-v1.5.onnx)+ /// Filename for the downloaded ONNX model (default: paraphrase-multilingual-MiniLM-L12-v2.onnx)#[arg(long)]model_name: Option<String>,},@@ -754,15 +765,15 @@ async fn async_main() -> anyhow::Result<()> {let onnx_url = _model_url.as_deref().or(cfg.model_url.as_deref())- .unwrap_or(open_ontologies::embed::BGE_SMALL_ONNX_URL);+ .unwrap_or(open_ontologies::embed::DEFAULT_MODEL_ONNX_URL);let tok_url = _tokenizer_url.as_deref().or(cfg.tokenizer_url.as_deref())- .unwrap_or(open_ontologies::embed::BGE_SMALL_TOKENIZER_URL);+ .unwrap_or(open_ontologies::embed::DEFAULT_MODEL_TOKENIZER_URL);let onnx_filename = _model_name.as_deref().or(cfg.model_name.as_deref())- .unwrap_or("bge-small-en-v1.5.onnx");+ .unwrap_or(open_ontologies::embed::DEFAULT_MODEL_FILENAME);let model_path = models_dir.join(onnx_filename);let tokenizer_path = models_dir.join("tokenizer.json");diff --git a/src/runtime.rs b/src/runtime.rsindex 55cdd22e..d876acca 100644--- a/src/runtime.rs+++ b/src/runtime.rs@@ -13,9 +13,11 @@//! observe the legacy behaviour.use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU64, AtomicUsize, Ordering};+use std::sync::RwLock;use crate::config::{- self, Config, FeedbackConfig, ImportsConfig, ReasonerConfig, RepoConfig, WebhookConfig,+ self, Config, FeedbackConfig, ImportsConfig, LanguageConfig, ReasonerConfig, RepoConfig,+ WebhookConfig,};// ── Defaults match the previous hardcoded constants exactly ─────────────@@ -45,6 +47,10 @@ static IMPORTS_MAX_DEPTH: AtomicUsize = AtomicUsize::new(DEFAULT_IMPORTS_MAX_DEPstatic IMPORTS_TIMEOUT: AtomicU64 = AtomicU64::new(DEFAULT_IMPORTS_TIMEOUT);static IMPORTS_FOLLOW_REMOTE: AtomicBool = AtomicBool::new(true);static WEBHOOK_TIMEOUT: AtomicU64 = AtomicU64::new(DEFAULT_WEBHOOK_TIMEOUT);+/// Preferred natural-language tags for label matching. Empty (the default)+/// means "keep all languages" — fully multilingual. Populated from+/// `[language] preferred` / `OPEN_ONTOLOGIES_LANGUAGES` at startup.+static PREFERRED_LANGUAGES: RwLock<Vec<String>> = RwLock::new(Vec::new());/// Initialise all runtime knobs from a loaded `Config`. Idempotent — calling/// this multiple times simply overwrites the current values, which is fine@@ -56,6 +62,14 @@ pub fn init_from_config(cfg: &Config) {apply_repo(&cfg.repo);apply_imports(&cfg.imports);apply_webhook(&cfg.webhook);+ apply_language(&cfg.language);+}++fn apply_language(l: &LanguageConfig) {+ let resolved = config::resolve_languages(l);+ if let Ok(mut guard) = PREFERRED_LANGUAGES.write() {+ *guard = resolved;+ }}fn apply_reasoner(r: &ReasonerConfig) {@@ -107,6 +121,16 @@ pub fn imports_request_timeout_secs() -> u64 { IMPORTS_TIMEOUT.load(Ordering::Repub fn imports_follow_remote() -> bool { IMPORTS_FOLLOW_REMOTE.load(Ordering::Relaxed) }pub fn webhook_request_timeout_secs() -> u64 { WEBHOOK_TIMEOUT.load(Ordering::Relaxed) }+/// Preferred natural-language tags for label matching. An empty vector means+/// "keep all languages" (multilingual mode). Cloned per call so callers hold no+/// lock; alignment runs are infrequent relative to this cost.+pub fn preferred_languages() -> Vec<String> {+ PREFERRED_LANGUAGES+ .read()+ .map(|g| g.clone())+ .unwrap_or_default()+}+#[cfg(test)]mod tests {use super::*;