Skip to content

Plan, apply and roll back changes to a production ontology, with a blast radius report and a proof an auditor can re-check.

feat(align): experimental multilingual / cross-lingual alignment logic

FABIOTESS46fe0a63e93d
diff --git a/CLAUDE.md b/CLAUDE.md
index e1047a70..8dc7294f 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -82,14 +82,14 @@ Claude dynamically decides the next tool call based on what the previous tool re
| `onto_crosswalk` | To look up clinical terminology mappings (ICD-10 ↔ SNOMED ↔ MeSH) |
| `onto_enrich` | To add skos:exactMatch triples linking classes to clinical codes |
| `onto_validate_clinical` | To check class labels against clinical crosswalk terminology |
-| `onto_align` | To detect alignment candidates (equivalentClass, exactMatch, subClassOf) between two ontologies using 7 weighted signals (6 structural + embedding similarity when embeddings are loaded) |
+| `onto_align` | To detect alignment candidates (equivalentClass, exactMatch, subClassOf) between two ontologies using 7 weighted signals (6 structural + embedding similarity when embeddings are loaded). Labels are matched with their parsed BCP-47 language tag; with the multilingual embedder loaded, cross-lingual pairs that share no surface tokens (e.g. `Dog`↔`Chien`) are admitted via the embedding signal. Restrict the languages consulted with `[language] preferred = [...]` / `OPEN_ONTOLOGIES_LANGUAGES` (empty = all) |
| `onto_align_feedback` | To accept/reject alignment candidates for self-calibrating confidence weights |
| `onto_lineage` | To view the session's lineage trail (plan → enforce → apply → monitor → drift) |
| `onto_lint_feedback` | To accept/dismiss a lint issue — teaches lint to suppress repeatedly dismissed warnings |
| `onto_enforce_feedback` | To accept/dismiss an enforce violation — teaches enforce to suppress repeatedly dismissed violations |
| `onto_dl_explain` | To explain why a class is unsatisfiable using DL tableaux reasoning — returns clash trace |
| `onto_dl_check` | To check if one class is subsumed by another using DL tableaux reasoning |
-| `onto_embed` | After loading an ontology — generates text + Poincaré structural embeddings for all classes. Honours `[embeddings] provider = "local" \| "openai"` in `config.toml`; OpenAI-compatible gateways (Azure, Ollama, vLLM, LocalAI, …) are supported via `OPEN_ONTOLOGIES_EMBEDDINGS_*` env vars |
+| `onto_embed` | After loading an ontology — generates text + Poincaré structural embeddings for all classes. The default local model is **multilingual** (`paraphrase-multilingual-MiniLM-L12-v2`), so labels in different natural languages embed into a shared space. Honours `[embeddings] provider = "local" \| "openai"` in `config.toml`; OpenAI-compatible gateways (Azure, Ollama, vLLM, LocalAI, …) are supported via `OPEN_ONTOLOGIES_EMBEDDINGS_*` env vars |
| `onto_search` | To find classes by natural language description — requires onto_embed first |
| `onto_similarity` | To compute embedding similarity between two specific IRIs |
| `onto_unload` | To unload the active ontology from memory. Optional `name` targets a specific cached entry; `delete_cache=true` also removes the on-disk N-Triples cache file |
@@ -183,6 +183,26 @@ When exploring or aligning ontologies using semantic embeddings:
7. When running `onto_align`, embedding similarity is automatically used as signal #7 if embeddings are loaded
8. This catches semantically equivalent classes that have different labels (e.g., Vehicle ↔ Automobile)
+### Cross-Lingual Alignment
+
+The default `onto_embed` model is multilingual, so labels in different natural
+languages embed into a shared vector space. `onto_align` parses the BCP-47
+language tag off each label and, when label similarity is near zero (as it is
+across languages — `Dog` vs `Chien` share no tokens), lets a strong embedding
+match bypass the label pre-filter so the pair is still scored. Such pairs
+typically surface as `borderline` candidates with their language-tagged labels
+in the `context` block, for review via `onto_align_feedback`.
+
+Control which languages are consulted with the `[language]` config section:
+
+```toml
+[language]
+# Empty (default) = keep ALL languages — multilingual matching.
+# Restrict to a set to pin matching to specific languages (untagged labels are
+# always kept). Override with OPEN_ONTOLOGIES_LANGUAGES=en,fr
+preferred = []
+```
+
### Borderline-Candidate Review (LLM-as-Oracle Pattern)
`onto_align` partitions candidates into three buckets by confidence:
diff --git a/src/align.rs b/src/align.rs
index e419a526..5b51840a 100644
--- a/src/align.rs
+++ b/src/align.rs
@@ -42,7 +42,11 @@ impl AlignmentEngine {
}
}
- fn extract_classes(store: &GraphStore) -> Vec<ClassInfo> {
+ /// Extract class IRIs and their labels, parsing the BCP-47 language tag off
+ /// each literal and retaining only labels permitted by `preferred` (empty =
+ /// keep all languages — multilingual mode). Untagged literals are always
+ /// kept.
+ fn extract_classes(store: &GraphStore, preferred: &[String]) -> Vec<ClassInfo> {
let query = r#"
SELECT ?class ?label ?altLabel WHERE {
?class a <http://www.w3.org/2002/07/owl#Class> .
@@ -85,25 +89,26 @@ impl AlignmentEngine {
labels: Vec::new(),
});
- if let Some(label) = row["label"].as_str() {
- let l = label.trim_matches('"').to_string();
- if !entry.labels.contains(&l) {
- entry.labels.push(l);
- }
- }
- if let Some(alt) = row["altLabel"].as_str() {
- let a = alt.trim_matches('"').to_string();
- if !entry.labels.contains(&a) {
- entry.labels.push(a);
+ for key in ["label", "altLabel"] {
+ if let Some(raw) = row[key].as_str() {
+ let label = crate::language::parse_literal(raw);
+ if !label.text.is_empty()
+ && crate::language::label_matches_policy(&label, preferred)
+ && !entry.labels.contains(&label)
+ {
+ entry.labels.push(label);
+ }
}
}
}
}
- // If no label found, use IRI local name
+ // If no policy-matching label was found, fall back to the IRI local
+ // name as a language-neutral label.
for info in class_map.values_mut() {
if info.labels.is_empty() {
- info.labels.push(local_name(&info.iri));
+ info.labels
+ .push(crate::language::Label::new(local_name(&info.iri), None));
}
}
@@ -271,8 +276,8 @@ impl AlignmentEngine {
let mut best = 0.0f64;
for la in &a.labels {
for lb in &b.labels {
- let na = normalize_label(la);
- let nb = normalize_label(lb);
+ let na = normalize_label(&la.text);
+ let nb = normalize_label(&lb.text);
// Jaro-Winkler on full normalized strings
let jw = jaro_winkler(&na, &nb);
@@ -301,6 +306,13 @@ impl AlignmentEngine {
#[cfg(feature = "embeddings")]
const DEFAULT_WEIGHTS: [f64; 7] = [0.20, 0.15, 0.12, 0.12, 0.12, 0.09, 0.20];
+ /// Minimum embedding cosine for a pair to bypass the label-similarity
+ /// pre-filter. This is the gate that admits cross-lingual matches
+ /// (e.g. `Dog`↔`Chien`), which share no surface tokens and so score ~0 on
+ /// label similarity but high on a multilingual embedding model.
+ #[cfg(feature = "embeddings")]
+ const CROSS_LINGUAL_EMBED_MIN: f64 = 0.80;
+
#[cfg(feature = "embeddings")]
/// Compute embedding similarity score using cosine similarity on text vectors.
pub fn embedding_similarity_score(vec_a: &[f32], vec_b: &[f32]) -> f64 {
@@ -360,6 +372,8 @@ impl AlignmentEngine {
) -> anyhow::Result<String> {
// Clamp: degenerate range means "everything above high_threshold; nothing borderline".
let low_threshold = low_threshold.min(high_threshold);
+ // Active language policy (empty = keep all languages / multilingual).
+ let preferred = crate::runtime::preferred_languages();
// For RRF the inline weighted-sum confidence is throwaway (recomputed after
// collection). We collect all label-prefiltered candidates and only apply
// `low_threshold` to the FINAL fused score post-rerank.
@@ -372,7 +386,7 @@ impl AlignmentEngine {
let format = Self::detect_content_format(source);
source_store.load_content(source, format)?;
}
- let source_classes = Self::extract_classes(&source_store);
+ let source_classes = Self::extract_classes(&source_store, &preferred);
// Load target into a temporary graph (or use the main store)
let target_store_owned;
@@ -389,7 +403,7 @@ impl AlignmentEngine {
} else {
target_ref = &*self.graph;
}
- let target_classes = Self::extract_classes(target_ref);
+ let target_classes = Self::extract_classes(target_ref, &preferred);
// Get learned weights (or defaults)
let weights = self.get_learned_weights();
@@ -461,19 +475,9 @@ impl AlignmentEngine {
let label_sim = Self::label_similarity(sc, tc);
- // Pre-filter: skip pairs where label similarity is too low.
- // Raised from 0.7 to 0.75 to reduce false positives on anatomy-style
- // ontologies where many terms share token overlap (e.g., "bone" variants).
- if label_sim < 0.75 {
- continue;
- }
-
- let prop_overlap = Self::property_overlap(&source_store, &sc.iri, target_ref, &tc.iri);
- let parent_ovlp = Self::parent_overlap(&source_store, &sc.iri, target_ref, &tc.iri);
- let inst_overlap = Self::instance_overlap(&source_store, &sc.iri, target_ref, &tc.iri);
- let restr_sim = Self::restriction_similarity(&source_store, &sc.iri, target_ref, &tc.iri);
- let neigh_sim = Self::neighborhood_similarity(&source_store, &sc.iri, target_ref, &tc.iri);
-
+ // Embedding similarity is computed BEFORE the label pre-filter so
+ // it can vouch for cross-lingual pairs that share no surface
+ // tokens (label_sim ~ 0) but embed close in a multilingual model.
#[cfg(feature = "embeddings")]
let embedding_sim = {
if let Some(ref vs) = self.vecstore {
@@ -487,6 +491,26 @@ impl AlignmentEngine {
}
};
+ // Pre-filter: skip pairs where label similarity is too low.
+ // Raised from 0.7 to 0.75 to reduce false positives on anatomy-style
+ // ontologies where many terms share token overlap (e.g., "bone" variants).
+ // With embeddings, a strong embedding match bypasses the gate so
+ // cross-lingual equivalences are not discarded before scoring.
+ #[cfg(feature = "embeddings")]
+ if label_sim < 0.75 && embedding_sim < Self::CROSS_LINGUAL_EMBED_MIN {
+ continue;
+ }
+ #[cfg(not(feature = "embeddings"))]
+ if label_sim < 0.75 {
+ continue;
+ }
+
+ let prop_overlap = Self::property_overlap(&source_store, &sc.iri, target_ref, &tc.iri);
+ let parent_ovlp = Self::parent_overlap(&source_store, &sc.iri, target_ref, &tc.iri);
+ let inst_overlap = Self::instance_overlap(&source_store, &sc.iri, target_ref, &tc.iri);
+ let restr_sim = Self::restriction_similarity(&source_store, &sc.iri, target_ref, &tc.iri);
+ let neigh_sim = Self::neighborhood_similarity(&source_store, &sc.iri, target_ref, &tc.iri);
+
#[cfg(feature = "embeddings")]
let signals = [label_sim, prop_overlap, parent_ovlp, inst_overlap, restr_sim, neigh_sim, embedding_sim];
@@ -494,13 +518,21 @@ impl AlignmentEngine {
let signals = [label_sim, prop_overlap, parent_ovlp, inst_overlap, restr_sim, neigh_sim];
// Compute confidence. When structural signals are all zero
- // (common in lightweight OWL files), use label similarity with a
- // penalty rather than the weighted sum (which would be ~0.25 * label_sim
- // and too low to pass any threshold).
+ // (common in lightweight OWL files), fall back to the strongest
+ // lexical/semantic signal with a penalty rather than the weighted
+ // sum (which would be ~0.25 * label_sim and too low to pass any
+ // threshold).
let structural_sum: f64 = signals[1..6].iter().sum();
let confidence: f64 = if structural_sum == 0.0 {
- // No structural evidence: use label_sim but apply 15% penalty
- label_sim * 0.85
+ // No structural evidence: apply a 15% penalty to the best
+ // available signal. With embeddings this lets a strong
+ // cross-lingual embedding match still surface (typically as a
+ // borderline candidate for review).
+ #[cfg(feature = "embeddings")]
+ let best_signal = label_sim.max(embedding_sim);
+ #[cfg(not(feature = "embeddings"))]
+ let best_signal = label_sim;
+ best_signal * 0.85
} else {
signals.iter().zip(weights.iter()).map(|(s, w)| s * w).sum()
};
@@ -612,11 +644,11 @@ impl AlignmentEngine {
let t_iri = candidate["target_iri"].as_str().unwrap_or("").to_string();
let s_labels: Vec<String> = source_class_map
.get(s_iri.as_str())
- .map(|c| c.labels.clone())
+ .map(|c| c.labels.iter().map(|l| l.tagged()).collect())
.unwrap_or_default();
let t_labels: Vec<String> = target_class_map
.get(t_iri.as_str())
- .map(|c| c.labels.clone())
+ .map(|c| c.labels.iter().map(|l| l.tagged()).collect())
.unwrap_or_default();
let s_parents = Self::extract_parents(&source_store, &s_iri);
let t_parents = Self::extract_parents(target_ref, &t_iri);
@@ -925,7 +957,11 @@ fn jaccard_similarity(a: &[String], b: &[String]) -> f64 {
#[derive(Debug, Clone)]
pub struct ClassInfo {
pub iri: String,
- pub labels: Vec<String>,
+ /// Labels with their parsed language tags. Populated from rdfs:label /
+ /// skos:prefLabel / skos:altLabel (and OBO synonyms), filtered by the
+ /// active language policy. Falls back to a single language-neutral label
+ /// derived from the IRI local name when no policy-matching label exists.
+ pub labels: Vec<crate::language::Label>,
}
/// Extract local name from an IRI (after last # or /).
@@ -975,11 +1011,14 @@ mod tests {
fn test_label_similarity() {
let a = ClassInfo {
iri: "http://ex.org/Dog".into(),
- labels: vec!["Dog".into()],
+ labels: vec![crate::language::Label::new("Dog", None)],
};
let b = ClassInfo {
iri: "http://other.org/Canine".into(),
- labels: vec!["Dog".into(), "Canine".into()],
+ labels: vec![
+ crate::language::Label::new("Dog", None),
+ crate::language::Label::new("Canine", None),
+ ],
};
// Exact label match should give 1.0
let sim = AlignmentEngine::label_similarity(&a, &b);
@@ -1117,16 +1156,49 @@ mod tests {
fn test_label_similarity_camelcase() {
let a = ClassInfo {
iri: "http://ex.org/DomesticCat".into(),
- labels: vec!["DomesticCat".into()],
+ labels: vec![crate::language::Label::new("DomesticCat", None)],
};
let b = ClassInfo {
iri: "http://other.org/HouseCat".into(),
- labels: vec!["Domestic Cat".into()],
+ labels: vec![crate::language::Label::new("Domestic Cat", None)],
};
let sim = AlignmentEngine::label_similarity(&a, &b);
assert!(sim > 0.95, "CamelCase split should match: {}", sim);
}
+ #[test]
+ fn extract_classes_parses_language_tags_and_policy() {
+ let gs = GraphStore::new();
+ gs.load_content(
+ r#"
+ @prefix owl: <http://www.w3.org/2002/07/owl#> .
+ @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+ @prefix ex: <http://example.org/> .
+ ex:Dog a owl:Class ; rdfs:label "Dog"@en, "Chien"@fr, "Perro"@es .
+ "#,
+ RdfFormat::Turtle,
+ )
+ .unwrap();
+
+ // Multilingual policy (empty) keeps all three, each with its tag parsed
+ // off — no more `Dog"@en` mangling.
+ let all = AlignmentEngine::extract_classes(&gs, &[]);
+ let dog = all.iter().find(|c| c.iri.ends_with("Dog")).unwrap();
+ let texts: std::collections::HashSet<&str> =
+ dog.labels.iter().map(|l| l.text.as_str()).collect();
+ assert!(texts.contains("Dog"));
+ assert!(texts.contains("Chien"));
+ assert!(texts.contains("Perro"));
+ let chien = dog.labels.iter().find(|l| l.text == "Chien").unwrap();
+ assert_eq!(chien.lang.as_deref(), Some("fr"));
+
+ // English-only policy drops the fr/es labels.
+ let en = AlignmentEngine::extract_classes(&gs, &["en".to_string()]);
+ let dog_en = en.iter().find(|c| c.iri.ends_with("Dog")).unwrap();
+ let texts_en: Vec<&str> = dog_en.labels.iter().map(|l| l.text.as_str()).collect();
+ assert_eq!(texts_en, vec!["Dog"]);
+ }
+
#[test]
fn test_align_feedback() {
let tmp = tempfile::NamedTempFile::new().unwrap();
diff --git a/src/config.rs b/src/config.rs
index 74ada7f7..0296ec5c 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -8,6 +8,7 @@ use std::path::Path;
pub struct Config {
pub general: GeneralConfig,
pub embeddings: EmbeddingsConfig,
+ pub language: LanguageConfig,
pub cache: CacheConfig,
pub tools: ToolsConfig,
pub webhook: WebhookConfig,
@@ -104,16 +105,17 @@ pub struct EmbeddingsConfig {
/// `OPEN_ONTOLOGIES_EMBEDDINGS_PROVIDER`.
pub provider: Option<String>,
/// Path to the ONNX model file (provider = "local" only).
- /// Default: ~/.open-ontologies/models/bge-small-en-v1.5.onnx
+ /// Default: ~/.open-ontologies/models/paraphrase-multilingual-MiniLM-L12-v2.onnx
+ /// (falls back to a legacy bge-small-en-v1.5.onnx if that is what is on disk).
pub model_path: Option<String>,
/// Path to the tokenizer.json file (provider = "local" only).
/// Default: ~/.open-ontologies/models/tokenizer.json
pub tokenizer_path: Option<String>,
- /// URL to download the ONNX model from. Default: BGE-small-en-v1.5 from Hugging Face
+ /// URL to download the ONNX model from. Default: multilingual MiniLM-L12-v2 from Hugging Face
pub model_url: Option<String>,
- /// URL to download the tokenizer from. Default: BGE-small-en-v1.5 tokenizer from Hugging Face
+ /// URL to download the tokenizer from. Default: multilingual MiniLM-L12-v2 tokenizer from Hugging Face
pub tokenizer_url: Option<String>,
- /// Filename for the downloaded model. Default: bge-small-en-v1.5.onnx
+ /// Filename for the downloaded model. Default: paraphrase-multilingual-MiniLM-L12-v2.onnx
pub model_name: Option<String>,
// ─── OpenAI-compatible provider (provider = "openai") ───────────────
@@ -198,6 +200,50 @@ pub fn resolve_embeddings_model(cfg: &EmbeddingsConfig) -> String {
.unwrap_or_else(|| "text-embedding-3-small".to_string())
}
+/// `[language]` — natural-language policy for ontology labels used in
+/// alignment / matching.
+#[derive(Debug, Default, Deserialize, Clone)]
+#[serde(default)]
+pub struct LanguageConfig {
+ /// Preferred natural-language tags (BCP-47, e.g. `["en", "cy"]`) for the
+ /// `rdfs:label` / `skos:prefLabel` / `skos:altLabel` values consulted by
+ /// `onto_align`. Untagged (plain or datatyped) literals are always kept.
+ ///
+ /// An empty list (the default) keeps **all** languages — i.e. fully
+ /// multilingual matching. This only pays off when paired with a
+ /// multilingual embedding model (the default `onto_embed` model is
+ /// multilingual), because cross-lingual matches (`Dog`↔`Chien`) are carried
+ /// by the embedding signal, not by string similarity.
+ ///
+ /// Override at runtime with `OPEN_ONTOLOGIES_LANGUAGES` (comma / colon /
+ /// semicolon separated, e.g. `OPEN_ONTOLOGIES_LANGUAGES=en,fr`).
+ #[serde(alias = "preferred_languages")]
+ pub preferred: Vec<String>,
+}
+
+/// Resolve the preferred-language policy.
+///
+/// Precedence: `OPEN_ONTOLOGIES_LANGUAGES` env var > config field. Entries are
+/// trimmed, lowercased, and emptied entries dropped. An empty result means
+/// "keep all languages" (multilingual mode).
+pub fn resolve_languages(cfg: &LanguageConfig) -> Vec<String> {
+ let from_env = std::env::var("OPEN_ONTOLOGIES_LANGUAGES").ok();
+ let raw: Vec<String> = match from_env {
+ Some(v) if !v.trim().is_empty() => v
+ .split([',', ':', ';'])
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .collect(),
+ _ => cfg
+ .preferred
+ .iter()
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .collect(),
+ };
+ raw.into_iter().map(|s| s.to_lowercase()).collect()
+}
+
#[derive(Debug, Deserialize, Clone)]
#[serde(default)]
pub struct CacheConfig {
@@ -576,6 +622,31 @@ mod tests {
assert_eq!(resolved, "local");
}
+ #[test]
+ fn language_section_parses_and_defaults_to_multilingual() {
+ // Empty / absent => keep all languages.
+ let cfg: Config = toml::from_str("").expect("parse empty");
+ assert!(cfg.language.preferred.is_empty());
+
+ let toml_src = r#"
+ [language]
+ preferred = ["en", "CY"]
+ "#;
+ let cfg: Config = toml::from_str(toml_src).expect("parse");
+ assert_eq!(cfg.language.preferred, vec!["en", "CY"]);
+ // Resolver lowercases (env var unset in this path).
+ if std::env::var("OPEN_ONTOLOGIES_LANGUAGES").is_err() {
+ assert_eq!(resolve_languages(&cfg.language), vec!["en", "cy"]);
+ }
+
+ // Alias `preferred_languages` also populates the field.
+ let aliased: Config = toml::from_str(
+ "[language]\npreferred_languages = [\"fr\"]\n",
+ )
+ .expect("parse alias");
+ assert_eq!(aliased.language.preferred, vec!["fr"]);
+ }
+
#[test]
fn base_url_alias_accepted() {
// The legacy/alternative `base_url` key should also populate
diff --git a/src/embed.rs b/src/embed.rs
index 7c47a207..660939b9 100644
--- a/src/embed.rs
+++ b/src/embed.rs
@@ -8,11 +8,31 @@ use tract_onnx::prelude::*;
use crate::poincare::l2_normalize;
-/// Model download URLs for bge-small-en-v1.5
-pub const BGE_SMALL_ONNX_URL: &str =
- "https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx";
-pub const BGE_SMALL_TOKENIZER_URL: &str =
- "https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json";
+/// Default embedding model: multilingual MiniLM (384-dim, BERT-style with
+/// `token_type_ids`, mean pooling — a drop-in match for this loader's input
+/// signature and pooling). Replaces the previous English-only
+/// `bge-small-en-v1.5` so that labels in different natural languages embed into
+/// a *shared* vector space, which is what makes cross-lingual alignment
+/// (`Dog` ↔ `Chien` ↔ `Perro`) possible via the embedding signal in
+/// `onto_align`. Override with `[embeddings] model_url/tokenizer_url/model_name`
+/// or by switching to the OpenAI provider. The `Xenova/*` repo ships the
+/// `onnx/model.onnx` + `tokenizer.json` layout this loader expects.
+pub const DEFAULT_MODEL_ONNX_URL: &str =
+ "https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/onnx/model.onnx";
+pub const DEFAULT_MODEL_TOKENIZER_URL: &str =
+ "https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/tokenizer.json";
+/// On-disk filename for the default downloaded model.
+pub const DEFAULT_MODEL_FILENAME: &str = "paraphrase-multilingual-MiniLM-L12-v2.onnx";
+/// Legacy English-only model filename. Still loaded as a fallback when present
+/// so existing installs that downloaded it before the multilingual switch keep
+/// working (with English-only embeddings) until they re-run `init`.
+pub const LEGACY_EN_MODEL_FILENAME: &str = "bge-small-en-v1.5.onnx";
+
+// Back-compat aliases for the previous public constant names.
+#[deprecated(note = "use DEFAULT_MODEL_ONNX_URL")]
+pub const BGE_SMALL_ONNX_URL: &str = DEFAULT_MODEL_ONNX_URL;
+#[deprecated(note = "use DEFAULT_MODEL_TOKENIZER_URL")]
+pub const BGE_SMALL_TOKENIZER_URL: &str = DEFAULT_MODEL_TOKENIZER_URL;
pub struct TextEmbedder {
#[allow(clippy::type_complexity)]
@@ -154,9 +174,17 @@ impl TextEmbedderProvider {
.clone()
.map(|p| std::path::PathBuf::from(crate::config::expand_tilde(&p)))
.or_else(|| {
- default_model_dir
- .as_ref()
- .map(|d| d.join("bge-small-en-v1.5.onnx"))
+ default_model_dir.as_ref().map(|d| {
+ // Prefer the multilingual default; fall back to a
+ // legacy English model only if it is the one present
+ // on disk (older installs).
+ let preferred = d.join(DEFAULT_MODEL_FILENAME);
+ if preferred.exists() {
+ return preferred;
+ }
+ let legacy = d.join(LEGACY_EN_MODEL_FILENAME);
+ if legacy.exists() { legacy } else { preferred }
+ })
});
let tokenizer_path = cfg
diff --git a/src/language.rs b/src/language.rs
new file mode 100644
index 00000000..0a228efc
--- /dev/null
+++ b/src/language.rs
@@ -0,0 +1,203 @@
+//! Natural-language handling for ontology labels.
+//!
+//! RDF labels carry an optional BCP-47 language tag (`"Dog"@en`). Before this
+//! module, the alignment engine matched the raw N-Triples string returned by
+//! `Term::to_string()`, which (a) silently mangled tagged literals
+//! (`"Dog"@en` survived `trim_matches('"')` as `Dog"@en`) and (b) gave callers
+//! no way to either pin matching to a preferred language or opt into fully
+//! multilingual matching.
+//!
+//! This module parses the language tag off the literal and provides a small,
+//! configurable policy so callers can keep all languages (default — pairs with
+//! a multilingual embedding model for cross-lingual alignment) or restrict to a
+//! preferred set. Untagged / datatyped literals are always treated as
+//! language-neutral and retained regardless of policy.
+
+/// A label with its optional (lowercased) BCP-47 language tag.
+///
+/// `lang == None` means a plain or datatyped literal — language-neutral, always
+/// retained under any policy.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct Label {
+ pub text: String,
+ pub lang: Option<String>,
+}
+
+impl Label {
+ pub fn new(text: impl Into<String>, lang: Option<String>) -> Self {
+ Self {
+ text: text.into(),
+ lang: lang.map(|l| l.trim().to_lowercase()).filter(|l| !l.is_empty()),
+ }
+ }
+
+ /// Human/LLM-readable form: `"text@lang"` when tagged, else just `"text"`.
+ /// Used to surface the language of borderline alignment candidates so the
+ /// reviewing orchestrator can judge cross-lingual pairs.
+ pub fn tagged(&self) -> String {
+ match &self.lang {
+ Some(lang) => format!("{}@{}", self.text, lang),
+ None => self.text.clone(),
+ }
+ }
+}
+
+/// Parse an oxigraph `Term::to_string()` literal into its lexical value and
+/// language tag.
+///
+/// Handles the N-Triples literal forms:
+/// - `"value"@en` → `(value, Some("en"))`
+/// - `"value"^^<datatype>` → `(value, None)`
+/// - `"value"` → `(value, None)`
+/// - bare (IRI / local name) → `(raw, None)`
+///
+/// The closing quote is located as the last `"` in the string, which is correct
+/// for well-formed literals because neither the `@lang` suffix nor the
+/// `^^<datatype>` suffix contains a double quote. Common escapes inside the
+/// lexical value (`\"`, `\\`, `\n`, `\t`, `\r`) are unescaped.
+pub fn parse_literal(raw: &str) -> Label {
+ let s = raw.trim();
+ if !s.starts_with('"') {
+ // Bare IRI or unquoted local name — no language tag.
+ return Label::new(s.to_string(), None);
+ }
+ if let Some(close) = s.rfind('"') {
+ if close > 0 {
+ let value = unescape(&s[1..close]);
+ let suffix = &s[close + 1..];
+ let lang = suffix.strip_prefix('@').map(|l| l.to_string());
+ return Label::new(value, lang);
+ }
+ }
+ // Degenerate input (e.g. a lone `"`). Fall back to a best-effort strip.
+ Label::new(s.trim_matches('"').to_string(), None)
+}
+
+/// Unescape the common N-Triples/Turtle string escapes that appear inside a
+/// literal's lexical form. Anything else is passed through verbatim.
+fn unescape(s: &str) -> String {
+ if !s.contains('\\') {
+ return s.to_string();
+ }
+ let mut out = String::with_capacity(s.len());
+ let mut chars = s.chars();
+ while let Some(c) = chars.next() {
+ if c == '\\' {
+ match chars.next() {
+ Some('"') => out.push('"'),
+ Some('\\') => out.push('\\'),
+ Some('n') => out.push('\n'),
+ Some('t') => out.push('\t'),
+ Some('r') => out.push('\r'),
+ Some(other) => {
+ out.push('\\');
+ out.push(other);
+ }
+ None => out.push('\\'),
+ }
+ } else {
+ out.push(c);
+ }
+ }
+ out
+}
+
+/// Should this label be retained under the given preferred-language policy?
+///
+/// An empty `preferred` list means "keep all languages" (multilingual mode).
+/// Untagged labels (`lang == None`) are always kept. Otherwise the label's tag
+/// must be present in `preferred`. Comparison is case-insensitive on the prefix
+/// so that `"en"` matches `"en-GB"` / `"en-us"` etc.
+pub fn label_matches_policy(label: &Label, preferred: &[String]) -> bool {
+ if preferred.is_empty() {
+ return true;
+ }
+ match &label.lang {
+ None => true,
+ Some(lang) => preferred.iter().any(|p| {
+ let p = p.to_lowercase();
+ lang == &p || lang.starts_with(&format!("{p}-"))
+ }),
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn parse_tagged_literal() {
+ let l = parse_literal("\"Dog\"@en");
+ assert_eq!(l.text, "Dog");
+ assert_eq!(l.lang.as_deref(), Some("en"));
+ }
+
+ #[test]
+ fn parse_region_tag_is_lowercased() {
+ let l = parse_literal("\"Colour\"@en-GB");
+ assert_eq!(l.text, "Colour");
+ assert_eq!(l.lang.as_deref(), Some("en-gb"));
+ }
+
+ #[test]
+ fn parse_plain_literal() {
+ let l = parse_literal("\"Dog\"");
+ assert_eq!(l.text, "Dog");
+ assert_eq!(l.lang, None);
+ }
+
+ #[test]
+ fn parse_typed_literal_has_no_lang() {
+ let l = parse_literal("\"42\"^^<http://www.w3.org/2001/XMLSchema#integer>");
+ assert_eq!(l.text, "42");
+ assert_eq!(l.lang, None);
+ }
+
+ #[test]
+ fn parse_non_english_value() {
+ assert_eq!(parse_literal("\"Chien\"@fr").text, "Chien");
+ assert_eq!(parse_literal("\"Hund\"@de").lang.as_deref(), Some("de"));
+ // Non-Latin scripts survive intact.
+ assert_eq!(parse_literal("\"犬\"@ja").text, "犬");
+ }
+
+ #[test]
+ fn parse_bare_value() {
+ let l = parse_literal("Dog");
+ assert_eq!(l.text, "Dog");
+ assert_eq!(l.lang, None);
+ }
+
+ #[test]
+ fn unescape_quotes_and_backslashes() {
+ let l = parse_literal("\"say \\\"hi\\\"\"@en");
+ assert_eq!(l.text, "say \"hi\"");
+ assert_eq!(l.lang.as_deref(), Some("en"));
+ }
+
+ #[test]
+ fn policy_empty_keeps_all() {
+ assert!(label_matches_policy(&Label::new("Chien", Some("fr".into())), &[]));
+ assert!(label_matches_policy(&Label::new("Dog", None), &[]));
+ }
+
+ #[test]
+ fn policy_untagged_always_kept() {
+ let pref = vec!["en".to_string()];
+ assert!(label_matches_policy(&Label::new("Dog", None), &pref));
+ }
+
+ #[test]
+ fn policy_filters_by_preferred() {
+ let pref = vec!["en".to_string(), "cy".to_string()];
+ assert!(label_matches_policy(&Label::new("Dog", Some("en".into())), &pref));
+ assert!(label_matches_policy(&Label::new("Ci", Some("cy".into())), &pref));
+ assert!(!label_matches_policy(&Label::new("Chien", Some("fr".into())), &pref));
+ }
+
+ #[test]
+ fn policy_region_subtag_matches_base() {
+ let pref = vec!["en".to_string()];
+ assert!(label_matches_policy(&Label::new("Colour", Some("en-gb".into())), &pref));
+ }
+}
diff --git a/src/lib.rs b/src/lib.rs
index 7668a484..6ee09426 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -35,6 +35,7 @@ pub mod hnsw_index;
pub mod ingest;
pub mod inputs;
pub mod kgcl;
+pub mod language;
pub mod lineage;
pub mod mapping;
pub mod marketplace;
diff --git a/src/main.rs b/src/main.rs
index 02763b35..cb6ac222 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -52,15 +52,26 @@ data_dir = "~/.open-ontologies"
#
# ── Local provider (provider = "local", default) ────────────────────────
# Paths to a local ONNX model and tokenizer (loaded at runtime).
-# model_path = "~/.open-ontologies/models/bge-small-en-v1.5.onnx"
+# Default is the multilingual MiniLM model (see model_url below); labels in
+# different natural languages embed into a shared space for cross-lingual
+# alignment.
+# model_path = "~/.open-ontologies/models/paraphrase-multilingual-MiniLM-L12-v2.onnx"
# tokenizer_path = "~/.open-ontologies/models/tokenizer.json"
#
# URLs used by `open-ontologies init` to download the model.
-# Override these to use a different sentence-transformer model (e.g. for non-English text).
+# Override these to use a different sentence-transformer model. To go back to
+# the smaller English-only model set these to BAAI/bge-small-en-v1.5.
# The model must be exported to ONNX and use a Hugging Face tokenizer.json.
-# model_url = "https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx"
-# tokenizer_url = "https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json"
-# model_name = "bge-small-en-v1.5.onnx"
+# model_url = "https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/onnx/model.onnx"
+# tokenizer_url = "https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/tokenizer.json"
+# model_name = "paraphrase-multilingual-MiniLM-L12-v2.onnx"
+#
+# [language]
+# Preferred natural-language tags for the labels onto_align consults. Untagged
+# literals are always kept. Empty (default) = keep ALL languages (multilingual
+# matching, paired with the multilingual embedding model above). Override with
+# OPEN_ONTOLOGIES_LANGUAGES=en,fr
+# preferred = []
#
# ── OpenAI-compatible provider (provider = "openai") ────────────────────
# Works with the official OpenAI API, Azure OpenAI, Ollama, vLLM, LocalAI,
@@ -157,13 +168,13 @@ enum Commands {
Init {
#[arg(long, default_value = "~/.open-ontologies")]
data_dir: String,
- /// Custom ONNX model URL (default: BGE-small-en-v1.5 from Hugging Face)
+ /// Custom ONNX model URL (default: multilingual MiniLM-L12-v2 from Hugging Face)
#[arg(long)]
model_url: Option<String>,
- /// Custom tokenizer URL (default: BGE-small-en-v1.5 tokenizer from Hugging Face)
+ /// Custom tokenizer URL (default: multilingual MiniLM-L12-v2 tokenizer from Hugging Face)
#[arg(long)]
tokenizer_url: Option<String>,
- /// Filename for the downloaded ONNX model (default: bge-small-en-v1.5.onnx)
+ /// Filename for the downloaded ONNX model (default: paraphrase-multilingual-MiniLM-L12-v2.onnx)
#[arg(long)]
model_name: Option<String>,
},
@@ -754,15 +765,15 @@ async fn async_main() -> anyhow::Result<()> {
let onnx_url = _model_url
.as_deref()
.or(cfg.model_url.as_deref())
- .unwrap_or(open_ontologies::embed::BGE_SMALL_ONNX_URL);
+ .unwrap_or(open_ontologies::embed::DEFAULT_MODEL_ONNX_URL);
let tok_url = _tokenizer_url
.as_deref()
.or(cfg.tokenizer_url.as_deref())
- .unwrap_or(open_ontologies::embed::BGE_SMALL_TOKENIZER_URL);
+ .unwrap_or(open_ontologies::embed::DEFAULT_MODEL_TOKENIZER_URL);
let onnx_filename = _model_name
.as_deref()
.or(cfg.model_name.as_deref())
- .unwrap_or("bge-small-en-v1.5.onnx");
+ .unwrap_or(open_ontologies::embed::DEFAULT_MODEL_FILENAME);
let model_path = models_dir.join(onnx_filename);
let tokenizer_path = models_dir.join("tokenizer.json");
diff --git a/src/runtime.rs b/src/runtime.rs
index 55cdd22e..d876acca 100644
--- a/src/runtime.rs
+++ b/src/runtime.rs
@@ -13,9 +13,11 @@
//! observe the legacy behaviour.
use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU64, AtomicUsize, Ordering};
+use std::sync::RwLock;
use crate::config::{
- self, Config, FeedbackConfig, ImportsConfig, ReasonerConfig, RepoConfig, WebhookConfig,
+ self, Config, FeedbackConfig, ImportsConfig, LanguageConfig, ReasonerConfig, RepoConfig,
+ WebhookConfig,
};
// ── Defaults match the previous hardcoded constants exactly ─────────────
@@ -45,6 +47,10 @@ static IMPORTS_MAX_DEPTH: AtomicUsize = AtomicUsize::new(DEFAULT_IMPORTS_MAX_DEP
static IMPORTS_TIMEOUT: AtomicU64 = AtomicU64::new(DEFAULT_IMPORTS_TIMEOUT);
static IMPORTS_FOLLOW_REMOTE: AtomicBool = AtomicBool::new(true);
static WEBHOOK_TIMEOUT: AtomicU64 = AtomicU64::new(DEFAULT_WEBHOOK_TIMEOUT);
+/// Preferred natural-language tags for label matching. Empty (the default)
+/// means "keep all languages" — fully multilingual. Populated from
+/// `[language] preferred` / `OPEN_ONTOLOGIES_LANGUAGES` at startup.
+static PREFERRED_LANGUAGES: RwLock<Vec<String>> = RwLock::new(Vec::new());
/// Initialise all runtime knobs from a loaded `Config`. Idempotent — calling
/// this multiple times simply overwrites the current values, which is fine
@@ -56,6 +62,14 @@ pub fn init_from_config(cfg: &Config) {
apply_repo(&cfg.repo);
apply_imports(&cfg.imports);
apply_webhook(&cfg.webhook);
+ apply_language(&cfg.language);
+}
+
+fn apply_language(l: &LanguageConfig) {
+ let resolved = config::resolve_languages(l);
+ if let Ok(mut guard) = PREFERRED_LANGUAGES.write() {
+ *guard = resolved;
+ }
}
fn apply_reasoner(r: &ReasonerConfig) {
@@ -107,6 +121,16 @@ pub fn imports_request_timeout_secs() -> u64 { IMPORTS_TIMEOUT.load(Ordering::Re
pub fn imports_follow_remote() -> bool { IMPORTS_FOLLOW_REMOTE.load(Ordering::Relaxed) }
pub fn webhook_request_timeout_secs() -> u64 { WEBHOOK_TIMEOUT.load(Ordering::Relaxed) }
+/// Preferred natural-language tags for label matching. An empty vector means
+/// "keep all languages" (multilingual mode). Cloned per call so callers hold no
+/// lock; alignment runs are infrequent relative to this cost.
+pub fn preferred_languages() -> Vec<String> {
+ PREFERRED_LANGUAGES
+ .read()
+ .map(|g| g.clone())
+ .unwrap_or_default()
+}
+
#[cfg(test)]
mod tests {
use super::*;