Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
feat(ontology): bootstrap a draft domain schema from extracted data (PR 2/3) (#1537)
Kevin7c4d62841bc9
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 293480e9..52aaee9a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -24,6 +24,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- New `ExtractionSchema` (`semantica.semantic_extract`, lazy export): a lightweight, read-only view over a domain ontology (allowed concepts + predicates with optional `domain` / `range`). Reuses the project's existing OWL ontology representation rather than a parallel type — build one from a `generate_ontology`-style dict (`ExtractionSchema.from_ontology`) or an OWL/Turtle file/string (`ExtractionSchema.from_owl`, via the existing `rdflib` dependency). An empty `domain`/`range` means unconstrained, matching OWL- Implements the deterministic core of ontology-based information extraction (OBIE; Wimalasuriya & Dou, 2010). No new runtime dependencies- New `tests/semantic_extract/test_schema_validator.py`+- **Schema bootstrap** (#1510) by @pkupt+ - New `bootstrap_schema` (`semantica.ontology`): induces a draft domain ontology from a sample of extracted `{entities, relationships}` so you can ratify it by hand and then use it (via `SchemaValidator`) to gate future extraction. Complements `SchemaValidator`: where the validator *constrains* output against an existing ontology, bootstrap *induces* the draft from data+ - Threads the existing `min_occurrences` frequency gate through as a class- and predicate-level filter: types/predicates seen fewer times are omitted from the draft — not just entity-type (class) inference+ - Never auto-applies: returns `draft: True` plus a Turtle serialization for human review, per the OBIE/ontology-learning convention that frequency alone yields proposals, not final ontologies+ - New `tests/test_bootstrap_schema.py`## [0.7.0] - 2026-09-07diff --git a/semantica/ontology/__init__.py b/semantica/ontology/__init__.pyindex 93b6c2f2..76f62398 100644--- a/semantica/ontology/__init__.py+++ b/semantica/ontology/__init__.py@@ -133,6 +133,7 @@ License: MITfrom typing import Any, Dict, List, Optional, Unionfrom .associative_class import AssociativeClass, AssociativeClassBuilder+from .bootstrap_schema import bootstrap_schemafrom .class_inferrer import ClassInferrerfrom .competency_questions import CompetencyQuestion, CompetencyQuestionsManagerfrom .config import OntologyConfig, ontology_config@@ -182,6 +183,7 @@ from .methods import ingest_ontology__all__ = [# Main generators"OntologyGenerator",+ "bootstrap_schema","ClassInferrer","ClassInferencer", # Legacy alias"PropertyGenerator",diff --git a/semantica/ontology/bootstrap_schema.py b/semantica/ontology/bootstrap_schema.pynew file mode 100644index 00000000..6d31ccc6--- /dev/null+++ b/semantica/ontology/bootstrap_schema.py@@ -0,0 +1,122 @@+"""Bootstrap a draft domain ontology from a sample of extracted data.++Part of #1510 (Schema-guided extraction). Complements ``SchemaValidator``:+where the validator *constrains* extraction output against an existing+ontology, ``bootstrap_schema`` *induces* a draft ``Ontology`` from a sample of+``{entities, relationships}`` so you can ratify it by hand and then use it to+gate future extraction.++The draft is never applied automatically: this module emits a TTL serialization+(for human review) and marks the result as a draft. Auto-application is+explicitly a human decision, per the OBIE / ontology-learning convention that+frequency alone yields proposals, not final ontologies.+"""++from typing import Any, Dict, List, Optional++from ..utils.logging import get_logger+from .ontology_generator import OntologyGenerator+from .owl_generator import OWLGenerator++logger = get_logger("bootstrap_schema")+++def _align_endpoints(ontology: Dict[str, Any]) -> Dict[str, Any]:+ """Drop property endpoints that never made it into ``classes``.++ Class inference and predicate inference share a frequency gate, but a rare+ endpoint type can still appear on a frequent relationship's property. If we+ left it in ``domain``/``range``, the emitted TTL would reference an undeclared+ class and ``ExtractionSchema.from_ontology`` would fold the filtered type back+ into its concept vocabulary, defeating the gate. Unconstrained is the honest+ reading of "did not clear the gate", so such endpoints become ``owl:Thing``.+ """+ declared = set()+ for c in ontology.get("classes", []):+ if not isinstance(c, dict):+ continue+ name = c.get("name") or c.get("label")+ if name:+ declared.add(name)+ # Property endpoints carry the *raw* entity type while class names are+ # normalized ("person" -> "Person"); match both spellings.+ inferred_from = (c.get("metadata") or {}).get("inferred_from")+ if isinstance(inferred_from, str):+ declared.add(inferred_from)+ for prop in ontology.get("properties", []):+ if not isinstance(prop, dict):+ continue+ for key in ("domain", "range"):+ values = prop.get(key)+ if not isinstance(values, list):+ continue+ prop[key] = [+ v+ for v in values+ if not isinstance(v, str) or v in declared or v == "owl:Thing"+ ] or ["owl:Thing"]+ return ontology+++def bootstrap_schema(+ entities: List[Dict[str, Any]],+ relationships: List[Dict[str, Any]],+ min_occurrences: int = 2,+ base_uri: Optional[str] = None,+ **options: Any,+) -> Dict[str, Any]:+ """Induce a draft ontology from a sample of extracted entities/relationships.++ Thin wrapper over ``OntologyGenerator``: the entities/relationships ->+ concepts/properties induction, the ``domain``/``range`` inference (via+ ``PropertyInferencer``) and the TTL export all live inside the existing+ pipeline. This function only (a) threads ``min_occurrences`` through as a+ frequency gate, and (b) wraps the result as a *draft* that must be+ human-ratified before use.++ Args:+ entities: Extracted entities as dicts. ``type`` is used as the class+ label; per ``semantica/semantic_extract/`` the common fields are+ ``type``/``text``/``confidence``/``metadata``.+ relationships: Extracted relationships as dicts, with ``type`` as the+ predicate and ``source``/``target`` endpoints.+ min_occurrences: Minimum occurrence count for a class/property to be+ induced. Passed through to the pipeline's frequency gate+ (``ClassInferrer``/``PropertyGenerator`` default to 2).+ base_uri: Base IRI for generated classes/properties; defaults to the+ generator's own.+ **options: Forwarded to ``OntologyGenerator.generate_ontology``.++ Returns:+ A dictionary (never auto-applied):+ - ``ontology``: the induced ontology dict from the pipeline+ - ``ttl``: Turtle serialization for human review+ - ``draft``: always ``True``+ - ``min_occurrences``: the frequency gate actually used+ """+ gen_options: Dict[str, Any] = dict(options)+ if base_uri is not None:+ gen_options["base_uri"] = base_uri+ gen_options["min_occurrences"] = min_occurrences++ generator = OntologyGenerator(**gen_options)+ data: Dict[str, List[Dict[str, Any]]] = {+ "entities": entities,+ "relationships": relationships,+ }+ ontology = generator.generate_ontology(data, **gen_options)+ ontology = _align_endpoints(ontology)++ # Serialize with the generator's own namespace manager so declared classes,+ # properties and domain/range endpoints resolve under one consistent IRI+ # scheme (a fresh OWLGenerator would mint a second, inconsistent set).+ ttl = OWLGenerator(namespace_manager=generator.namespace_manager).generate_owl(+ ontology, format="turtle"+ )++ return {+ "ontology": ontology,+ "ttl": ttl,+ "draft": True,+ "min_occurrences": min_occurrences,+ }diff --git a/semantica/ontology/class_inferrer.py b/semantica/ontology/class_inferrer.pyindex 16c8e912..923a3acd 100644--- a/semantica/ontology/class_inferrer.py+++ b/semantica/ontology/class_inferrer.py@@ -110,6 +110,10 @@ class ClassInferrer:**options: Additional options:- build_hierarchy: Whether to build class hierarchy (default: True)- namespace_manager: Optional namespace manager for URI generation+ - min_occurrences: Per-call frequency gate, overriding the+ constructor default for this invocation (default: constructor+ value). Mirrors ``PropertyGenerator`` so a single per-call+ threshold consistently gates both classes and predicates.Returns:List of inferred class definition dictionaries, each containing:@@ -142,6 +146,10 @@ class ClassInferrer:self.progress_tracker.update_tracking(tracking_id, message="Grouping entities by type...")+ # Per-call gate overrides the constructor value (PropertyGenerator+ # already reads options; this keeps class and predicate gating aligned).+ min_occurrences = options.get("min_occurrences", self.min_occurrences)+# Group entities by typeentity_types = defaultdict(list)for entity in entities:@@ -150,7 +158,7 @@ class ClassInferrer:normalized_types = defaultdict(list)for entity_type, type_entities in entity_types.items():- if len(type_entities) >= self.min_occurrences:+ if len(type_entities) >= min_occurrences:normalized_name = self.naming_conventions.normalize_class_name(str(entity_type))@@ -175,7 +183,7 @@ class ClassInferrer:)classes = []for entity_type, type_entities in entity_types.items():- if len(type_entities) >= self.min_occurrences:+ if len(type_entities) >= min_occurrences:class_def = self._create_class_from_entities(entity_type, type_entities, **options)diff --git a/semantica/ontology/ontology_generator.py b/semantica/ontology/ontology_generator.pyindex baa4035b..709a73dd 100644--- a/semantica/ontology/ontology_generator.py+++ b/semantica/ontology/ontology_generator.py@@ -83,7 +83,10 @@ class OntologyGenerator:**kwargs: Additional configuration options:- base_uri: Base URI for ontology (default: "https://semantica.dev/ontology/")- namespace_manager: Optional namespace manager instance- - min_occurrences: Minimum occurrences for class inference (default: 2)+ - min_occurrences: Minimum occurrences for a class or predicate to+ be inferred. Gates both entity-type (class) inference and+ relationship-type (object property) inference; types/predicates+ seen fewer times are omitted from the ontology (default: 2)Example:```python@@ -138,6 +141,8 @@ class OntologyGenerator:- name: Ontology name (default: "GeneratedOntology")- build_hierarchy: Whether to build class hierarchy (default: True)- namespace_manager: Optional namespace manager instance+ - min_occurrences: Frequency gate for classes and predicates+ (same semantics as __init__; default: 2)Returns:Generated ontology dictionary containing:diff --git a/semantica/ontology/owl_generator.py b/semantica/ontology/owl_generator.pyindex 85c1cff7..135573c6 100644--- a/semantica/ontology/owl_generator.py+++ b/semantica/ontology/owl_generator.py@@ -192,7 +192,13 @@ class OWLGenerator:def _resolve_class_uri(self, value: Any, ns_manager: NamespaceManager) -> str:if self._is_http_uri(value):return value- return ns_manager.generate_class_iri(str(value))+ # Preserve standard vocabulary terms (owl:Thing, rdfs:Class, ...) as their+ # canonical IRIs instead of minting a local class for the prefixed name.+ text = str(value)+ prefix, _, local = text.partition(":")+ if prefix in ns_manager.get_all_namespaces() and local:+ return f"{ns_manager.get_all_namespaces()[prefix]}{local}"+ return ns_manager.generate_class_iri(text)def _resolve_property_identifier(self, prop: Dict[str, Any]) -> str:return prop.get("label") or prop.get("name")diff --git a/tests/test_bootstrap_schema.py b/tests/test_bootstrap_schema.pynew file mode 100644index 00000000..2fe86be2--- /dev/null+++ b/tests/test_bootstrap_schema.py@@ -0,0 +1,265 @@+"""Tests for bootstrap_schema (#1510).++Covers the draft-induction contract: it wraps the existing+``OntologyGenerator`` pipeline, threads ``min_occurrences`` through as a+frequency gate, and emits a draft TTL that is never auto-applied.+"""++from semantica.ontology import bootstrap_schema+++def _sample(dup: bool = True):+ """A small extraction sample with a common 'Person'/'worksFor' core."""+ entities = [+ {"type": "Person", "text": "John"},+ {"type": "Person", "text": "Jane"},+ {"type": "Org", "text": "Acme"},+ {"type": "Org", "text": "Globex"},+ {"type": "Product", "text": "Widget"},+ ]+ rels = [+ {"type": "worksFor", "source": "John", "target": "Acme"},+ {"type": "worksFor", "source": "Jane", "target": "Globex"},+ ]+ return entities, rels+++def test_returns_a_draft_never_auto_applied():+ entities, rels = _sample()+ result = bootstrap_schema(entities, rels)+ assert result["draft"] is True+++def test_induces_classes_and_properties_with_domain_range():+ entities, rels = _sample()+ result = bootstrap_schema(entities, rels)+ ontology = result["ontology"]+ class_names = {c.get("name") for c in ontology.get("classes", [])}+ assert "Person" in class_names+ props = ontology.get("properties", [])+ assert any(+ p.get("name") == "worksFor"+ and "Person" in p.get("domain", [])+ and "Org" in p.get("range", [])+ for p in props+ )+++def test_min_occurrences_gates_sparse_types():+ # 'Product' appears once; with a higher gate it should be dropped from+ # the draft's classes, while the frequent core survives.+ entities, rels = _sample()+ result = bootstrap_schema(entities, rels, min_occurrences=2)+ class_names = {c.get("name") for c in result["ontology"].get("classes", [])}+ assert "Product" not in class_names+ assert "Person" in class_names+++def test_min_occurrences_gates_sparse_predicates_too():+ # A predicate seen fewer than min_occurrences is dropped from the draft's+ # properties — not just types. This is the contract clarified against+ # #1510's "class inference only" framing.+ entities = _sample()[0]+ # Two worksFor (frequent) and a single locatedIn (sparse).+ rels = [+ {"type": "worksFor", "source": "John", "target": "Acme"},+ {"type": "worksFor", "source": "Jane", "target": "Acme"},+ {"type": "locatedIn", "source": "John", "target": "Acme"},+ ]+ result = bootstrap_schema(entities, rels, min_occurrences=2)+ prop_names = {p.get("name") for p in result["ontology"].get("properties", [])}+ assert "worksFor" in prop_names+ assert "locatedIn" not in prop_names+++def test_emits_ttl_for_human_review():+ entities, rels = _sample()+ result = bootstrap_schema(entities, rels)+ ttl = result["ttl"]+ assert isinstance(ttl, str) and len(ttl) > 0+ assert "@prefix" in ttl+++def test_exposes_min_occurrences_used():+ entities, rels = _sample()+ result = bootstrap_schema(entities, rels, min_occurrences=3)+ assert result["min_occurrences"] == 3+++def test_empty_input_returns_empty_draft():+ # Nothing extracted yet: still a valid (empty) draft with a parseable+ # TTL, so callers can run it early in the pipeline without guarding.+ result = bootstrap_schema([], [])+ assert result["draft"] is True+ assert result["ontology"]["classes"] == []+ assert result["ontology"]["properties"] == []+ assert "@prefix" in result["ttl"]+++def test_owl_thing_serializes_as_standard_owl_iri():+ # An unresolved relationship endpoint gets owl:Thing; serializing it must+ # mint the standard OWL IRI, not a generated local "Thing" class that would+ # over-constrain the draft and reject later typed relationships.+ entities, rels = _sample()+ # Two worksFor (passes the predicate gate) whose range endpoint is not an+ # entity type, so the range stays unresolved -> owl:Thing.+ rels = [+ {"type": "worksFor", "source": "John", "target": "SomeStranger"},+ {"type": "worksFor", "source": "Jane", "target": "AnotherStranger"},+ ]+ result = bootstrap_schema(entities, rels)+ # The standard OWL term must appear (rdflib writes it as the qname prefix),+ # and the ontology must not mint a local Thing class under its own base.+ assert (+ "owl:Thing" in result["ttl"]+ ), "owl:Thing must serialize to the standard OWL term"+ assert "class/Thing" not in result["ttl"]+ from rdflib import Graph, Namespace+ from rdflib.namespace import OWL++ g = Graph().parse(data=result["ttl"], format="turtle")+ base = Namespace(result["ontology"]["uri"].rstrip("/") + "/")+ local_things = list(g.subjects(OWL.Class, None))+ assert base["Thing"] not in local_things, "must not mint a local Thing class"+++def test_sparse_endpoint_types_not_folded_back_by_from_ontology():+ # A rare type that rides in on a frequent relationship's endpoint must not+ # be smuggled back into the concept vocabulary via from_ontology: the+ # frequency gate should stay decisive.+ from semantica.semantic_extract.schema import ExtractionSchema++ entities, rels = _sample()+ # 'Product' appears once (fails class gate) but is a worksFor endpoint.+ rels = [+ {"type": "worksFor", "source": "John", "target": "Widget"},+ {"type": "worksFor", "source": "Jane", "target": "Widget"},+ ]+ result = bootstrap_schema(entities, rels, min_occurrences=2)+ ontology = result["ontology"]+ class_names = {c.get("name") for c in ontology.get("classes", [])}+ assert "Product" not in class_names+ schema = ExtractionSchema.from_ontology(ontology)+ assert "Product" not in schema.concepts+++def test_ttl_iris_consistent_with_declared_classes():+ # Finding 2: with hash IRIs (use_speaking_iris=False), a serializer that+ # ignores the generator's namespace manager resolves domain/range to+ # speaking IRIs while declared classes use hashed ones, leaving properties+ # pointing at undeclared resources. Parse the TTL and require every+ # domain/range endpoint to be a declared owl:Class node.+ from rdflib import Graph+ from rdflib.namespace import OWL, RDF, RDFS++ entities, rels = _sample()+ result = bootstrap_schema(entities, rels, use_speaking_iris=False)+ g = Graph().parse(data=result["ttl"], format="turtle")+ class_nodes = set(g.subjects(RDF.type, OWL.Class))+ endpoints = set(g.objects(None, RDFS.domain)) | set(g.objects(None, RDFS.range))+ # xsd datatypes may appear as ranges of data properties; only class-like+ # URIRef endpoints must be declared.+ from rdflib import URIRef+ from rdflib.namespace import XSD++ xsd = str(XSD)+ class_endpoints = {+ e for e in endpoints if isinstance(e, URIRef) and not str(e).startswith(xsd)+ }+ assert class_endpoints, "expected at least one class endpoint in TTL"+ undeclared = class_endpoints - class_nodes - {OWL.Thing}+ assert not undeclared, f"endpoints not declared as owl:Class: {undeclared}"+++def test_raw_endpoint_types_survive_alignment():+ # Property endpoints carry the raw entity type ("person") while class names+ # are normalized ("Person"); a declared type must not be dropped from+ # domain/range just because its spelling differs from the class name.+ entities = [+ {"type": "person", "text": "John"},+ {"type": "person", "text": "Jane"},+ {"type": "organization", "text": "Acme"},+ {"type": "organization", "text": "Globex"},+ ]+ rels = [+ {"type": "worksFor", "source": "John", "target": "Acme"},+ {"type": "worksFor", "source": "Jane", "target": "Globex"},+ ]+ result = bootstrap_schema(entities, rels)+ works_for = next(+ p for p in result["ontology"]["properties"] if p["name"] == "worksFor"+ )+ assert "person" in works_for["domain"]+ assert "organization" in works_for["range"]+++def test_per_call_min_occurrences_gates_classes_too():+ # Finding 3: a per-call threshold must gate classes as well as predicates,+ # not silently apply only to predicates.+ from semantica.ontology import OntologyGenerator++ entities, rels = _sample()+ gen = OntologyGenerator() # constructor default gate = 2+ ontology = gen.generate_ontology(+ {"entities": entities, "relationships": rels}, min_occurrences=3+ )+ class_names = {c.get("name") for c in ontology.get("classes", [])}+ assert "Product" not in class_names, "per-call gate must drop sparse class"+++def test_draft_ttl_round_trips_through_from_owl():+ # The emitted TTL is not only a human-readable artifact: it is an input to+ # ExtractionSchema.from_owl. Parsing it back must reproduce the induced+ # vocabulary, so the two documented entry points -- from_ontology on the+ # returned dict, from_owl on the returned TTL -- agree on a sample whose+ # entity types are already spelled the way the generator normalizes them.+ from semantica.semantic_extract.schema import ExtractionSchema++ entities, rels = _sample()+ result = bootstrap_schema(entities, rels)+ from_ttl = ExtractionSchema.from_owl(result["ttl"])+ from_dict = ExtractionSchema.from_ontology(result["ontology"])++ assert "Person" in from_ttl.concepts+ assert "Org" in from_ttl.concepts+ assert from_ttl.concepts == from_dict.concepts+ assert set(from_ttl.predicates) == set(from_dict.predicates)+ assert from_ttl.predicates["worksFor"].domain == frozenset({"Person"})+ assert from_ttl.predicates["worksFor"].range == frozenset({"Org"})+++def test_unresolved_endpoint_not_resurrected_by_parsing_the_ttl():+ # An endpoint that never cleared the frequency gate is serialized as+ # owl:Thing. Reading the draft back must not turn that into a concept --+ # neither as the filtered raw type nor as a literal "Thing" -- or the gate+ # would be undone by the very artifact handed to a reviewer.+ from semantica.semantic_extract.schema import ExtractionSchema++ entities, rels = _sample()+ # 'Product' appears once (fails the class gate) yet is a worksFor target.+ rels = [+ {"type": "worksFor", "source": "John", "target": "Widget"},+ {"type": "worksFor", "source": "Jane", "target": "Widget"},+ ]+ result = bootstrap_schema(entities, rels, min_occurrences=2)+ assert "owl:Thing" in result["ttl"]++ schema = ExtractionSchema.from_owl(result["ttl"])+ assert "Product" not in schema.concepts+ assert "Thing" not in schema.concepts+ assert "owl:Thing" not in schema.concepts+ works_for = schema.predicates["worksFor"]+ assert works_for.domain == frozenset({"Person"})+ # Unconstrained, expressed as the empty set -- not the raw endpoint name.+ assert works_for.range == frozenset()+++def test_empty_draft_ttl_parses_to_an_empty_schema():+ # A draft induced before any extraction has run is still a legal artifact:+ # from_owl must accept it and yield an empty vocabulary rather than raise.+ from semantica.semantic_extract.schema import ExtractionSchema++ result = bootstrap_schema([], [])+ schema = ExtractionSchema.from_owl(result["ttl"])+ assert schema.concepts == frozenset()+ assert schema.predicates == {}