Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
Verified RDF export: integrations/open_ontologies (#1108) (#1166)
Fabio Rovaiea6deed79228
diff --git a/integrations/open_ontologies/__init__.py b/integrations/open_ontologies/__init__.pynew file mode 100644index 00000000..296726c4--- /dev/null+++ b/integrations/open_ontologies/__init__.py@@ -0,0 +1,139 @@+"""+Semantica × Open Ontologies Integration+=======================================++Verified RDF export: write the graph, then have an independent engine read it+back before the file is trusted.++Public surface+--------------+verify_rdf — check an RDF string, return a ``VerificationReport``+verified_export_rdf — export, verify, and refuse to leave an unsound file+VerificationReport — what the independent engine found+VerificationError — raised when a verified export would have written junk+register — install the ``"verified"`` method in the export registry++Quick start+-----------+ pip install semantica[open-ontologies]++ >>> from integrations.open_ontologies import register+ >>> register()+ >>> from semantica.export.methods import export_rdf+ >>> export_rdf(kg, "graph.ttl", method="verified", ontology=vocab_ttl)++Nothing in the core changes. The method is registered through Semantica's own+``method_registry`` and is opt-in per call, so an existing pipeline behaves+exactly as it did until it asks for ``method="verified"``.++Why an independent engine+-------------------------+A generator cannot check its own output: it shares the assumptions that+produced the bug. If the code that writes an identifier has the wrong idea of+what a valid IRI is, the code that reads it back has the same wrong idea, and+the tests pass.++Concretely, and this is why the check earns its place. rdflib is lenient by+design: given a subject written as ``<Acme Corp>``, which the IRI grammar+forbids, it resolves the reference against the current working directory,+yields ``file:///…/Acme%20Corp``, and logs a warning. Oxigraph refuses the+file outright. One reader hands you a graph whose identifiers depend on which+directory the job ran in and reports success; the other tells you there is no+graph. This integration puts the strict reader after the export.++Three checks run in the order in which a failure in one makes the next+meaningless:++1. **Syntax, strictly.** Is it RDF 1.1 at all.+2. **Vocabulary, closed-world.** Which terms were used and declared nowhere.+ RDF is open-world, so an invented predicate is unknown rather than wrong and+ passes both parsing and SHACL untouched. Closing the world against a+ declared vocabulary is the only way to tell a real term from a plausible+ one, which matters here because inventing plausible terms is what an+ extractor does when it is unsure.+3. **Shapes, with an audit of what was examined.** Not only whether constraints+ held, but over how many nodes. A shapes graph whose targets are absent from+ the data reports conformance having checked nothing, and that report is+ indistinguishable from a real pass.++Compatibility+-------------+Requires ``open-ontologies-lite >= 0.5.0``, a pure-Python package over the+Oxigraph engine. No Rust toolchain, no service to run, no network. SHACL needs+the ``open-ontologies-lite[shacl]`` extra. Every symbol imports without it+installed; the import happens inside the call.++This integration depends on #1108, fixed in #1127. Before that, a custom method+could not refuse anything: ``method_registry`` caught every exception and fell+back to the default, so a gate that rejected invalid RDF and deleted the file+was followed by the fallback writing it straight back.+"""++from __future__ import annotations++from pathlib import Path+from typing import Any, Optional++from .verify import VerificationError, VerificationReport, verify_rdf++__all__ = [+ "VerificationError",+ "VerificationReport",+ "verify_rdf",+ "verified_export_rdf",+ "register",+]+++def verified_export_rdf(+ data: Any,+ file_path: str,+ format: str = "turtle",+ *,+ ontology: Optional[str] = None,+ shapes: Optional[str] = None,+ policed_namespaces: Optional[list] = None,+ raise_on_failure: bool = True,+ **kwargs: Any,+) -> VerificationReport:+ """Export RDF, then verify it before the file is trusted.++ Returns the :class:`VerificationReport`. With ``raise_on_failure`` (the+ default) an export that produced something unsound raises+ :class:`VerificationError` and the file is removed, so a failed run leaves+ no artifact that looks like a successful one.++ Pass ``raise_on_failure=False`` to keep the file and inspect the report,+ which is the right mode for a survey of an existing corpus.+ """+ from semantica.export.rdf_exporter import RDFExporter++ path = Path(file_path)+ encoding = kwargs.get("encoding", "utf-8")+ RDFExporter().export(data, path, format=format, **kwargs)++ try:+ report = verify_rdf(+ path.read_text(encoding=encoding),+ fmt=format,+ ontology=ontology,+ shapes=shapes,+ policed_namespaces=policed_namespaces,+ )+ except Exception:+ if raise_on_failure:+ path.unlink(missing_ok=True)+ raise++ if raise_on_failure and not report.ok:+ path.unlink(missing_ok=True)+ raise VerificationError(report)++ return report+++def register() -> None:+ """Register the ``"verified"`` method with Semantica's export registry."""+ from semantica.export.registry import method_registry++ method_registry.register("rdf", "verified", verified_export_rdf)diff --git a/integrations/open_ontologies/verify.py b/integrations/open_ontologies/verify.pynew file mode 100644index 00000000..867e43e3--- /dev/null+++ b/integrations/open_ontologies/verify.py@@ -0,0 +1,159 @@+"""Verification of Semantica's RDF exports.++Three checks a generation pipeline cannot run on itself, in the order that a+failure in one makes the next meaningless:++1. **Syntax, strictly.** Whether the output is RDF 1.1 at all. rdflib is lenient+ by design and will resolve a relative reference like ``<Acme Corp>`` against+ the file location, producing a ``file://`` IRI and a warning rather than an+ error, so a graph can "load" and still be nothing like what was meant.+ Oxigraph refuses it, which is the answer you want before publishing.++2. **Vocabulary, closed-world.** Whether the terms used were declared. RDF is+ open-world, so an invented predicate is unknown rather than wrong and passes+ both parsing and SHACL untouched. Closing the world against a declared+ vocabulary is the only way to separate a real term from a plausible one, and+ it is exactly the check an extraction pipeline needs, because inventing+ plausible terms is what extractors do when they are unsure.++3. **Shapes, with an audit of what was examined.** Whether constraints held, and+ how many nodes they held over. A shapes graph whose targets are absent from+ the data reports conformance having checked nothing, and that report is+ indistinguishable from a real pass. Every verdict here carries the focus-node+ count behind it.++Runs on ``open-ontologies-lite``, a pure-Python package over the Oxigraph engine.+No Rust toolchain, no service to run, no network.+"""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import Any+++@dataclass+class VerificationReport:+ """What an independent engine says about an export."""++ parses: bool+ triples: int = 0+ parse_error: str | None = None++ undeclared_terms: list[str] = field(default_factory=list)+ vocabulary_checked: bool = False++ conforms: bool | None = None+ violations: list[dict] = field(default_factory=list)+ focus_nodes: int | None = None+ unmatched_shapes: list[dict] = field(default_factory=list)++ @property+ def ok(self) -> bool:+ """True only when every check that ran gave a positive answer.++ A withheld SHACL verdict (``conforms is None``) is not a pass. Nothing+ was examined, so there is nothing to pass.+ """+ if not self.parses:+ return False+ if self.undeclared_terms:+ return False+ if self.conforms is False or (self.conforms is None and self.unmatched_shapes):+ return False+ return True++ def summary(self) -> str:+ if not self.parses:+ return f"not valid RDF: {self.parse_error}"+ parts = [f"{self.triples} triples"]+ if self.undeclared_terms:+ parts.append(f"{len(self.undeclared_terms)} undeclared term(s)")+ elif self.vocabulary_checked:+ parts.append("vocabulary clean")+ if self.conforms is False:+ parts.append(f"{len(self.violations)} SHACL violation(s)")+ elif self.conforms is None and self.unmatched_shapes:+ parts.append(+ f"SHACL verdict withheld: {len(self.unmatched_shapes)} shape(s) "+ f"matched nothing"+ )+ elif self.conforms is True:+ parts.append(f"conforms over {self.focus_nodes} focus node(s)")+ return ", ".join(parts)++ def to_dict(self) -> dict[str, Any]:+ return {+ "ok": self.ok,+ "parses": self.parses,+ "triples": self.triples,+ "parse_error": self.parse_error,+ "undeclared_terms": self.undeclared_terms,+ "conforms": self.conforms,+ "focus_nodes": self.focus_nodes,+ "unmatched_shapes": self.unmatched_shapes,+ "violations": self.violations,+ "summary": self.summary(),+ }+++class VerificationError(RuntimeError):+ """Raised when a verified export would have written something unsound."""++ def __init__(self, report: VerificationReport):+ self.report = report+ super().__init__(report.summary())+++def verify_rdf(+ rdf: str,+ *,+ fmt: str = "turtle",+ ontology: str | None = None,+ shapes: str | None = None,+ policed_namespaces: list[str] | None = None,+) -> VerificationReport:+ """Verify an RDF string and report what an independent engine finds.++ `ontology` enables the closed-world vocabulary check. Without it, or without+ `policed_namespaces`, that check is skipped rather than passed: there is+ nothing to check against, and a green light from an empty vocabulary is the+ failure the check exists to prevent.++ `shapes` enables SHACL. It needs `open-ontologies-lite[shacl]`.+ """+ from open_ontologies_lite import OntologyEngine, vocab_check++ report = VerificationReport(parses=False)++ # 1. Strict RDF 1.1 syntax.+ result = OntologyEngine.validate(rdf, fmt)+ report.parses = result.ok+ report.triples = result.triples+ report.parse_error = result.error+ if not result.ok:+ # Nothing downstream can be trusted about a graph that did not parse.+ return report++ # 2. Closed-world vocabulary.+ if ontology and policed_namespaces:+ vocab = vocab_check(+ ontology,+ rdf,+ data_format=fmt,+ extra_namespaces=policed_namespaces,+ )+ report.vocabulary_checked = "warning" not in vocab+ report.undeclared_terms = vocab["undeclared_terms"]++ # 3. Shapes, with the focus-node audit.+ if shapes:+ from open_ontologies_lite.shacl import shacl_validate++ shacl = shacl_validate(rdf, shapes, data_format=fmt)+ report.conforms = shacl["conforms"]+ report.violations = shacl["violations"]+ report.focus_nodes = shacl["focus_nodes"]+ report.unmatched_shapes = shacl["unmatched_shapes"]++ return reportdiff --git a/pyproject.toml b/pyproject.tomlindex 8382ffe7..da1d62db 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -279,6 +279,19 @@ crewai = ["crewai>=0.80.0; python_version >= '3.10'"]langchain = ["langchain-core>=0.3.0"]google-adk = ["google-adk>=1.27.0; python_version >= '3.10'"]+# ---- Verification ----+# The Oxigraph-backed engine behind integrations/open_ontologies. Pure Python,+# no Rust toolchain and no service to run. The `shacl` extra of that package is+# what the shapes half of the check needs; the syntax and closed-world+# vocabulary checks work without it.+#+# The 0.5.0 floor is not cosmetic. Up to 0.4.0 that package declared `mcp` as a+# a core dependency, and `mcp` pulls a starlette major that `fastapi>=0.109.2`+# above does not accept, so installing it into this environment UNINSTALLED+# fastapi and every import of `semantica/explorer` then failed. 0.5.0 moves+# `mcp` behind a `server` extra, which the library API does not use.+open-ontologies = ["open-ontologies-lite>=0.5.0"]+# ---- File Watching ----watch = ["watchdog>=6.0.0"]diff --git a/tests/integrations/open_ontologies/__init__.py b/tests/integrations/open_ontologies/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/tests/integrations/open_ontologies/test_verified_export.py b/tests/integrations/open_ontologies/test_verified_export.pynew file mode 100644index 00000000..090543ae--- /dev/null+++ b/tests/integrations/open_ontologies/test_verified_export.py@@ -0,0 +1,198 @@+"""Verified RDF export: an independent engine reads the file back.++The case that matters is the first one. Semantica's GraphBuilder defaults an+entity's id to its surface text, so the documented path emits `<Acme Corp> a+<ORG>`, and a space is not allowed in an IRI. rdflib parses it anyway by+resolving against the working directory; Oxigraph refuses it. A pipeline that+writes a file which parses nowhere has not succeeded, and this is the check+that says so before anything downstream reads it.++These tests are also the regression guard for #1108: until #1127, the registry+caught every exception from a custom method and fell back to the default, so a+gate that rejected the export and deleted the file was followed by the default+writing it straight back. Test one asserts the file is gone.+"""++import pytest++pytest.importorskip("open_ontologies_lite")++from integrations.open_ontologies import ( # noqa: E402+ VerificationError,+ register,+ verified_export_rdf,+ verify_rdf,+)++# An unsound export: rdflib leniency accepts invalid absolute IRIs (e.g. invalid+# second fragment '#' or malformed IPv6 host), while strict RDF 1.1 (Oxigraph)+# refuses them.+UNSOUND = {+ "entities": [{"id": "http://example.com/##", "text": "Acme Corp", "type": "ORG"}],+ "relationships": [],+}++SOUND = {+ "entities": [+ {+ "id": "https://example.org/acme",+ "text": "Acme Corp",+ "type": "https://example.org/Org",+ }+ ],+ "relationships": [],+}++VOCAB = """+@prefix owl: <http://www.w3.org/2002/07/owl#> .+@prefix ex: <https://example.org/> .+ex:Org a owl:Class .+"""++# Targets a namespace the data never uses, which is the defect that makes a+# validator report a pass having examined nothing.+SHAPES_THAT_MATCH_NOTHING = """+@prefix sh: <http://www.w3.org/ns/shacl#> .+@prefix ex: <https://example.org/shapes/> .+ex:PersonShape a sh:NodeShape ;+ sh:targetClass ex:Person ;+ sh:property [ sh:path ex:name ; sh:minCount 1 ] .+"""+++def test_an_unsound_export_raises_and_leaves_no_file(tmp_path):+ path = tmp_path / "graph.ttl"+ with pytest.raises(VerificationError) as excinfo:+ verified_export_rdf(UNSOUND, str(path))+ assert "Invalid IRI" in str(excinfo.value) or "not valid RDF" in str(excinfo.value)+ assert not path.exists(), "a failed run must not leave an artifact behind"+++def test_the_same_export_succeeds_without_verification(tmp_path):+ """The bytes are unchanged; only the verdict on them is new."""+ from semantica.export.rdf_exporter import RDFExporter++ path = tmp_path / "graph.ttl"+ RDFExporter().export(UNSOUND, path, format="turtle")+ assert path.exists() and path.stat().st_size > 0+++def test_a_sound_export_keeps_its_file_and_reports(tmp_path):+ path = tmp_path / "graph.ttl"+ report = verified_export_rdf(SOUND, str(path))+ assert report.ok+ assert report.triples > 0+ assert path.exists()+++def test_raise_on_failure_false_keeps_the_file_and_still_reports(tmp_path):+ path = tmp_path / "graph.ttl"+ report = verified_export_rdf(UNSOUND, str(path), raise_on_failure=False)+ assert not report.ok+ assert not report.parses+ assert path.exists(), "surveying a corpus needs the file kept"+++def test_a_term_declared_nowhere_is_reported(tmp_path):+ rdf = """+ @prefix ex: <https://example.org/> .+ ex:acme a ex:Org ; ex:hasInventedProperty "x" .+ """+ report = verify_rdf(rdf, ontology=VOCAB, policed_namespaces=["https://example.org/"])+ assert "https://example.org/hasInventedProperty" in report.undeclared_terms+ assert not report.ok+++def test_shapes_that_match_nothing_do_not_report_a_pass():+ rdf = """+ @prefix ex: <https://example.org/> .+ ex:acme a ex:Org .+ """+ report = verify_rdf(rdf, shapes=SHAPES_THAT_MATCH_NOTHING)+ assert report.conforms is None, "a vacuous run is not a pass"+ assert report.focus_nodes == 0+ assert report.unmatched_shapes+ assert not report.ok+++def test_registration_routes_export_rdf_through_the_gate(tmp_path):+ """The whole point of #1127: a registered method can refuse."""+ from semantica.export.methods import export_rdf++ register()+ path = tmp_path / "graph.ttl"+ with pytest.raises(VerificationError):+ export_rdf(UNSOUND, str(path), method="verified")+ assert not path.exists()+++def test_registration_leaves_the_default_method_alone(tmp_path):+ from semantica.export.methods import export_rdf++ register()+ path = tmp_path / "graph.ttl"+ export_rdf(UNSOUND, str(path))+ assert path.exists(), "an existing pipeline must behave exactly as before"+++def test_verified_export_unlinks_file_on_verify_exception(tmp_path, monkeypatch):+ """If verification itself raises an exception, the file must be removed when raise_on_failure=True."""+ path = tmp_path / "graph.ttl"++ def mock_verify(*args, **kwargs):+ raise RuntimeError("simulated engine failure")++ monkeypatch.setattr("integrations.open_ontologies.verify_rdf", mock_verify)+ with pytest.raises(RuntimeError, match="simulated engine failure"):+ verified_export_rdf(SOUND, str(path), raise_on_failure=True)+ assert not path.exists(), "failed verification must not leave an unverified file behind"+++def test_verified_export_keeps_file_on_verify_exception_when_raise_on_failure_false(tmp_path, monkeypatch):+ """When raise_on_failure=False, exceptions during verification still keep the file for debugging."""+ path = tmp_path / "graph.ttl"++ def mock_verify(*args, **kwargs):+ raise RuntimeError("simulated engine failure")++ monkeypatch.setattr("integrations.open_ontologies.verify_rdf", mock_verify)+ with pytest.raises(RuntimeError, match="simulated engine failure"):+ verified_export_rdf(SOUND, str(path), raise_on_failure=False)+ assert path.exists(), "file must be kept for surveys/debugging when raise_on_failure=False"+++def test_verified_export_honors_caller_encoding(tmp_path, monkeypatch):+ """The read-back verification must respect the encoding used for export."""+ from pathlib import Path++ path = tmp_path / "graph.ttl"+ read_encodings = []++ original_read_text = Path.read_text++ def spy_read_text(self, *args, **kwargs):+ read_encodings.append(kwargs.get("encoding"))+ return original_read_text(self, *args, **kwargs)++ monkeypatch.setattr(Path, "read_text", spy_read_text)+ report = verified_export_rdf(SOUND, str(path), encoding="utf-8")+ assert report.ok+ assert "utf-8" in read_encodings+++def test_vocabulary_check_skipped_when_only_ontology_or_namespaces_provided():+ """Vocabulary check must be skipped unless BOTH ontology and policed_namespaces are provided."""+ rdf = """+ @prefix ex: <https://example.org/> .+ ex:acme a ex:Org .+ """+ # Only ontology provided -> skipped+ report_onto_only = verify_rdf(rdf, ontology=VOCAB)+ assert not report_onto_only.vocabulary_checked+ assert report_onto_only.undeclared_terms == []++ # Only policed_namespaces provided -> skipped+ report_ns_only = verify_rdf(rdf, policed_namespaces=["https://example.org/"])+ assert not report_ns_only.vocabulary_checked+ assert report_ns_only.undeclared_terms == []+