Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
docs: improve SHACL validation guide onboarding and workflow guidance
Sameer63059bda47784750
diff --git a/docs/guides/shacl-validation.md b/docs/guides/shacl-validation.mdindex 5cce9817..c56d31fd 100644--- a/docs/guides/shacl-validation.md+++ b/docs/guides/shacl-validation.md@@ -4,14 +4,109 @@ description: "Generate W3C SHACL shapes from OWL ontologies, validate RDF knowleicon: "shield-check"----`SHACLGenerator` produces W3C SHACL constraint shapes from an OWL ontology, and `_run_pyshacl` validates your knowledge graph against them, returning a structured violation report. Use this to gate graph data before analytics, ISAC sharing, or regulatory submission — catching missing required properties, datatype violations, and cardinality breaches before they propagate.+## What Is SHACL Validation?-<Info>-SHACL shapes are produced from the same ontology dict that `OntologyGenerator` builds. The full workflow is: graph → ontology → SHACL shapes → validation report. Each stage is one function call. `NodeShape`, `PropertyShape`, and `SHACLGraph` import from `semantica.ontology`. `SHACLValidationReport`, `SHACLViolation`, and `_run_pyshacl` import from `semantica.ontology.ontology_validator`.-</Info>+SHACL (Shapes Constraint Language) is a standard for validating graph-based data. While an ontology defines the conceptual *schema* (the "what" exists in your domain), SHACL defines the structural *rules and constraints* (the "how" it should be structured).++In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and `_run_pyshacl` evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated.++## Why Use SHACL Validation?++Data validation is critical before running analytics, exporting data, or feeding it into production models. SHACL acts as a **data quality gate** that ensures your graph data is structurally sound. Use it to catch:+- Missing required properties (e.g., a customer without an email address).+- Datatype mismatches (e.g., a string where a number was expected).+- Cardinality breaches (e.g., a person with three primary addresses).++## When To Use / When Not To Use++- **When to Use**: You have a complex, interconnected knowledge graph and need to validate the *relationships* and structural integrity of the nodes across the graph. SHACL excels at ensuring that merged, highly connected data conforms to your business rules.+- **When NOT to Use**: If you are simply validating a flat JSON payload or a single incoming API request. For flat data or single records, use simpler, faster libraries like Pydantic or JSONSchema.++---++## Key Terms Explained++Before diving in, here are a few concepts you'll encounter:++- **RDF (Resource Description Framework)**: A standard way of representing data as a graph. It treats information as connected "triplets" (Subject → Predicate → Object).+- **OWL (Web Ontology Language)**: A language used to build ontologies. It defines the classes and properties that exist in your domain.+- **SHACL Shapes**: The actual validation rules. A "Shape" targets a specific class in your data (like `Person`) and defines the constraints it must follow (like "must have one birthdate").+- **Turtle (.ttl)**: A popular, human-readable file format for storing RDF graph data and SHACL shapes.++---++## Typical Workflow++A typical SHACL validation pipeline follows this lifecycle:++1. **Ontology**: Build an ontology representing your domain.+2. **SHACL Shapes**: Generate shapes from that ontology.+3. **Data Graph**: Prepare your knowledge graph.+4. **Validation**: Validate the knowledge graph against the SHACL shapes.+5. **Violation Report**: Analyze the report for errors.+6. **Remediation**: Fix the data or pipeline and re-validate.---+## Universal Example: Employee & Department++Let's look at a simple, universally understood example: ensuring every `Employee` belongs to a `Department` and has an `employee_id`.++```python+from semantica.context import ContextGraph+from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape+from semantica.ontology.ontology_validator import _run_pyshacl+from semantica.export import export_rdf+import tempfile, os++# 1. Prepare your data graph+graph = ContextGraph()+graph.add_node("emp-1", "Employee", "Alice", employee_id="E001")+graph.add_node("emp-2", "Employee", "Bob") # Missing employee_id, will cause a violation!++# 2. Build the ontology+ontology = (+ OntologyGenerator(base_uri="https://company.example.com/ontology/", min_occurrences=1)+ .generate_from_graph(graph.to_dict(), name="CompanyOntology")+)++# 3. Generate SHACL Shapes+shacl_gen = SHACLGenerator(base_uri="https://company.example.com/shapes/", severity="Violation")+shacl_graph = shacl_gen.generate(ontology)++# Inject mandatory constraints+BASE = "https://company.example.com/ontology/"+for ns in shacl_graph.node_shapes:+ if "Employee" in ns.target_class:+ ns.property_shapes.append(+ PropertyShape(path=f"{BASE}employee_id", min_count=1, severity="Violation")+ )++# Serialize shapes to Turtle+shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")++# 4. Export graph data to RDF (Turtle) via a temporary file+tmp = tempfile.NamedTemporaryFile(suffix=".ttl", delete=False, mode="w")+export_rdf(graph.to_dict(), tmp.name, format="turtle")+with open(tmp.name) as f:+ data_ttl = f.read()+os.unlink(tmp.name)++# 5. Run Validation+report = _run_pyshacl(data_ttl, shacl_ttl)++# 6. Analyze the Report+print(f"Graph conforms: {report.conforms}")+if not report.conforms:+ report.explain_violations() # Populates human-readable explanations+ for v in report.violations:+ print(f"Violation: {v.explanation}")+```++---++Now, let's explore the workflow in more depth.+## Step 1 — Build the ontology from your merged graphSHACL shapes are derived from an ontology. If you already have one from a previous run, skip this step.@@ -214,8 +309,12 @@ Each `SHACLViolation` identifies the node, property path, and fix required.```pythonif not report.conforms:- # Print plain-English explanations for every violation+ # Populate plain-English explanations for every violationreport.explain_violations()++ # Iterate and print the explanations+ for v in report.violations:+ print(v.explanation)# Node <https://cti.example.org/data/malware-002> is missing required property# <https://cti.example.org/ontology/family>. At least 1 value(s) are required.# Node <https://cti.example.org/data/vuln-003> is missing required property@@ -272,6 +371,16 @@ print(f"Violations after remediation: {report2.violation_count}")---+## Common Pitfalls++- **Assuming the ontology automatically enforces data quality**: `SHACLGenerator` generates shapes based on what it observes in the data. If your data is missing a field, the generator won't know it was mandatory unless you explicitly inject the constraint (as shown in Step 3).+- **Passing `ContextGraph` directly to SHACL validators**: The `_run_pyshacl` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.+- **Forgetting RDF serialization**: You must serialize your graph (often via a temporary file using `export_rdf`) before validating it.+- **Treating validation as a one-time step**: Validation should be integrated as an automated step in your CI/CD pipeline or data ingestion flow, acting as a recurring gatekeeper rather than a one-off script.+- **Ignoring validation reports**: A graph that does not conform must be remediated. Failing to review the `violation_count` and address the issues negates the purpose of SHACL validation.++---+## Domain Examples<Tabs>@@ -342,6 +451,8 @@ print(f"Warnings : {report.warning_count}")if not report.conforms:report.explain_violations()+ for v in report.violations:+ print(v.explanation)# Blocks the nightly ISAC share until violations are resolved```@@ -561,6 +672,8 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:if not report.conforms:print(f"Graph validation FAILED — {report.violation_count} violation(s)")report.explain_violations()+ for v in report.violations:+ print(v.explanation)sys.exit(1)print(f"Graph validation PASSED ({report.warning_count} warning(s))")@@ -575,3 +688,4 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `_run_pyshacl` input- [Conflict Resolution](conflict-resolution) — detect and resolve data conflicts before SHACL validation- [Change Management](change-management) — version-gate SHACL shapes alongside ontology versions+