Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
docs(provenance): align lineage and provenance APIs with implementation
Sameer6305777deaa4e335
diff --git a/docs/reference/provenance.md b/docs/reference/provenance.mdindex 6702df62..aebda7e6 100644--- a/docs/reference/provenance.md+++ b/docs/reference/provenance.md@@ -1,36 +1,48 @@---title: "Provenance Module"-description: "W3C PROV-O compliant lineage tracking, source attribution, tamper-evident checksums, and audit trails across all modules."+description: "W3C PROV-O lineage tracking, source attribution, tamper-evident checksums, and audit trails across all modules."icon: "link"----`semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, reasoning, and export. Compliant with W3C PROV-O, suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 environments.+`semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, chunking, and relationship building. Designed for high-stakes domains requiring complete traceability.## Exported Classes| Class | Role || --- | --- |-| `ProvenanceManager` | Track entities and get lineage: `track_entity`, `get_lineage`, `export_provenance` |-| `ProvenanceEntry` | Single record: `{entity_id, source, method, confidence, timestamp, checksum}` |-| `SourceReference` | Rich source pointer: `{url, doi, page, quote, author, publication_date}` |+| `ProvenanceManager` | Central tracker: `track_entity`, `track_relationship`, `track_chunk`, `get_lineage`, `get_statistics` |+| `ProvenanceEntry` | Single lineage record: `{entity_id, entity_type, activity_id, source_document, confidence, checksum, ...}` |+| `SourceReference` | Rich source pointer: `{document, page, section, line, confidence, metadata}` |+| `ProvenanceStorage` | Abstract storage interface || `InMemoryStorage` | Default backend — fast, not persisted across restarts |-| `SQLiteStorage` | Production backend — persists to a local SQLite file |-| `compute_checksum()` | Returns SHA-256 fingerprint of a provenance entry |-| `verify_checksum()` | Detects tampering by comparing stored vs recomputed hash |+| `SQLiteStorage` | Persistent backend — persists to a local SQLite file |+| `compute_checksum` | Returns SHA-256 fingerprint of a `ProvenanceEntry` |+| `verify_checksum` | Detects tampering by comparing stored vs recomputed hash |-## ProvenanceManager+## Getting Started```python-from semantica.provenance import ProvenanceManager, InMemoryStorage, SQLiteStorage+from semantica.provenance import ProvenanceManager, SQLiteStorage-# In-memory (default) — fast, not persisted across restarts-manager = ProvenanceManager(storage=InMemoryStorage())+# In-memory (default) — no arguments required+manager = ProvenanceManager()-# SQLite — persisted, production-ready+# SQLite — persists to diskmanager = ProvenanceManager(storage=SQLiteStorage("provenance.db"))-# Track an extracted entity (with rich source reference)-manager.track_entity(+# Shorthand for SQLite — equivalent to the line above+manager = ProvenanceManager(storage_path="provenance.db")+```++Track an entity with a source document, retrieve its lineage, then verify integrity:++```python+from semantica.provenance import ProvenanceManager, compute_checksum, verify_checksum++manager = ProvenanceManager()++# track_entity returns a ProvenanceEntry+entry = manager.track_entity(entity_id="apple_inc",source="annual_report_2023.pdf",source_location="Page 12, Section 3.1",@@ -38,132 +50,431 @@ manager.track_entity(confidence=0.98,)-# Track an extracted relationship-manager.track_entity(- entity_id="steve_jobs_founded_apple",- source="annual_report_2023.pdf",+print(entry.entity_id) # "apple_inc"+print(entry.source_document) # "annual_report_2023.pdf"+print(entry.confidence) # 0.98+print(entry.checksum) # SHA-256 hex string (set automatically)++# Verify the entry has not been tampered with+is_valid = verify_checksum(entry)+print(is_valid) # True+```++## ProvenanceManager++### Constructor++```python+ProvenanceManager(+ storage=None, # ProvenanceStorage instance; defaults to InMemoryStorage+ storage_path=None, # str path — creates SQLiteStorage if provided+)+```++If both `storage` and `storage_path` are omitted, an `InMemoryStorage` is used.++### Tracking Methods++```python+from semantica.provenance import ProvenanceManager, SourceReference++manager = ProvenanceManager()++# Track an entity+entry = manager.track_entity(+ entity_id="apple_inc", # required+ source="annual_report.pdf", # required: document ID, DOI, file path+ source_location="Page 12", # optional kwarg+ source_quote="Incorporated on January 3, 1977.", # optional kwarg+ confidence=0.98, # optional kwarg, default 1.0+ entity_type="organization", # optional kwarg, default "entity"+ metadata={"sector": "tech"}, # optional metadata dict+)++# Track a relationship+rel_entry = manager.track_relationship(+ relationship_id="jobs_founded_apple",+ source="annual_report.pdf",+ confidence=0.95,+ metadata={"type": "founded"},+)++# Track a document chunk (after splitting)+chunk_entry = manager.track_chunk(+ chunk_id="chunk_001",+ source_document="report.pdf",+ source_path="/docs/report.pdf",+ start_index=0,+ end_index=500,+ parent_chunk_id=None,+)++# Track a property with a SourceReference+source_ref = SourceReference(+ document="DOI:10.1038/s41586-021-03371-z",+ page=4,+ section="Table S4",confidence=0.92,)+prop_entry = manager.track_property_source(+ entity_id="cabo_pulmo",+ property_name="biomass_increase",+ value="463%",+ source=source_ref,+)+```++### Batch Tracking++```python+entities = [+ {"id": "entity_1", "confidence": 0.9},+ {"id": "entity_2", "confidence": 0.85},+]+count = manager.track_entities_batch(entities, source="doc_1")+# Returns the number of entities successfully tracked++chunks = [+ {"id": "chunk_0", "start_index": 0, "end_index": 500},+ {"id": "chunk_1", "start_index": 500, "end_index": 1000},+]+count = manager.track_chunks_batch(chunks, source_document="doc_1")+```++### Retrieving Lineage++```python+# get_lineage returns a dict — not a ProvenanceEntry+lineage = manager.get_lineage("apple_inc")++print(lineage["entity_id"]) # "apple_inc"+print(lineage["source_documents"]) # ["annual_report.pdf"]+print(lineage["first_seen"]) # ISO timestamp string+print(lineage["last_updated"]) # ISO timestamp string+print(lineage["entity_count"]) # number of entries in chain+print(lineage["lineage_chain"]) # list of entry dicts (full history)+print(lineage["metadata"]) # merged metadata dict++# trace_lineage returns the raw ProvenanceEntry objects+entries = manager.trace_lineage("apple_inc")+for entry in entries:+ print(entry.entity_id, entry.source_document, entry.confidence)++# get_all_sources returns a list of source dicts+sources = manager.get_all_sources("apple_inc")+for s in sources:+ print(s["source"], s["location"], s["confidence"])++# get_provenance returns the most recent entry as a dict (or None)+prov = manager.get_provenance("apple_inc")+if prov:+ print(prov["source_document"])+```++### Utility Methods++```python+# Statistics about all tracked entries+stats = manager.get_statistics()+# {"total_entries": 42, "entity_types": {"entity": 30, "chunk": 12}, "unique_sources": 5}++# Clear all provenance data; returns count of cleared entries+cleared = manager.clear()+```++### ProvenanceManager Methods Reference++| Method | Returns | Description |+| ------ | ------- | ----------- |+| `track_entity(entity_id, source, metadata, **kwargs)` | `ProvenanceEntry` | Record entity provenance; checksum set automatically |+| `track_relationship(relationship_id, source, metadata, **kwargs)` | `ProvenanceEntry` | Record relationship provenance |+| `track_chunk(chunk_id, source_document, ...)` | `ProvenanceEntry` | Record chunk provenance with char offsets |+| `track_property_source(entity_id, property_name, value, source)` | `ProvenanceEntry` | Record property-level source attribution |+| `track_entities_batch(entities, source)` | `int` | Batch-track entities; returns success count |+| `track_chunks_batch(chunks, source_document)` | `int` | Batch-track chunks; returns success count |+| `get_lineage(entity_id)` | `Dict[str, Any]` | Full lineage as aggregated dict |+| `trace_lineage(entity_id)` | `List[ProvenanceEntry]` | Full lineage as raw `ProvenanceEntry` objects |+| `get_all_sources(entity_id)` | `List[Dict]` | All source documents for an entity |+| `get_provenance(entity_id)` | `Dict \| None` | Most recent provenance entry as dict |+| `get_statistics()` | `Dict[str, Any]` | Count of entries by type and unique sources |+| `clear()` | `int` | Clear all records; returns count cleared |++## ProvenanceEntry Fields++`ProvenanceEntry` is the core dataclass. Every tracking method returns one:++```python+from semantica.provenance import ProvenanceEntry++# All fields with their types and defaults+entry = ProvenanceEntry(+ entity_id="entity_001", # str — required+ entity_type="entity", # str — required (entity, chunk, relationship, property)+ activity_id="ner_extraction", # str — required+ agent_id="semantica", # str — default "semantica"+ source_document="report.pdf", # str — default ""+ source_location="Page 4", # Optional[str] — default None+ source_quote="Relevant text...", # Optional[str] — default None+ timestamp="2024-01-01T12:00:00", # str — auto-set to utcnow()+ first_seen=None, # Optional[str] — ISO timestamp+ last_updated=None, # Optional[str] — ISO timestamp+ confidence=0.9, # float — default 1.0+ checksum=None, # Optional[str] — set by compute_checksum()+ parent_entity_id=None, # Optional[str] — prov:wasDerivedFrom+ used_entities=[], # List[str] — prov:used+ start_index=None, # Optional[int] — for chunks+ end_index=None, # Optional[int] — for chunks+ credibility=None, # Optional[float] — source credibility+ metadata={}, # Dict[str, Any]+ version="1.0", # str+)-# Retrieve full lineage for any entity-entry = manager.get_lineage("apple_inc")-print(f"Source: {entry.source}")-print(f"Quote: {entry.source_quote}")-print(f"Confidence: {entry.confidence}")-print(f"Tracked at: {entry.tracked_at}")+# Convert to dict+d = entry.to_dict()++# Reconstruct from dict+entry2 = ProvenanceEntry.from_dict(d)```-## SourceReference+## SourceReference Fields-`SourceReference` provides a rich, citable pointer to the exact location in a source document:+`SourceReference` provides a citable pointer to a location within a source document:```pythonfrom semantica.provenance import SourceReferenceref = SourceReference(- document_id="annual_report_2023.pdf",- page=12,- section="3.1",- quote="Apple Inc. was incorporated on January 3, 1977.",- url="https://investor.apple.com/sec-filings/annual-reports/",- doi="10.0000/example.doi",+ document="DOI:10.1038/s41586-021-03371-z", # str — required (DOI, URL, file path)+ page=4, # Optional[int]+ section="Table S4", # Optional[str]+ line=None, # Optional[int]+ timestamp=None, # Optional[datetime]+ confidence=0.92, # float — default 1.0+ metadata={"credibility": "peer-reviewed"}, # Dict[str, Any])-manager.track_entity(- entity_id="apple_inc",- source_reference=ref,- confidence=0.98,+# Use with track_property_source+manager.track_property_source(+ entity_id="cabo_pulmo",+ property_name="biomass_increase",+ value="463%",+ source=ref,)```-## ProvenanceManager Methods+## Storage Backends-| Method | Returns | Description |-| ------ | ------- | ----------- |-| `track_entity(entity_id, source_reference, confidence)` | `str` | Record a provenance entry, returns entry ID |-| `get_lineage(entity_id)` | `ProvenanceEntry` | Retrieve full lineage for an entity |-| `export_prov_o(entity_id, format)` | `str` | Export single entity as W3C PROV-O Turtle/JSON-LD |-| `export_all(path, format)` | `None` | Export full provenance graph to file |-| `verify_checksum(entry, checksum)` | `bool` | Verify entry hasn't been tampered with |+### InMemoryStorage++Fast, no persistence. Suitable for development, tests, and short-lived processes:++```python+from semantica.provenance import InMemoryStorage, ProvenanceManager++manager = ProvenanceManager(storage=InMemoryStorage())+```++### SQLiteStorage++Persists to disk. Suitable for production, audit trails, and regulatory compliance:++```python+from semantica.provenance import SQLiteStorage, ProvenanceManager++manager = ProvenanceManager(storage=SQLiteStorage("provenance.db"))++# Or use the shorthand+manager = ProvenanceManager(storage_path="provenance.db")+```++`SQLiteStorage` creates the database and indexes automatically on first use.## Tamper-Evident Checksums-Verify that provenance records have not been modified after creation:+`compute_checksum` and `verify_checksum` are auto-used by `track_entity` and all other tracking methods. You can also call them directly:```pythonfrom semantica.provenance import compute_checksum, verify_checksum-entry = manager.get_lineage("apple_inc")+entry = manager.trace_lineage("apple_inc")[0]-# Compute and store a checksum on first write+# Recompute checksum from entry fieldschecksum = compute_checksum(entry)-# Later: verify the entry hasn’t been altered-is_valid = verify_checksum(entry, checksum)+# Verify using stored checksum (entry.checksum)+is_valid = verify_checksum(entry)++# Or verify against a separately stored expected checksum+is_valid = verify_checksum(entry, expected_checksum=checksum)+if not is_valid:- raise RuntimeError("Provenance record has been tampered with!")+ raise RuntimeError("Provenance record has been tampered with.")```-## W3C PROV-O Export+The checksum covers `entity_id`, `entity_type`, `activity_id`, `source_document`, `timestamp`, and `confidence`.++## Bridge Axiom Translation Chains-Export lineage as W3C PROV-O Turtle for compliance reporting:+`BridgeAxiom` and `TranslationChain` are available in `semantica.provenance.bridge_axiom` for tracking multi-layer domain translations with full coefficient attribution:```python-# Single entity lineage-prov_ttl = manager.export_prov_o("apple_inc", format="turtle")+from semantica.provenance.bridge_axiom import BridgeAxiom, create_translation_chain+from semantica.provenance import ProvenanceManager++manager = ProvenanceManager()++# Define a bridge axiom with DOI-backed coefficient+axiom = BridgeAxiom(+ axiom_id="BA-001",+ name="biomass_tourism_elasticity",+ rule="1% biomass increase -> 0.346% tourism revenue increase",+ coefficient=0.346,+ source_doi="10.1038/s41586-021-03371-z",+ source_page="Table S4",+ confidence=0.92,+ input_domain="ecological",+ output_domain="financial",+)-# Full provenance graph for all tracked entities-manager.export_all(path="provenance.ttl", format="turtle")+# Apply to a value with provenance tracking+result = axiom.apply(+ input_entity="cabo_pulmo_biomass",+ input_value=463.0,+ prov_manager=manager,+)+print(result["output_value"]) # 463.0 * 0.346 = 160.098-# Compliance-ready JSON-LD export-manager.export_all(path="provenance.jsonld", format="json-ld")+# Build a multi-step translation chain+input_data = {"entity_id": "cabo_pulmo", "value": 463.0, "source": "DOI:10.1371/..."}+chain = create_translation_chain(input_data, [axiom], prov_manager=manager)+print(chain.confidence) # 0.92```## Integration with GraphBuilder-`GraphBuilderWithProvenance` (from `semantica.kg`) automatically records provenance for every node and edge constructed:+`GraphBuilderWithProvenance` (from `semantica.kg`) automatically records provenance for every node and edge:```pythonfrom semantica.kg import GraphBuilderWithProvenancefrom semantica.provenance import ProvenanceManager, SQLiteStorageprov_manager = ProvenanceManager(storage=SQLiteStorage("provenance.db"))-builder = GraphBuilderWithProvenance(provenance_manager=prov_manager)-kg = builder.build_single_source(graph_data)+builder = GraphBuilderWithProvenance(provenance_manager=prov_manager)+kg = builder.build_single_source(graph_data)-# Every node and edge now has full source attribution-entry = prov_manager.get_lineage("apple_inc")-print(f"Source document: {entry.source}")-print(f"Confidence: {entry.confidence}")+# Retrieve lineage — get_lineage returns a dict+lineage = prov_manager.get_lineage("apple_inc")+print(lineage["source_documents"]) # list of source document IDs+print(lineage["first_seen"]) # ISO timestamp```-## Enable Provenance in Extractors+## Integration with NERExtractor++`NERExtractor` and other extractors accept `provenance=True` to embed provenance metadata on each extracted entity. You must track the results manually using `ProvenanceManager`:```pythonfrom semantica.semantic_extract import NERExtractorfrom semantica.provenance import ProvenanceManager-prov_manager = ProvenanceManager()+manager = ProvenanceManager()-ner = NERExtractor(method="llm", llm_provider=llm, provenance=True)-entities = ner.extract(text)+ner = NERExtractor(method="ml", provenance=True)+entities = ner.extract("Steve Jobs founded Apple Inc.")-# Retrieve lineage for the first extracted entity-entry = prov_manager.get_lineage(entities[0]["id"])-print(f"Source: {entry.source}")-```+# Track each extracted entity manually+for entity in entities:+ manager.track_entity(+ entity_id=entity.id,+ source="source_document.txt",+ confidence=entity.confidence,+ entity_type=entity.type,+ )-## Compliance Standards--Provenance tracking in Semantica is designed to satisfy:+# Now retrieve lineage+lineage = manager.get_lineage(entities[0].id)+print(lineage["source_documents"])+```-| Standard | Requirement Met |-| -------- | --------------- |-| **W3C PROV-O** | Full PROV-O compliant serialization (Turtle and JSON-LD) |-| **HIPAA** | Complete audit trail linking clinical facts to source documents |-| **SOX** | Immutable change history with timestamps and actor IDs |-| **GDPR** | Data lineage supporting right-to-erasure impact analysis |-| **FDA 21 CFR Part 11** | Electronic records with origination timestamp and extraction method |+## Common Workflows++<Tabs>+ <Tab title="Entity Tracking">+ ```python+ from semantica.provenance import ProvenanceManager++ manager = ProvenanceManager(storage_path="provenance.db")++ # Track an entity extracted from a document+ entry = manager.track_entity(+ entity_id="entity_001",+ source="report_2024.pdf",+ source_location="Page 5",+ source_quote="Revenue grew 12% year-over-year.",+ confidence=0.95,+ )+ # entry.checksum is set automatically++ # Retrieve full lineage+ lineage = manager.get_lineage("entity_001")+ print(lineage["source_documents"])+ ```+ </Tab>+ <Tab title="Chunk Tracking">+ ```python+ from semantica.provenance import ProvenanceManager++ manager = ProvenanceManager()++ # Track chunks produced by the split module+ manager.track_chunk(+ chunk_id="chunk_0001",+ source_document="report.pdf",+ start_index=0,+ end_index=512,+ )++ # Batch-track all chunks at once+ chunks = [+ {"id": "c0", "start_index": 0, "end_index": 512},+ {"id": "c1", "start_index": 512, "end_index": 1024},+ ]+ count = manager.track_chunks_batch(chunks, source_document="report.pdf")+ ```+ </Tab>+ <Tab title="Integrity Check">+ ```python+ from semantica.provenance import (+ ProvenanceManager, compute_checksum, verify_checksum+ )++ manager = ProvenanceManager(storage_path="provenance.db")+ manager.track_entity("e1", source="doc.pdf", confidence=0.9)++ entries = manager.trace_lineage("e1")+ entry = entries[0]++ # Verify the stored checksum is still valid+ if not verify_checksum(entry):+ raise RuntimeError("Provenance tampered: " + entry.entity_id)+ ```+ </Tab>+</Tabs>++## Compliance Notes++Provenance tracking in Semantica produces the following audit artifacts:++| Standard | Available |+| -------- | --------- |+| **W3C PROV-O** | Compliant data model; `to_dict()` and `from_dict()` for serialization |+| **HIPAA** | Audit trail: entity → source document → timestamp → confidence |+| **SOX** | Tamper-evident checksums; timestamps on every entry |+| **GDPR** | Lineage graph supports data erasure impact analysis |+| **FDA 21 CFR Part 11** | Electronic record with `timestamp`, `agent_id`, `activity_id`, `checksum` |++<Note>+ `ProvenanceManager` does not include built-in Turtle or JSON-LD serialization. Use `entry.to_dict()` and `get_lineage()` to retrieve provenance data, then serialize with your preferred RDF library if W3C PROV-O RDF output is required.+</Note><CardGroup cols={2}><Card title="Change Management" icon="clock-rotate-left" href="change_management">