Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

Merge branch 'main' into feat/incremental-delta-processing

Mohd Kaif1405f85d62be
diff --cc CHANGELOG.md
index 96537c22,1d9490cd..9df11248
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,19 -7,117 +7,130 @@@ and this project adheres to [Semantic V
## [Unreleased]
+- **Incremental/Delta Processing Feature** (PR #349 by @ZohaibHassan16, reviewed and fixed by @KaifAhmad1):
+ - Native delta computation between graph snapshots using SPARQL queries
+ - Delta-aware pipeline execution with `delta_mode` configuration for processing only changed data
+ - Version snapshot management with graph URI tracking and metadata storage
+ - Snapshot retention policies with automatic cleanup via `prune_versions()` method
+ - Integration with pipeline execution engine for incremental workflows
+ - Significant performance improvements: processes only changes instead of full datasets
+ - Cost optimization: dramatically reduces compute and storage requirements for large-scale operations
+ - Production-ready for near real-time pipelines and frequent deployment scenarios
+ - Bug fixes: corrected SPARQL variable order, fixed class references, resolved duplicate dictionary keys
+ - Comprehensive test coverage including delta mode integration tests
+ - Complete documentation with usage examples and API references
+ - Essential for enterprise-grade, large-scale semantic infrastructure
+ - **Deduplication v2 Migration Guide** (PR #344 by @ZohaibHassan16, fixes by @KaifAhmad1):
+ - Added comprehensive MIGRATION_V2.md documentation for Deduplication v2 Epic #333
+ - Documented Candidate Generation V2 with multi-key blocking and phonetic matching
+ - Documented Two-Stage Scoring prefilter with configurable thresholds
+ - Documented Semantic Relationship Deduplication v2 with synonym mapping
+ - Added practical code examples for all V2 features with opt-in configuration
+ - Fixed critical infinite recursion bug in dedup_triplets() function
+ - Completed Epic #333 with comprehensive migration path and documentation
+ - Performance: 5.86x speedup confirmed (129ms vs 754ms) for semantic deduplication
+ - Full backward compatibility maintained with legacy mode as default
+ - **Semantic Relationship Deduplication v2** (PR #340 by @ZohaibHassan16, fixes by @KaifAhmad1):
+ - Implemented opt-in semantic relationship deduplication mode (`semantic_v2`) with 6.98x performance improvement
+ - Added canonicalization engine with predicate synonym mapping (`works_for` → `employed_by`)
+ - Implemented fast-path O(1) hash matching for exact canonical signature comparisons
+ - Added weighted semantic scoring (60% predicate + 40% object composition) with explainable `semantic_match_score` metadata
+ - Enhanced `dedup_triplets()` function as first-class API in `methods.py`
+ - Integrated semantic deduplication into merge strategy with canonical key generation
+ - Added literal normalization for whitespace cleanup in object matching
+ - Maintained full backward compatibility with legacy mode as default
+ - Fixed critical infinite recursion bug in `dedup_triplets()` function via registry name checking
+ - Performance: Semantic V2 (~83ms) vs Legacy (~579ms) - 6.98x speedup confirmed
+ - All 13 deduplication benchmarks passing with comprehensive test coverage
+ - **Two-Stage Scoring Prefilter** (PR #339 by @ZohaibHassan16):
+ - Implemented opt-in two-stage scoring with fast prefilter gates to eliminate expensive semantic scoring for obvious non-matches
+ - Prefilter gates: type mismatch detection, name length ratio validation, token overlap requirements
+ - Performance improvements: 18-25% faster batch processing with prefilter enabled
+ - Configurable thresholds: `min_length_ratio`, `min_token_overlap_ratio`, `required_shared_token`
+ - Enhanced explainability with score breakdown and rejection reasons in metadata
+ - Complete backward compatibility with default `prefilter_enabled=False`
+
+ - **Candidate Generation v2 with Multi-Key Blocking** (PR #338 by @ZohaibHassan16):
+ - Implemented opt-in candidate generation strategies (`legacy`, `blocking_v2`, `hybrid_v2`) to address O(N²) pair explosion during deduplication
+ - Multi-key blocking with normalized token prefixes, type-aware keys, and optional phonetic (Soundex) blocking
+ - Deterministic candidate budgeting with `max_candidates_per_entity` limit using stable sorting
+ - Efficient pair generation with set-based deduplication across overlapping blocks
+ - Performance improvements: 63.6% faster in worst-case scenarios (0.259s → 0.094s for 100 entities)
+ - Complete backward compatibility with default `candidate_strategy="legacy"`
+ - Added configuration options: `blocking_keys`, `enable_phonetic_blocking`, `max_candidates_per_entity`
+
+ - **ArangoDB AQL Export Support** (PR #342 by @tibisabau):
+ ### Added
+
+ - **ArangoDB AQL Export Support** (PR #342 by @tibisabau)
+ - Full-featured ArangoDB AQL exporter with 642 lines of production-ready code
+ - Comprehensive AQL INSERT statement generation for vertices and edges
+ - Configurable collection names with validation and sanitization
+ - Batch processing support for large knowledge graphs (default: 1000)
+ - Added export_arango() convenience function for easy access
+ - Enhanced unified export with AQL format support and .aql auto-detection
+ - Added `export_arango()` convenience function for easy access
+ - Enhanced unified export with AQL format support and `.aql` auto-detection
+ - Integrated with method registry for extensibility
+ - 17 comprehensive test cases with 100% pass rate
+ - Enterprise-grade ArangoDB multi-model database integration
+
+ - **Apache Parquet Export Support** (PR #343 by @tibisabau):
+ - **Apache Parquet Export Support** (PR #343 by @tibisabau)
+ - Full-featured Apache Parquet exporter with 701 lines of production-ready code
+ - Columnar storage format optimized for analytics and data warehousing
+ - Configurable compression codecs (snappy, gzip, brotli, zstd, lz4, none)
+ - Explicit Arrow schemas with type safety and consistency
+ - Field normalization for varied entity and relationship naming conventions
+ - Structured metadata handling using Parquet struct fields
+ - Added export_parquet() convenience function for easy access
+ - Enhanced unified export with Parquet format support and .parquet auto-detection
+ - Added `export_parquet()` convenience function for easy access
+ - Enhanced unified export with Parquet format support and `.parquet` auto-detection
+ - Integrated with method registry for extensibility
+ - 25 comprehensive test cases with 100% pass rate
+ - Enterprise-grade analytics integration with pandas, Spark, Snowflake, BigQuery, Databricks
+
+ ### Fixed
+ - **Fixed NameError**: missing Type import in utils/helpers.py
+
+ - Fixed NameError: missing Type import in utils/helpers.py
+ - Added Type to typing imports to fix retry_on_error decorator
+ - Removed unused Type import from config_manager.py
+ - Resolves ImportError when importing semantica modules
+ - Fixes capability gap analysis notebook execution
+
+ ## [0.3.0-alpha] - 2026-02-19
+
+ ### Added / Changed
+
+ - **Decision Tracking System**: Complete decision lifecycle management with audit trails and provenance tracking
+ - **Advanced KG Algorithms**: Node2Vec embeddings, centrality analysis, community detection for decision insights
+ - **Enhanced Context Module**: Unified AgentContext with granular feature flags and decision tracking integration
+ - **Vector Store Features**: Hybrid search combining semantic, structural, and category similarity
+ - **Policy Management**: Versioning, compliance checking, and exception handling
+ - **Production Ready Architecture**: Scalable design with comprehensive error handling and validation
+
+ ### Fixed
+
+ - Fixed import issues in test suite (ProvenanceTracker location fixes)
+ - Fixed causal analyzer validation (max_depth bounds checking)
+ - Fixed test compatibility with updated method signatures
+ - Fixed mock object setup in test suites
+ - Comprehensive test suite fixes for decision tracking features
+
+ ### Testing
+
+ - 113+ tests passing across context and core modules
+ - Comprehensive decision tracking test coverage
+ - Enhanced error handling and edge case testing
+ - Fixed all critical test failures for release readiness
+
+ ### Documentation
+
+ - Enhanced context module documentation
+ - Updated API references for decision tracking features
+ - Comprehensive usage guides and examples
- Fixed: Context Graphs decision tracking bugs and added comprehensive test coverage (PR #315 by @KaifAhmad1)
- Fixed empty/None decision ID handling in ContextGraph.add_decision()