Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
docs: add Dedupliaction v2 migration guide
ZohaibHassan16bb1ac5eb996b
diff --git a/docs/MIGRATION_V2.md b/docs/MIGRATION_V2.mdnew file mode 100644index 00000000..5bfaf8be--- /dev/null+++ b/docs/MIGRATION_V2.md@@ -0,0 +1,152 @@+## Semantica Deduplication V2: Migration & Performance Guide++Welcome to the Deduplication V2 engine!! This release specifically targets severe CI delays and production bottlenecks caused by massive knowledge graph deduplication workloads. By introducing smarter candidate generation, fast-fail prefilters, and semantic triplet canonicalization, we have reduced worst-case execution times by up to **80%**.++**Note:** This upgrade is **100% backward compatible.** All existing scripts, tests, and API signatures will continue to work exactly as they did before.++++To utilize this new addition, you must explicitly **opt-in** using the new configuration keys detailed below.++---++### 1. Candidate Generation V2 (Beating the $O(N^2)$ Pair Explosion)++**The Problem:** The legacy engine relied on a naive first-character blocking strategy. If your dataset contained 5,000 companies starting with letter "A", the engine generated nearly 12.5 million candidate pairs.++**The V2 Solution:** Multi-key token blocking, prefix matching, and deterministic candidate budgeting.++++**How to Opt-In**++Pass the keys into the `similarity`configuration dictionary when initializing the `DuplicateDetector`:++++```python+from semantica.deduplication import DuplicateDetector++detector = DuplicateDetector(+ similarity_threshold=0.8,+ similarity = {+ # Switches from legacy to v2+ "candidate_strategy": "blocking_v2",++ # Highly recommended: Limits the max number of comparisons+ # per entity to prevent adversarial latency spikes.+ "max_candidates_per_entity": 50,++ # Optional: Generates blocks using Soundex algorithm to catch+ # phonetic misspellings (e.g, "Jon" vs "John")+ "enable_phonetic_blocking": True+}+)+```++++### 2. Two-Stage scoring (The Fast Prefilter)++**The Problem**: Calculating multi-factor semantic scores (Levenshtein, Jaro-Winkler, property intersections, and Embeddings) is computationally expensive. Running these++calculations on two entities that share absolutely zero words or have vastly different string lengths is a waste of resources.++**The V2 Solution:** A lightning-fast prefilter gate that instantly drops obvious non-matches before they ever reach the heavy semantic scorers.++++**How to Opt-In**++Enable the prefilter and define your rejection thresholds:++++```python+from semantica.deduplication import DuplicateDetector++detector = DuplicateDetector(+ similarity_threshold=0.8,+ similarity={+ "candidate_strategy": "blocking_v2",++ # Enable prefilter+ "prefilter_enabled": True,++ "prefilter_thresholds": {+ # Rejects pairs if shortest string is less than 40% the length+ # of the longest+ "min_length_ratio": 0.4,++ # Instantly rejects pairs if they don't share at least one+ # valid word token+ "required_shared_token": True+},+ # Optional Explainability: Injects a 'score_breakdown' dict into+ # the candidate metadata so you can see exactly how the string,+ # property, and relationships scores contributed.++ "score_breakdown_enabled": True+}+)+```++++### 3. Semantic Relationship & Triplet Deduplication++**The problem:** The legacy relationship deduplication relied on exact `(Subject, Predicate, Object)` string matches. It couldn't recognize that `(Person, "works_for", Company)` is semantically identical to `(Person, "employed_by", Company)` .++**The V2 Solution:** A new `semantic_v2` mode that introduces predicate synonym mapping, literal normalization (cleaning up rogue spaces/casing), and a highly optimized $O(1)$ canonical hash path for fast matching.++++**How to Opt-In**++When calling relationship-specific dedup methods, pass the new configuration keys:++```python+from semantica.deduplication import DuplicateDetector+from semantica.deduplication.methods import dedup_triplets+++# Approach A: Using the Detector explicitly+detector = DuplicateDetector()+duplicates = detector.detect_relationship_duplicates(+ relationship_list,+ relationship_dedup_mode="semantic_v2",++ # Cleans up messy object strings+ # (e.g., " Apple Inc. " -> "apple inc.")+ literal_normalization_enabled=True,++ # Maps various synonyms to a single canonical predicate+ # before hashing+ predicate_synonym_map={+ "works_for": "employed_by",+ "is_employee_of": "employed_by",+ "has_employer": "employed_by"+}+)+++# Approach B: Using the new simplified wrapper in methods.py+duplicates = dedup_triplets(+ relationships_list,+ mode="semantic_v2",+ literal_normalization_enabled=True,+ predicate_synonym_map={"works_for": "employed_by"}+)+```++++###### Note on Merge Strategies++When using `semantic_v2` for relationships, the `MergeStrategyManager` will now automatically respect your canonicalized keys. If two entities share a relationship that differs only by a mapped synonym, the engine will correctly identify them as the same relationship and prevent duplicate graph edges during the merge phase.++++### Need Help?++If you experience any unexpected behavior when switching from `legacy` to `blocking_v2` or `semantic_v2`, please check the explainability metadata (by setting `"score_breakdown_enabled": True`) to audit the exact scoring process, or open an issue on GitHub.\ No newline at end of file