Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

docs: add Dedupliaction v2 migration guide

ZohaibHassan16bb1ac5eb996b
diff --git a/docs/MIGRATION_V2.md b/docs/MIGRATION_V2.md
new file mode 100644
index 00000000..5bfaf8be
--- /dev/null
+++ b/docs/MIGRATION_V2.md
@@ -0,0 +1,152 @@
+## Semantica Deduplication V2: Migration & Performance Guide
+
+Welcome to the Deduplication V2 engine!! This release specifically targets severe CI delays and production bottlenecks caused by massive knowledge graph deduplication workloads. By introducing smarter candidate generation, fast-fail prefilters, and semantic triplet canonicalization, we have reduced worst-case execution times by up to **80%**.
+
+**Note:** This upgrade is **100% backward compatible.** All existing scripts, tests, and API signatures will continue to work exactly as they did before.
+
+
+
+To utilize this new addition, you must explicitly **opt-in** using the new configuration keys detailed below.
+
+---
+
+### 1. Candidate Generation V2 (Beating the $O(N^2)$ Pair Explosion)
+
+**The Problem:** The legacy engine relied on a naive first-character blocking strategy. If your dataset contained 5,000 companies starting with letter "A", the engine generated nearly 12.5 million candidate pairs.
+
+**The V2 Solution:** Multi-key token blocking, prefix matching, and deterministic candidate budgeting.
+
+
+
+**How to Opt-In**
+
+Pass the keys into the `similarity`configuration dictionary when initializing the `DuplicateDetector`:
+
+
+
+```python
+from semantica.deduplication import DuplicateDetector
+
+detector = DuplicateDetector(
+ similarity_threshold=0.8,
+ similarity = {
+ # Switches from legacy to v2
+ "candidate_strategy": "blocking_v2",
+
+ # Highly recommended: Limits the max number of comparisons
+ # per entity to prevent adversarial latency spikes.
+ "max_candidates_per_entity": 50,
+
+ # Optional: Generates blocks using Soundex algorithm to catch
+ # phonetic misspellings (e.g, "Jon" vs "John")
+ "enable_phonetic_blocking": True
+}
+)
+```
+
+
+
+### 2. Two-Stage scoring (The Fast Prefilter)
+
+**The Problem**: Calculating multi-factor semantic scores (Levenshtein, Jaro-Winkler, property intersections, and Embeddings) is computationally expensive. Running these
+
+calculations on two entities that share absolutely zero words or have vastly different string lengths is a waste of resources.
+
+**The V2 Solution:** A lightning-fast prefilter gate that instantly drops obvious non-matches before they ever reach the heavy semantic scorers.
+
+
+
+**How to Opt-In**
+
+Enable the prefilter and define your rejection thresholds:
+
+
+
+```python
+from semantica.deduplication import DuplicateDetector
+
+detector = DuplicateDetector(
+ similarity_threshold=0.8,
+ similarity={
+ "candidate_strategy": "blocking_v2",
+
+ # Enable prefilter
+ "prefilter_enabled": True,
+
+ "prefilter_thresholds": {
+ # Rejects pairs if shortest string is less than 40% the length
+ # of the longest
+ "min_length_ratio": 0.4,
+
+ # Instantly rejects pairs if they don't share at least one
+ # valid word token
+ "required_shared_token": True
+},
+ # Optional Explainability: Injects a 'score_breakdown' dict into
+ # the candidate metadata so you can see exactly how the string,
+ # property, and relationships scores contributed.
+
+ "score_breakdown_enabled": True
+}
+)
+```
+
+
+
+### 3. Semantic Relationship & Triplet Deduplication
+
+**The problem:** The legacy relationship deduplication relied on exact `(Subject, Predicate, Object)` string matches. It couldn't recognize that `(Person, "works_for", Company)` is semantically identical to `(Person, "employed_by", Company)` .
+
+**The V2 Solution:** A new `semantic_v2` mode that introduces predicate synonym mapping, literal normalization (cleaning up rogue spaces/casing), and a highly optimized $O(1)$ canonical hash path for fast matching.
+
+
+
+**How to Opt-In**
+
+When calling relationship-specific dedup methods, pass the new configuration keys:
+
+```python
+from semantica.deduplication import DuplicateDetector
+from semantica.deduplication.methods import dedup_triplets
+
+
+# Approach A: Using the Detector explicitly
+detector = DuplicateDetector()
+duplicates = detector.detect_relationship_duplicates(
+ relationship_list,
+ relationship_dedup_mode="semantic_v2",
+
+ # Cleans up messy object strings
+ # (e.g., " Apple Inc. " -> "apple inc.")
+ literal_normalization_enabled=True,
+
+ # Maps various synonyms to a single canonical predicate
+ # before hashing
+ predicate_synonym_map={
+ "works_for": "employed_by",
+ "is_employee_of": "employed_by",
+ "has_employer": "employed_by"
+}
+)
+
+
+# Approach B: Using the new simplified wrapper in methods.py
+duplicates = dedup_triplets(
+ relationships_list,
+ mode="semantic_v2",
+ literal_normalization_enabled=True,
+ predicate_synonym_map={"works_for": "employed_by"}
+)
+```
+
+
+
+###### Note on Merge Strategies
+
+When using `semantic_v2` for relationships, the `MergeStrategyManager` will now automatically respect your canonicalized keys. If two entities share a relationship that differs only by a mapped synonym, the engine will correctly identify them as the same relationship and prevent duplicate graph edges during the merge phase.
+
+
+
+### Need Help?
+
+If you experience any unexpected behavior when switching from `legacy` to `blocking_v2` or `semantic_v2`, please check the explainability metadata (by setting `"score_breakdown_enabled": True`) to audit the exact scoring process, or open an issue on GitHub.
\ No newline at end of file