Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
docs(deduplication): align entity resolution APIs with implementation
Sameer63054275d6ee34a4
diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.mdindex 79e420e5..4d47d464 100644--- a/docs/reference/deduplication.md+++ b/docs/reference/deduplication.md@@ -1,114 +1,212 @@---title: "Deduplication Module"-description: "Entity deduplication v1/v2 — similarity scoring, blocking, merging, and cluster-based batch processing."+description: "Entity deduplication — similarity scoring, blocking, merging, and cluster-based batch processing."icon: "copy"----`semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. **v2 strategies** (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to **7x faster** than v1 with fine-grained result control.+`semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. All deduplication workflows operate on plain Python dicts with `id`, `name`, `type`, `properties`, and `relationships` keys.## Exported Classes| Class | Role || --- | --- |-| `DuplicateDetector` | Pairwise and batch detection with configurable strategy and threshold |-| `EntityMerger` | Merge duplicate groups with per-property merge policies (`KEEP_FIRST`, `UNION`, `VOTING`, ...) |-| `SimilarityCalculator` | Levenshtein, Jaro-Winkler, cosine, Jaccard, and embedding similarity |+| `DuplicateDetector` | Pairwise and batch detection — returns `DuplicateCandidate` or `DuplicateGroup` lists |+| `EntityMerger` | Merge duplicate groups — returns `List[MergeOperation]` |+| `SimilarityCalculator` | Multi-factor similarity: string, property, relationship, and embedding || `ClusterBuilder` | Union-Find and hierarchical clustering for large-scale batch deduplication |-| `PropertyMergeRule` | Enum of merge policies used by `EntityMerger` |-| `MergeStrategyManager` | Manage and apply named merge strategies across entity types |-| `detect_duplicates()` | Quick function: `detect_duplicates(entities, method="semantic_v2")` |-| `merge_entities()` | Quick function: `merge_entities(entities, duplicates, method="union")` |--**Available `method=` values:**--| Method | Speed | Notes |-| --- | --- | --- |-| `exact` | Fastest | Exact string match only |-| `fuzzy` | Fast | Levenshtein + Jaro-Winkler |-| `semantic` | Moderate | Embedding cosine similarity |-| `blocking_v2` | Fast | Sorted neighborhood blocking (v2) |-| `hybrid_v2` | Balanced | Blocking + multi-feature scoring (v2) |-| `semantic_v2` | Best accuracy | Embedding + structural features, up to 7× faster than v1 |+| `MergeStrategy` | Enum of merge strategies: `KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL` |+| `PropertyMergeRule` | Dataclass holding per-property merge rule: `{property_name, strategy, conflict_resolution, priority}` |+| `MergeStrategyManager` | Manage and apply named merge strategies; accepts per-property rules |+| `detect_duplicates()` | Convenience function — `detect_duplicates(entities, method="pairwise", similarity_threshold=0.7)` |+| `merge_entities()` | Convenience function — `merge_entities(entities, method="keep_most_complete")` |+| `calculate_similarity()` | Convenience function — `calculate_similarity(entity_a, entity_b, method="multi_factor")` |++## Getting Started++```python+from semantica.deduplication import DuplicateDetector, EntityMerger++entities = [+ {"id": "1", "name": "Apple Inc.", "type": "Company"},+ {"id": "2", "name": "Apple", "type": "Company"},+ {"id": "3", "name": "Microsoft", "type": "Company"},+]++# 1. Detect duplicates — returns List[DuplicateCandidate]+detector = DuplicateDetector(similarity_threshold=0.7)+candidates = detector.detect_duplicates(entities)++for dup in candidates:+ print(+ "{} vs {} — sim: {:.2f}, confidence: {:.2f}".format(+ dup.entity1.get("name"),+ dup.entity2.get("name"),+ dup.similarity_score,+ dup.confidence,+ )+ )++# 2. Merge duplicates — returns List[MergeOperation]+merger = EntityMerger()+operations = merger.merge_duplicates(entities, strategy="keep_most_complete")++for op in operations:+ print("Merged {} entities → {}".format(+ len(op.source_entities), op.merged_entity.get("name")+ ))+```## DuplicateDetector-Find duplicate entity pairs with configurable strategies and result filtering:+Find duplicate entity pairs:```pythonfrom semantica.deduplication import DuplicateDetector-detector = DuplicateDetector(similarity_threshold=0.85)-duplicates = detector.detect_duplicates(entities)+detector = DuplicateDetector(+ similarity_threshold=0.7, # default 0.7 — minimum score to include a candidate+ confidence_threshold=0.6, # default 0.6 — minimum confidence to include a candidate+ max_results=100, # optional — hard cap on total candidates returned+ top_k_per_entity=3, # optional — max candidates per entity+ min_similarity=0.75, # optional — additional floor applied after sorting+ sort_by="confidence", # "confidence" (default) | "similarity_score"+)-for dup in duplicates:- print(f"{dup.entity_a} ≈ {dup.entity_b} ({dup.similarity:.2f})")+# Returns List[DuplicateCandidate]+candidates = detector.detect_duplicates(entities)++for c in candidates:+ print(c.entity1.get("name"), "vs", c.entity2.get("name"))+ print(" similarity_score:", c.similarity_score)+ print(" confidence: ", c.confidence)+ print(" reasons: ", c.reasons)++# Detect duplicate groups with union-find (returns List[DuplicateGroup])+groups = detector.detect_duplicate_groups(entities)+for g in groups:+ print("Group of {} — confidence: {:.2f} — representative: {}".format(+ len(g.entities), g.confidence, g.representative and g.representative.get("name")+ ))++# Incremental — compare new entities against existing (returns List[DuplicateCandidate])+new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}]+candidates = detector.incremental_detect(new_entities, entities)```-Fine-grained control over strategy, thresholds, and result size:+### `detect_duplicates()` detection methods-```python-duplicates = detector.detect_duplicates(- entities,- strategy="semantic_v2", # see strategies table below- min_similarity=0.85, # minimum score to consider a match- top_k_per_entity=3, # max candidates per entity- max_results=100, # total result cap- sort_by="similarity", # "similarity" | "entity_id" | "cluster_size"-)-```+The `method=` parameter of the `detect_duplicates()` convenience function controls how+the comparison is performed. These are independent of the `SimilarityCalculator` string+method used internally:-### Detection Strategies+| `method=` | Algorithm | Returns |+| --------- | --------- | ------- |+| `"pairwise"` (default) | O(n²) all-pairs comparison | `List[DuplicateCandidate]` |+| `"batch"` | Batch similarity calculation | `List[DuplicateCandidate]` |+| `"incremental"` | New vs existing entities | `List[DuplicateCandidate]` |+| `"group"` | Union-find group formation | `List[DuplicateGroup]` |-| Strategy | Algorithm | Speed | Accuracy |-| -------- | --------- | ----- | -------- |-| `jaro_winkler` | String similarity (v1) | Fast | Medium |-| `blocking_v2` | Blocking + Jaro-Winkler (v2) | Very fast | Medium |-| `hybrid_v2` | Blocking + semantic + string (v2) | Fast | High |-| `semantic_v2` | Embedding similarity (v2) | Medium | Highest |+### DuplicateCandidate fields-<Note>- **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions.-</Note>+| Field | Type | Description |+| ----- | ---- | ----------- |+| `entity1` | `Dict` | First entity |+| `entity2` | `Dict` | Second entity |+| `similarity_score` | `float` | Similarity score (0–1) |+| `confidence` | `float` | Confidence score (0–1) |+| `reasons` | `List[str]` | Why they are considered duplicates |+| `metadata` | `Dict` | Additional metadata |++### DuplicateGroup fields++| Field | Type | Description |+| ----- | ---- | ----------- |+| `entities` | `List[Dict]` | All entities in the group |+| `similarity_scores` | `Dict` | Pair → score mapping |+| `representative` | `Optional[Dict]` | Most complete entity in group |+| `confidence` | `float` | Group confidence score |+| `metadata` | `Dict` | Additional metadata |## EntityMerger-Merges detected duplicate groups into canonical entities, preserving provenance:+Merges detected duplicate groups into canonical entities:```python-from semantica.deduplication import EntityMerger--merger = EntityMerger()-merged_entities = merger.merge_duplicates(- entities,- strategy="keep_most_complete", # see strategies table below- preserve_provenance=True, # keep source references after merge-)+from semantica.deduplication import EntityMerger, MergeStrategy++# Basic usage+merger = EntityMerger(preserve_provenance=True)+operations = merger.merge_duplicates(entities, strategy="keep_most_complete")++# operations is List[MergeOperation]+for op in operations:+ merged = op.merged_entity # the resulting merged entity dict+ sources = op.source_entities # list of original entities that were merged+ conflicts = op.merge_result.conflicts # unresolved conflicts, if any++# Merge a known group directly (no duplicate detection)+pair = [+ {"id": "1", "name": "Apple Inc.", "type": "Company"},+ {"id": "2", "name": "Apple", "type": "Company"},+]+op = merger.merge_entity_group(pair, strategy="keep_most_complete")+print(op.merged_entity)++# Retrieve full merge history+history = merger.get_merge_history()+print("Total merges performed:", len(history))```-### Merge Strategies+### Merge strategies++Pass as a string to `strategy=` on `merge_duplicates()` or `merge_entity_group()`:| Strategy | Behavior || -------- | -------- |-| `keep_first` | Keep the first entity in each duplicate group |-| `keep_last` | Keep the most recently seen entity |-| `keep_most_complete` | Keep the entity with the most non-null properties |-| `union` | Merge all properties — non-conflicting fields combined |-| `voting` | Most common property value wins |+| `"keep_first"` | Keep the first entity in each duplicate group |+| `"keep_last"` | Keep the most recently seen entity |+| `"keep_most_complete"` | Keep the entity with the most non-null properties + relationships |+| `"keep_highest_confidence"` | Keep the entity with the highest `.confidence` value |+| `"merge_all"` | Combine all properties — conflicts resolved to lists |++### Per-property merge rules-Fine-grained per-property merge rules:+Per-property rules are set on `EntityMerger.merge_strategy_manager` using+`add_property_rule()`. Rules take a `MergeStrategy` enum value:```python-from semantica.deduplication import EntityMerger, PropertyMergeRule--merger = EntityMerger(- property_rules={- "name": PropertyMergeRule.KEEP_FIRST,- "aliases": PropertyMergeRule.UNION,- "description": PropertyMergeRule.KEEP_LONGEST,- }+from semantica.deduplication import EntityMerger, MergeStrategy++merger = EntityMerger()++# Add per-property rules+merger.merge_strategy_manager.add_property_rule(+ "name", MergeStrategy.KEEP_FIRST+)+merger.merge_strategy_manager.add_property_rule(+ "aliases", MergeStrategy.MERGE_ALL+)++# Custom conflict resolution function+def keep_longest(val1, val2):+ return val1 if len(str(val1)) >= len(str(val2)) else val2++merger.merge_strategy_manager.add_property_rule(+ "description", MergeStrategy.KEEP_FIRST, conflict_resolution=keep_longest)++operations = merger.merge_duplicates(entities)```+### MergeOperation fields++| Field | Type | Description |+| ----- | ---- | ----------- |+| `source_entities` | `List[Dict]` | Original entities that were merged |+| `merged_entity` | `Dict` | Resulting merged entity |+| `merge_result` | `MergeResult` | Detailed result with conflicts |+| `metadata` | `Dict` | Group confidence, similarity scores, strategy used |+## SimilarityCalculatorCompute multi-factor similarity scores between entity pairs:@@ -116,25 +214,44 @@ Compute multi-factor similarity scores between entity pairs:```pythonfrom semantica.deduplication import SimilarityCalculator-calc = SimilarityCalculator()--score = calc.calculate_similarity(entity_a, entity_b)-# → SimilarityResult(score=0.91, components={...})+calc = SimilarityCalculator(+ string_weight=0.6, # default 0.6+ property_weight=0.2, # default 0.2+ relationship_weight=0.2, # default 0.2+ embedding_weight=0.0, # default 0.0 (used only when "embedding" key is present)+ similarity_threshold=0.7,+)-print(score.score) # overall score 0.0–1.0-print(score.components["label"]) # label similarity-print(score.components["embedding"]) # semantic similarity-print(score.components["property"]) # property overlap+result = calc.calculate_similarity(entity_a, entity_b)+# result is a SimilarityResult+print(result.score) # overall score 0.0–1.0+print(result.method) # e.g. "multi_factor"+print(result.components["string"]) # string similarity component+print(result.components["property"]) # property overlap component+print(result.components["relationship"]) # relationship jaccard component+# result.components["embedding"] is present only when embeddings are supplied++# String similarity methods — method= accepts "levenshtein", "jaro_winkler", "cosine"+lev = calc.calculate_string_similarity("Apple Inc.", "Apple Inc", method="levenshtein")+jaro = calc.calculate_string_similarity("Steve Jobs", "Steven Jobs", method="jaro_winkler")+cos = calc.calculate_string_similarity("apple", "apples", method="cosine")++# Embedding cosine similarity (vector inputs)+emb_score = calc.calculate_embedding_similarity(embedding_a, embedding_b)++# Property and relationship similarity (entity dict inputs)+prop_score = calc.calculate_property_similarity(entity_a, entity_b)+rel_score = calc.calculate_relationship_similarity(entity_a, entity_b)```-Individual string and vector metrics:+### SimilarityResult fields-```python-lev = calc.levenshtein("Apple Inc.", "Apple Inc")-jaro = calc.jaro_winkler("Steve Jobs", "Steven Jobs")-cos = calc.cosine_similarity(embedding_a, embedding_b)-jacc = calc.jaccard({"founded", "tech"}, {"founded", "technology"})-```+| Field | Type | Description |+| ----- | ---- | ----------- |+| `score` | `float` | Overall weighted similarity score (0–1) |+| `method` | `str` | Method used (e.g. `"multi_factor"`, `"levenshtein"`) |+| `components` | `Dict[str, float]` | Per-component scores: `"string"`, `"property"`, `"relationship"`, `"embedding"` |+| `metadata` | `Dict` | Weights used and optional score breakdown |## ClusterBuilder@@ -143,56 +260,167 @@ Build entity clusters for large-scale batch deduplication:```pythonfrom semantica.deduplication import ClusterBuilder-builder = ClusterBuilder(algorithm="union_find") # or "hierarchical"-result = builder.build_clusters(entities, similarity_threshold=0.85)+builder = ClusterBuilder(+ similarity_threshold=0.8, # minimum similarity to be in same cluster+ min_cluster_size=2, # minimum entities per valid cluster+ max_cluster_size=100, # maximum entities per cluster+ use_hierarchical=False, # True for hierarchical, False (default) for union-find+)+result = builder.build_clusters(entities)-print(f"Clusters: {len(result.clusters)}")+print("Clusters found:", len(result.clusters))for cluster in result.clusters:- print(f" [{cluster.id}] {cluster.members} — cohesion: {cluster.cohesion:.2f}")+ print(" [{}] {} entities — quality: {:.2f}".format(+ cluster.cluster_id,+ len(cluster.entities),+ cluster.quality_score,+ ))++print("Unclustered: ", len(result.unclustered))+print("Quality metrics:", result.quality_metrics)+# {"average_size": ..., "average_quality": ..., "total_clusters": ..., "high_quality_clusters": ...}```-## Blocking Strategies+### Cluster fields++| Field | Type | Description |+| ----- | ---- | ----------- |+| `cluster_id` | `str` | Unique cluster identifier |+| `entities` | `List[Dict]` | Entities in the cluster |+| `centroid` | `Optional[Dict]` | Representative entity (optional) |+| `quality_score` | `float` | Intra-cluster average similarity |+| `metadata` | `Dict` | Similarity scores and other metadata |-Blocking reduces the O(n²) pairwise comparison problem to a manageable candidate set:+## Convenience Functions```python-detector = DuplicateDetector(- blocking_strategy="token", # "token" | "phonetic" | "ngram"- blocking_threshold=0.6,- similarity_threshold=0.85+from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity++# Detect — method= accepts "pairwise" (default), "batch", "incremental", "group"+candidates = detect_duplicates(+ entities,+ method="pairwise",+ similarity_threshold=0.8,+ confidence_threshold=0.6,)++# Merge — method= accepts the strategy strings, same as EntityMerger+operations = merge_entities(entities, method="keep_most_complete", preserve_provenance=True)+# Returns List[MergeOperation]; access .merged_entity on each++# Similarity — method= accepts "exact", "levenshtein", "jaro_winkler", "cosine",+# "property", "relationship", "embedding", "multi_factor" (default)+result = calculate_similarity(entity_a, entity_b, method="multi_factor")+print(result.score)```## Custom Similarity Functions-Register domain-specific similarity logic:+Register domain-specific similarity logic and use it via the method registry:```python-from semantica.deduplication import method_registry+from semantica.deduplication import method_registry, SimilarityResult-def drug_name_similarity(entity_a, entity_b):- # Match drug names by active compound- return score # 0.0 to 1.0+def drug_name_similarity(entity_a, entity_b, **kwargs):+ """Match drug names by active compound prefix."""+ name_a = entity_a.get("name", "").lower()+ name_b = entity_b.get("name", "").lower()+ score = 1.0 if name_a[:5] == name_b[:5] else 0.0+ return SimilarityResult(score=score, method="drug_name")method_registry.register("similarity", "drug_name", drug_name_similarity)-detector = DuplicateDetector(similarity_method="drug_name")+# Now callable via calculate_similarity+from semantica.deduplication import calculate_similarity+result = calculate_similarity(entity_a, entity_b, method="drug_name")```-## Convenience Functions--```python-from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity--# Quick detection-duplicates = detect_duplicates(entities, method="semantic_v2", similarity_threshold=0.85)--# Quick merge-merged = merge_entities(entities, duplicates, method="keep_most_complete")--# Quick similarity-score = calculate_similarity(entity_a, entity_b, method="hybrid_v2")-```+## Common Workflows++<Tabs>+ <Tab title="Basic Deduplication">+ ```python+ from semantica.deduplication import DuplicateDetector, EntityMerger++ detector = DuplicateDetector(similarity_threshold=0.8)+ candidates = detector.detect_duplicates(entities)++ print("Duplicate pairs found:", len(candidates))++ merger = EntityMerger(preserve_provenance=True)+ operations = merger.merge_duplicates(entities, strategy="keep_most_complete")++ merged_entities = [op.merged_entity for op in operations]+ ```+ </Tab>+ <Tab title="Group-based Batch">+ ```python+ from semantica.deduplication import DuplicateDetector, EntityMerger++ detector = DuplicateDetector(similarity_threshold=0.75)+ # detect_duplicate_groups uses union-find internally+ groups = detector.detect_duplicate_groups(entities)++ merger = EntityMerger()+ for group in groups:+ op = merger.merge_entity_group(group.entities, strategy="keep_most_complete")+ print("Merged into:", op.merged_entity.get("name"))+ ```+ </Tab>+ <Tab title="Large-scale Clustering">+ ```python+ from semantica.deduplication import ClusterBuilder, EntityMerger++ # Build clusters first — more efficient for large entity sets+ builder = ClusterBuilder(similarity_threshold=0.8, min_cluster_size=2)+ result = builder.build_clusters(entities)++ merger = EntityMerger()+ for cluster in result.clusters:+ op = merger.merge_entity_group(cluster.entities, strategy="keep_most_complete")+ print("Cluster merged into:", op.merged_entity.get("name"))+ ```+ </Tab>+ <Tab title="Incremental">+ ```python+ from semantica.deduplication import EntityMerger++ existing_entities = [...] # already in the graph+ new_entities = [...] # arriving in a batch++ merger = EntityMerger()+ operations = merger.incremental_merge(new_entities, existing_entities)++ print("New merges performed:", len(operations))+ ```+ </Tab>+</Tabs>++## Tips and Common Pitfalls++<Warning>+ **`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score` — not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`.+</Warning>++<Warning>+ **`merge_entities()` and `EntityMerger.merge_duplicates()` return `List[MergeOperation]`, not a list of entity dicts.** Access `.merged_entity` on each operation to get the merged dict.+</Warning>++<Warning>+ **`PropertyMergeRule` is a dataclass, not an Enum.** The merge strategy Enum is `MergeStrategy` (`KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL`). Per-property rules are added via `merger.merge_strategy_manager.add_property_rule(name, strategy)`.+</Warning>++<Tip>+ **Tune `similarity_threshold` before `confidence_threshold`.** The similarity threshold gates which entity pairs are even considered. The confidence threshold further filters those pairs based on multi-factor scoring. Start with `similarity_threshold=0.7` and raise it to reduce false positives.+</Tip>++<Tip>+ **Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters — if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity.+</Tip>++<Tip>+ **Normalize entity names before deduplication.** Canonical forms such as `"Apple Inc."` vs `"apple inc"` may score below threshold due to case alone. Run `EntityNormalizer` or `TextNormalizer` first for reliable matching.+</Tip><CardGroup cols={2}><Card title="Conflicts" icon="triangle-exclamation" href="conflicts">