Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

docs(deduplication): align entity resolution APIs with implementation

Sameer63054275d6ee34a4
diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md
index 79e420e5..4d47d464 100644
--- a/docs/reference/deduplication.md
+++ b/docs/reference/deduplication.md
@@ -1,114 +1,212 @@
---
title: "Deduplication Module"
-description: "Entity deduplication v1/v2 — similarity scoring, blocking, merging, and cluster-based batch processing."
+description: "Entity deduplication — similarity scoring, blocking, merging, and cluster-based batch processing."
icon: "copy"
---
-`semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. **v2 strategies** (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to **7x faster** than v1 with fine-grained result control.
+`semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. All deduplication workflows operate on plain Python dicts with `id`, `name`, `type`, `properties`, and `relationships` keys.
## Exported Classes
| Class | Role |
| --- | --- |
-| `DuplicateDetector` | Pairwise and batch detection with configurable strategy and threshold |
-| `EntityMerger` | Merge duplicate groups with per-property merge policies (`KEEP_FIRST`, `UNION`, `VOTING`, ...) |
-| `SimilarityCalculator` | Levenshtein, Jaro-Winkler, cosine, Jaccard, and embedding similarity |
+| `DuplicateDetector` | Pairwise and batch detection — returns `DuplicateCandidate` or `DuplicateGroup` lists |
+| `EntityMerger` | Merge duplicate groups — returns `List[MergeOperation]` |
+| `SimilarityCalculator` | Multi-factor similarity: string, property, relationship, and embedding |
| `ClusterBuilder` | Union-Find and hierarchical clustering for large-scale batch deduplication |
-| `PropertyMergeRule` | Enum of merge policies used by `EntityMerger` |
-| `MergeStrategyManager` | Manage and apply named merge strategies across entity types |
-| `detect_duplicates()` | Quick function: `detect_duplicates(entities, method="semantic_v2")` |
-| `merge_entities()` | Quick function: `merge_entities(entities, duplicates, method="union")` |
-
-**Available `method=` values:**
-
-| Method | Speed | Notes |
-| --- | --- | --- |
-| `exact` | Fastest | Exact string match only |
-| `fuzzy` | Fast | Levenshtein + Jaro-Winkler |
-| `semantic` | Moderate | Embedding cosine similarity |
-| `blocking_v2` | Fast | Sorted neighborhood blocking (v2) |
-| `hybrid_v2` | Balanced | Blocking + multi-feature scoring (v2) |
-| `semantic_v2` | Best accuracy | Embedding + structural features, up to 7× faster than v1 |
+| `MergeStrategy` | Enum of merge strategies: `KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL` |
+| `PropertyMergeRule` | Dataclass holding per-property merge rule: `{property_name, strategy, conflict_resolution, priority}` |
+| `MergeStrategyManager` | Manage and apply named merge strategies; accepts per-property rules |
+| `detect_duplicates()` | Convenience function — `detect_duplicates(entities, method="pairwise", similarity_threshold=0.7)` |
+| `merge_entities()` | Convenience function — `merge_entities(entities, method="keep_most_complete")` |
+| `calculate_similarity()` | Convenience function — `calculate_similarity(entity_a, entity_b, method="multi_factor")` |
+
+## Getting Started
+
+```python
+from semantica.deduplication import DuplicateDetector, EntityMerger
+
+entities = [
+ {"id": "1", "name": "Apple Inc.", "type": "Company"},
+ {"id": "2", "name": "Apple", "type": "Company"},
+ {"id": "3", "name": "Microsoft", "type": "Company"},
+]
+
+# 1. Detect duplicates — returns List[DuplicateCandidate]
+detector = DuplicateDetector(similarity_threshold=0.7)
+candidates = detector.detect_duplicates(entities)
+
+for dup in candidates:
+ print(
+ "{} vs {} — sim: {:.2f}, confidence: {:.2f}".format(
+ dup.entity1.get("name"),
+ dup.entity2.get("name"),
+ dup.similarity_score,
+ dup.confidence,
+ )
+ )
+
+# 2. Merge duplicates — returns List[MergeOperation]
+merger = EntityMerger()
+operations = merger.merge_duplicates(entities, strategy="keep_most_complete")
+
+for op in operations:
+ print("Merged {} entities → {}".format(
+ len(op.source_entities), op.merged_entity.get("name")
+ ))
+```
## DuplicateDetector
-Find duplicate entity pairs with configurable strategies and result filtering:
+Find duplicate entity pairs:
```python
from semantica.deduplication import DuplicateDetector
-detector = DuplicateDetector(similarity_threshold=0.85)
-duplicates = detector.detect_duplicates(entities)
+detector = DuplicateDetector(
+ similarity_threshold=0.7, # default 0.7 — minimum score to include a candidate
+ confidence_threshold=0.6, # default 0.6 — minimum confidence to include a candidate
+ max_results=100, # optional — hard cap on total candidates returned
+ top_k_per_entity=3, # optional — max candidates per entity
+ min_similarity=0.75, # optional — additional floor applied after sorting
+ sort_by="confidence", # "confidence" (default) | "similarity_score"
+)
-for dup in duplicates:
- print(f"{dup.entity_a} ≈ {dup.entity_b} ({dup.similarity:.2f})")
+# Returns List[DuplicateCandidate]
+candidates = detector.detect_duplicates(entities)
+
+for c in candidates:
+ print(c.entity1.get("name"), "vs", c.entity2.get("name"))
+ print(" similarity_score:", c.similarity_score)
+ print(" confidence: ", c.confidence)
+ print(" reasons: ", c.reasons)
+
+# Detect duplicate groups with union-find (returns List[DuplicateGroup])
+groups = detector.detect_duplicate_groups(entities)
+for g in groups:
+ print("Group of {} — confidence: {:.2f} — representative: {}".format(
+ len(g.entities), g.confidence, g.representative and g.representative.get("name")
+ ))
+
+# Incremental — compare new entities against existing (returns List[DuplicateCandidate])
+new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}]
+candidates = detector.incremental_detect(new_entities, entities)
```
-Fine-grained control over strategy, thresholds, and result size:
+### `detect_duplicates()` detection methods
-```python
-duplicates = detector.detect_duplicates(
- entities,
- strategy="semantic_v2", # see strategies table below
- min_similarity=0.85, # minimum score to consider a match
- top_k_per_entity=3, # max candidates per entity
- max_results=100, # total result cap
- sort_by="similarity", # "similarity" | "entity_id" | "cluster_size"
-)
-```
+The `method=` parameter of the `detect_duplicates()` convenience function controls how
+the comparison is performed. These are independent of the `SimilarityCalculator` string
+method used internally:
-### Detection Strategies
+| `method=` | Algorithm | Returns |
+| --------- | --------- | ------- |
+| `"pairwise"` (default) | O(n²) all-pairs comparison | `List[DuplicateCandidate]` |
+| `"batch"` | Batch similarity calculation | `List[DuplicateCandidate]` |
+| `"incremental"` | New vs existing entities | `List[DuplicateCandidate]` |
+| `"group"` | Union-find group formation | `List[DuplicateGroup]` |
-| Strategy | Algorithm | Speed | Accuracy |
-| -------- | --------- | ----- | -------- |
-| `jaro_winkler` | String similarity (v1) | Fast | Medium |
-| `blocking_v2` | Blocking + Jaro-Winkler (v2) | Very fast | Medium |
-| `hybrid_v2` | Blocking + semantic + string (v2) | Fast | High |
-| `semantic_v2` | Embedding similarity (v2) | Medium | Highest |
+### DuplicateCandidate fields
-<Note>
- **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions.
-</Note>
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| `entity1` | `Dict` | First entity |
+| `entity2` | `Dict` | Second entity |
+| `similarity_score` | `float` | Similarity score (0–1) |
+| `confidence` | `float` | Confidence score (0–1) |
+| `reasons` | `List[str]` | Why they are considered duplicates |
+| `metadata` | `Dict` | Additional metadata |
+
+### DuplicateGroup fields
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| `entities` | `List[Dict]` | All entities in the group |
+| `similarity_scores` | `Dict` | Pair → score mapping |
+| `representative` | `Optional[Dict]` | Most complete entity in group |
+| `confidence` | `float` | Group confidence score |
+| `metadata` | `Dict` | Additional metadata |
## EntityMerger
-Merges detected duplicate groups into canonical entities, preserving provenance:
+Merges detected duplicate groups into canonical entities:
```python
-from semantica.deduplication import EntityMerger
-
-merger = EntityMerger()
-merged_entities = merger.merge_duplicates(
- entities,
- strategy="keep_most_complete", # see strategies table below
- preserve_provenance=True, # keep source references after merge
-)
+from semantica.deduplication import EntityMerger, MergeStrategy
+
+# Basic usage
+merger = EntityMerger(preserve_provenance=True)
+operations = merger.merge_duplicates(entities, strategy="keep_most_complete")
+
+# operations is List[MergeOperation]
+for op in operations:
+ merged = op.merged_entity # the resulting merged entity dict
+ sources = op.source_entities # list of original entities that were merged
+ conflicts = op.merge_result.conflicts # unresolved conflicts, if any
+
+# Merge a known group directly (no duplicate detection)
+pair = [
+ {"id": "1", "name": "Apple Inc.", "type": "Company"},
+ {"id": "2", "name": "Apple", "type": "Company"},
+]
+op = merger.merge_entity_group(pair, strategy="keep_most_complete")
+print(op.merged_entity)
+
+# Retrieve full merge history
+history = merger.get_merge_history()
+print("Total merges performed:", len(history))
```
-### Merge Strategies
+### Merge strategies
+
+Pass as a string to `strategy=` on `merge_duplicates()` or `merge_entity_group()`:
| Strategy | Behavior |
| -------- | -------- |
-| `keep_first` | Keep the first entity in each duplicate group |
-| `keep_last` | Keep the most recently seen entity |
-| `keep_most_complete` | Keep the entity with the most non-null properties |
-| `union` | Merge all properties — non-conflicting fields combined |
-| `voting` | Most common property value wins |
+| `"keep_first"` | Keep the first entity in each duplicate group |
+| `"keep_last"` | Keep the most recently seen entity |
+| `"keep_most_complete"` | Keep the entity with the most non-null properties + relationships |
+| `"keep_highest_confidence"` | Keep the entity with the highest `.confidence` value |
+| `"merge_all"` | Combine all properties — conflicts resolved to lists |
+
+### Per-property merge rules
-Fine-grained per-property merge rules:
+Per-property rules are set on `EntityMerger.merge_strategy_manager` using
+`add_property_rule()`. Rules take a `MergeStrategy` enum value:
```python
-from semantica.deduplication import EntityMerger, PropertyMergeRule
-
-merger = EntityMerger(
- property_rules={
- "name": PropertyMergeRule.KEEP_FIRST,
- "aliases": PropertyMergeRule.UNION,
- "description": PropertyMergeRule.KEEP_LONGEST,
- }
+from semantica.deduplication import EntityMerger, MergeStrategy
+
+merger = EntityMerger()
+
+# Add per-property rules
+merger.merge_strategy_manager.add_property_rule(
+ "name", MergeStrategy.KEEP_FIRST
+)
+merger.merge_strategy_manager.add_property_rule(
+ "aliases", MergeStrategy.MERGE_ALL
+)
+
+# Custom conflict resolution function
+def keep_longest(val1, val2):
+ return val1 if len(str(val1)) >= len(str(val2)) else val2
+
+merger.merge_strategy_manager.add_property_rule(
+ "description", MergeStrategy.KEEP_FIRST, conflict_resolution=keep_longest
)
+
+operations = merger.merge_duplicates(entities)
```
+### MergeOperation fields
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| `source_entities` | `List[Dict]` | Original entities that were merged |
+| `merged_entity` | `Dict` | Resulting merged entity |
+| `merge_result` | `MergeResult` | Detailed result with conflicts |
+| `metadata` | `Dict` | Group confidence, similarity scores, strategy used |
+
## SimilarityCalculator
Compute multi-factor similarity scores between entity pairs:
@@ -116,25 +214,44 @@ Compute multi-factor similarity scores between entity pairs:
```python
from semantica.deduplication import SimilarityCalculator
-calc = SimilarityCalculator()
-
-score = calc.calculate_similarity(entity_a, entity_b)
-# → SimilarityResult(score=0.91, components={...})
+calc = SimilarityCalculator(
+ string_weight=0.6, # default 0.6
+ property_weight=0.2, # default 0.2
+ relationship_weight=0.2, # default 0.2
+ embedding_weight=0.0, # default 0.0 (used only when "embedding" key is present)
+ similarity_threshold=0.7,
+)
-print(score.score) # overall score 0.0–1.0
-print(score.components["label"]) # label similarity
-print(score.components["embedding"]) # semantic similarity
-print(score.components["property"]) # property overlap
+result = calc.calculate_similarity(entity_a, entity_b)
+# result is a SimilarityResult
+print(result.score) # overall score 0.0–1.0
+print(result.method) # e.g. "multi_factor"
+print(result.components["string"]) # string similarity component
+print(result.components["property"]) # property overlap component
+print(result.components["relationship"]) # relationship jaccard component
+# result.components["embedding"] is present only when embeddings are supplied
+
+# String similarity methods — method= accepts "levenshtein", "jaro_winkler", "cosine"
+lev = calc.calculate_string_similarity("Apple Inc.", "Apple Inc", method="levenshtein")
+jaro = calc.calculate_string_similarity("Steve Jobs", "Steven Jobs", method="jaro_winkler")
+cos = calc.calculate_string_similarity("apple", "apples", method="cosine")
+
+# Embedding cosine similarity (vector inputs)
+emb_score = calc.calculate_embedding_similarity(embedding_a, embedding_b)
+
+# Property and relationship similarity (entity dict inputs)
+prop_score = calc.calculate_property_similarity(entity_a, entity_b)
+rel_score = calc.calculate_relationship_similarity(entity_a, entity_b)
```
-Individual string and vector metrics:
+### SimilarityResult fields
-```python
-lev = calc.levenshtein("Apple Inc.", "Apple Inc")
-jaro = calc.jaro_winkler("Steve Jobs", "Steven Jobs")
-cos = calc.cosine_similarity(embedding_a, embedding_b)
-jacc = calc.jaccard({"founded", "tech"}, {"founded", "technology"})
-```
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| `score` | `float` | Overall weighted similarity score (0–1) |
+| `method` | `str` | Method used (e.g. `"multi_factor"`, `"levenshtein"`) |
+| `components` | `Dict[str, float]` | Per-component scores: `"string"`, `"property"`, `"relationship"`, `"embedding"` |
+| `metadata` | `Dict` | Weights used and optional score breakdown |
## ClusterBuilder
@@ -143,56 +260,167 @@ Build entity clusters for large-scale batch deduplication:
```python
from semantica.deduplication import ClusterBuilder
-builder = ClusterBuilder(algorithm="union_find") # or "hierarchical"
-result = builder.build_clusters(entities, similarity_threshold=0.85)
+builder = ClusterBuilder(
+ similarity_threshold=0.8, # minimum similarity to be in same cluster
+ min_cluster_size=2, # minimum entities per valid cluster
+ max_cluster_size=100, # maximum entities per cluster
+ use_hierarchical=False, # True for hierarchical, False (default) for union-find
+)
+result = builder.build_clusters(entities)
-print(f"Clusters: {len(result.clusters)}")
+print("Clusters found:", len(result.clusters))
for cluster in result.clusters:
- print(f" [{cluster.id}] {cluster.members} — cohesion: {cluster.cohesion:.2f}")
+ print(" [{}] {} entities — quality: {:.2f}".format(
+ cluster.cluster_id,
+ len(cluster.entities),
+ cluster.quality_score,
+ ))
+
+print("Unclustered: ", len(result.unclustered))
+print("Quality metrics:", result.quality_metrics)
+# {"average_size": ..., "average_quality": ..., "total_clusters": ..., "high_quality_clusters": ...}
```
-## Blocking Strategies
+### Cluster fields
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| `cluster_id` | `str` | Unique cluster identifier |
+| `entities` | `List[Dict]` | Entities in the cluster |
+| `centroid` | `Optional[Dict]` | Representative entity (optional) |
+| `quality_score` | `float` | Intra-cluster average similarity |
+| `metadata` | `Dict` | Similarity scores and other metadata |
-Blocking reduces the O(n²) pairwise comparison problem to a manageable candidate set:
+## Convenience Functions
```python
-detector = DuplicateDetector(
- blocking_strategy="token", # "token" | "phonetic" | "ngram"
- blocking_threshold=0.6,
- similarity_threshold=0.85
+from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity
+
+# Detect — method= accepts "pairwise" (default), "batch", "incremental", "group"
+candidates = detect_duplicates(
+ entities,
+ method="pairwise",
+ similarity_threshold=0.8,
+ confidence_threshold=0.6,
)
+
+# Merge — method= accepts the strategy strings, same as EntityMerger
+operations = merge_entities(entities, method="keep_most_complete", preserve_provenance=True)
+# Returns List[MergeOperation]; access .merged_entity on each
+
+# Similarity — method= accepts "exact", "levenshtein", "jaro_winkler", "cosine",
+# "property", "relationship", "embedding", "multi_factor" (default)
+result = calculate_similarity(entity_a, entity_b, method="multi_factor")
+print(result.score)
```
## Custom Similarity Functions
-Register domain-specific similarity logic:
+Register domain-specific similarity logic and use it via the method registry:
```python
-from semantica.deduplication import method_registry
+from semantica.deduplication import method_registry, SimilarityResult
-def drug_name_similarity(entity_a, entity_b):
- # Match drug names by active compound
- return score # 0.0 to 1.0
+def drug_name_similarity(entity_a, entity_b, **kwargs):
+ """Match drug names by active compound prefix."""
+ name_a = entity_a.get("name", "").lower()
+ name_b = entity_b.get("name", "").lower()
+ score = 1.0 if name_a[:5] == name_b[:5] else 0.0
+ return SimilarityResult(score=score, method="drug_name")
method_registry.register("similarity", "drug_name", drug_name_similarity)
-detector = DuplicateDetector(similarity_method="drug_name")
+# Now callable via calculate_similarity
+from semantica.deduplication import calculate_similarity
+result = calculate_similarity(entity_a, entity_b, method="drug_name")
```
-## Convenience Functions
-
-```python
-from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity
-
-# Quick detection
-duplicates = detect_duplicates(entities, method="semantic_v2", similarity_threshold=0.85)
-
-# Quick merge
-merged = merge_entities(entities, duplicates, method="keep_most_complete")
-
-# Quick similarity
-score = calculate_similarity(entity_a, entity_b, method="hybrid_v2")
-```
+## Common Workflows
+
+<Tabs>
+ <Tab title="Basic Deduplication">
+ ```python
+ from semantica.deduplication import DuplicateDetector, EntityMerger
+
+ detector = DuplicateDetector(similarity_threshold=0.8)
+ candidates = detector.detect_duplicates(entities)
+
+ print("Duplicate pairs found:", len(candidates))
+
+ merger = EntityMerger(preserve_provenance=True)
+ operations = merger.merge_duplicates(entities, strategy="keep_most_complete")
+
+ merged_entities = [op.merged_entity for op in operations]
+ ```
+ </Tab>
+ <Tab title="Group-based Batch">
+ ```python
+ from semantica.deduplication import DuplicateDetector, EntityMerger
+
+ detector = DuplicateDetector(similarity_threshold=0.75)
+ # detect_duplicate_groups uses union-find internally
+ groups = detector.detect_duplicate_groups(entities)
+
+ merger = EntityMerger()
+ for group in groups:
+ op = merger.merge_entity_group(group.entities, strategy="keep_most_complete")
+ print("Merged into:", op.merged_entity.get("name"))
+ ```
+ </Tab>
+ <Tab title="Large-scale Clustering">
+ ```python
+ from semantica.deduplication import ClusterBuilder, EntityMerger
+
+ # Build clusters first — more efficient for large entity sets
+ builder = ClusterBuilder(similarity_threshold=0.8, min_cluster_size=2)
+ result = builder.build_clusters(entities)
+
+ merger = EntityMerger()
+ for cluster in result.clusters:
+ op = merger.merge_entity_group(cluster.entities, strategy="keep_most_complete")
+ print("Cluster merged into:", op.merged_entity.get("name"))
+ ```
+ </Tab>
+ <Tab title="Incremental">
+ ```python
+ from semantica.deduplication import EntityMerger
+
+ existing_entities = [...] # already in the graph
+ new_entities = [...] # arriving in a batch
+
+ merger = EntityMerger()
+ operations = merger.incremental_merge(new_entities, existing_entities)
+
+ print("New merges performed:", len(operations))
+ ```
+ </Tab>
+</Tabs>
+
+## Tips and Common Pitfalls
+
+<Warning>
+ **`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score` — not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`.
+</Warning>
+
+<Warning>
+ **`merge_entities()` and `EntityMerger.merge_duplicates()` return `List[MergeOperation]`, not a list of entity dicts.** Access `.merged_entity` on each operation to get the merged dict.
+</Warning>
+
+<Warning>
+ **`PropertyMergeRule` is a dataclass, not an Enum.** The merge strategy Enum is `MergeStrategy` (`KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL`). Per-property rules are added via `merger.merge_strategy_manager.add_property_rule(name, strategy)`.
+</Warning>
+
+<Tip>
+ **Tune `similarity_threshold` before `confidence_threshold`.** The similarity threshold gates which entity pairs are even considered. The confidence threshold further filters those pairs based on multi-factor scoring. Start with `similarity_threshold=0.7` and raise it to reduce false positives.
+</Tip>
+
+<Tip>
+ **Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters — if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity.
+</Tip>
+
+<Tip>
+ **Normalize entity names before deduplication.** Canonical forms such as `"Apple Inc."` vs `"apple inc"` may score below threshold due to case alone. Run `EntityNormalizer` or `TextNormalizer` first for reliable matching.
+</Tip>
<CardGroup cols={2}>
<Card title="Conflicts" icon="triangle-exclamation" href="conflicts">