Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
Merge branch 'main' into feat/optimize-search
Mohd Kaif66c8431eee55
diff --cc CHANGELOG.mdindex f3e5af33,c5575565..00984f03--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,14 -7,27 +7,35 @@@ and this project adheres to [Semantic V## [Unreleased]+- **Performance: Indexed search for large knowledge graphs** (closes #467, PR #481 by @ZohaibHassan16, review fixes by @KaifAhmad1):+ - **Root cause**: The previous `GraphSession.search()` ran a full O(n) scan over all nodes per query, serializing every node's properties to JSON for string matching. On a 118 k-node graph warm queries took 24–471 ms; a session with 500 k nodes was effectively unusable.+ - **New `semantica/explorer/search_index.py`**: Purpose-built in-memory inverted index with three lookup tiers — exact-term index (full normalized strings), token index (individual words), and prefix index (2–12 character prefixes of every token). A linear secondary-scan fallback (capped at 12 k nodes) handles queries that miss all three tiers. An LRU result cache (128 slots, `OrderedDict`) serves repeat queries at zero cost. Warm query times on the same 118 k-node graph: 24 ms → 0.004 ms (exact), 471 ms → 0.009 ms (ID lookup), 475 ms → 0.002 ms (no-match).+ - **`IndexedNodeDocument`** frozen dataclass stores per-node primary text (ID, content, curated alias keys: `label`, `name`, `pref_label`, `aliases`, `synonyms`, `display_name`, etc.), secondary text (remaining properties), token set, prefix expansions, confidence, and tags. Primary text prioritizes human-readable fields; secondary text covers the full property bag up to a 48-fragment cap.+ - **Scoring**: exact ID match → 140, exact term → 120, primary-text substring → 78 + length bonus, token hit → 18, prefix hit → 10, multi-token bonus → 4 per hit. Ties broken deterministically on `(score, exactness, token_hits, node_id)`.+ - **Mutation sync**: `GraphSession.add_node()`, `add_nodes()`, `add_edges()` and `add_node()` update the index incrementally. `handle_graph_mutation()` integrates with the WebSocket mutation bridge for live updates during reasoning, enrichment, and remote graph changes. `rebuild_search_index()` performs a full O(n) rebuild when needed (session init, merge, reload). `enrich.py` routes node/edge additions through `session.add_node`/`session.add_edge` so reasoning-inferred nodes are indexed immediately.+ - **Review fixes applied**: replaced `list.sort()` per upsert with `bisect.insort()` (O(log n) vs O(n log n)); replaced `list.remove()` with `bisect.bisect_left` + `pop()` (O(log n) vs O(n)); added `with self._lock` in `handle_graph_mutation()` to prevent index races from the WebSocket thread; removed unnecessary source/target upserts on `add_edge()` (edges don't affect node text); sorted tag values in `_cache_key()` so `["a","b"]` and `["b","a"]` share a cache entry.+ - 3 new tests: `test_search_exact_and_prefix` (exact match + prefix match), `test_search_filters_and_cache_stability` (type + confidence filter, identical repeated requests), `test_search_sees_new_nodes_after_mutation` (node added via `session.add_node()` immediately visible in search).+ - **Fix: Provenance traversal now includes multi-hop upstream ancestors + edge direction classification** (closes #470, PR #480 by @Sameer6305, review fixes by @KaifAhmad1):+ - **Bug — upstream ancestors silently excluded**: `_build_provenance()` built a directed `nx.DiGraph` and seeded first-hop neighbors correctly, but the final subgraph extraction called `nx.ego_graph(..., undirected=False)`. With directed traversal, ego-graph expansion only follows outgoing edges from the focus node, so any node that *points into* the focus node (i.e. an upstream ancestor at depth ≥ 2) was invisible. For the chain `Source → Intermediate → node_id`, `Intermediate` appeared at hop 1 but `Source` was silently dropped. Fixed by changing to `undirected=True` — the radius expansion now traverses both incoming and outgoing edges while the underlying `DiGraph` is preserved, so edge source/target semantics remain correct.+ - **Enhancement — edge direction classification**: `ProvenanceEdge` gains a `direction: str` field. Each edge in the provenance subgraph is classified relative to the focus node: `"upstream"` when `target == node_id` (edge flows into the focus node), `"downstream"` when `source == node_id` (edge flows out), and `"lateral"` for all other edges between non-focus neighbors. This lets consumers distinguish ancestor provenance from descendant impact without re-traversing the graph.+ - **Enhancement — grouped markdown report**: `_render_markdown()` now groups lineage edges under separate `## Upstream`, `## Downstream`, and `## Lateral` sections instead of a flat `## Lineage Edges` list. Empty sections are omitted. This improves readability of exported provenance reports.+ - **Schema consolidation**: `ProvenanceNode`, `ProvenanceEdge`, and `ProvenanceResponse` moved from inline definitions in `routes/provenance.py` to the shared `semantica/explorer/schemas.py`, matching the convention used by all other Explorer routes. `ProvenanceNode.parent_id` is now `Optional[str] = None`.+ - **Merge conflicts resolved**: Resolved all conflict markers in `provenance.py`, `app.py`, and `.gitignore`; restored the complete router import set in `app.py` (`graph`, `sparql`, `temporal`, `vocabulary`) that the conflict had dropped.+ - 2 new tests in `tests/explorer/test_provenance_route.py`: `test_build_provenance_direction_classification_chain` (asserts `Source` and `Intermediate` both appear for `Source → Intermediate → node_id`; verifies `Intermediate → node_id` classified as `"upstream"`) and `test_render_markdown_groups_edges_by_direction` (asserts grouped section headings and correct edge lines in output).++ - **Fix: `OWLExporter._export_owl_turtle` invalid Turtle syntax and silent data-property omission** (closes #478 by @KaifAhmad1):+ - **Bug 1 — Invalid Turtle syntax**: `_export_owl_turtle` unconditionally wrote `rdfs:label` with a closing period (`.`), then appended `rdfs:subClassOf`, `rdfs:domain`, and `rdfs:range` triples after the closed block. Any RDF parser would reject the output. Fixed by introducing `_ttl_block(subject_uri, rdf_type, predicates)` — all predicate-object pairs for a subject are accumulated first, then joined with ` ;\n ` and terminated with a single ` .`, producing valid Turtle in all cases.+ - **Bug 2 — Data properties silently dropped**: `_export_owl_turtle` had loops for `classes` and `object_properties` but no loop for `data_properties`, so all `owl:DatatypeProperty` declarations were silently omitted. Added the missing loop, mirroring the existing object-property loop.+ - **String escaping**: User-provided strings (`name`, `description`, `comment`, version) were embedded directly into Turtle string literals without escaping. A class named `John"s Class` or a comment containing a backslash or newline produced unparseable output. Added `_escape_ttl_str()` static method (escapes `"`, `\`, `\n`, `\r`, `\t`) applied at every `rdfs:label`, `rdfs:comment`, and `owl:versionInfo` site.+ - **Null-check consistency**: All optional field reads now use `x = prop.get("field"); if x:` uniformly — eliminates the mixed pattern of `.get()` guards followed by direct `[]` access.+ - 43 tests added in `tests/export/test_owl_exporter.py` across five suites: `TestTurtleSyntaxValidity` (5), `TestDataPropertiesInTurtle` (8), `TestTurtleHeader` (4), `TestTurtleStringEscaping` (16), `TestNullFieldHandling` (7), plus `TestObjectPropertyListDomainRange` (2) and `TestEquivalentClass` (1).++ - **Enhancement: Node distance semantics in path responses** (closes #472 by @KaifAhmad1): `PathResponse` now surfaces two new first-class fields — `hop_count: int` (equal to `len(path) - 1`; `0` for self-paths) and `distance_band: str` — so callers no longer need to count hops or implement band classification themselves. Four bands are defined: `"direct"` (0–1 hops), `"near"` (2–3), `"mid-range"` (4–6), `"distant"` (7+). The classification function `classify_path_distance()` lives in `semantica/utils/helpers.py` as the single source of truth; both the Explorer route and the visualizer import from it. `KGVisualizer.visualize_network()` gains an optional `highlight_path: list[str]` parameter: when provided, path edges are rendered as a separate orange trace with opacity and stroke width scaled to the distance band (direct: 1.0 / 4 px → distant: 0.35 / 1.5 px), while non-path edges render at reduced opacity underneath. Edge direction is respected — only the forward pairs `(A, B)` along the path are matched; reverse back-edges in directed graphs are not incorrectly highlighted. A logger warning is emitted when any node ID in `highlight_path` has no layout position, surfacing silent no-op mismatches. Frontend `PathResponse` type in `GraphInspectorPanel.tsx` and `GraphWorkspaceShell.tsx` extended with `hop_count: number` and `distance_band: "direct" | "near" | "mid-range" | "distant"`. All changes are additive; no existing fields removed. 10 new tests: 2 API-level (`test_response_includes_hop_count_and_distance_band`, `test_one_hop_path_is_direct`) and 8 unit tests covering all four band boundaries (0, 1, 2, 3, 4, 6, 7, 20 hops).++ - **Enhancement: Bidirectional path finding in Knowledge Explorer** (closes #469 by @KaifAhmad1): Path queries in the Explorer were direction-sensitive — querying B→A when only the edge A→B existed always returned no result, because `PathFinder._get_neighbors()` called `graph.neighbors(node)` which on a `nx.DiGraph` yields only successors. Added a `directed: bool = True` parameter to `bfs_shortest_path()` and `dijkstra_shortest_path()`. When `directed=False` a lightweight undirected view is built via `graph.to_undirected()` for the traversal pass only; the original directed edges are preserved and returned in the response. A `_make_undirected_view()` helper encapsulates the conversion and falls back safely for non-NetworkX graph types. The `/api/graph/node/{id}/path` route exposes the parameter as a query string flag (`?directed=false`); `PathResponse` gains a `directed: bool` field that echoes the mode used. Default is `True`, so all existing callers are unaffected. The route also gained an empty-path 404 guard — previously a traversal that found no path returned `200` with `path: []` instead of `404`. 21 new tests: 12 unit tests in `TestBidirectionalPathFinding` (`tests/kg/test_path_finder.py`) and 9 API-level tests in `TestBidirectionalPathRoute` (`tests/explorer/test_explorer_api.py`).++ - **Enhancement: Native `KnowledgeGraph` type support in `KGVisualizer`** (PR `kg` by @KaifAhmad1, closes #471): Added `semantica/kg/knowledge_graph.py` — a formal `KnowledgeGraph` dataclass (`entities`, `relationships`, `metadata`) that is now the canonical in-memory type produced and consumed by the Semantica KG pipeline. Exported from `semantica.kg`. `KGVisualizer` gains `_convert_knowledge_graph()` — an authoritative, non-mutating conversion path from `KnowledgeGraph` to the internal dict format — and `_normalize_graph()` now routes `isinstance(graph, KnowledgeGraph)` through it as an explicit fast-path before duck-typing. All five public entry points (`visualize_network`, `visualize_communities`, `visualize_centrality`, `visualize_entity_types`, `visualize_relationship_matrix`) accept `KnowledgeGraph` directly; no manual conversion required. All existing callers passing dicts or duck-typed objects are unaffected. 15 new tests in `TestFormalKnowledgeGraphType` (conversion shape, non-mutation, determinism, routing, all five entry points, import availability).+ - **Fix: `KGVisualizer` now accepts `KnowledgeGraph` objects in all `visualize_*` methods** (PR `visualization` by @KaifAhmad1, closes #458): All five public methods (`visualize_network`, `visualize_communities`, `visualize_centrality`, `visualize_entity_types`, `visualize_relationship_matrix`) previously called `graph.get("entities", [])`, silently producing no output when passed a non-dict object. Added `_normalize_graph()` which duck-types the input — dicts pass through unchanged; any object exposing `.entities` / `.relationships` attributes (e.g. the result of `GraphBuilder.build()`) is converted to the canonical dict form; anything else raises a clear `ProcessingError` naming the offending type. 21 tests added in `tests/visualization/test_kg_visualizer_normalize_graph.py`.- **Security: 12 vulnerability fixes across CRITICAL → LOW severity** (PR `security-enhancement` by @KaifAhmad1):