Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

chore: resolve CHANGELOG.md merge conflict with main

KaifAhmad1a2b6a481dcad
diff --cc CHANGELOG.md
index e84e43dd,04029d23..73b6e4e8
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -5,915 -5,157 +5,172 @@@ All notable changes to this project wil
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+ ---
+
## [Unreleased]
+ ### Added
+
+ **Ontology Hub** (part of #517)
+
+ - **Alignments tab** (PR #524, @KaifAhmad1 @ZohaibHassan16) — cross-ontology alignment authoring UI:
+ - Create/edit/delete alignments with source URI, target URI, relation selector (owl:equivalentClass, all five skos:*Match variants), confidence slider, provenance, and reviewer fields.
+ - Pairwise alignment matrix: scrollable table for all loaded ontology pairs; clicking a badge pre-fills the form.
+ - Alignment suggestions via `POST /api/ontology/suggest-alignments` — blended score (0.4×label + 0.6×TF-IDF char-ngram cosine); one-click accept.
+ - Ephemeral-storage banner; all handlers wrapped in `useCallback`.
+ - **Health Dashboard** (PR #524) — per-ontology quality scoring across 5 dimensions:
+ - Completeness, Consistency, SHACL (stub), Alignment, Documentation.
+ - Total score computed as mean of scoreable dimensions only (SHACL excluded when unavailable).
+ - Issue list with severity badges (error/warning/info), entity URI chip, "Fix in Editor" deep-link.
+ - Downloadable JSON health report; `GET /api/ontology/health` with `_MAX_ANALYSIS_NODES = 5 000` OOM cap.
+ - **SHACL Studio** (PR #524) — interactive SHACL shape authoring:
+ - Shape generation via `POST /api/ontology/shacl/generate` (permissive/standard/strict tiers).
+ - Shape library panel with per-shape Turtle extraction; "View all" restores full document.
+ - Monaco editor with custom Monarch tokenizer for Turtle syntax.
+ - Validation stub via `POST /api/ontology/shacl/validate`; rejects empty/invalid Turtle with HTTP 422.
+ - **Visual Ontology Editor** (PR #519, @KaifAhmad1) — @xyflow/react canvas for authoring classes/properties/individuals without hand-writing OWL/Turtle:
+ - Context menus on nodes (rename, add super/subclass, restrictions, SKOS metadata, deprecation, delete with impact count) and edges (toggle functional/symmetric/transitive/inverse-functional, add inverse).
+ - All edits debounced and staged as pending diffs via `PATCH /api/ontology/draft`; nothing commits until proposal publish.
+ - **Versions & Proposals tab** (PR #519) — version timeline, proposal review (approve/reject/publish), SHACL pre-validation, side-by-side diff via `VersionManager.diff_ontologies()`.
+ - **Ontology Registry** (PR #518, @KaifAhmad1) — full CRUD with status/format badges, per-ontology stats, live search, filter pills (All/OWL/SKOS/Internal/External), action feedback auto-hide.
+ - **Ontology Loader** (PR #518) — three-mode modal: URL import (fetch preview + load), file upload (.ttl/.rdf/.owl/.nt/.jsonld/.n3), create new (scratch/from-data/from-text).
+ - **Entity Search panel** (PR #518) — debounced 320 ms search across all loaded ontologies; type filter pills; result detail panel with super/subclasses, domain/range, instance count.
+ - **SKOS Vocabulary Manager** (PR #518) — hierarchical concept browser with recursive `ConceptTreeNode`, client-side `filterConcepts()`, full SKOS annotation detail (definition, scopeNote, broader/narrower/related/exactMatch).
+ - **16 backend endpoints** under `/api/ontology` — registry, preview, load, create, search, entity, skos/schemes, skos/concept, draft, proposals CRUD, versions, alignments, health, shacl/generate, shacl/shapes, shacl/validate (PRs #518, #519, #524).
+ - **Explorer landing page redesign** (PR #516, @ZohaibHassan16) — hero section, animated SVG graph preview, live `/api/graph/stats` metrics, workspace launcher; `Space Grotesk` / `IBM Plex Sans` fonts; `prefers-reduced-motion` support.
+ - **Distance Intelligence** (PR #502, @KaifAhmad1):
+ - `ContextGraph.get_neighbors(include_distance_metadata)` — adds `distance_band`, `confidence_decay`, `path_to_anchor` per result.
+ - `AgentContext.retrieve()` / `find_precedents()` blend graph proximity with semantic score (`combined_score = (1−w)×semantic + w×proximity`).
+ - 5 new API endpoints: `POST /api/graph/distance-matrix` (N×N, upper-triangle mirrored), `GET /api/graph/node/{id}/semantic-neighborhood`, `GET /api/decisions/causal-distance`, `GET /api/temporal/distance-history`, `POST /api/export/distance-enriched` (CSV/JSONL, capped at 200 nodes).
+ - Explorer UI: Ego Mode (BFS depth-of-field fading, depth slider 1–8), Structural overlay, Semantic overlay, Heatmap (green→red by hop); Path inspector with distance band chip, metric cards, bottleneck node highlight.
+ - 57 new tests in `tests/context/test_distance_intelligence.py`.
+ - **Graph Explorer visual refresh** (PR #503, @ZohaibHassan16) — structured `ui.*` design-token namespace; per-shape biomolecule/condition/compound config; decomposed toolbar memos; typed sub-components (`SearchCommandBar`, `ToolbarCluster`, etc.); deterministic LOD edge classification via `GraphFullEdgeClass`.
+ - **Graph Workspace declutter** (PR #483, @ZohaibHassan16) — calmer default presentation for dense graphs, display-edge aggregation with raw-edge bundle retention, grouped community view, neighborhood collapse/expand.
+ - **Bidirectional path finding** (closes #469, @KaifAhmad1) — `directed=false` query param on BFS and Dijkstra; undirected view built via `graph.to_undirected()` for traversal only; empty-path 404 guard; `PathResponse.directed` field.
+ - **Node distance semantics in path responses** (closes #472) — `PathResponse` gains `hop_count` and `distance_band` ("direct"/"near"/"mid-range"/"distant"); `classify_path_distance()` in `semantica/utils/helpers.py`; `KGVisualizer.visualize_network(highlight_path)` with band-scaled edge rendering.
+ - **Native `KnowledgeGraph` type support in `KGVisualizer`** (closes #471) — formal `KnowledgeGraph` dataclass (`entities`, `relationships`, `metadata`); `_normalize_graph()` routes it through `_convert_knowledge_graph()` as an explicit fast-path in all 5 `visualize_*` methods.
+ - **Indexed search for large graphs** (PR #481, @ZohaibHassan16) — purpose-built inverted index with exact/token/prefix lookup tiers; LRU cache (128 slots); O(log n) mutation sync via `bisect.insort`; warm-query time 24 ms → 0.004 ms on 118 k-node graph.
+ - **Provenance traversal multi-hop fix** (PR #480, @Sameer6305) — undirected ego-graph expansion so upstream ancestors at depth ≥ 2 are no longer silently excluded; `ProvenanceEdge.direction` field (upstream/downstream/lateral); grouped markdown report under `## Upstream/Downstream/Lateral` sections.
+ - **TripletStore ontology namespace** (PR #447, @KaifAhmad1) — `_resolve_iri()` applies `base_uri` before `urn:` fallback; W3C prefix expansion table (owl/xsd/rdf/rdfs/skos) expands to canonical IRIs regardless of `base_uri`.
+ - **Blazegraph literal serialization** (PR #448, @KaifAhmad1) — `_format_object_for_sparql()` selects IRI/typed-literal/language-tagged-literal/plain-literal token; `_resolve_datatype_iri()` with prefix expansion; RFC 5646 language-tag validation; `_escape_literal()` for string escaping.
+ - **DeepSeek provider via OpenAI SDK** (PR #482, @liling) — `_init_client` rewritten using `openai.OpenAI(base_url=self.base_url)` instead of defunct `deepseek` package; `verbose_mode` assignment fix; `pyproject.toml` updated to `openai>=1.0.0`.
+
+ ### Fixed
+
+- **Fix: Lazy-load optional ingest backends; address Qodo review bugs** (issue #527, PR #535, by @ZohaibHassan16, review fixes by @KaifAhmad1):
+ - `semantica/ingest/__init__.py` — core exports (`FileIngestor`, `ingest_file`, config, registry) remain eagerly imported; all optional backends (`WebIngestor`, `FeedIngestor`, `RepoIngestor`, `EmailIngestor`, `StreamIngestor`, `DBIngestor`, `MCPIngestor`, `OntologyIngestor`, `SnowflakeIngestor`) are now deferred behind a module-level `__getattr__`, so `from semantica.ingest import FileIngestor` no longer fails when GitPython or BeautifulSoup4 are absent.
+ - `semantica/ingest/methods.py` — backend imports relocated into their respective ingestion functions (`ingest_web`, `ingest_feed`, `ingest_repository`, `ingest_email`) with helper `_missing_optional_dependency()` / `_is_missing_dependency()` for consistent, actionable error messages.
+ - **Review fix (Bug 1 — overbroad missing-dep detection)**: replaced `except ImportError` with `except ModuleNotFoundError` in all four function-level import guards and in `__getattr__`. `ImportError` catches failures thrown by code *inside* a successfully found module, masking real bugs with a misleading "package not installed" message; `ModuleNotFoundError` (its subclass) is specific to absent modules. Simplified `_is_missing_dependency` to rely solely on `exc.name` now that `ModuleNotFoundError` always sets it.
+ - **Review fix (Bug 2 — expected errors logged as failures)**: added `except ConfigurationError: raise` before the blanket `except Exception` handlers in `ingest_web`, `ingest_feed`, `ingest_repository`, and `ingest_email`. Missing optional dependencies are expected user-configuration issues and must not produce error-level log entries.
++ - **Review fix (Bug 3 — test blocker not setting `exc.name`)**: `OptionalDependencyBlocker.find_spec` now sets `err.name = root_name` on the manually constructed `ModuleNotFoundError`, matching what Python's import machinery does, so `_is_missing_dependency` correctly identifies the missing package in tests.
+ - Added regression tests (`tests/ingest/test_optional_imports.py`) that block the `git` and `bs4` modules via a custom meta path finder and assert core imports succeed and backends raise `ConfigurationError` with an actionable message.
-
+- **Fix: Ontology Hub post-review bug fixes and security hardening** (follow-up to #518, closes security advisory #23, by @KaifAhmad1):
+ - **Broken registry filters** — `fetchRegistry` was sending toolbar filter values (`owl`, `skos`, `internal`, `external`) to the backend as the `status` query param, which only accepts `published|draft|external`, causing those filters to return empty lists. Removed the spurious `status` param; all format/kind filtering is now applied client-side via `filteredEntries`, which already had the correct logic.
+ - **Toggle/refresh URI corruption** — `toggle_ontology` and `refresh_ontology` applied `.removesuffix("/toggle")` / `.removesuffix("/refresh")` to the captured path parameter, which would silently corrupt any ontology URI that legitimately ends with those strings. Starlette's route regex (`/{uri:path}/toggle`) already strips the literal suffix via backtracking, so the `removesuffix` calls were removed and the raw `ontology_uri` parameter is used directly.
+ - **SSRF in URL fetch** — `_fetch_url_sync()` accepted arbitrary user-supplied URLs and called `requests.get()` with no validation, enabling server-side request forgery against internal services. Added `_validate_fetch_url()` which rejects non-`http`/`https` schemes and resolves the hostname via `socket.getaddrinfo`, blocking loopback, private, link-local, reserved, and multicast addresses. Applied to all three fetch sites: preview, load, and refresh.
+ - **File upload format misdetected** — the file picker accepted `.xml` and `.json` but `fmtMap` had no entries for those extensions, causing them to default to `turtle`. Added `xml: "xml"` and `json: "json-ld"` mappings. Changed the unknown-extension fallback from `|| "turtle"` to `?? ""` (empty string), and omit the `format` key from the request body when empty so the backend `_detect_format()` runs instead of receiving a forced incorrect value. Also added `.n3` to the accepted extension list and dropzone hint.
+ - **Inconsistent XML hardening** — `_parse_rdf_sync()` called `rdflib.Graph().parse()` directly, bypassing the `defusedxml`-based XXE protection already present in `semantica/explorer/utils/rdf_parser.py`. Now routes through `_safe_parse_rdf()` from that module, applying consistent protection for all RDF/XML parse paths.
+ - **Search scans whole graph** (`GET /api/ontology/search`) — the endpoint fetched up to 999 999 nodes and performed a linear Python substring scan on every request. Replaced with `session.search(q, limit * 6)` which uses the `GraphSearchIndex`; results are then post-filtered by `_SEARCHABLE_TYPES` and `entity_type` before being returned up to the requested limit.
+ - **ReDoS in format detector** (security advisory #23, CodeQL `py/polynomial-redos`, CWE-1333/730/400) — `_detect_format()` used `re.match(r"_:\w+|<[^>]+>\s+<[^>]+>", ...)` to detect N-Triples content. The `<[^>]+>\s+<[^>]+>` alternative was flagged as a polynomial regular expression on uncontrolled data. The URI-subject branch was already unreachable (strings starting with `<` return `"xml"` two lines above), so the entire regex was replaced with two O(1) string operations: `stripped.startswith("_:")` and `" <" in stripped`. `import re` removed as now unused.
+ - **OWLExporter Turtle syntax** (closes #478) — invalid multi-block output fixed via `_ttl_block()`; data properties no longer silently dropped; `_escape_ttl_str()` applied to all label/comment/version sites. 43 tests added.
+ - **OWLGenerator schema compatibility** (Issue #446) — label-first IRI fallback, list-typed datatype ranges, per-call namespace consistency, `subClassOf`/`subclassOf` parity.
+ - **TripletStore IRI regressions** (PR #447 follow-up) — non-string IDs coerced to `str()`; W3C prefix expansion now correct regardless of `base_uri`.
+ - **`KGVisualizer` accepts `KnowledgeGraph` objects** (closes #458) — `_normalize_graph()` duck-types input; raises clear `ProcessingError` on unknown types. 21 tests added.
+ - **Semantic Distance UI slash-safe routes** (PR #515, @ZohaibHassan16) — query-param routes `/api/graph/semantic-neighborhood?node_id=` and `/api/graph/path?source=&target=` bypass FastAPI's `%2F` pre-decode; legacy path-segment routes kept as deprecated aliases.
+ - **Explorer Distance Intelligence rendering** (PR #513, @ZohaibHassan16) — distance state flows through Sigma reducer/theme pipeline instead of mutating raw graph attributes; `restoreNodeColors()` race eliminated by merging ego/heatmap `useEffect` hooks.
+ - **Distance Intelligence code review regressions** (PR #502 follow-up, @KaifAhmad1) — `top_k` param name fix; `include_distance_metadata` gated behind `False` default; `weakest_link` key standardized; temporal sampling uses `timedelta` not `timetuple`; O(E×L) decay replaced with O(E) index; `AgentContext._apply_proximity_metadata` stores `graph_node_id` separately; sweep animation `sweepGeneration` counter fix; HTTP 413 for >200 node subsets; upper-triangle distance matrix.
+ - **Knowledge Explorer blockers** (PR #420, @ZohaibHassan16):
+ - `Dockerfile`: renamed `DockerFile` → `Dockerfile`; fixed `CMD` module path; added `app = create_app()` at module level.
+ - CORS: default origins narrowed from `"*"` to `localhost:5173` only.
+ - `get_ws_manager()` now raises HTTP 503 instead of unhandled `AttributeError`.
+ - SPARQL: read-only enforcement — `INSERT`/`DELETE`/`UPDATE`/`LOAD`/`DROP` rejected.
+ - Vocabulary: 10 MB upload cap; JSON-LD format auto-detection for `.jsonld`/`.json-ld`/`.json`.
+ - Annotation `O(1)` lookup via `GraphSession.get_annotation(id)`.
+ - Self-loop guard in `batchMergeEdges` prevents Graphology crash.
+ - Static build artifacts removed from git; `semantica/static/` added to `.gitignore`.
+ - **Ontology Hub post-review hardening** (PR #518 follow-up, @KaifAhmad1):
+ - Registry filter: `status` param removed from `fetchRegistry`; filtering applied client-side.
+ - Toggle/refresh URI: removed `.removesuffix()` calls that corrupted URIs ending with those strings.
+ - Format detector: `_detect_format()` ReDoS eliminated — `re.match` replaced with two O(1) string ops.
+ - Broken `fmtMap` entries: added `xml`/`json` mappings; unknown-extension fallback changed from `|| "turtle"` to `?? ""`.
+ - XML hardening: `_parse_rdf_sync()` now routes through `_safe_parse_rdf()` for consistent defusedxml XXE protection.
+ - Search: replaced O(999 999) linear scan with `GraphSearchIndex`-backed `session.search()`.
- - **Feature: Ontology Hub — Registry, Loader, Entity Search & SKOS Vocabulary Manager** (closes #518, part of #517, by @KaifAhmad1):
- - Added a sixth workspace, **Ontology Hub** (`ontology-hub`), to the Knowledge Explorer sidebar with a `GitMerge` icon and "Schema Governance" kicker. The workspace shell hosts six tabs — Registry, Editor, Versions, Alignments, Health, and SHACL — with the active tab persisted in the `ontologyTab` URL search parameter via `window.history.replaceState`.
- - **Registry tab (`OntologyManager`)** — full CRUD interface for loaded ontologies. Lists entries with color-coded status badges (published / draft / external), format badges (Turtle / XML / JSON-LD / N-Triples), per-ontology stats (class count, concept count, property count), source URL link, and enable/disable toggle, refresh, and remove (with confirmation) actions. Toolbar provides a live search input, All / OWL / SKOS / INTERNAL / EXTERNAL filter pills, an Entity Search button, and a "Load Ontology" button. Empty state surfaces a prominent CTA. Action feedback bar auto-hides after 3 seconds.
- - **Load Ontology modal (`OntologyLoader`)** — three-tab modal overlay for importing ontologies:
- - *URL Import*: paste any HTTP(S) URL, click "Fetch Preview" to call `POST /api/ontology/preview` (fetches up to 20 MB, parses with rdflib, returns title / namespace / version / license / format / triple count), then "Load Ontology" (`POST /api/ontology/load`). Advanced options toggle exposes format override, custom display name, description, and tags fields.
- - *File Upload*: drag-and-drop zone (or browse) accepting `.ttl`, `.rdf`, `.owl`, `.nt`, `.jsonld` files; format auto-detected from extension; multipart `POST /api/ontology/load`.
- - *Create New*: three modes — From Scratch (namespace + name + description + tags), From Data (sample data textarea for schema inference via `OntologyEngine.from_data()`), From Text (free-text textarea for LLM-assisted schema generation via `OntologyEngine.from_text()`); calls `POST /api/ontology/create`.
- - **Entity Search panel (`OntologySearch`)** — slide-in right panel with debounced 320 ms search across all loaded ontologies via `GET /api/ontology/search`. Type filter pills: All, Class, Property, Individual, Concept, Scheme. Result rows show label, type badge, URI, definition snippet, and source ontology. Selecting a result opens a detail panel that fetches `GET /api/ontology/entity/{uri}` and renders label, URI, definition, superclasses, subclasses, domain, range, instance count, and external URI link. Long lists use a `CollapsibleList` expanding up to 12 items.
- - **SKOS Vocabulary Manager (`SKOSVocabularyManager`)** — hierarchical SKOS concept browser activated when a SKOS ontology is selected in the registry. Fetches scheme hierarchy from `GET /api/vocabulary/hierarchy`, renders a recursive `ConceptTreeNode` tree with depth-based indentation, expand/collapse, and selection highlight. Client-side `filterConcepts()` matches label, altLabels, and description. Detail panel fetches `GET /api/ontology/skos/concept/{uri}` and displays all SKOS annotation properties (definition, scopeNote, example, historyNote, editorialNote, changeNote) plus broader / narrower / related / exactMatch / closeMatch lists with clickable navigation.
- - **Backend (`semantica/explorer/routes/ontology.py`)** — 12 FastAPI endpoints under `GET|POST /api/ontology`:
- - `GET /registry` — returns the in-memory `app.state.ontology_registry` dict as a list, with optional `q` search and `status` filter query params.
- - `POST /preview` — streams up to 20 MB from a URL via `requests.get` in `asyncio.to_thread`, parses RDF with rdflib (auto-detects format or accepts `format` param), returns `OntologyPreview` metadata.
- - `POST /load` — URL or multipart file load; stores parsed nodes/edges into the active graph session and registers an `OntologyEntry` in the registry.
- - `POST /create` — creates an ontology from scratch, sample data, or natural-language text; falls back to a minimal ontology shell if `OntologyEngine` is unavailable.
- - `GET /search` — full-text entity search with optional `type` filter across all nodes whose `node_type` maps to class, property, individual, concept, or scheme.
- - `GET /entity/{uri:path}` — entity detail: label, type, definition, superclasses, subclasses, domain, range, instance count.
- - `GET /skos/schemes` — lists all `skos:ConceptScheme` nodes in the active session.
- - `GET /skos/concept/{uri:path}` — full SKOS concept detail including all annotation properties and relation sets.
- - `DELETE /{uri:path}`, `PATCH /{uri:path}/toggle`, `POST /{uri:path}/refresh` — remove, enable/disable toggle, and re-fetch/re-parse for registered ontologies. Route ordering places all literal paths before the `:path` wildcards to avoid shadowing.
- - Helper internals: `_parse_rdf_sync()` (rdflib parse → nodes/edges/metadata), `_fetch_url_sync()` (streaming requests with 20 MB cap), `_classify_node_type()` (maps raw RDF types to canonical categories), `_uri_to_prefix()` (URI → prefixed form for display).
- - Editor, Versions (Subissue 2) and Alignments, Health, SHACL (Subissue 3) tabs render descriptive stub cards with amber subissue badges as placeholders for upcoming implementations.
- - TypeScript compiled with zero errors; Vite dev server starts cleanly with the new workspace lazy-loaded via `React.lazy` + `Suspense`.
-
- - **Feature: Explorer landing page redesign** (PR #516 by @ZohaibHassan16, review fixes by @KaifAhmad1):
- - Replaced the plain welcome screen with a full landing composition: premium hero section, product preview mock with animated SVG graph, live graph status metrics, intelligence capability band, and consolidated workspace launcher.
- - `WelcomeScreen` fetches `/api/graph/stats` on mount with `AbortController` cleanup and displays live node and edge counts; falls back to `"Live"` / `"Ready"` labels when the endpoint is unavailable.
- - Workspace launcher surfaces Network Explorer as the primary path and provides direct one-click entry into Vocabulary, Analyze, Decisions, Enrich, and Manage workspaces.
- - Added `LandingMetric`, `LandingAction`, and `GraphStatsPayload` TypeScript types; `getNumberStat` handles three API key shapes (`node_count`, `nodeCount`, `nodes` and equivalents) for forward-compatibility.
- - Added `Space Grotesk` and `IBM Plex Sans` fonts (replacing `Inter`); `JetBrains Mono` used for kickers, badges, and metadata labels.
- - Added `prefers-reduced-motion` media query suppressing `landing-float` animation and launcher hover transitions.
- - **Review fixes** (follow-up by @KaifAhmad1 and @ZohaibHassan16): replaced invalid `inset-left` CSS property with `inset: 0 0 0 72px` on `.landing-page::before` in the `≤680px` breakpoint; added the same `inset` correction to `.landing-page::after` which was still offset at `88px` after rail narrowing; merged duplicate `.landing-capability-band` CSS rule blocks into one; corrected non-standard `font-weight: 850` to `800` on `.landing-launcher-item-title`; removed unused `eyebrow` field from `LandingAction` type and all data entries; extracted the static 42-dot SVG background array to a module-level `PREVIEW_DOTS` constant to avoid recomputing it on every render.
- - **Fix: Semantic Distance UI slash-safe node IDs** (issue #514, PR #515 by @ZohaibHassan16, review fixes by @KaifAhmad1):
- - **Root cause** — FastAPI decodes `%2F` before route matching, so node IDs containing `/` (e.g. `gene/protein:6164`) split the path-segment route and return 404. The frontend encoded slashes correctly but they were decoded server-side before the router matched the pattern.
- - Added slash-safe query-param routes: `GET /api/graph/semantic-neighborhood?node_id=...` and `GET /api/graph/path?source=...&target=...`. Legacy path-segment routes (`/node/{id}/semantic-neighborhood`, `/node/{id}/path`) are kept as deprecated backward-compatible aliases with docstrings documenting the limitation.
- - Frontend (`GraphWorkspace.tsx`, `GraphWorkspaceShell.tsx`) now builds all distance API calls via `URLSearchParams` so node IDs with slashes or other special characters are never embedded in URL path segments.
- - `_semantic_neighborhood_impl` now returns HTTP 503 (instead of a silent 200 with zero neighbors) when semantic similarity is unavailable or the graph has no node embeddings, and HTTP 404 only when the anchor node itself does not exist. Frontend error messages updated to distinguish the two cases.
- - Fixed a pre-existing bug where `find_most_similar` was called with `(graph_dict, node_id_string)` instead of the correct `(embeddings_dict, query_vector)` signature; added `_extract_node_embeddings` and `_coerce_embedding_vector` helpers to build the embeddings dict before the call.
- - **Review fixes** (follow-up by @KaifAhmad1 and @ZohaibHassan16): aligned `_coerce_embedding_vector` inner dict-probe key list (added `"embeddings"`, reordered generic-first) with `_extract_node_embeddings` outer key list; added `TODO` comment on per-session embedding cache; extracted `_FakeSimilarity` test stub to module level to eliminate duplication; rewrote `test_legacy_semantic_neighborhood_still_works_for_simple_ids` as a fully isolated `TestClient` session instead of mutating the shared module-scoped `client` fixture.
-
- - **Fix: Explorer Distance Intelligence visible rendering** (PR #513 by @ZohaibHassan16, review fixes by @KaifAhmad1):
- - Distance Intelligence now renders as a first-class visual state through the Sigma reducer/theme pipeline instead of mutating raw graph attributes directly.
- - Ego mode fades and scales nodes by structural distance from the selected anchor; nodes outside `maxHops` are dimmed and label-suppressed.
- - Heatmap mode renders a sampled local lens capped per ring (1-hop: ≤120, 2-hop: ≤650, 3-hop: ≤900 nodes shown); true counts remain visible in the status strip. Saturation detection reduces alpha for dense outer rings automatically.
- - Structural mode highlights distance-aware context edges (colored by hop band) without breaking existing edge LOD.
- - Semantic mode surfaces loading, unavailable, and error states visibly; edges colored by cosine similarity score.
- - Trace Path inspector shows a distance band chip, hop count, and optional metric cards (confidence decay, semantic similarity, path coherence, bottleneck node) when path data is available.
- - Added `GraphDistanceVisualState`, `GraphDistanceBucketCounts`, and `GraphHeatmapRenderSnapshot` types in `types.ts`; distance state flows through `GraphCanvas` → `buildReducerSceneState` → Sigma node/edge reducers.
- - Added `buildStructuralDistanceSnapshot` (bounded BFS), `summarizeDistanceBuckets`, `buildHeatmapRenderSnapshot` (ring-capped deterministic sampling via `hashString` tiebreaker), `resolveDistanceNodeStyle`, and `resolveDistanceEdgeStyle` in `graphSceneState.ts`.
- - Distance Intelligence status strip shows active mode, anchor label, per-ring node counts, sampled status, and a color legend.
- - **Review blockers fixed** (follow-up by @KaifAhmad1 and @ZohaibHassan16): removed dead `if (anchorNodeId)` conditional in `buildHeatmapRenderSnapshot` (anchor always truthy past early-return guard); replaced O(n) `.includes()` call in the Sigma reducer hot path with a `WeakMap`-cached `Set.has()` lookup; renamed `GraphDistanceBucketCounts.threeHop → threeHopPlus` so the field accurately reflects ≥ 3 hops and updated status strip labels to "3+ hop"; restored `hasMetrics` guard in `PathDistanceIntelPanel` to suppress the empty metric grid `<div>` when a path result carries no optional metric fields.
-
- - **Feature: Graph Explorer visual refresh** (PR #503 by @ZohaibHassan16, conflict resolution by @KaifAhmad1):
- - Extracted all hardcoded `rgba(...)` color literals into a structured `ui.*` design-token namespace in `graphTheme.ts` — covering `ui.text`, `ui.surface`, `ui.scene`, `ui.control`, `ui.timeline`, and `ui.interaction`. Future theming is now a one-file change.
- - Added `GraphEntityShapeVariant` type and per-shape config (`fillAlpha`, `shellAlpha`, `coreScale`, `borderBoost`, `minSize`) for biomolecule, condition, compound, process, community, and entity node kinds. Shell and fill colors now derive from per-entity-shape config rather than uniform overrides.
- - Decomposed the monolithic `coreToolbarGroups` useMemo into focused per-cluster memos (`viewModeItems`, `cameraToolbarItems`, `layoutToolbarItems`, `localToolbarItems`, `analysisToolbarItems`, `utilityToolbarItems`) each with minimal deps arrays. Distance Intelligence controls (ego mode, heatmap, structural/semantic overlay) ported into a new `distanceToolbarItems` cluster, gated on node selection.
- - Replaced the raw `<input>` search bar and inline `<button>` loop with typed sub-components: `SearchCommandBar`, `SegmentedModeControl`, `ToolbarCluster`, `ToolbarButton`, and `EntityVisualKey`. All carry `aria-label`, `role`, and `disabled` attributes.
- - Added `GraphFullEdgeClass` union (`hidden | backbone | bridge | local-context | selected | path | muted`), `classifyFullGraphEdge`, and `resolveEdgeVisibilityPolicy` for deterministic per-mode LOD edge classification. Full-graph mode visibility and context caps are now declared as data (`edges.visibility`, `edges.contextCaps`) per view mode × zoom tier.
- - Added `GraphRuntimeDiagnosticsSnapshot` type; `onDiagnosticsChange` callback now emits `{ effectAvailability, edgeClasses, structureLayer }` instead of the internal `effectAvailability` sub-object.
- - Edge visual state weights (size multipliers, min sizes) tuned for quieter large-graph rendering: default edges dropped from `0.96×` to `0.48×`; muted edges from `0.6×` to `0.24×`; path/selected edges raised slightly to maintain hierarchy contrast.
- - Scene grid updated to a two-frequency pattern (minor 48 px, major 240 px) with tokens sourced from `GRAPH_THEME.ui.scene`.
- - `focusNode` early-return now clears `selectedNodeId`, `selectedEdgeId`, `pathResult`, `searchResults`, and `searchError` when called with an empty string.
- - Added 15 new display-state and edge-classification tests in `explorer/tests/graphSceneState.display.test.ts`.
-
- - **Feature: Distance Intelligence** (closes #502 by @KaifAhmad1):
- - **Context layer** — `ContextGraph.get_neighbors()` gains `include_distance_metadata=False`; when enabled adds `distance_band`, `confidence_decay`, and `path_to_anchor` per result. New `get_neighbor_distances()` returns neighbors sorted by `(hop, -decay)` with optional `min_confidence` filter. `AgentContext.retrieve()` / `find_precedents()` accept `anchor_node`, `max_hops`, `proximity_weight`, `min_confidence_decay` and blend graph proximity with semantic score as `combined_score = (1 − w) × semantic + w × proximity`.
- - **Path enrichment (FR-4)** — `GET /api/graph/node/{id}/path` now returns `semantic_similarity`, `path_coherence_score`, `confidence_decay` (O(L) via pre-built edge-weight index), `bottleneck_node`, `alternative_path_count`, and `interpretation`. All fields optional; zero breaking changes.
- - **Distance matrix (FR-6)** — `POST /api/graph/distance-matrix` accepts up to 50 nodes and metric `hops | weighted | semantic`. Returns N × N matrix (upper-triangle computed, lower mirrored), unreachable pairs, and `computation_time_ms`.
- - **Semantic neighborhood (FR-3 backend)** — `GET /api/graph/node/{id}/semantic-neighborhood?top_k=N` returns the N most similar nodes with `id`, `type`, `content`, `similarity`, `hop_distance`.
- - **Causal distance (FR-8)** — `GET /api/decisions/causal-distance?source=&target=` traverses only causal-typed edges and returns `CausalDistanceReport` with path, hop count, `confidence_decay`, `weakest_link`, and interpretation.
- - **Temporal distance history (FR-9)** — `GET /api/temporal/distance-history` samples 11 evenly-spaced snapshots across the graph's time range and emits `convergence | divergence | disconnected | reconnected` events.
- - **Distance-enriched export (FR-10)** — `POST /api/export/distance-enriched` streams pairwise hop/weighted/semantic/band/centrality metrics as CSV or JSONL. `node_subset` capped at 200 nodes.
- - **Explorer UI** — Path inspector panel (`GraphInspectorPanel.tsx`) shows a distance band chip, progress-bar metric cards (decay, similarity, coherence), bottleneck node highlight, and interpretation text. Toolbar gains Ego Mode (client-side BFS depth-of-field fading, depth slider 1–8), Structural overlay (edges colored by hop distance), Semantic overlay (edges colored by cosine similarity), and Heatmap (nodes colored green → red by hop distance). Ego and heatmap share a single merged `useEffect` to prevent `restoreNodeColors()` races.
- - **Tests** — 57 new tests in `tests/context/test_distance_intelligence.py`; 18 targeted regression tests in `tests/_smoke_review_fixes.py`.
-
- - **Fix: Distance Intelligence — code review regressions** (PR #502 follow-up by @KaifAhmad1):
- - `GraphWorkspace.tsx` semantic fetch used `?limit=50`; corrected to `?top_k=50` to match the backend param (bug_001). Response type widened to full `SemanticNeighborhoodResponse` shape (bug_002).
- - `ContextGraph.get_neighbors()` was embedding distance metadata unconditionally, breaking existing callers; gated behind `include_distance_metadata=False` default (bug_003).
- - `weakest_link` dict key standardised from `weight` → `edge_weight` across `CausalChainAnalyzer` and `CausalDistanceReport` (bug_004).
- - Temporal distance history sampling replaced `timetuple()[:6]` reconstruction with `min_bound + timedelta(seconds=...)` (bug_005).
- - Confidence decay in `find_path` was O(E × L); replaced with a single O(E) edge-weight index built before the hop loop, with undirected mirroring (bug_006).
- - `AgentContext._apply_proximity_metadata()` was overwriting the original record `"id"` with the graph node id; stored as `"graph_node_id"` instead (bug_007).
- - Path highlight sweep animation used a shared `sweepTimer`; stale callbacks fired after cancellation. Added `sweepGeneration` counter — callbacks no-op if generation no longer matches (bug_008).
- - `POST /api/export/distance-enriched` now rejects `node_subset` larger than 200 nodes with HTTP 413 (sec_001).
- - `POST /api/graph/distance-matrix` now computes only the upper triangle and mirrors results, halving computation cost (sec_002).
- - Ego mode and heatmap `useEffect` hooks merged into one to eliminate concurrent `restoreNodeColors()` race (qual_001).
- - Bare `except Exception: pass` blocks in `find_path` and `semantic_neighborhood` replaced with `logger.debug(...)` (qual_002).
- - Duplicated `_distance_band()` static method removed from `CausalChainAnalyzer` and `AgentContext`; both now use `classify_path_distance` from `semantica.utils.helpers` (qual_003).
-
- - **Feature: Graph Workspace declutter + calmer structural exploration** (PR #483 by @ZohaibHassan16, follow-up by @KaifAhmad1):
- - Added a calmer default presentation for dense graphs: reduced label pressure, stronger inactive-state muting, and tuned zoom-tier visibility to improve readability during overview and structure navigation.
- - Added display-edge aggregation with raw-edge bundle metadata retention, enabling cleaner visuals while preserving drill-down context for selected edges.
- - Added grouped community view and neighborhood collapse/expand controls for high-degree local structures in Graph Workspace and Neighborhood panel flows.
- - Extended graph selection/runtime state with display-state metadata (`groupedViewAvailable`, visible/collapsed neighbor counts, aggregated edge descriptors) for plugin and panel introspection.
- - Added regression coverage for `resolveDisplayGraph` behavior in `explorer/tests/graphSceneState.display.test.ts`:
- - parallel-edge aggregation in full view
- - collapse behavior preserving active-path neighbors
- - grouped community-node/community-edge projection behavior
- - Follow-up merge resolution synced the PR branch with `main` after Explorer path migration (`semantica-explorer` -> `explorer`) and preserved PR #483 behavior in conflicted Graph Workspace files.
-
- - **Fix: DeepSeekProvider now uses OpenAI SDK instead of unmaintained deepseek SDK** (closes #482, PR #482 by @liling, review fixes by @KaifAhmad1):
- - **Root cause**: The `deepseek` PyPI package has no `deepseek.Client`, causing `AttributeError` on every `DeepSeekProvider` instantiation. The DeepSeek API is OpenAI-compatible, so the `openai` SDK is the correct client.
- - **`_init_client` rewritten**: Replaced `import deepseek; deepseek.Client(api_key=...)` with `from openai import OpenAI; OpenAI(api_key=..., base_url=self.base_url)`, matching the pattern already used by `NovitaProvider`.
- - **`self.base_url` added to `__init__`**: Set to `"https://api.deepseek.com/v1"` (with `/v1` suffix required by the OpenAI SDK for correct endpoint resolution). This was missing from the original PR, causing a second `AttributeError` at `_init_client` call time.
- - **`generate_typed` `verbose_mode` fix**: `verbose_mode` was referenced before assignment inside the instructor path. Added assignment `verbose_mode = kwargs.get("verbose", False) or self.config.get("verbose", False)` at the correct scope.
- - **`pyproject.toml` updated**: `llm-deepseek` extra now declares `openai>=1.0.0` instead of the defunct `deepseek>=0.1.0`.
- - **Warning message updated**: `_init_client` ImportError warning now references the `openai` library and `llm-openai` extra.
- - **Instructor path improved**: Since `self.client` is now an `OpenAI` instance, the `isinstance(self.client, OpenAI)` check in `generate_typed` passes correctly, avoiding a redundant second client construction.
- - 19 new tests in `tests/semantic_extract/test_pr482_deepseek_openai.py` across five suites: `TestDeepSeekProviderInit` (8 — covers `base_url`, OpenAI instantiation, no `deepseek` import, ImportError handling, `is_available`), `TestDeepSeekProviderGenerate` (5 — `generate`, `generate_structured`, no-client error paths), `TestDeepSeekInstructorPath` (1 — `isinstance` check), `TestVerboseModeAssignment` (4 — no NameError, verbose kwarg, config verbose, no-print default), `TestDeepSeekGenerateTypedInstructorIntegration` (1 — end-to-end instructor path reuses existing client).
-
- - **Performance: Indexed search for large knowledge graphs** (closes #467, PR #481 by @ZohaibHassan16, review fixes by @KaifAhmad1):
- - **Root cause**: The previous `GraphSession.search()` ran a full O(n) scan over all nodes per query, serializing every node's properties to JSON for string matching. On a 118 k-node graph warm queries took 24–471 ms; a session with 500 k nodes was effectively unusable.
- - **New `semantica/explorer/search_index.py`**: Purpose-built in-memory inverted index with three lookup tiers — exact-term index (full normalized strings), token index (individual words), and prefix index (2–12 character prefixes of every token). A linear secondary-scan fallback (capped at 12 k nodes) handles queries that miss all three tiers. An LRU result cache (128 slots, `OrderedDict`) serves repeat queries at zero cost. Warm query times on the same 118 k-node graph: 24 ms → 0.004 ms (exact), 471 ms → 0.009 ms (ID lookup), 475 ms → 0.002 ms (no-match).
- - **`IndexedNodeDocument`** frozen dataclass stores per-node primary text (ID, content, curated alias keys: `label`, `name`, `pref_label`, `aliases`, `synonyms`, `display_name`, etc.), secondary text (remaining properties), token set, prefix expansions, confidence, and tags. Primary text prioritizes human-readable fields; secondary text covers the full property bag up to a 48-fragment cap.
- - **Scoring**: exact ID match → 140, exact term → 120, primary-text substring → 78 + length bonus, token hit → 18, prefix hit → 10, multi-token bonus → 4 per hit. Ties broken deterministically on `(score, exactness, token_hits, node_id)`.
- - **Mutation sync**: `GraphSession.add_node()`, `add_nodes()`, `add_edges()` and `add_node()` update the index incrementally. `handle_graph_mutation()` integrates with the WebSocket mutation bridge for live updates during reasoning, enrichment, and remote graph changes. `rebuild_search_index()` performs a full O(n) rebuild when needed (session init, merge, reload). `enrich.py` routes node/edge additions through `session.add_node`/`session.add_edge` so reasoning-inferred nodes are indexed immediately.
- - **Review fixes applied**: replaced `list.sort()` per upsert with `bisect.insort()` (O(log n) vs O(n log n)); replaced `list.remove()` with `bisect.bisect_left` + `pop()` (O(log n) vs O(n)); added `with self._lock` in `handle_graph_mutation()` to prevent index races from the WebSocket thread; removed unnecessary source/target upserts on `add_edge()` (edges don't affect node text); sorted tag values in `_cache_key()` so `["a","b"]` and `["b","a"]` share a cache entry.
- - **Follow-up fix by @KaifAhmad1 and @ZohaibHassan16**: restored `_ordered_node_ids` maintenance during upserts so `secondary_scan` fallback works for terms that are only present in non-curated properties; added regression test coverage to lock this behavior.
- - 3 new tests: `test_search_exact_and_prefix` (exact match + prefix match), `test_search_filters_and_cache_stability` (type + confidence filter, identical repeated requests), `test_search_sees_new_nodes_after_mutation` (node added via `session.add_node()` immediately visible in search).
- - 1 additional regression test: `test_search_secondary_scan_fallback_matches_non_curated_properties` (verifies fallback matching when a query term appears only in non-curated properties).
- - **Fix: Provenance traversal now includes multi-hop upstream ancestors + edge direction classification** (closes #470, PR #480 by @Sameer6305, review fixes by @KaifAhmad1):
- - **Bug — upstream ancestors silently excluded**: `_build_provenance()` built a directed `nx.DiGraph` and seeded first-hop neighbors correctly, but the final subgraph extraction called `nx.ego_graph(..., undirected=False)`. With directed traversal, ego-graph expansion only follows outgoing edges from the focus node, so any node that *points into* the focus node (i.e. an upstream ancestor at depth ≥ 2) was invisible. For the chain `Source → Intermediate → node_id`, `Intermediate` appeared at hop 1 but `Source` was silently dropped. Fixed by changing to `undirected=True` — the radius expansion now traverses both incoming and outgoing edges while the underlying `DiGraph` is preserved, so edge source/target semantics remain correct.
- - **Enhancement — edge direction classification**: `ProvenanceEdge` gains a `direction: str` field. Each edge in the provenance subgraph is classified relative to the focus node: `"upstream"` when `target == node_id` (edge flows into the focus node), `"downstream"` when `source == node_id` (edge flows out), and `"lateral"` for all other edges between non-focus neighbors. This lets consumers distinguish ancestor provenance from descendant impact without re-traversing the graph.
- - **Enhancement — grouped markdown report**: `_render_markdown()` now groups lineage edges under separate `## Upstream`, `## Downstream`, and `## Lateral` sections instead of a flat `## Lineage Edges` list. Empty sections are omitted. This improves readability of exported provenance reports.
- - **Schema consolidation**: `ProvenanceNode`, `ProvenanceEdge`, and `ProvenanceResponse` moved from inline definitions in `routes/provenance.py` to the shared `semantica/explorer/schemas.py`, matching the convention used by all other Explorer routes. `ProvenanceNode.parent_id` is now `Optional[str] = None`.
- - **Merge conflicts resolved**: Resolved all conflict markers in `provenance.py`, `app.py`, and `.gitignore`; restored the complete router import set in `app.py` (`graph`, `sparql`, `temporal`, `vocabulary`) that the conflict had dropped.
- - 2 new tests in `tests/explorer/test_provenance_route.py`: `test_build_provenance_direction_classification_chain` (asserts `Source` and `Intermediate` both appear for `Source → Intermediate → node_id`; verifies `Intermediate → node_id` classified as `"upstream"`) and `test_render_markdown_groups_edges_by_direction` (asserts grouped section headings and correct edge lines in output).
-
- - **Fix: `OWLExporter._export_owl_turtle` invalid Turtle syntax and silent data-property omission** (closes #478 by @KaifAhmad1):
- - **Bug 1 — Invalid Turtle syntax**: `_export_owl_turtle` unconditionally wrote `rdfs:label` with a closing period (`.`), then appended `rdfs:subClassOf`, `rdfs:domain`, and `rdfs:range` triples after the closed block. Any RDF parser would reject the output. Fixed by introducing `_ttl_block(subject_uri, rdf_type, predicates)` — all predicate-object pairs for a subject are accumulated first, then joined with ` ;\n ` and terminated with a single ` .`, producing valid Turtle in all cases.
- - **Bug 2 — Data properties silently dropped**: `_export_owl_turtle` had loops for `classes` and `object_properties` but no loop for `data_properties`, so all `owl:DatatypeProperty` declarations were silently omitted. Added the missing loop, mirroring the existing object-property loop.
- - **String escaping**: User-provided strings (`name`, `description`, `comment`, version) were embedded directly into Turtle string literals without escaping. A class named `John"s Class` or a comment containing a backslash or newline produced unparseable output. Added `_escape_ttl_str()` static method (escapes `"`, `\`, `\n`, `\r`, `\t`) applied at every `rdfs:label`, `rdfs:comment`, and `owl:versionInfo` site.
- - **Null-check consistency**: All optional field reads now use `x = prop.get("field"); if x:` uniformly — eliminates the mixed pattern of `.get()` guards followed by direct `[]` access.
- - 43 tests added in `tests/export/test_owl_exporter.py` across five suites: `TestTurtleSyntaxValidity` (5), `TestDataPropertiesInTurtle` (8), `TestTurtleHeader` (4), `TestTurtleStringEscaping` (16), `TestNullFieldHandling` (7), plus `TestObjectPropertyListDomainRange` (2) and `TestEquivalentClass` (1).
-
- - **Enhancement: Node distance semantics in path responses** (closes #472 by @KaifAhmad1): `PathResponse` now surfaces two new first-class fields — `hop_count: int` (equal to `len(path) - 1`; `0` for self-paths) and `distance_band: str` — so callers no longer need to count hops or implement band classification themselves. Four bands are defined: `"direct"` (0–1 hops), `"near"` (2–3), `"mid-range"` (4–6), `"distant"` (7+). The classification function `classify_path_distance()` lives in `semantica/utils/helpers.py` as the single source of truth; both the Explorer route and the visualizer import from it. `KGVisualizer.visualize_network()` gains an optional `highlight_path: list[str]` parameter: when provided, path edges are rendered as a separate orange trace with opacity and stroke width scaled to the distance band (direct: 1.0 / 4 px → distant: 0.35 / 1.5 px), while non-path edges render at reduced opacity underneath. Edge direction is respected — only the forward pairs `(A, B)` along the path are matched; reverse back-edges in directed graphs are not incorrectly highlighted. A logger warning is emitted when any node ID in `highlight_path` has no layout position, surfacing silent no-op mismatches. Frontend `PathResponse` type in `GraphInspectorPanel.tsx` and `GraphWorkspaceShell.tsx` extended with `hop_count: number` and `distance_band: "direct" | "near" | "mid-range" | "distant"`. All changes are additive; no existing fields removed. 10 new tests: 2 API-level (`test_response_includes_hop_count_and_distance_band`, `test_one_hop_path_is_direct`) and 8 unit tests covering all four band boundaries (0, 1, 2, 3, 4, 6, 7, 20 hops).
-
- - **Enhancement: Bidirectional path finding in Knowledge Explorer** (closes #469 by @KaifAhmad1): Path queries in the Explorer were direction-sensitive — querying B→A when only the edge A→B existed always returned no result, because `PathFinder._get_neighbors()` called `graph.neighbors(node)` which on a `nx.DiGraph` yields only successors. Added a `directed: bool = True` parameter to `bfs_shortest_path()` and `dijkstra_shortest_path()`. When `directed=False` a lightweight undirected view is built via `graph.to_undirected()` for the traversal pass only; the original directed edges are preserved and returned in the response. A `_make_undirected_view()` helper encapsulates the conversion and falls back safely for non-NetworkX graph types. The `/api/graph/node/{id}/path` route exposes the parameter as a query string flag (`?directed=false`); `PathResponse` gains a `directed: bool` field that echoes the mode used. Default is `True`, so all existing callers are unaffected. The route also gained an empty-path 404 guard — previously a traversal that found no path returned `200` with `path: []` instead of `404`. 21 new tests: 12 unit tests in `TestBidirectionalPathFinding` (`tests/kg/test_path_finder.py`) and 9 API-level tests in `TestBidirectionalPathRoute` (`tests/explorer/test_explorer_api.py`).
-
- - **Enhancement: Native `KnowledgeGraph` type support in `KGVisualizer`** (PR `kg` by @KaifAhmad1, closes #471): Added `semantica/kg/knowledge_graph.py` — a formal `KnowledgeGraph` dataclass (`entities`, `relationships`, `metadata`) that is now the canonical in-memory type produced and consumed by the Semantica KG pipeline. Exported from `semantica.kg`. `KGVisualizer` gains `_convert_knowledge_graph()` — an authoritative, non-mutating conversion path from `KnowledgeGraph` to the internal dict format — and `_normalize_graph()` now routes `isinstance(graph, KnowledgeGraph)` through it as an explicit fast-path before duck-typing. All five public entry points (`visualize_network`, `visualize_communities`, `visualize_centrality`, `visualize_entity_types`, `visualize_relationship_matrix`) accept `KnowledgeGraph` directly; no manual conversion required. All existing callers passing dicts or duck-typed objects are unaffected. 15 new tests in `TestFormalKnowledgeGraphType` (conversion shape, non-mutation, determinism, routing, all five entry points, import availability).
- - **Fix: `KGVisualizer` now accepts `KnowledgeGraph` objects in all `visualize_*` methods** (PR `visualization` by @KaifAhmad1, closes #458): All five public methods (`visualize_network`, `visualize_communities`, `visualize_centrality`, `visualize_entity_types`, `visualize_relationship_matrix`) previously called `graph.get("entities", [])`, silently producing no output when passed a non-dict object. Added `_normalize_graph()` which duck-types the input — dicts pass through unchanged; any object exposing `.entities` / `.relationships` attributes (e.g. the result of `GraphBuilder.build()`) is converted to the canonical dict form; anything else raises a clear `ProcessingError` naming the offending type. 21 tests added in `tests/visualization/test_kg_visualizer_normalize_graph.py`.
-
- - **Security: 12 vulnerability fixes across CRITICAL → LOW severity** (PR `security-enhancement` by @KaifAhmad1):
-
- **Critical**
- - **Eval injection eliminated** (`semantica/parse/media_parser.py`): Replaced `eval(stream.get("r_frame_rate", ...))` — which executed arbitrary Python from ffprobe JSON output — with a `_safe_parse_fps()` helper using `fractions.Fraction`. No code execution possible regardless of ffprobe output content. (CWE-95)
- - **Unsafe pickle deserialization replaced** (`semantica/context/agent_memory.py`): `AgentMemory.save()` / `load()` previously used `pickle.dump` / `pickle.load`, allowing RCE if an attacker could write the `.pkl` file. Replaced with `json.dump` / `json.load`. `MemoryItem.to_dict()` / `MemoryItem.from_dict()` added for safe round-trip serialization — `timestamp` via `isoformat()`, `embedding` dropped (not JSON-safe, regenerated on demand). Legacy `.pkl` files are detected and refused with a migration message. (CWE-502)
-
- **High**
- - **SQL injection hardened** (`semantica/ingest/snowflake_ingestor.py`): `WHERE`, `ORDER BY`, `LIMIT`, and `OFFSET` were f-string interpolated directly into Snowflake queries. `LIMIT` / `OFFSET` now use parameterized `%s` placeholders; `ORDER BY` is validated against a strict `^[A-Za-z_][A-Za-z0-9_]*(\s+(ASC|DESC))?` regex; `WHERE` clauses containing semicolons are rejected before execution. (CWE-89)
- - **XXE protection added for RDF/XML parsing** (`semantica/explorer/utils/rdf_parser.py`): `rdflib.Graph().parse()` on XML-based RDF formats had no external-entity restrictions. Added `_safe_parse_rdf()` wrapper that calls `defusedxml.defuse_stdlib()` before parsing, neutralising Billion Laughs and local file-read XXE attacks. Falls back gracefully with a `UserWarning` if `defusedxml` is not installed. (CWE-611)
- - **Security headers and CORS added to main server** (`semantica/server.py`): No CORS policy, no response security headers, and raw `Exception` details were returned to clients. Added `CORSMiddleware` (origins from `SEMANTICA_CORS_ORIGINS` env var, defaults to `localhost` only); `_SecurityHeadersMiddleware` emitting `X-Content-Type-Options`, `X-Frame-Options`, `X-XSS-Protection`, `Referrer-Policy`, `Permissions-Policy`, and HSTS (HTTPS only) on every response; global error handler that logs internally and returns a generic `500`. (CWE-346, CWE-200)
- - **CORS and WebSocket hardened in Explorer app** (`semantica/explorer/app.py`): `allow_methods=["*"]` and `allow_headers=["*"]` narrowed to `GET, POST, DELETE, OPTIONS` and `Content-Type, Authorization` only. `KeyError` / `ValueError` exception handlers now log the real message server-side and return generic text to clients. WebSocket messages larger than 64 KB trigger close with code `1009` (Message Too Big), preventing memory exhaustion via large frame injection. (CWE-346, CWE-400)
-
- **Medium**
- - **Algorithm parameter validated by enum** (`semantica/explorer/routes/graph.py`): The `algorithm` query parameter previously accepted any string; unknown values silently fell back to BFS. Replaced with `_PathAlgorithm(str, Enum)` — FastAPI now returns `422 Unprocessable Entity` for any value other than `bfs` or `dijkstra`. (CWE-20)
- - **RDF upload extension allowlist** (`semantica/explorer/routes/vocabulary.py`): No file extension check was performed before reading RDF uploads. Extension is now validated against `{".ttl", ".rdf", ".owl", ".xml", ".jsonld", ".json-ld", ".json"}` before any content is read. (CWE-434)
- - **Prompt injection mitigated** (`semantica/semantic_extract/llm_extraction.py`): User-supplied text and entity/relation labels were embedded directly into LLM prompts via f-string interpolation — a crafted input like `"\n\nIgnore all above instructions..."` could override system instructions. All user-supplied content is now passed through `json.dumps()` before embedding, neutralising newlines, quotes, and instruction-override attempts. (CWE-1336)
- - **Dynamic `__import__()` removed** (`semantica/pipeline/pipeline_validator.py`): `__import__("collections").Counter(...)` replaced with a proper `from collections import Counter` module-level import. (CWE-95)
- - **ReDoS eliminated** (`semantica/explorer/routes/enrich.py`): `re.split(r"\s+AND\s+", antecedent_text, re.IGNORECASE)` on user-supplied rule strings could exhibit polynomial backtracking. Fixed by normalising whitespace first with `" ".join(text.split())` (no regex) then splitting on the literal `" AND "`. Closes CodeQL alert #12. (CWE-1333)
- - **Path traversal blocked** (`semantica/server.py`): SPA catch-all route used `STATIC_DIR / full_path` without validation. Added `Path.resolve()` + `relative_to()` check that returns `400 Bad Request` for any path that escapes `STATIC_DIR`. Closes CodeQL alerts #13 and #14. (CWE-22)
-
- **Low**
- - **SPARQL result cap and timeout** (`semantica/explorer/routes/sparql.py`): SPARQL queries ran to completion with no row limit or timeout; expensive queries could exhaust memory or block indefinitely. Results are now capped at 5 000 rows; `asyncio.wait_for(..., timeout=30)` abandons the await after 30 seconds and returns a structured error response. A module-level `asyncio.Semaphore(4)` caps concurrent in-flight `graph.query` calls so that timed-out threads (which continue running in the pool) cannot crowd out other requests by exhausting executor workers. `SparqlResponse` gains a `truncated: bool` field so callers can detect a capped result set. (CWE-400)
- - **Import upload size limit and extension allowlist** (`semantica/explorer/routes/export_import.py`): No file size or type checks were enforced before reading import uploads. Extension is now validated against `{".json", ".csv"}` (the formats the handler actually parses — allowlist trimmed to match implementation); a hard 50 MB cap is enforced before content is read. (CWE-434)
-
- **CodeQL / scanning infrastructure**
- - Added `.github/codeql/codeql-config.yml` with `paths-ignore` for `cookbook/**/*.html` and `cookbook/**/*.js`. Notebook-exported HTML files embed entire minified third-party bundles (Plotly + MapLibre GL JS v4.7.1) that triggered false-positive JS alerts #15–#18. The Advanced Setup workflow now references this config via `config-file:`. Closes CodeQL alerts #15, #16, #17, #18.
- - Removed blanket rule-ID auto-dismiss job from `.github/workflows/codeql.yml`. The previous job dismissed every open alert whose `rule.id` matched a fixed list on each `main` push — this would silently suppress any future real vulnerability of the same type. Replaced with a commented template for pinning specific alert numbers when manual dismissal is genuinely required.
-
- - **Fix: Knowledge Explorer — blockers and security hardening** (PR #420 by @ZohaibHassan16, review fixes by @KaifAhmad1):
- - **Dockerfile**: Renamed `DockerFile` → `Dockerfile` (case-sensitive filename caused Docker build failures on Linux CI). Fixed `CMD` module path from the non-existent `semantica.server:app` to `semantica.explorer.app:app`, which caused the Docker image to crash on startup. Added `app = create_app()` at module level in `semantica/explorer/app.py` so uvicorn can reference the ASGI app instance directly.
- - **CORS hardening**: Changed `EXPLORER_CORS_ORIGINS` default from `"*"` to `"http://localhost:5173,http://127.0.0.1:5173"`. Any deployment that does not explicitly set the env var no longer exposes the API to all origins. The env var override continues to work as before.
- - **`get_ws_manager()` guard** (`semantica/explorer/dependencies.py`): `get_ws_manager()` now raises HTTP 503 if `app.state.ws_manager` is absent, matching the existing `get_session()` guard. Previously raised an unhandled `AttributeError` during testing or if the lifespan had not completed.
- - **SPARQL read-only enforcement** (`semantica/explorer/routes/sparql.py`): Added `_is_read_only_query()` regex guard — rejects any query whose first keyword is not `SELECT`, `ASK`, `CONSTRUCT`, or `DESCRIBE`. `INSERT`, `DELETE`, `UPDATE`, `LOAD`, and `DROP` queries now return a structured `SparqlResponse` error instead of being executed against the rdflib projection.
- - **Vocabulary import size limit** (`semantica/explorer/routes/vocabulary.py`): Added 10 MB upload cap for both file and raw-text payloads — returns HTTP 413 with a human-readable message before calling `parse_skos_file()`. Prevents memory exhaustion from oversized RDF uploads.
- - **JSON-LD format auto-detection** (`semantica/explorer/routes/vocabulary.py`): Import route now detects `.jsonld`, `.json-ld`, and `.json` file extensions and passes `"json-ld"` to `parse_skos_file()`. Previously these extensions fell through to `"turtle"` and failed silently despite JSON-LD being listed as a supported format.
- - **Annotation O(1) lookup** (`semantica/explorer/routes/annotations.py`, `semantica/explorer/session.py`): `create_annotation` previously called `get_annotations()` and scanned the full list to find the just-created annotation (O(N)). Added `GraphSession.get_annotation(annotation_id)` — O(1) dict lookup — and updated the route to use it directly.
- - **Self-loop guard in `batchMergeEdges`** (`semantica-explorer/src/store/graphStore.ts`): Added `if (source === target) continue` guard at the start of the loop. The Graphology instance is initialised with `allowSelfLoops: false`; a self-loop edge from reasoning inferences or provenance cycles previously caused an uncaught Graphology error that silently broke graph loading.
- - **Static build artifacts removed from git** (`.gitignore`): Added `semantica/static/` to `.gitignore` and removed all pre-built Vite bundles from version control. The Docker multi-stage build already rebuilds the frontend from source; committing minified bundles bloated repository history and caused merge conflicts on every frontend change.
-
- - **Fix: TripletStore.store() IRI resolution regressions** (PR #447 follow-up by @KaifAhmad1):
- - Fixed `AttributeError` crash when entity or relationship IDs are non-string types (e.g. integers emitted by `GraphBuilder`). `_resolve_iri()` previously called `.startswith()` directly on the raw ID; it now coerces any value to `str()` at entry, restoring the implicit stringification that the old f-string URN minting provided.
- - Fixed W3C vocabulary prefixes (`owl:Thing`, `xsd:date`, `rdfs:Literal`, `skos:Concept`, etc.) being incorrectly re-namespaced under the ontology `base_uri` (e.g. `https://example.com/owl:Thing`) when `base_uri` was present. `_resolve_iri()` now consults a known-prefix expansion table (`xsd`, `rdf`, `rdfs`, `owl`, `skos`, `semantica`) before applying `base_uri`, matching the same prefix map already used in `BlazegraphStore`. Standard vocabulary IRIs are always expanded to their canonical W3C forms regardless of what `base_uri` is set to.
- - Added 5 regression tests: integer IDs with and without `base_uri`, `owl:Thing` domain/range, `xsd:date` range, and `skos:Concept` parent class expansion. Total tests in `TestTripletStoreOntologyNamespace`: 14.
-
- - **Fix: TripletStore.store() ignores ontology namespace base_uri** (PR #447 by @KaifAhmad1):
- - `store(knowledge_graph, ontology)` was minting `urn:entity:{id}`, `urn:class:{type}`, and `urn:property:{name}` URIs for all bare local names, even when `ontology.namespace.base_uri` was present. This made instance data and ontology class data irreconcilable in SPARQL joins.
- - Extracts `base_uri` once from `ontology["namespace"]["base_uri"]` (with `ontology["uri"]` as fallback) and ensures a trailing separator so concatenation is always a valid IRI path.
- - Introduced `_resolve_iri(local, kind)` closure applied to all 7 IRI-minting sites: entity URIs, entity types, relationship predicates, ontology class URIs, parent class URIs, property URIs, and domain/range URIs. Explicit `entity["uri"]` values are never overridden. Falls back to `urn:` only when no `base_uri` is available.
- - Added 9 tests in `TestTripletStoreOntologyNamespace` covering all expansion paths, `urn:` fallback, explicit URI passthrough, top-level `uri` key fallback, and trailing-slash safety.
-
- - **Fix: Blazegraph literal serialization and SPARQL injection hardening** (PR #448 by @KaifAhmad1):
- - Fixed `_build_ntriples()`, `_build_insert_data()`, `find_triplets()`, and `delete_triplet()` in `BlazegraphStore` — all four methods previously unconditionally wrapped every triplet object in `<...>` as an IRI, causing Blazegraph to reject or misparse any triple whose object was a plain string, typed literal, or language-tagged literal.
- - Added `_format_object_for_sparql(triplet)` — central formatter that selects the correct SPARQL/N-Triples token: IRI (`<uri>`), typed literal (`"value"^^<datatype>`), language-tagged literal (`"value"@lang`), or plain literal (`"value"`).
- - Added `_resolve_datatype_iri(datatype)` — expands prefixed datatype names (`xsd:integer`, `rdf:langString`, `rdfs:Literal`, `owl:real`, `skos:notation`) to their full IRIs instead of producing invalid `<xsd:integer>` tokens. Accepts full `http/https/urn` IRIs and already-bracketed IRIs after whitespace validation. Rejects unknown prefixes and bare local names with a clear `ValueError`.
- - Added language-tag validation against RFC 5646 (`^[a-zA-Z]{1,8}(-[a-zA-Z0-9]{1,8})*$`) — values containing whitespace, dots, or other punctuation (e.g. `"en . CLEAR ALL #"`) raise `ValueError` before interpolation, closing a SPARQL injection vector in `metadata["lang"]` / `metadata["language"]`.
- - Added datatype-string validation — whitespace and SPARQL-delimiting characters inside `metadata["datatype"]` / `metadata["literal_datatype"]` raise `ValueError`, closing the parallel injection vector for typed literals.
- - Added `_is_uri_value(value)` — URI detection using `urlparse`; rejects strings that only start with a URI scheme but contain whitespace (e.g. `"http not a uri"` is serialised as a literal, not an IRI).
- - Added `_escape_literal(value)` — escapes `\`, `"`, `\n`, `\r`, `\t` inside literal strings before SPARQL interpolation.
- - New test file `tests/triplet_store/test_blazegraph_store.py` — 15 offline unit tests covering URI serialization, plain/typed/language-tagged/escaped literals, prefix expansion, IRI passthrough, injection rejection, and `_build_insert_data` delegation; all run without a live Blazegraph instance.
-
- - **OWLGenerator user-facing schema compatibility fixes** (Issue #446):
- - Fixed OWL class/property IRI identifier fallback order to prefer `label` and then `name`.
- - Fixed datatype property handling to accept scalar and list `range` values in rdflib path (including `xsd:*`, full IRIs, and local names), preventing list-based `.startswith()` crashes.
- - Fixed generated class/property/domain/range IRIs to use the current ontology dict `uri` namespace for each generation call (instead of drifting to default namespace manager base URI when per-entity `uri` is omitted).
- - Fixed `subClassOf` / `subclassOf` parent resolution so local class names are expanded to ontology IRIs consistently with domain/range behavior.
- - Added/expanded regression coverage in `tests/ontology/test_ontology_comprehensive.py` (`test_owl_generator_user_facing_schema_compatibility`) for label-first fallback, lowercase `subclassOf`, datatype range lists, and ontology namespace consistency.
-
- - **SKOS Vocabulary Module** (PR #319 by @KaifAhmad1):
- - **Namespace helpers** (`semantica/ontology/namespace_manager.py`): Added `get_skos_uri(local_name)` — returns the full `http://www.w3.org/2004/02/skos/core#<local_name>` URI for any SKOS term. Added `build_concept_scheme_uri(name)` — slugifies a human-readable vocabulary name (spaces/special chars → hyphens, lower-cased) and anchors the result at the configured base URI as `<base>/vocab/<slug>`.
- - **Triplet-store SKOS helpers** (`semantica/triplet_store/triplet_store.py`): Added `add_skos_concept(concept_uri, scheme_uri, pref_label, alt_labels, broader, narrower, related, definition, notation)` — assembles and stores all required SKOS triples (auto-declares the `skos:ConceptScheme`, asserts `rdf:type skos:Concept`, `skos:inScheme`, `skos:prefLabel`, and all optional predicates) via the existing `add_triplets()` API; no new storage paths introduced. Added `get_skos_concepts(scheme_uri=None)` — issues a SPARQL `SELECT` via `execute_query()` and collapses multi-valued `altLabel`/`broader`/`narrower`/`related` bindings into structured concept dicts; optional `scheme_uri` restricts results to one vocabulary.
- - **OntologyEngine vocabulary APIs** (`semantica/ontology/engine.py`): Added three public methods that delegate to `QueryEngine` via `self.store.execute_query()` — `list_vocabularies()` returns all `skos:ConceptScheme` instances with labels; `list_concepts(scheme_uri)` returns every `skos:Concept` in a scheme with `pref_label` and `alt_labels`; `search_concepts(query, scheme_uri=None)` performs case-insensitive substring matching across `skos:prefLabel` and `skos:altLabel` with optional scheme scoping.
- - **Security**: `search_concepts` sanitises user input (escapes `\`, `"`, newlines) before embedding it in the SPARQL string literal. All URI interpolation uses the existing `_sanitize_uri` helper.
- - **Tests**: Added `TestSKOSOntologyEngine` (14 tests) to `tests/ontology/test_ontology_comprehensive.py` and `TestSKOSTripletStore` (6 tests) to `tests/triplet_store/test_triplet_store.py`. Coverage: URI helpers, vocabulary listing + deduplication, concept listing with multi-value alt-label collapse, search with/without scheme filter, injection sanitisation, empty results, and no-store error paths. 20 new tests, 0 failures, 1162 total passing, 0 regressions.
- - **Docs** (`docs/reference/ontology.md`): Added "SKOS Vocabulary Management" section with SKOS data-model reference table, `add_skos_concept` usage example, bulk import via rdflib + `add_triplets`, `list_vocabularies` / `list_concepts` / `search_concepts` usage examples, and `NamespaceManager` URI helper examples.
- - No new top-level Python package created; all code extends existing `semantica/ontology/` and `semantica/triplet_store/` packages. Fully opt-in and non-breaking.
-
- - **SHACL Shape Generation & Validation** (PR #318 by @KaifAhmad1):
- - **Phase 1 — Generation**: Added `SHACLGenerator` to `semantica/ontology/ontology_generator.py` — 6-stage internal pipeline: `_build_class_index` → `_generate_node_shapes` → `_attach_property_shapes` → `_propagate_inheritance` → `_apply_quality_tier` → `serialize`. Derives SHACL node and property shapes from any Semantica ontology dict; zero hand-authoring. Three output formats: Turtle, JSON-LD, N-Triples. Three quality tiers: `"basic"` (structure + cardinality), `"standard"` (+ `sh:in`, `sh:pattern`, inheritance; default), `"strict"` (+ `sh:closed true` + `sh:ignoredProperties` on all non-empty shapes). Iterative inheritance propagation up to 3+ levels, cycle-safe (max 20 passes), no duplicate property shapes per shape. No-domain properties attach to all node shapes. Added `PropertyShape`, `NodeShape`, `SHACLGraph` dataclasses.
- - **Phase 1 — Engine API**: Added `OntologyEngine.to_shacl(ontology, *, format, base_uri, shapes_uri, include_inherited, severity, quality_tier, validate_output)` and `OntologyEngine.export_shacl(ontology, path, format, encoding)` to `semantica/ontology/engine.py`. Added `RDFExporter.export_shacl(shacl_string, file_path, format, encoding)` to `semantica/export/rdf_exporter.py` with extension validation (`.ttl`, `.jsonld`, `.nt`, `.shacl`).
- - **Phase 2 — Runtime Validation**: Added `SHACLViolation` (8 fields: `focus_node`, `result_path`, `constraint`, `severity`, `message`, `value`, `shape`, `explanation`; `to_dict()`) and `SHACLValidationReport` (`conforms`, `violations`, `warnings`, `infos`, `raw_report`; `violation_count`/`warning_count` properties; `summary()`, `explain_violations()`, `to_dict()`) to `semantica/ontology/ontology_validator.py`. Added `_run_pyshacl(data_graph_str, shacl_str, data_graph_format, shacl_format)` — thin wrapper around `pyshacl.validate()` returning typed `SHACLValidationReport`. `pyshacl` and `rdflib` are optional deferred imports (`pip install semantica[shacl]`); `ImportError` with install hint raised if absent. Added `OntologyEngine.validate_graph(data_graph, shacl=None, *, ontology=None, data_graph_format, shacl_format, explain, abort_on_first)` — exactly one of `shacl`/`ontology` must be provided (`ValueError` otherwise); `explain=True` populates plain-English explanations via rule-based templates for all 7 SHACL constraint types (`MinCount`, `MaxCount`, `Datatype`, `Class`, `In`, `Pattern`, `Closed`).
- - **Exports**: `SHACLGenerator`, `SHACLGraph`, `NodeShape`, `PropertyShape`, `SHACLValidationReport`, `SHACLViolation` added to `semantica/ontology/__init__.py`.
- - **Security & reliability fixes**:
- - **High** (`engine.py`): Replaced path-vs-content heuristic (`len < 500 and "\n" not in s`) with `os.path.exists()` — prevents attacker-controlled SHACL strings from being silently interpreted as file paths.
- - **High** (`ontology_generator.py`): `_propagate_inheritance` now uses `dataclasses.replace(pps)` instead of appending parent `PropertyShape` objects by reference — mutations on a child's inherited property no longer silently affect the parent.
- - **Medium** (`engine.py` / `ontology_validator.py`): Added `shacl_format` parameter to `validate_graph` and `_run_pyshacl`; full format alias map (`"ttl"→"turtle"`, `"jsonld"→"json-ld"`, `"ntriples"→"nt"`) in both `to_shacl` validate-output and `_run_pyshacl` — JSON-LD and N-Triples shapes no longer fail parsing.
- - **Medium** (`ontology_generator.py`): `sh:ignoredProperties` now emits full URI `<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>` instead of prefixed `rdf:type` — eliminates prefix-dependency in strict-tier Turtle output.
- - **Low** (`ontology_generator.py`): `_prefix_decls` now iterates `sorted(graph.prefixes.items())` — deterministic Turtle output for reproducible CI `git diff` checks.
- - **Tests**: Added `TestSHACLGeneration` (16 tests) to `tests/ontology/test_ontology_comprehensive.py` and `TestSHACLHierarchicalAndValidation` (18 tests) to `tests/ontology/test_ontology_advanced.py`. 34 new tests, 0 failures, 1111 total passing, 0 regressions.
- - **README**: Added `## Unreleased / Coming Next` section, SHACL bullet points under Features → Ontology and Export Formats, updated Modules table, full Phase 1 + Phase 2 code examples under `## Ontology`, `pip install semantica[shacl]` under Installation.
-
- - **Temporal GraphRAG Integration** (PR #402 by @KaifAhmad1):
- - Added `TemporalGraphRetriever` to `semantica/context/context_retriever.py` — drop-in wrapper for any `ContextRetriever`; calls `base_retriever.retrieve(query)` then filters `related_entities`/`related_relationships` via `reconstruct_at_time()`; `at_time=None` is a true passthrough; returns new `RetrievedContext` objects via `dataclasses.replace()` (no in-place mutation); temporal modules guarded with `try/except` at import time.
- - Extended `ContextRetriever._generate_reasoned_response()` and `query_with_reasoning()` with `at_time` and `header_template` parameters — when `at_time` is set a structured temporal header (`[Graph context valid as of: … UTC | Source: KnowledgeGraph snapshot]`) is prepended to the LLM context block; omitted when `at_time=None` (prompt byte-identical to previous behaviour); naive datetimes normalised to UTC; header built via `str.replace` not `.format` (format-string injection guard).
- - Added `TemporalQueryRewriter` and `TemporalQueryResult` in `semantica/kg/temporal_query_rewriter.py` — extracts `temporal_intent` (`"before"`, `"after"`, `"at"`, `"during"`, `"between"`, `None`), `at_time`, `start_time`, `end_time`, and `rewritten_query` from natural-language queries; regex-only by default (zero LLM calls), optional LLM-assisted mode; datetime resolution always delegated to `TemporalNormalizer`; word-boundary guards prevent false matches (`at` inside `that`); year fallback handles noun-phrase dates like `"the 2021 merger"`; never calls `reconstruct_at_time`.
- - Exported `TemporalGraphRetriever` from `semantica.context`; exported `TemporalQueryRewriter`, `TemporalQueryResult` from `semantica.kg`.
- - **Security fixes**: format-string injection in header template (medium); unconditional temporal module import at package init (low).
- - **Bug fixes**: in-place mutation of `RetrievedContext` (high); naive datetime formatted without timezone (low); missing `timezone` import causing `NameError` (low).
- - Added 99 tests across `tests/context/test_temporal_retriever.py` (56) and `tests/kg/test_temporal_query_rewriter.py` (43); 0 failures, 0 regressions.
-
- - **Temporal Provenance & Export** (PR #401 by @KaifAhmad1):
- - **Transaction time on provenance records** (`semantica/kg/provenance_tracker.py`): `track_entity()` now automatically attaches `recorded_at = datetime.now(UTC).isoformat()` to every new record — no opt-in required. Existing records without `recorded_at` continue to work in all existing query methods (treated as unknown, not an error). Added `query_recorded_between(start, end) -> list` returning all provenance records whose `recorded_at` falls within the inclusive range; accepts `datetime` objects or ISO strings including trailing `Z`.
- - **Fact revision audit trail** (`semantica/kg/provenance_tracker.py`): Added `revision_history(fact_id) -> list` returning the complete revision chain ordered by `recorded_at` ascending; each entry includes `version` (int, 1-based), `valid_from`, `valid_until`, `recorded_at`, `author`, and optionally `revision_type`/`supersedes`; returns `[]` for unknown facts (never raises). Added `export_audit_log(fact_ids, format) -> str` supporting `"json"` (pretty-printed) and `"csv"` (with header row) formats.
- - **OWL-Time RDF export** (`semantica/export/rdf_exporter.py`): `export_to_rdf()` gains `include_temporal: bool = False` and `time_axis: str = "valid"` parameters. When `include_temporal=True`, emits OWL-Time triples (`http://www.w3.org/2006/time#`) for every relationship carrying `valid_from`/`valid_until` — a `time:Interval` node linked via `time:hasTime`, `time:hasBeginning`/`time:hasEnd` with `time:Instant` nodes, and `time:inXSDDateTimeStamp` values. `time_axis` controls which axis is exported: `"valid"`, `"transaction"`, or `"both"`. Relationships without temporal metadata are unaffected. Default `include_temporal=False` produces output identical to current behavior. **Design decision for `TemporalBound.OPEN`**: OWL-Time has no standard predicate for "no known end date" — `time:hasEnd` is omitted and `semantica:openEndedInterval "true"^^xsd:boolean` is emitted on the interval node instead. Output parses without errors in rdflib.
- - **Stable snapshot serialization format** (`semantica/kg/temporal_query.py`, new `semantica/kg/schemas/temporal_snapshot_v1.json`): `create_snapshot()` now stamps `"format_version": "1.0"` on every snapshot. Added `validate_snapshot(snapshot) -> bool` — validates required fields (`format_version`, `label`, `timestamp`, `author`, `description`, `entities`, `relationships`, `checksum`); returns `False` with structured DEBUG-level error details on failure, never raises. Added `migrate_snapshot(snapshot) -> dict` — deep-copies and upgrades old-format snapshots to v1.0, populating missing required fields with `None`; already-v1.0 snapshots returned unchanged with no data loss. New `semantica/kg/schemas/temporal_snapshot_v1.json` — JSON Schema (draft 2020-12) defining required and optional fields, types, and constraints.
- - Added 28 new tests in `tests/test_401_temporal_provenance_export.py` covering every acceptance criterion; 451 related tests pass, 0 regressions.
-
- - **Temporal Metadata Extraction from Text** (PR #400 by @KaifAhmad1):
- - Added `extract_temporal_bounds: bool = False` parameter to `extract_relations_llm()`. When `True`, the LLM prompt is extended with a calibrated confidence scale and four few-shot examples; each returned `Relation` gains `valid_from`, `valid_until`, `temporal_confidence` (0.0–1.0), and `temporal_source_text` in its `metadata` dict. Default `False` preserves 100% backward compatibility.
- - Confidence scale anchors baked into the prompt: `1.00` = full ISO date, `0.90` = year+month, `0.85` = year only, `0.75` = quarter, `0.65` = named season/approximate range, `0.50` = vague relative with computable anchor, `0.35` = highly vague, `0.00` = no temporal signal. LLMs self-report certainty rather than clustering near 1.0.
- - Low temporal confidence (< 0.5) with a non-null date logs a `WARNING`; signal is never suppressed — callers decide how to filter.
- - Cache key now includes the `extract_temporal_bounds` flag to prevent cross-mode cache pollution.
- - Flag propagated through `_extract_relations_chunked()` so long-text chunked extraction also carries temporal metadata.
- - Added `RelationWithTemporalOut` and `RelationsWithTemporalResponse` Pydantic schemas in `semantica/semantic_extract/schemas.py`. A separate schema is required because `RelationOut` uses `extra="ignore"`, which silently drops any undeclared field including the four temporal fields.
- - New `semantica/kg/temporal_normalizer.py` — `TemporalNormalizer` class (zero LLM calls, pure regex + `dateutil` arithmetic):
- - `normalize(value)` → `(valid_from, valid_until)` UTC `datetime` tuple or `None`. Resolution order: ISO 8601 full parse → partial-date regex (year-only, month+year, YYYY-MM, Q[1-4] YYYY) → ambiguous-slash-date detection → domain phrase map → relative phrase resolution via `relativedelta`.
- - `normalize_phrase(phrase)` → metadata dict `{"maps_to": ..., "type": ..., "domain": [...]}` or `None` — exact match then regex-pattern keys.
- - Ambiguous `DD/MM/YYYY`-style inputs issue `TemporalAmbiguityWarning` and return `None` — never silently guesses locale.
- - Unparseable inputs return `None` with a debug log — never raise.
- - Relative phrases (`"last year"`, `"three months ago"`, etc.) raise `ValueError` if `reference_date` is `None` rather than guessing.
- - Default phrase map covers 13 domains: General/Policy (`effective date`, `effective from/as of/beginning`, `in force until`, `retroactive to`, `sunset clause`), Healthcare (`approval date`, `expiry date`, `market authorization`), Cybersecurity (`incident window`, `campaign period`), Supply Chain (`certification valid through`), Finance (`trading halt`), Energy (`commissioned date`, `decommissioned date`).
- - User-supplied `phrase_map` is merged over defaults at construction (`{**defaults, **user_map}`) — custom entries win without forking the library.
- - Added `TemporalAmbiguityWarning(UserWarning)` to `semantica/utils/exceptions.py`.
- - Exported `TemporalNormalizer` from `semantica/kg/__init__.py`.
- - Added 53 new tests in `tests/semantic_extract/test_temporal_extraction.py`; zero real LLM calls, suite runs in ~3.5 s. All 873 existing tests continue to pass.
-
- - **Fix: OllamaProvider ignores `base_url`** (PR #408 by @AlexeyMyslin, fixed by @KaifAhmad1):
- - `OllamaProvider._init_client()` was assigning the raw `ollama` module to `self.client` instead of instantiating `ollama.Client(host=self.base_url)`, causing all requests to silently hit `localhost:11434` regardless of the `base_url` passed by the user
- - Fixed by replacing `self.client = ollama` with `self.client = ollama.Client(host=self.base_url)` — remote Ollama servers (e.g. `http://192.168.1.3:11434`) are now reachable
- - Added 3 regression tests: default URL forwarded as host, custom URL forwarded as host, and guard ensuring `self.client` is never the raw module
-
- - **Temporal Awareness in Context Graph** (PR #399 by @KaifAhmad1):
- - Added `valid_from` and `valid_until` fields to the `Decision` dataclass and `record_decision()` — decisions now carry explicit validity windows; superseded decisions remain in the graph (history is immutable)
- - Added `include_superseded=False` and `as_of=None` parameters to `find_precedents_by_scenario()` — defaults exclude expired decisions; `as_of` enables point-in-time precedent queries
- - Added `ContextGraph.state_at(timestamp)` — returns a serializable point-in-time snapshot of all nodes, edges, and decisions whose validity windows include `timestamp`; source graph is never mutated
- - Stamped `recorded_at` on causal relationship edges created via `add_causal_relationship()` — enables transaction-time filtering
- - Added `CausalChainAnalyzer.trace_at_time(event_id, at_time)` — reconstructs a causal chain using only edges recorded up to `at_time` (transaction time); returns an empty list when `at_time` predates all facts, never raises
- - Added `AgentContext.checkpoint(label)`, `diff_checkpoints(label1, label2)`, and `flush_checkpoint(label)` — named in-memory context snapshots with structured diffs (`decisions_added`, `decisions_removed`, `relationships_added`, `relationships_removed`) and optional persistence via `TemporalVersionManager`
- - **Review fixes applied in the same PR**:
- - Fixed `max_depth` error message in `trace_at_time` to match actual bound (1–100)
- - Fixed Cypher `at_time` query parameter to RFC3339 UTC (`Z` suffix) for unambiguous external DB comparisons
- - `_normalize_temporal_input` now raises `ValueError` on unparseable strings instead of silently returning raw input
- - Replaced `datetime.now()` with `datetime.utcnow()` for all `recorded_at` and checkpoint timestamps — aligns with codebase convention and avoids wrong local time on Windows
- - `flush_checkpoint` wraps `TemporalVersionManager()` construction in a `try/except` and re-raises as `RuntimeError` with a clear actionable message
- - Added 7 new tests (93 total across context modules, 0 failures)
-
- - **spaCy Runtime Fallback for NER Benchmarks**:
- - Hardened `NERExtractor` spaCy initialization so installed-but-broken spaCy environments no longer crash during extractor construction.
- - Updated ML entity extraction fallback behavior to catch runtime spaCy initialization failures, not just missing-model errors.
- - Added regression coverage for the "spaCy present but unusable at runtime" initialization path.
- - **Deterministic Temporal Reasoning Engine** (PR #398 by @KaifAhmad1, implemented and follow-up fixes by OpenAI Codex):
- - Added `semantica.kg.temporal_reasoning` as the single source of truth for deterministic, LLM-free temporal reasoning with an explicit zero-LLM module contract
- - Implemented `TemporalInterval`, full Allen interval algebra via `IntervalRelation`, and `TemporalReasoningEngine`
- - Added deterministic helpers for interval overlap/containment checks, open-ended activity checks, interval merging, gap analysis, coverage calculation, timelines, retroactive coverage, and temporal normalization
- - Integrated temporal query interval logic with the reasoning engine in `TemporalGraphQuery`
- - Preserved `semantica.reasoning` access via re-exports without making it the canonical implementation source
- - Fixed open-ended `query_time_range(..., end_time=None)` handling so temporal range queries no longer crash on `TemporalBound.OPEN`
- - Restored `temporal_granularity` behavior for point-in-time checks in `query_at_time()`
- - Eliminated the `semantica.reasoning` / `semantica.kg` circular import risk introduced during the initial module move
- - Added regression coverage for all 13 Allen relations, open-ended intervals, month-granularity point queries, open-ended range queries, retroactive coverage, and normalization idempotence
-
- - **Temporal Query Engine: Point-in-Time Correctness** (PR #397 by @KaifAhmad1, implemented and follow-up fixes by OpenAI Codex):
- - Added `reconstruct_at_time(graph, at_time)` to `TemporalGraphQuery` to build a self-consistent point-in-time subgraph without mutating the input graph
- - Updated `query_at_time()` to use point-in-time reconstruction internally so returned subgraphs exclude dangling edges when entity lifetimes are available
- - Added `TemporalConsistencyIssue` and `TemporalConsistencyReport` plus temporal consistency validation for:
- - inverted relationship intervals
- - relationships outside entity lifetimes
- - missing source/target entities
- - overlapping same-type relationships on the same edge
- - temporal gaps where a fact ends and restarts later
- - Added a module-level `validate_temporal_consistency(graph)` API alongside the query-engine method
- - Implemented sequence and cycle pattern detection with structured outputs containing `pattern_type`, `signature`, `frequency`, and per-occurrence node/edge/time details
- - Implemented calendar-aligned temporal evolution bucketing based on `temporal_granularity`
- - Added causal ordering controls to `find_temporal_paths()` via `enforce_causal_ordering` and `ordering_strategy` (`strict`, `overlap`, `loose`)
- - **Follow-up fixes applied in the same PR**:
- - Made `validate_temporal_consistency()` non-throwing on malformed temporal fields and return report errors instead of raising
- - Enforced exclusive `valid_until` semantics for point-in-time checks (`valid_from <= at_time < valid_until`)
- - Kept `query_time_range(..., temporal_aggregation="evolution")` backward-compatible by returning the flat relationship list plus a new `relationship_buckets` field
- - Hardened temporal pattern detection for open-ended intervals (`TemporalBound.OPEN`) to avoid datetime arithmetic/comparison crashes
- - Normalized relationship endpoints during point-in-time reconstruction so mixed-type IDs like `1` and `"1"` do not silently drop valid edges
- - Added in-code design comments documenting the sequence/cycle output structure required by the checklist
- - Added and expanded regression coverage for point-in-time reconstruction, exclusive end bounds, non-throwing validation, module-level validator access, pattern detection with gap tolerance/open bounds, evolution bucketing, causal ordering, and mixed-type IDs
-
- - **Core Temporal Data Model Overhaul** (PR #396 by @KaifAhmad1, implemented and follow-up fixes by OpenAI Codex):
- - Added `semantica.kg.temporal_model` with shared helpers for parsing, normalizing, serializing, and deserializing temporal relationship fields
- - Exported `TemporalBound` and `BiTemporalFact` from `semantica.kg` for backward-compatible temporal relationship handling
- - Updated `TemporalGraphQuery` to use shared temporal parsing/model helpers instead of ad hoc string handling
- - Added support for `valid`, `transaction`, and `both` time axes in temporal query filtering
- - Standardized temporal normalization on `timezone.utc` for better cross-version portability
- - Added `TemporalValidationError` to utils exports and made invalid temporal inputs consistently raise it
- - Added history-preserving temporal revisions in `TemporalVersionManager.apply_revision()` with provenance metadata and supersession semantics
- - Added safer snapshot persistence by serializing revision metadata before storage and surfacing storage failures as `ProcessingError`
- - **Follow-up fixes applied in the same PR**:
- - Added a default factory for `BiTemporalFact.recorded_at` and preserved legacy transaction-axis behavior by falling back to `valid_from` when `recorded_at` is missing
- - Treated `TemporalBound.OPEN` as an unbounded value in shared query parsing so open-ended facts do not fail in public APIs like `analyze_evolution()` and path filtering
- - Recomputed snapshot checksums before persisting revised snapshots and any original snapshot inserted during revision flow
- - Replaced second-based revision suffixes with collision-resistant revision IDs/labels to avoid duplicate save failures under rapid revisions
- - Removed warning spam caused by canonical serialized open bounds represented as `None`
- - Added and expanded regression coverage for UTC normalization, transaction-axis queries, open-ended bounds, revision integrity, checksum verification, and collision-resistant revision identifiers
-
- - **Audit Trail, Named Tags, and Rollback Protection** (PR #394 by @ZohaibHassan16, reviewed by @KaifAhmad1, follow-up fixes by OpenAI Codex):
- - Added mutation-level audit tracking for `ContextGraph` node and edge changes via `TemporalVersionManager.attach_to_graph()` and persistent mutation logging backends
- - Added named version tags in both in-memory and SQLite storage so human-readable tags can point to saved snapshots
- - Added rollback protection to `restore_snapshot()` so destructive graph restores require explicit confirmation
- - Added `get_node_history()` for per-entity audit inspection and `diff()` as a Git-like alias over version comparisons
- - Preserved backward compatibility for snapshot payloads and diff outputs by supporting both `nodes`/`edges` and `entities`/`relationships`
- - Fixed mixed-schema snapshot comparison and version metadata counts after the audit-trail feature landed on top of PR #393
- - Fixed restore replay so rollback does not generate synthetic mutation events in the audit log
- - Added version-label assignment for previously unlabeled mutations when a snapshot is created
- - Resolved merge conflicts against updated `main` in `managers.py`, `version_storage.py`, `context_graph.py`, and `test_managers.py`
- - Added and updated regression coverage for audit history, rollback safety, version-label persistence, and snapshot compatibility
-
- - **Snapshot Schema Compatibility Fix** (PR #393 by @ZohaibHassan16, reviewed by @KaifAhmad1, follow-up fixes by OpenAI Codex):
- - Fixed silent snapshot restore failures caused by the `ContextGraph` `nodes`/`edges` schema not matching the version manager's legacy `entities`/`relationships` expectations
- - Updated temporal snapshot handling to accept both `nodes`/`edges` and `entities`/`relationships`
- - Preserved both schema shapes in stored snapshots to maintain backward compatibility during migration
- - Fixed temporal diffing and detailed comparison paths so new-format and mixed-format snapshots compare correctly
- - Fixed version metadata counts so `entity_count` and `relationship_count` remain accurate for both snapshot schemas
- - Restored ontology snapshot compatibility fields removed during the PR follow-up iteration
- - Added regression coverage for new-format snapshot creation, metadata counts, and mixed-schema diffing
-
- - **ContextGraph Traversal Fallbacks for DecisionQuery & DecisionRecorder** (PR #386 by @ZohaibHassan16, reviewed and fixed by @KaifAhmad1):
- - Added native `ContextGraph` fallback execution paths to all 7 `DecisionQuery` methods (`_find_precedents_basic`, `find_by_category`, `find_by_entity`, `find_by_time_range`, `multi_hop_reasoning`, `trace_decision_path`, `find_similar_exceptions`) — resolves issue #379 where hardcoded Cypher queries broke in-memory usage
- - Added native `ContextGraph` fallback paths to 4 `DecisionRecorder` methods (`link_entities`, `record_exception`, `link_precedents`, `_store_decision_node`, `_store_exception_node`) using `add_node` / `add_edge` primitives
- - Implemented undirected BFS in `multi_hop_reasoning` fallback — traverses both outgoing and incoming edges so decisions are reachable from linked entities (matches Cypher `(start)-[*1..N]-(d:Decision)` semantics)
- - Fixed `isinstance(graph_store, ContextGraph)` guards → `type(graph_store) is ContextGraph` — prevents `Mock(spec=ContextGraph)` from triggering fallback branches and breaking 2 existing tests
- - Fixed `add_node(properties=metadata)` call in `_store_decision_node` and `_store_exception_node` — changed to `**metadata` so all decision fields are stored flat and remain readable via `_dict_to_decision`; previous form silently nested every field under a `"properties"` key
- - Fixed spurious `properties={}` keyword argument in all `add_edge` fallback calls — argument did not match the actual `add_edge(**properties)` signature
- - Fixed tz-aware / naive `datetime` mismatch in `find_by_time_range` fallback — strips `tzinfo` from aware bounds when stored timestamps are naive, preventing `TypeError` at comparison time
- - Hoisted `find_edges()` calls out of the BFS `while` loop in `trace_decision_path` — edges are now fetched once per call instead of once per visited node, eliminating O(nodes × total_edges) repeated full-graph scans
- - Removed duplicate `from ..embeddings import EmbeddingGenerator` import in `decision_query.py`
- - Added `tests/context/test_decision_query_fallback.py` with 14 tests: full integration test covering the complete fallback flow end-to-end, plus 13 targeted unit tests covering each `DecisionQuery` and `DecisionRecorder` fallback method individually, tz-aware/naive datetime mixing, and `Mock` guard correctness
-
- - **ContextGraph Thread Safety & Pagination** (PR #385, Issues #378 #376 by @ZohaibHassan16, review & fixes by @KaifAhmad1):
- - `ContextGraph`: added `threading.RLock` (`self._lock`) to `__init__`; all mutation paths (`add_nodes`, `add_edges`, `add_node`, `add_edge`, `save_to_file`, `load_from_file`, `link_graph`) and all read/query paths (`find_nodes`, `find_edges`, `find_node`, `find_active_nodes`, `get_neighbors`, `query`, `stats`, `density`) now protected with `with self._lock:` to prevent race-condition corruption under concurrent FastAPI workers
- - `find_nodes` and `find_edges` gained native `skip`/`limit` pagination parameters so the explorer layer never loads the full collection into memory to slice it
- - `GraphSession` (`session.py`): introduced session-level `RLock` wrapping all graph access; all 8 lazy analytics properties (`centrality`, `community`, `connectivity`, `path_finder`, `node_embedder`, `similarity`, `link_predictor`, `validator`) initialised under the lock (thread-safe double-checked); `get_nodes()` and `get_edges()` delegate pagination to the graph layer when no in-memory filter is needed
- - `pyproject.toml`: removed duplicate entry and added missing comma in the `all` optional-dependency array that caused `ERROR Failed to parse pyproject.toml: Unclosed array` in CI
- - **Fixes applied post-review (by @KaifAhmad1)**:
- - Fixed `/api/graph/search` returning empty `content` and `properties` — `ContextGraph.query()` wraps results in `node.to_dict()` which uses a `"properties"` envelope, but `_node_dict_to_response` expected a flat `{id, type, content, metadata}` shape; `session.search()` now normalises the envelope before returning
- - Fixed edge metadata silently dropped on import — `add_edges()` read only from the `"properties"` key, but edges produced by `find_edges()` and `build_graph_dict()` use `"metadata"`; fixed with `edge.get("properties") or edge.get("metadata", {})` fallback
- - Fixed `POST /api/enrich/links` blocking the asyncio event loop — the O(n) `score_link` scoring loop ran inline in the `async` handler; wrapped in `asyncio.to_thread(_score_all)`
- - Removed merge-artifact dead code in `session.py`: duplicate `self.annotations` assignment, duplicate un-locked property set, and double-query logic in `get_nodes()`/`get_edges()` that recomputed results outside the lock and threw away the correctly-paginated result computed inside it
- - Removed merge-artifact dead code in `enrich.py`: unreachable second `predict_links` implementation block after early `return`, and duplicate `nodes, _` fetch in `detect_duplicates`
-
- - **Knowledge Explorer API Backend** (PR #384, Issue #377 by @ZohaibHassan16, review & fixes by @KaifAhmad1):
- - Added `semantica.explorer` package — a full FastAPI backend for the Semantica Knowledge Explorer dashboard
- - `app.py`: `create_app(session)` factory with CORS middleware, custom exception handlers (`KeyError→404`, `ValueError→422`), and HTML5 static-file fallback routing; generic `Exception` handler correctly re-raises `HTTPException` so dependency-injection 503s are not swallowed
- - `session.py`: `GraphSession` — thread-safe container wrapping a `ContextGraph` with 8 lazily-initialised analytics components (`CentralityCalculator`, `CommunityDetector`, `ConnectivityAnalyzer`, `PathFinder`, `NodeEmbedder`, `SimilarityCalculator`, `LinkPredictor`, `GraphValidator`); all lazy properties initialised under `RLock` to prevent double-instantiation under concurrent requests; shared `build_graph_dict(node_ids=None)` method eliminates duplication across route files; `from_file(path)` classmethod loads from JSON
- - `ws.py`: `ConnectionManager` — thread-safe WebSocket manager with `connect()`, `disconnect()`, `broadcast(event_type, data)`, and `send_personal()` support; safe disconnection cleanup during broadcast
- - `dependencies.py`: `get_session(request)` and `get_ws_manager(request)` FastAPI `Depends`-compatible callables; `get_session` raises `HTTP 503` when no session is attached
- - 7 modular route files, all using `asyncio.to_thread` for sync graph operations:
- - `routes/graph.py`: `GET /api/graph/nodes` (type/keyword filter, pagination), `GET /api/graph/node/{id}`, `GET /api/graph/node/{id}/neighbors` (BFS, depth 1–5), `GET /api/graph/edges` (type/source/target filter), `GET /api/graph/node/{id}/path` (BFS or Dijkstra — algorithm param now correctly dispatched), `POST /api/graph/search`, `GET /api/graph/stats`
- - `routes/analytics.py`: `GET /api/analytics` (centrality, community, connectivity — comma-separated metrics param), `GET /api/analytics/validation`
- - `routes/decisions.py`: `GET /api/decisions` (category filter, pagination), `GET /api/decisions/{id}`, `GET /api/decisions/{id}/chain` (BFS causal chain up to 5 hops), `GET /api/decisions/{id}/precedents` (category + scenario keyword ranking), `GET /api/decisions/{id}/compliance` (in-graph check over `violates`/`non_compliant`/`breaches` edges — no longer a stub)
- - `routes/temporal.py`: `GET /api/temporal/snapshot` (ISO-8601 `at` param), `GET /api/temporal/diff` (added/removed node sets between two timestamps), `GET /api/temporal/patterns` (graceful fallback when `TemporalPatternDetector` unavailable, with warning log for unexpected errors)
- - `routes/enrich.py`: `POST /api/enrich/extract` (NLP entity/relation extraction), `POST /api/enrich/links` (per-node link prediction via `score_link` against all non-adjacent candidates — fixed from broken `predict_links` call), `POST /api/enrich/dedup` (duplicate detection — fixed missing `asyncio.to_thread` that was blocking the event loop), `POST /api/reason` (forward/backward inference via `Reasoner`)
- - `routes/export_import.py`: `POST /api/export` (12 formats: JSON, Turtle, RDF-XML, N-Triples, CSV, GraphML, GEXF, OWL, Cypher, AQL, YAML — temp file always cleaned up via `try/finally`), `POST /api/import` (JSON/JSON-LD multipart upload with WebSocket progress events)
- - `routes/annotations.py`: `GET /api/annotations`, `POST /api/annotations` (validates node exists; `add_annotation` mutates dict in-place so no extra roundtrip), `DELETE /api/annotations/{id}`
- - `schemas.py`: 28 Pydantic v2 request/response models covering all endpoint shapes including pagination, temporal, enrichment, compliance, and annotation types
- - `__init__.py`: `semantica-explorer` CLI entry point — `--graph`, `--host`, `--port`, `--no-browser` args; validates graph file exists; checks for `uvicorn`; opens browser after 1.5 s delay
- - `pyproject.toml`: added `[project.optional-dependencies] explorer` group (`fastapi`, `uvicorn[standard]`, `websockets`, `python-multipart`); registered `semantica-explorer` script entry point; fixed missing comma in `all` extra that broke `pip install semantica[all]`
- - **Fixes applied post-review (by @KaifAhmad1)**:
- - Fixed `predict_links` endpoint — was calling `predictor.predict_links(graph_dict, node_id, top_n=...)` with wrong type (`dict` as `graph_store`), wrong positional arg (`node_id` as `node_labels`), and wrong kwarg (`top_n` vs `top_k`); rewrote to iterate all non-adjacent candidate nodes and call `predictor.score_link(session.graph, source, candidate)` directly
- - Fixed `detect_duplicates` endpoint — `session.get_nodes()` was called directly in an `async def` handler without `asyncio.to_thread`, blocking the event loop
- - Fixed temp file leak in `export_graph` — file was not deleted on exception from `export_fn` or `open()`; wrapped in `try/finally`; moved `import os` to module level
- - Fixed `pyproject.toml` `all` extra — two consecutive strings with no comma between them caused a TOML syntax error
- - Fixed generic `Exception` handler swallowing `HTTPException(503)` raised by `get_session`
- - Fixed compliance endpoint — imported `PolicyEngine` then discarded it, always returning `compliant=True`; replaced with in-graph edge scan
- - Fixed `temporal_patterns` bare `except Exception` silently hiding bugs — split into `ImportError` (silent graceful) and `Exception` (warning log)
- - Fixed all 8 lazy analytics properties to initialise under `_lock` (thread-safe double-checked)
- - Fixed `find_path` ignoring the `algorithm` query param — now dispatches to `dijkstra_shortest_path` or `bfs_shortest_path`
- - Removed unnecessary `get_annotations()` round-trip in `create_annotation`
- - Removed `import traceback` unused import in `app.py`
- - Deduplicated `_build_graph_dict` (was copied identically in `graph.py`, `analytics.py`, `export_import.py`) into `GraphSession.build_graph_dict()`
- - 49 integration tests in `tests/explorer/test_explorer_api.py` using `starlette.testclient.TestClient` — all passing; covers health, nodes, edges, search, stats, decisions, causal chains, precedents, compliance (including violation detection), temporal snapshots/diff/patterns, analytics, reasoning, entity extraction, link prediction, deduplication, annotations, export (JSON + node-subset), and import (JSON + edges + unsupported format)
- - **Reasoning Dead Code Removal** (PR #387, Issue #382 by @ZohaibHassan16):
- - Removed lines 357–358 in `semantica/reasoning/reasoner.py` that silently overwrote the sophisticated `_match_pattern` regex (which handles pre-bound variable embedding, repeated-variable backreferences via `(?P=var)`, and non-greedy named capture groups) with a simpler `re.escape`-based pattern, making all the prior logic unreachable dead code
- - Removed duplicate unreachable `return None` on line 368 (syntactically dead, appearing immediately after another `return None` in the same branch)
- - Surfaced `re.error` exceptions instead of swallowing them with `except Exception: pass`, preventing silent failures when malformed patterns were passed to `re.match`
- - Before this fix, any rule using the same variable twice (e.g. `rel(?x, ?x)`) generated a duplicate named group error that was silently caught, causing the match to return `None` regardless of the fact — breaking transitivity, symmetry, and self-join rule patterns entirely
-
- - **Agno Agentic Framework Integration** (Issue #249):
- - Added `AgnoContextStore` — graph-backed agent memory implementing the `agno.memory.db.base.MemoryDb` protocol; wraps `AgentContext` + `VectorStore`; supports `create()`, `table_exists()`, `memory_exists()`, `read_memories()`, `upsert_memory()`, `delete_memory()`, `drop_table()`, `clear()` plus extended `record_decision()`, `find_precedents()`, `retrieve()` methods
- - Added `AgnoKnowledgeGraph` — multi-hop GraphRAG knowledge base implementing `agno.knowledge.base.AgentKnowledge`; ingests files, directories, URLs, and raw text via NER → relation extraction → graph build → vector index pipeline; `search()` returns `AgnoDocument` objects; `get_graph_context(entity)` returns text summary of entity's graph neighbourhood
- - Added `AgnoDecisionKit` — Agno `Toolkit` subclass exposing 6 decision-intelligence tools: `record_decision`, `find_precedents`, `trace_causal_chain`, `analyze_impact`, `check_policy`, `get_decision_summary`
- - Added `AgnoKGToolkit` — Agno `Toolkit` subclass exposing 7 KG pipeline tools: `extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`, `infer_facts`, `export_subgraph`
- - Added `AgnoSharedContext` — team-level coordinator with a single shared `ContextGraph`; `bind_agent(role)` returns a role-scoped `_AgentScopedStore` with cross-agent memory visibility; thread-safe via `RLock`
- - All 5 components degrade gracefully when `agno` is not installed (`AGNO_AVAILABLE` flag); importable and functional without agno present
- - Added `agno = ["agno>=1.0.0"]` optional dependency in `pyproject.toml`; included in `all` extra
- - 110 integration tests in `tests/integrations/agno/` covering all public APIs, MemoryDb protocol compliance, GraphRAG search, tool registration, shared memory isolation, and thread-safety
- - 3 cookbook notebooks in `cookbook/integrations/`: `agno_decision_intelligence.ipynb` (loan underwriting), `agno_graphrag_context.ipynb` (regulatory compliance), `agno_multi_agent_shared_context.ipynb` (multi-agent team coordination)
- - Full reference documentation in `docs/integrations/agno.md`
-
- - **Novita AI Provider** (PR #374 by @Alex-wuhu):
- - Added `NovitaProvider` — OpenAI-compatible integration via `https://api.novita.ai/v1`; supports `generate()` and `generate_structured()` (JSON forced format)
- - Default model: `deepseek/deepseek-v3.2`; configurable via `NOVITA_API_KEY` environment variable
- - Registered `"novita"` in the built-in provider factory; usable via `create_provider("novita")`
- - Added integration tests in `tests/test_novita_integration.py` with proper assertions and graceful skip when `NOVITA_API_KEY` is unset
-
- - **Native Datalog Reasoning Engine** (PR #371, Issue #368 by @ZohaibHassan16, reviewed and fixed by @KaifAhmad1):
- - Added `DatalogReasoner` to `semantica.reasoning` — a pure-Python, bottom-up semi-naive fixpoint engine with guaranteed termination on finite graphs
- - Supports recursive Horn clause rules (e.g. `ancestor(X,Y) :- parent(X,Z), ancestor(Z,Y).`) that existing engines loop on indefinitely
- - Memory-optimized `_unify()` with deferred dict allocation — zero allocation on failed unifications
- - `O(1)` delta-index lookup per iteration eliminates redundant `O(N)` rule re-evaluations in semi-naive loop
- - `query("pred(?X, ?Y)")` returns variable-binding dicts; supports both uppercase `?Y` and lowercase `?y` variable syntax
- - `query(..., bindings={"Y": "val"})` pre-binds variables for exact-match verification
- - `load_from_graph(ContextGraph)` converts all edges and nodes to Datalog facts in one call; handles both `find_edges`/`find_nodes` and raw `edges`/`nodes` graph APIs
- - `add_fact()` accepts `"pred(a, b)"` strings and Semantica dicts (`subject/predicate/object`, `source/target/type`, `type/id` shapes); warns on unrecognised dict format instead of silently dropping
- - `_derived` cache flag — `derive_all()` skips re-evaluation when no facts or rules have changed since last run; `query()` respects the cache
- - Progress tracking wrapped in `try/finally` — `stop_tracking()` always called even on exception
- - `DatalogReasoner`, `DatalogFact`, `DatalogRule` exported from `semantica.reasoning`
- - 18 tests covering recursive rules, multi-hop inference, variable binding, graph integration, idempotency, and edge cases — all passing
- - **Ontology Diff & Migration** (PR #367 by @ZohaibHassan16, review & fixes by @KaifAhmad1):
- - `VersionManager.diff_ontologies(base, target)` — structured diff between two ontology dicts using hash-map lookups; handles URI-less items via `name` fallback; deep equality checks for unordered lists; now covers classes, properties, individuals, and axioms
- - `ChangeLogAnalyzer.analyze(diff)` — classifies each change by semantic impact: removed classes/properties → `CRITICAL/BREAKING`; narrowed domain/range/cardinality → `HIGH/BREAKING`; hierarchy modifications → `MEDIUM/POTENTIALLY_BREAKING`; added elements and annotation updates → `INFO/NON_BREAKING`
- - `ImpactReport` dataclass and `generate_change_report(diff)` public helper — returns a structured dict with `summary`, `impact_classification` (breaking / potentially_breaking / safe), `recommendations`, and the raw `diff`
- - `OntologyEngine.compare_versions(base_id, target_id, **options)` — end-to-end orchestrator: loads versions from `VersionManager`, runs `diff_ontologies`, generates impact report; accepts `base_dict`/`target_dict` overrides to bypass version store; `run_validation=True` triggers `OntologyValidator` on the target schema; `graph_data=...` additionally runs `GraphValidator` on instance data against the new schema
- - `OntologyEngine.get_ontology_version_dict(version_id)` — utility to load a registered version as a plain dict ready for diffing
- - Documentation added to `docs/reference/change_management.md`: "Ontology Diff & Migration" section with code example and full report format reference
- - 7 tests added to `tests/change_management/test_managers.py` covering: empty diff, unordered list equality, URI/name fallback, breaking class removal, narrowed domain (HIGH), safe additions and annotation changes, `compare_versions` dict override, version-not-found error path, individuals/axioms diff coverage, null constraint value flagged as breaking
- - **Fixes applied post-review (by @KaifAhmad1)**:
- - Fixed typo in `ChangeCategory` enum value: `"potenitally_breaking"` → `"potentially_breaking"`
- - Fixed missing space in impact description string: `f"New{entity_type}"` → `f"New {entity_type}"`
- - Added null-value guard in `_analyze_field_changes` — constraint fields with `None` old/new value are now correctly flagged as breaking instead of silently passing the subset check
- - Made `ChangeLogAnalyzer` stateless — `report` is now a local variable passed into `_generate_recommendations(report)` rather than stored as `self.report`; removes re-entrancy hazard
- - Removed no-op `__init__` from `ChangeLogAnalyzer`
- - Replaced non-portable emoji markers in recommendations (`✘✘✘`, `¤¤¤`, `☺☺☺`) with plain-text tags (`[BREAKING]`, `[WARNING]`, `[SAFE]`)
- - Extended `diff_ontologies` to cover `individuals` and `axioms` — previously only classes and properties were diffed; the public `compare_versions` path now returns all four element types
- - Fixed exception chaining in `compare_versions`: `raise ProcessingError(...) from e` to preserve original traceback
- - Removed silent `ImportError` swallow for `GraphValidator` — it is a first-party module; an `ImportError` indicates a broken install, not a graceful skip
- - Added comment on deferred `VersionManager` import in `OntologyEngine.__init__` explaining the circular-import constraint
- - Fixed import-before-docstring in `tests/change_management/test_managers.py`
- - Fixed broken Markdown link syntax in docs JSON example block: `"[http://...](http://...)"` → bare URI string
- - Updated docs recommendations example to match the new plain-text tag format
-
- - **Ontology Alignment API** (PR #361 by @ZohaibHassan16, review & fixes by @KaifAhmad1):
- - Alignment representation using standard RDF predicates: `owl:equivalentClass`, `owl:equivalentProperty`, `owl:sameAs`, `skos:exactMatch`, `skos:closeMatch`, `skos:broadMatch`, `skos:narrowMatch`, `skos:relatedMatch`
- - `OntologyEngine.create_alignment(source_uri, target_uri, predicate)` — store alignment triples in TripletStore
- - `OntologyEngine.get_alignments(entity_uri)` — bidirectional retrieval of all alignments for an entity
- - `OntologyEngine.list_alignments(ontology_uri=None)` — list all alignments, optionally filtered by ontology namespace
- - `NamespaceManager.get_alignment_predicates()` — expose standard OWL/SKOS alignment URIs as a convenience dict
- - `ReuseManager.suggest_alignments(target, source)` — O(N+M) hashmap heuristic to suggest alignments based on exact label matches across ontologies
- - `ReuseManager.merge_ontology_data(..., compute_alignments=True)` — optionally attach suggested alignments to merge output without auto-committing unverified triples
- - `QueryEngine.expand_entity_uri(uri, store, use_alignments=True)` — bidirectional SPARQL expansion to include aligned equivalents; no-ops when flag is False
- - `QueryEngine.build_values_clause(variable, uris)` — generate a SPARQL `VALUES` clause for injecting expanded URIs into queries
- - Alignment-aware queries section added to `docs/reference/triplet_store.md`
- - Ontology Alignment section added to `docs/reference/ontology.md`
- - **Fixes applied post-review (by @KaifAhmad1)**:
- - Fixed progress tracker leak in `expand_entity_uri` — `stop_tracking` was only called inside the `hasattr(execute_sparql)` branch; backends without it silently leaked a tracker entry
- - Fixed `relatedMatch` predicate gap — `get_alignment_predicates()` exposed `skos:relatedMatch` but all three SPARQL FILTER lists omitted it, making those alignments permanently invisible
- - Fixed SPARQL injection in `list_alignments` — previously only `"` was escaped; `\`, `{`, and `}` are now also percent-encoded to prevent WHERE block breakout
- - Fixed SPARQL injection in `build_values_clause` — URIs now run through `_sanitize_uri` before wrapping in angle-bracket literals
- - Added full-URI validation in `create_alignment` — raises `ProcessingError` if predicate is a CURIE instead of a full URI, preventing silent storage of unqueryable triples
- - Fixed E2E test `test_end_to_end_cross_ontology_uri_flow` — previously mocked the method under test; now uses a real mock backend with `execute_sparql` to exercise the actual expansion and VALUES clause injection flow
- - 19 tests added covering: `create_alignment`, `get_alignments`, `suggest_alignments`, merge with alignment computation, `expand_entity_uri` (enabled/disabled), `build_values_clause`, and full E2E cross-ontology query flow
- - **Context Explainability Output Fixes** (by @KaifAhmad1):
- - Fixed decision-node storage in `ContextGraph` so full human-readable `scenario`, `reasoning`, and decision metadata are preserved on graph nodes instead of degrading into opaque IDs or truncated display text
- - Fixed causal and precedent reconstruction paths in the context module so returned `Decision` objects prefer readable stored fields over raw node identifiers
- - Fixed context aggregate outputs to return enriched readable payloads for influence, causality, similarity, policy-impact, and entity-similarity workflows instead of bare UUID lists or tuple-only results
- - Fixed `PolicyEngine.get_affected_decisions()` so both Cypher and fallback branches return consistent decision metadata including `scenario`, `category`, `outcome`, and `confidence`
- - Fixed `EntityLinker` similarity flows so enriched similarity results are consumed correctly across internal linking paths and public search aliases
- - Fixed `CentralityCalculator._build_adjacency()` to handle `ContextGraph` edges (dataclass `ContextEdge` objects with `source_id`/`target_id`) so `calculate_degree_centrality()` and related centrality algorithms work correctly when a `ContextGraph` is passed as the graph store
- - Fixed downstream KG integrations in `node_embeddings`, `link_predictor`, `centrality_calculator`, `path_finder`, and context retrieval fallbacks to normalize enriched neighbor/node outputs without breaking graph algorithms
- - Added 23 regression tests in `tests/context/test_context_explainability_regression.py` covering readable decision text preservation, enriched causal/path outputs, policy-impact results, entity similarity payloads, and compatibility with KG consumers
+ ### Security
- ## [0.3.0] - 2026-03-10
+ - **12 vulnerability fixes** (PR security-enhancement, @KaifAhmad1):
+ - **[CRITICAL — CWE-95]** Eval injection in `media_parser.py`: replaced `eval(ffprobe_output)` with `fractions.Fraction`.
+ - **[CRITICAL — CWE-502]** Pickle deserialization in `agent_memory.py`: replaced with JSON; legacy `.pkl` files detected and refused with migration message.
+ - **[HIGH — CWE-89]** SQL injection in `snowflake_ingestor.py`: `LIMIT`/`OFFSET` parameterized; `ORDER BY` regex-validated; `WHERE` clauses containing semicolons rejected.
+ - **[HIGH — CWE-611]** XXE in `rdf_parser.py`: `defusedxml.defuse_stdlib()` before all RDF/XML parsing.
+ - **[HIGH — CWE-346/200]** Missing security headers in `server.py`: `CORSMiddleware`, `X-Content-Type-Options`, `X-Frame-Options`, HSTS, generic 500 handler.
+ - **[HIGH — CWE-346/400]** Overpermissive CORS in `explorer/app.py`: methods/headers narrowed; 64 KB WebSocket frame cap.
+ - **[MEDIUM — CWE-20]** Algorithm param unconstrained in `graph.py`: enum-validated `bfs|dijkstra` only.
+ - **[MEDIUM — CWE-434]** RDF upload without extension check in `vocabulary.py`: `.ttl/.rdf/.owl/.xml/.jsonld` allowlist enforced.
+ - **[MEDIUM — CWE-1336]** Prompt injection in `llm_extraction.py`: user-supplied content wrapped in `json.dumps()`.
+ - **[MEDIUM — CWE-95]** Dynamic `__import__()` in `pipeline_validator.py`: replaced with proper module-level import.
+ - **[MEDIUM — CWE-1333]** ReDoS in `enrich.py`: whitespace-normalize then split on literal `" AND "`.
+ - **[LOW — CWE-22]** Path traversal in `server.py` SPA route: `Path.resolve().relative_to()` guard; 400 on escape.
+ - **[LOW — CWE-400]** Unbounded SPARQL in `sparql.py`: 5 000-row cap, 30 s `asyncio.wait_for` timeout, `Semaphore(4)` concurrency cap; `SparqlResponse.truncated` field added.
+ - **[LOW — CWE-434]** Import upload in `export_import.py`: 50 MB cap; `{.json,.csv}` allowlist.
+ - CodeQL `paths-ignore` for `cookbook/**/*.html` to suppress false-positive JS alerts #15–18.
+ - **SSRF in Ontology Hub** (PR #518 follow-up): `_validate_fetch_url()` rejects non-http/https schemes and resolves hostname via `socket.getaddrinfo`, blocking loopback/private/link-local/multicast addresses.
- - **Context Graph Feature Completeness** (by @KaifAhmad1):
- - Added `valid_from` / `valid_until` temporal validity fields to `ContextNode` and `ContextEdge` dataclasses — both expose `is_active(at_time=None) -> bool`; nodes/edges without these fields are always considered active
- - Added `add_node(valid_from=..., valid_until=...)` and `add_edge(valid_from=..., valid_until=...)` support — validity windows are extracted from `**properties` and stored as first-class dataclass fields, not in metadata
- - Added `ContextGraph.find_active_nodes(node_type=None, at_time=None)` — returns only nodes whose validity window includes the given time (defaults to `datetime.utcnow()`); complements `find_nodes()` with temporal filtering
- - Added `min_weight: float = 0.0` parameter to `ContextGraph.get_neighbors()` — edges with weight below the threshold are skipped during BFS traversal, enabling weighted/confidence-filtered multi-hop navigation; fully backward-compatible (default 0.0 passes all edges)
- - Added `ContextGraph.link_graph(other_graph, source_node_id, target_node_id, link_type="CROSS_GRAPH") -> str` — creates a navigable bridge between two separate `ContextGraph` instances; records a marker edge internally and returns a `link_id`
- - Added `ContextGraph.navigate_to(link_id) -> (other_graph, target_node_id)` — resolves a `link_id` to the target graph and its entry node, enabling hierarchical cross-graph traversal (e.g. agent moving from a high-level decision graph into a domain-specific sub-graph)
- - Added `ContextGraph.resolve_links(registry)` — reconnects cross-graph links after `load_from_file()`; `save_to_file()` now persists a `links` section with `other_graph_id` so navigation survives the full save/load cycle
- - Added `graph_id` field to `ContextGraph` — stable UUID per instance, persisted to JSON, so separate graphs can identify each other after reload
- - Fixed `is_active()` on `ContextNode` and `ContextEdge` — tz-aware `datetime` inputs are now normalised to tz-naive UTC before comparison, preventing `TypeError` when callers pass `datetime.now(timezone.utc)`
- - Fixed `valid_from` / `valid_until` serialisation — `add_nodes()`, `add_edges()`, `to_dict()`, and `from_dict()` all now preserve and restore validity windows; previously these fields were silently lost
- - Fixed cross-graph link artifact — `link_graph()` now pre-creates a `"cross_graph_link"` typed `ContextNode` for the marker before inserting the marker edge, preventing `_add_internal_edge()` from auto-creating a phantom `"entity"` node
- - Added 14 tests in `tests/context/test_cross_graph_navigation.py` covering link creation, phantom-node prevention, and full save/load round-trips with `resolve_links()`
- - Fixed `pipeline_builder.add_step()` return type annotation from `"PipelineBuilder"` to `"PipelineStep"` — implementation was already correct per 0.3.0-beta changelog, only signature and docstring were stale
- - Fixed `test_hybrid_search_performance` timing computation — accumulated a real `search_times` list and compute true average; raised threshold to `< 5.0s` to account for real `sentence-transformers` (384-dim) latency
-
-
-
- - **0.3.0 Bug Fixes & Comprehensive Real-World Tests** (by @KaifAhmad1):
- - Fixed `ProvenanceTracker` missing from `semantica/kg/__init__.py` exports — `from semantica.kg import ProvenanceTracker` now works correctly
- - Fixed duplicate relation creation in `_parse_relation_result` — orphaned legacy block was appending every relation twice; removed the duplicate block
- - Added `extraction_method` parameter to `_parse_relation_result`; typed extraction path now correctly sets `"llm_typed"` instead of `"llm"` in relation metadata
- - Fixed cross-test cache pollution in `tests/semantic_extract/test_retry_logic.py` — module-level `_result_cache` now cleared in `setUp()` to prevent intermittent failures when tests share input text
- - Added `tests/test_030_realworld_comprehensive.py`: 85 real-world tests covering all 0.3.0-alpha/beta features with real data (tech companies, CEOs, products, investment chains, healthcare scenarios)
- - ContextGraph basic operations and decision tracking lifecycle
- - KG algorithms: centrality, community detection, embeddings, path finding, similarity, link prediction, connectivity
- - PolicyEngine, DecisionQuery, AgentContext, Decision model serialization
- - ProvenanceTracker with GraphBuilderWithProvenance and AlgorithmTrackerWithProvenance
- - Deduplication v2 with blocking strategies, RDF/TTL export, Reasoner inference
- - Pipeline builder/validator/failure handler with retry policies
- - Multi-hop investment chain (Microsoft→OpenAI, Google→Anthropic) end-to-end
- - Healthcare entity extraction and knowledge graph construction E2E
+ ---
- ## [0.3.0-beta] - 2026-03-07
+ ## [0.4.0] - 2026-04-08
- - **Multi-Founder LLM Extraction & Reasoner Inference Fix** (PR #354 by @KaifAhmad1):
- - Fixed `_parse_relation_result` in `methods.py` — unmatched subjects/objects now produce a synthetic `UNKNOWN` entity instead of silently dropping the relation; all LLM-returned co-founders are preserved
- - Rewrote `_match_pattern` in `reasoner.py` — splits pattern on `?var` placeholders first, then escapes only the literal segments; pre-bound variables resolve to exact literals, repeated variables use backreferences, non-greedy `.+?` prevents over-consumption of literal separators
- - Added `tests/reasoning/test_reasoner.py` with 4 tests covering multi-word value inference, pre-bound variables, binding conflicts, and single-word regression
- - Added `tests/semantic_extract/test_relation_extractor.py` with 6 tests covering all-founders returned, synthetic entity creation, matched entity integrity, predicate/confidence preservation, empty response, and malformed entries
- - **TTL Export Alias Fix** (PR #355 by @KaifAhmad1):
- - Added `_format_aliases` map in `RDFExporter` so `format="ttl"`, `"nt"`, `"xml"`, `"rdf"`, and `"json-ld"` resolve to their canonical counterparts without breaking existing callers
- - Alias resolution applied at the top of `export_to_rdf()` before format validation — zero public API changes
- - Added working TTL export cell to `cookbook/introduction/15_Export.ipynb` (Step 3: RDF Export)
- - Added `tests/export/test_rdf_exporter.py` with 8 tests covering all aliases, canonical formats, error handling, and file export
-
- - **Incremental/Delta Processing Feature** (PR #349 by @ZohaibHassan16, reviewed and fixed by @KaifAhmad1):
- - Native delta computation between graph snapshots using SPARQL queries
- - Delta-aware pipeline execution with `delta_mode` configuration for processing only changed data
- - Version snapshot management with graph URI tracking and metadata storage
- - Snapshot retention policies with automatic cleanup via `prune_versions()` method
- - Integration with pipeline execution engine for incremental workflows
- - Significant performance improvements: processes only changes instead of full datasets
- - Cost optimization: dramatically reduces compute and storage requirements for large-scale operations
- - Production-ready for near real-time pipelines and frequent deployment scenarios
- - Bug fixes: corrected SPARQL variable order, fixed class references, resolved duplicate dictionary keys
- - Comprehensive test coverage including delta mode integration tests
- - Complete documentation with usage examples and API references
- - Essential for enterprise-grade, large-scale semantic infrastructure
- - **Deduplication v2 Migration Guide** (PR #344 by @ZohaibHassan16, fixes by @KaifAhmad1):
- - Added comprehensive MIGRATION_V2.md documentation for Deduplication v2 Epic #333
- - Documented Candidate Generation V2 with multi-key blocking and phonetic matching
- - Documented Two-Stage Scoring prefilter with configurable thresholds
- - Documented Semantic Relationship Deduplication v2 with synonym mapping
- - Added practical code examples for all V2 features with opt-in configuration
- - Fixed critical infinite recursion bug in dedup_triplets() function
- - Completed Epic #333 with comprehensive migration path and documentation
- - Performance: 5.86x speedup confirmed (129ms vs 754ms) for semantic deduplication
- - Full backward compatibility maintained with legacy mode as default
- - **Semantic Relationship Deduplication v2** (PR #340 by @ZohaibHassan16, fixes by @KaifAhmad1):
- - Implemented opt-in semantic relationship deduplication mode (`semantic_v2`) with 6.98x performance improvement
- - Added canonicalization engine with predicate synonym mapping (`works_for` → `employed_by`)
- - Implemented fast-path O(1) hash matching for exact canonical signature comparisons
- - Added weighted semantic scoring (60% predicate + 40% object composition) with explainable `semantic_match_score` metadata
- - Enhanced `dedup_triplets()` function as first-class API in `methods.py`
- - Integrated semantic deduplication into merge strategy with canonical key generation
- - Added literal normalization for whitespace cleanup in object matching
- - Maintained full backward compatibility with legacy mode as default
- - Fixed critical infinite recursion bug in `dedup_triplets()` function via registry name checking
- - Performance: Semantic V2 (~83ms) vs Legacy (~579ms) - 6.98x speedup confirmed
- - All 13 deduplication benchmarks passing with comprehensive test coverage
- - **Two-Stage Scoring Prefilter** (PR #339 by @ZohaibHassan16):
- - Implemented opt-in two-stage scoring with fast prefilter gates to eliminate expensive semantic scoring for obvious non-matches
- - Prefilter gates: type mismatch detection, name length ratio validation, token overlap requirements
- - Performance improvements: 18-25% faster batch processing with prefilter enabled
- - Configurable thresholds: `min_length_ratio`, `min_token_overlap_ratio`, `required_shared_token`
- - Enhanced explainability with score breakdown and rejection reasons in metadata
- - Complete backward compatibility with default `prefilter_enabled=False`
-
- - **Candidate Generation v2 with Multi-Key Blocking** (PR #338 by @ZohaibHassan16):
- - Implemented opt-in candidate generation strategies (`legacy`, `blocking_v2`, `hybrid_v2`) to address O(N²) pair explosion during deduplication
- - Multi-key blocking with normalized token prefixes, type-aware keys, and optional phonetic (Soundex) blocking
- - Deterministic candidate budgeting with `max_candidates_per_entity` limit using stable sorting
- - Efficient pair generation with set-based deduplication across overlapping blocks
- - Performance improvements: 63.6% faster in worst-case scenarios (0.259s → 0.094s for 100 entities)
- - Complete backward compatibility with default `candidate_strategy="legacy"`
- - Added configuration options: `blocking_keys`, `enable_phonetic_blocking`, `max_candidates_per_entity`
-
- - **ArangoDB AQL Export Support** (PR #342 by @tibisabau):
### Added
- - **ArangoDB AQL Export Support** (PR #342 by @tibisabau)
- - Full-featured ArangoDB AQL exporter with 642 lines of production-ready code
- - Comprehensive AQL INSERT statement generation for vertices and edges
- - Configurable collection names with validation and sanitization
- - Batch processing support for large knowledge graphs (default: 1000)
- - Added export_arango() convenience function for easy access
- - Enhanced unified export with AQL format support and .aql auto-detection
- - Added `export_arango()` convenience function for easy access
- - Enhanced unified export with AQL format support and `.aql` auto-detection
- - Integrated with method registry for extensibility
- - 17 comprehensive test cases with 100% pass rate
- - Enterprise-grade ArangoDB multi-model database integration
-
- - **Apache Parquet Export Support** (PR #343 by @tibisabau):
- - **Apache Parquet Export Support** (PR #343 by @tibisabau)
- - Full-featured Apache Parquet exporter with 701 lines of production-ready code
- - Columnar storage format optimized for analytics and data warehousing
- - Configurable compression codecs (snappy, gzip, brotli, zstd, lz4, none)
- - Explicit Arrow schemas with type safety and consistency
- - Field normalization for varied entity and relationship naming conventions
- - Structured metadata handling using Parquet struct fields
- - Added export_parquet() convenience function for easy access
- - Enhanced unified export with Parquet format support and .parquet auto-detection
- - Added `export_parquet()` convenience function for easy access
- - Enhanced unified export with Parquet format support and `.parquet` auto-detection
- - Integrated with method registry for extensibility
- - 25 comprehensive test cases with 100% pass rate
- - Enterprise-grade analytics integration with pandas, Spark, Snowflake, BigQuery, Databricks
+ **Temporal Intelligence** (@KaifAhmad1, PRs #396–#402)
- ### Fixed
- - **Fixed NameError**: missing Type import in utils/helpers.py
-
- - Fixed NameError: missing Type import in utils/helpers.py
- - Added Type to typing imports to fix retry_on_error decorator
- - Removed unused Type import from config_manager.py
- - Resolves ImportError when importing semantica modules
- - Fixes capability gap analysis notebook execution
-
- - **Test Suite Fixes: 0.3.0-alpha & Unreleased Features** (PR utils by @KaifAhmad1):
-
- **Context Module (`semantica/context/`)**
- - Fixed `retrieve_decision_precedents` to gate entity extraction on `use_hybrid_search=True` — was incorrectly extracting entities when flag was `False`
- - Fixed `_extract_entities_from_query` to use `word[0].isupper()` instead of `word.istitle()` — correctly captures `CreditCard`, `CustomerID` etc.
- - Added missing `expand_context` method — BFS graph traversal via `knowledge_graph.get_neighbors`
- - Added missing `_get_decision_query` method — creates a `DecisionQuery` from the knowledge graph
- - Fixed `hybrid_retrieval` to call `expand_context(query)` once (not per-entity) and include `"query"` key in return dict
- - Fixed `dynamic_context_traversal` to call `expand_context` once per query instead of per entity
- - Fixed `multi_hop_context_assembly` to use `_get_decision_query()` for robust decision lookup
- - Fixed `_retrieve_from_vector` to fall back to `result["metadata"]["content"]` when `result["content"]` is absent — prevents empty content and negative similarity scores during semantic re-ranking
-
- **Knowledge Graph Module (`semantica/kg/`)**
- - Fixed `calculate_pagerank` — added `alpha` and `max_iter` parameter aliases; changed return format to structured dict `{"centrality": scores, "rankings": sorted_list}`
- - Fixed `community_detector._to_networkx` to return a NetworkX graph directly when one is passed (was converting to adjacency list, silently losing all edges)
- - Added `method` as alias for `algorithm` parameter in `detect_communities`
- - Fixed `_build_adjacency` to handle `"edges"` key (list of tuples) in addition to `"relationships"` (list of dicts)
- - Added `_track_generic` base method and 9 domain-specific tracking methods to `AlgorithmTrackerWithProvenance`: `track_influence_analysis`, `track_verification_analysis`, `track_supply_chain_paths`, `track_bottleneck_analysis`, `track_quality_analysis`, `track_lead_time_analysis`, `track_cross_domain_analysis`, `track_cross_domain_similarity`, `track_collaboration_potential`
- - Created new `provenance_tracker.py` module with `ProvenanceTracker` class (`track_entity`, `get_all_sources`, `clear`)
-
- **Pipeline Module (`semantica/pipeline/`)**
- - Fixed `execution_engine` retry loop to properly iterate up to `max_retries` (was only retrying once regardless of policy)
- - Added `RecoveryAction` dataclass and `handle_failure(error, policy, retry_count)` method to `FailureHandler` — implements LINEAR, EXPONENTIAL, and FIXED backoff strategies
- - Fixed `pipeline_builder.add_step` to return the created `PipelineStep` object instead of `self`
- - Added `validate` as a public alias for `validate_pipeline` in `PipelineValidator`
- - Updated missing-dependency error message to `"Missing dependency '{dep}' for step '{name}'"` for consistent test assertions
-
- **Vector Store (`semantica/vector_store/`)**
- - Relaxed `test_batch_processing_performance` threshold from `< 100ms` to `< 500ms` per decision — original threshold was too tight for development machines running a real `sentence-transformers` embedding model (384-dim)
-
- **Test File Fixes**
- - `test_end_to_end_context_integration.py` — replaced emoji characters (`✅`, `❌`, `🔄`, `⚠️`) with ASCII equivalents (`[OK]`, `[FAIL]`, `[...]`, `[WARN]`) to fix Windows cp1252 encoding error
- - `test_context_retriever_precedents.py` — moved `assert_called_once_with` inside `with patch.object` block; fixed assertion to use `decision.scenario` not `decision.decision_id`; removed `"iPhone"` (lowercase-first) from entity extraction assertion
- - `test_real_world_scenarios.py` — fixed duplicate `source=` keyword argument (renamed to `label=`); fixed cross-domain analysis loop to iterate over all social network users instead of only `academic_users`
- - `test_pipeline_comprehensive.py` — changed `test_pipeline_validator_missing_deps` to call `validator.validate(builder)` directly instead of `builder.build()` which raises `ValidationError` before validation can complete
-
- **Results: ~840 tests passing, 36 skipped (external services), 0 failed**
-
- ## [0.3.0-alpha] - 2026-02-19
+ - **Core Temporal Data Model** (PR #396) — `semantica.kg.temporal_model` with shared parsing/normalization/serialization helpers; `TemporalBound` and `BiTemporalFact` exported from `semantica.kg`; valid-time and transaction-time filtering; `TemporalValidationError` on invalid inputs; history-preserving revisions in `TemporalVersionManager.apply_revision()` with supersession semantics.
+ - **Point-in-Time Query Engine** (PR #397) — `TemporalGraphQuery.reconstruct_at_time(graph, at_time)` builds consistent point-in-time subgraphs without mutating source; `TemporalConsistencyReport` detects inverted intervals, relationships outside entity lifetimes, overlapping same-type relationships, and temporal gaps; sequence/cycle pattern detection; calendar-aligned evolution bucketing via `temporal_granularity`; causal ordering controls on `find_temporal_paths()` (strict/overlap/loose).
+ - **Deterministic Temporal Reasoning Engine** (PR #398) — `semantica.kg.temporal_reasoning`; full Allen interval algebra via `IntervalRelation` (all 13 relations); `TemporalReasoningEngine` with interval merging, gap analysis, coverage calculation, timelines, retroactive coverage; zero LLM calls; circular import risk between `semantica.reasoning` and `semantica.kg` eliminated.
+ - **Temporal Awareness in ContextGraph** (PR #399) — `Decision` dataclass gains `valid_from`/`valid_until`; superseded decisions remain in graph (immutable history); `find_precedents_by_scenario(include_superseded, as_of)`; `ContextGraph.state_at(timestamp)` serializable snapshot; `CausalChainAnalyzer.trace_at_time(event_id, at_time)`; `AgentContext.checkpoint(label)`, `diff_checkpoints()`, `flush_checkpoint()`.
+ - **Temporal Metadata Extraction from Text** (PR #400):
+ - `extract_relations_llm(extract_temporal_bounds=True)` — each `Relation` gains `valid_from`, `valid_until`, `temporal_confidence` (0.0–1.0), `temporal_source_text`; default `False` is 100% backward-compatible.
+ - Calibrated confidence anchors: 1.00 = full ISO date → 0.00 = no temporal signal.
+ - `TemporalNormalizer` (zero LLM calls, pure regex + dateutil): `normalize(value)` → UTC datetime tuple or `None`; `normalize_phrase(phrase)` → metadata dict or `None`; 13-domain default phrase map; `TemporalAmbiguityWarning` for ambiguous DD/MM/YYYY inputs (never silently guesses locale).
+ - **Temporal Provenance & OWL-Time Export** (PR #401):
+ - `ProvenanceTracker.track_entity()` auto-stamps `recorded_at` on every new record.
+ - `query_recorded_between(start, end)`, `revision_history(fact_id)`, `export_audit_log(fact_ids, format)` (JSON/CSV).
+ - `RDFExporter.export_to_rdf(include_temporal=True, time_axis="valid|transaction|both")` — emits OWL-Time triples for all temporally-annotated relationships.
+ - `create_snapshot()` stamps `"format_version": "1.0"`; `validate_snapshot()` and `migrate_snapshot()` for stable snapshot lifecycle.
+ - **Temporal GraphRAG Integration** (PR #402) — `TemporalGraphRetriever` filters retrieved context to a point in time; `ContextRetriever.query_with_reasoning(at_time, header_template)` prepends structured temporal header; `TemporalQueryRewriter` extracts temporal intent (before/after/at/during/between) from natural language; regex-only by default, optional LLM-assisted mode.
- ### Added / Changed
+ **Ontology** (@KaifAhmad1 @ZohaibHassan16)
- - **Decision Tracking System**: Complete decision lifecycle management with audit trails and provenance tracking
- - **Advanced KG Algorithms**: Node2Vec embeddings, centrality analysis, community detection for decision insights
- - **Enhanced Context Module**: Unified AgentContext with granular feature flags and decision tracking integration
- - **Vector Store Features**: Hybrid search combining semantic, structural, and category similarity
- - **Policy Management**: Versioning, compliance checking, and exception handling
- - **Production Ready Architecture**: Scalable design with comprehensive error handling and validation
+ - **SHACL Shape Generation & Validation** (PR #318) — `SHACLGenerator` derives SHACL node/property shapes from any ontology dict; three quality tiers (basic/standard/strict); Turtle/JSON-LD/N-Triples output; iterative multi-level inheritance propagation, cycle-safe; `OntologyEngine.to_shacl()`, `export_shacl()`, `validate_graph(explain=True)`; `SHACLValidationReport` with plain-English explanations for all 7 constraint types. `pip install semantica[shacl]`.
+ - **SKOS Vocabulary Module** (PR #319) — `TripletStore.add_skos_concept()` / `get_skos_concepts(scheme_uri)`; `OntologyEngine.list_vocabularies()`, `list_concepts()`, `search_concepts()`; `NamespaceManager.get_skos_uri()` / `build_concept_scheme_uri()`; SPARQL injection hardened.
+ - **Ontology Alignment API** (PR #361) — `OntologyEngine.create_alignment()`, `get_alignments()`, `list_alignments()`; OWL/SKOS standard predicates (`owl:equivalentClass`, all five `skos:*Match`); `ReuseManager.suggest_alignments()`; `QueryEngine.expand_entity_uri(use_alignments=True)` with SPARQL `VALUES` clause injection; SPARQL injection hardened.
+ - **Ontology Diff & Migration** (PR #367) — `VersionManager.diff_ontologies()` covering classes/properties/individuals/axioms; `ChangeLogAnalyzer.analyze()` classifying CRITICAL/HIGH/MEDIUM/INFO impact; `ImpactReport`, `generate_change_report()`; `OntologyEngine.compare_versions()` end-to-end orchestrator with optional validation and graph-instance checks.
- ### Fixed
+ **Knowledge Explorer API** (@ZohaibHassan16 @KaifAhmad1)
- - Fixed import issues in test suite (ProvenanceTracker location fixes)
- - Fixed causal analyzer validation (max_depth bounds checking)
- - Fixed test compatibility with updated method signatures
- - Fixed mock object setup in test suites
- - Comprehensive test suite fixes for decision tracking features
-
- ### Testing
-
- - 113+ tests passing across context and core modules
- - Comprehensive decision tracking test coverage
- - Enhanced error handling and edge case testing
- - Fixed all critical test failures for release readiness
-
- ### Documentation
-
- - Enhanced context module documentation
- - Updated API references for decision tracking features
- - Comprehensive usage guides and examples
-
- - Fixed: Context Graphs decision tracking bugs and added comprehensive test coverage (PR #315 by @KaifAhmad1)
- - Fixed empty/None decision ID handling in ContextGraph.add_decision()
- - Fixed None metadata handling to prevent TypeError
- - Fixed causal chain depth logic and node exclusion
- - Fixed nonexistent node handling in add_causal_relationship()
- - Added missing properties field in to_dict serialization
- - Added missing from_dict method for graph deserialization
- - Fixed precedent search direction in find_precedents()
- - Fixed UUID generation logic in all decision models
- - Added comprehensive test suite with 9 tests covering all features
- - All 71 context tests now passing (100% success rate)
-
- - Fixed: PolicyEngine latest version selection on ContextGraph; AgentContext fallback robustness and secure logging (PR #TBD by @KaifAhmad1)
- - Tests: Added ContextGraph fallback and AgentContext smoke tests; full suite passing
-
- - **Apache AGE Backend Security Fixes** (PR #311 by @Sameer6305, fixes by @KaifAhmad1):
- - Added AgeStore class with GraphStore API compatibility
- - Fixed SQL injection vulnerabilities with input validation
- - Added psycopg2-binary dependency and migration guide
- - Fixed parameter replacement and test mock leakage
- - Enhanced error handling and Unicode display issues
-
- - **Context Engineering Enhancement** (PR #307 by @KaifAhmad1):
- - Comprehensive decision tracking system with full lifecycle management (record → analyze → query → precedent → influence)
- - Advanced KG algorithm integration: centrality analysis, community detection, node embeddings with ContextGraph
- - Enhanced AgentContext with granular feature flags for decision tracking, KG algorithms, and vector store features
- - PolicyException model replacing conflicting Exception name for meaningful business domain modeling
- - GraphStore validation preventing runtime failures with explicit capability checking
- - Hybrid search combining semantic, structural, and category similarity with configurable weights
- - Decision influence analysis with centrality measures and causal chain tracking
- - Policy management with versioning, compliance checking, and exception handling
- - Production-ready architecture with audit trails, security, and scalability features
- - 9 critical bug fixes: logging, security, audit trails, API compatibility, Cypher queries, centrality access, validation, naming
- - Comprehensive documentation with usage guides, production examples, and API references
- - 100% test coverage with all validation tests passing (9/9 tests)
- - Enterprise-grade features for financial services, healthcare, legal, and business domains
- - Complete backward compatibility with existing semantica components
- - Performance optimizations: caching, indexing, and efficient graph operations
-
- - **Added PgVector Store Support** (PR #303 by @Sameer6305, @KaifAhmad1):
- - Native PostgreSQL vector storage using pgvector extension with full integration
- - Multiple distance metrics: cosine, L2/Euclidean, inner product with automatic score normalization
- - Advanced indexing: HNSW and IVFFlat for approximate nearest neighbor search with tunable parameters
- - JSONB metadata storage with flexible filtering capabilities and batch operations
- - Connection pooling support with psycopg3/psycopg2 fallback and efficient resource management
- - Comprehensive VectorStore integration with backend delegation and unified API
- - Idempotent index creation and table management with safe migration support
- - Production-ready security: SQL injection protection with psycopg_sql.SQL() and input validation
- - Performance optimizations: UUID4-based IDs, batch executemany operations, connection pooling
- - Full backward compatibility with existing vector store implementations
- - 36+ comprehensive test cases with Docker integration and dependency skipping
- - Complete documentation with setup guides, examples, and performance tuning
- - CI/CD integration: resolved benchmark compatibility and fixed documentation links
-
- - **Improved Vector Store for Decision Tracking** (PR #293 by @KaifAhmad1):
- - Comprehensive decision tracking capabilities with hybrid search combining semantic and structural embeddings
- - New DecisionEmbeddingPipeline for generating semantic and structural embeddings with KG algorithm integration
- - HybridSimilarityCalculator with configurable weights (semantic: 0.7, structural: 0.3)
- - DecisionContext high-level interface for decision management with explainable AI features
- - ContextRetriever with hybrid precedent search and multi-hop reasoning
- - User-friendly convenience API: quick_decision(), find_precedents(), explain(), similar_to(), batch_decisions(), filter_decisions()
- - Knowledge Graph algorithm integration: Node2Vec, PathFinder, CommunityDetector, CentralityCalculator, SimilarityCalculator, ConnectivityAnalyzer
- - Explainable AI with path tracing, confidence scoring, and comprehensive decision explanations
- - Performance optimizations: 0.028s per decision processing, 0.031s search performance, ~0.8KB per decision memory usage
- - 100% backward compatibility maintained with existing VectorStore functionality
- - 34+ comprehensive tests covering all functionality including end-to-end scenarios and performance benchmarks
- - Real-world validation examples for banking and insurance domains
- - Documentation with clear imports, examples, and API references
-
- - **Improved Graph Algorithms in KG Module** (PR #292 by @KaifAhmad1):
- - Complete algorithm suite with 30+ graph algorithms across 7 categories
- - Node Embeddings: Node2Vec, DeepWalk, Word2Vec for structural similarity analysis
- - Similarity Analysis: Cosine, Euclidean, Manhattan, Correlation metrics with batch processing
- - Path Finding: Dijkstra, A*, BFS, K-shortest paths for route and network analysis
- - Link Prediction: Preferential attachment, Jaccard, Adamic-Adar for network completion
- - Centrality Analysis: Degree, Betweenness, Closeness, PageRank for importance ranking
- - Community Detection: Louvain, Leiden, Label propagation for clustering analysis
- - Connectivity Analysis: Components, bridges, density for network robustness
- - Unified provenance tracking system with GraphBuilderWithProvenance and AlgorithmTrackerWithProvenance
- - Complete execution tracking with metadata, timestamps, and reproducibility IDs
- - Comprehensive test coverage with 5 test suites and 40+ test methods
- - Professional documentation overhaul for all modules and reference documentation
- - Enterprise-ready functionality with error handling and NetworkX compatibility
- - Performance optimizations with sparse matrix operations and batch processing
- - Full backward compatibility maintained with gradual migration support
-
- - **Improved Security Configuration with Dependabot**:
- - Configured bi-weekly security updates with manual review by @KaifAhmad1
- - Implemented automated security scans (Monday & Thursday at 7 AM IST) with Bandit, Safety, Semgrep
- - Added security-critical package grouping (cryptography, requests, urllib3, certifi, pyopenssl)
- - Enterprise-grade security with audit trail, compliance features, and zero auto-merge
- - Optimized IST timezone scheduling (Security scans: 7 AM IST, PRs: 9 AM IST)
- - Aligned with new Dependabot features: open-source proxy support, smart dependency grouping for Snowflake/Arrow/benchmark features, private registry support, semantic commit prefixes, and latest GitHub security best practices
-
- - **ResourceScheduler Deadlock Fix and Performance Improvements** (PR #299, #301 by @d4ndr4d3, @KaifAhmad1):
- - Fixed critical deadlock in ResourceScheduler by replacing `threading.Lock()` with `threading.RLock()`
- - Resolved nested lock acquisition issue in `allocate_resources()` → `allocate_cpu/memory/gpu()` calls
- - Added allocation validation with `ValidationError` when no resources can be allocated
- - Improved performance by moving progress tracking updates outside lock scope
- - Implemented comprehensive resource cleanup on allocation failures to prevent leaks
- - Added complete regression test suite (6 tests) for deadlock prevention and edge cases
- - Improved error handling and documentation for better operator visibility
- - Zero breaking changes, maintains thread safety and backward compatibility
+ - **Full FastAPI backend** (PR #384) — `semantica.explorer` package with graph, analytics, decisions, temporal, enrichment, export/import, annotations routes; 12 export formats; WebSocket progress for import; 99 integration tests. `pip install semantica[explorer]`; CLI: `semantica-explorer --graph my_graph.json`.
+ - **Thread safety** (PR #385) — `ContextGraph` and `GraphSession` protected with `threading.RLock`; 8 analytics components lazily initialized under lock.
+ - **In-memory fallbacks** (PR #386) — All 7 `DecisionQuery` and 4 `DecisionRecorder` methods have `ContextGraph` fallback paths for in-memory usage without a graph DB.
+ - **Snapshot schema compatibility** (PR #393) — accepts both `nodes`/`edges` and `entities`/`relationships` snapshot schemas transparently; metadata counts always accurate.
+ - **Audit trail & rollback protection** (PR #394) — mutation-level audit tracking, named version tags, `restore_snapshot()` requires explicit confirmation, `get_node_history()`, `diff()` Git-like alias.
+ - **SKOS Vocabulary REST API** (PR #426) — `GET /api/vocabulary/schemes`, `GET /api/vocabulary/hierarchy?scheme=<uri>` with cycle detection, `POST /api/vocabulary/import` (.ttl/.rdf/.owl; HTTP 422 on invalid).
+ - **O(N) → O(limit) Pagination** (PR #431) — `find_nodes`/`find_edges` use `itertools.islice` on generators; ghost-node fix (accepts `source_id`/`target_id` and `source`/`target` key names); deterministic page boundaries via `sorted()`; `stats()` applies same validity filters as pagination.
+ - **Named graph support** (PR #432, @Sameer6305) — `enable_named_graphs` flag forwarded correctly through `TripletStore.execute_query()`; duplicate `FROM`/`FROM NAMED` clauses prevented; graph URIs percent-encoded in DROP statements.
- ## [0.2.7] - 2026-02-09
+ **Integrations**
- ### Added / Changed
-
- - **Snowflake Connector for Data Ingestion** (PR #276 by @Sameer6305):
- - Native Snowflake connector with multi-authentication (password, OAuth, key-pair, SSO)
- - Table and query ingestion with pagination, schema introspection, batch processing
- - SQL injection prevention via identifier escaping, OAuth token validation
- - Progress tracking integration, context manager support, document export
- - 24 comprehensive unit tests with mocking, complete documentation and examples
- - Added as optional dependency `db-snowflake` with snowflake-connector-python>=3.0.0
-
- - **Apache Arrow Export Support** (PR #273 by @Sameer6305):
- - Added Apache Arrow exporter with explicit schemas, entity/relationship export, compression support
- - Integrated with export module and method registry, Pandas/DuckDB compatible
- - 20 unit tests + 1 integration test, complete documentation with examples
-
- - **Comprehensive Benchmark Suite with Regression CLI** (PR #289 by @ZohaibHassan16, @KaifAhmad1):
- - 137+ benchmarks across all 10 Semantica modules (Input, Core, Storage, Context, QA, Ontology, etc.)
- - Environment-agnostic design with robust mocking system for CI/CD compatibility
- - Statistical regression detection using Z-score analysis with configurable thresholds
- - Automated performance auditing via GitHub Actions workflow
- - Comprehensive documentation suite (benchmarks.md, architecture guides, usage examples)
- - Zero breaking changes, production-ready with ultra-fast text processing (>10,000 ops/s)
- - Added benchmark runner CLI: `python benchmarks/benchmark_runner.py`
+ - **Agno Agentic Framework** (Issue #249, @KaifAhmad1) — 5 components, all degrading gracefully when `agno` is not installed:
+ - `AgnoContextStore` — graph-backed agent memory implementing `agno.memory.db.base.MemoryDb`.
+ - `AgnoKnowledgeGraph` — multi-hop GraphRAG knowledge base implementing `agno.knowledge.base.AgentKnowledge`.
+ - `AgnoDecisionKit` — 6 decision-intelligence tools (record_decision, find_precedents, trace_causal_chain, analyze_impact, check_policy, get_decision_summary).
+ - `AgnoKGToolkit` — 7 KG pipeline tools (extract_entities, extract_relations, add_to_graph, query_graph, find_related, infer_facts, export_subgraph).
+ - `AgnoSharedContext` — team coordinator with single shared `ContextGraph`; `bind_agent(role)` returns role-scoped view; thread-safe via `RLock`.
+ - 110 integration tests; 3 cookbook notebooks. `pip install semantica[agno]`.
+ - **Novita AI Provider** (PR #374, @Alex-wuhu) — OpenAI-compatible; default model `deepseek/deepseek-v3.2`; `NOVITA_API_KEY`; `create_provider("novita")`.
- ## [0.2.6] - 2026-02-03
+ **Reasoning**
- ### Added / Changed
-
- - **W3C PROV-O Compliant Provenance Tracking** (#254, #246):
- - Comprehensive provenance tracking system with W3C PROV-O compliance across all 17 Semantica modules
- - **Core Module**: `ProvenanceManager`, W3C PROV-O schemas, storage backends (InMemory, SQLite), SHA-256 integrity verification
- - **Module Integrations**: Semantic Extract, LLMs (Groq, OpenAI, HuggingFace, LiteLLM), Pipeline, Context, Ingest, Embeddings, Graph/Vector/Triplet stores, Reasoning, Conflicts, Deduplication, Export, Parse, Normalize, Ontology, Visualization
- - **Features**: Complete lineage tracking (Document → Chunk → Entity → Relationship → Graph), LLM tracking (tokens, costs, latency), source tracking, bridge axioms for domain transformations
- - **Compliance Infrastructure**: W3C PROV-O, FDA 21 CFR Part 11, SOX, HIPAA, TNFD
- - **Testing**: 237 tests covering core functionality, all 17 module integrations, edge cases, backward compatibility
- - **Design**: Opt-in with `provenance=False` by default, zero breaking changes, no new dependencies
- - Contributed by @KaifAhmad1
-
- - **Enhanced Change Management Module** (#248, #243):
- - Enterprise-grade version control for knowledge graphs and ontologies with persistent storage and audit trails
- - **Core Classes**: `TemporalVersionManager` (KG versioning), `OntologyVersionManager` (ontology versioning), `ChangeLogEntry` (metadata)
- - **Storage**: SQLite (persistent) and in-memory backends with thread-safe operations
- - **Features**: SHA-256 checksums, detailed entity/relationship diffs, structural ontology comparison, email validation
- - **Compliance Infrastructure**: HIPAA, SOX, FDA 21 CFR Part 11 with immutable audit trails
- - **Testing**: 104 tests (100% pass) - unit, integration, compliance, performance, edge cases
- - **Performance**: 17.6ms for 10k entities, 510+ ops/sec concurrent, handles 5k+ entity graphs
- - **Migration**: Backward compatible, simplified class names, zero external dependencies
- - Contributed by @KaifAhmad1
-
- - CSV Ingestion Enhancements (PR #244 by @saloni0318)
- - Auto-detect CSV encoding (chardet) and delimiter (csv.Sniffer)
- - Tolerant decoding and malformed-row handling (`on_bad_lines='warn'`)
- - Optional chunked reading for large files; metadata tracks detected values
- - Expanded unit tests covering delimiters, quoted/multiline fields, header overrides, chunks, and NaN preservation
-
- - Tests: Comprehensive units for TextNormalizer (PR #242 by @ZohaibHassan16)
- - Added focused test coverage for TextNormalizer behavior across inputs
-
- - Tests: Register integration mark and tidy ingest test warnings (PR #241 by @KaifAhmad1)
- - Introduced integration test marker and reduced noisy warnings in ingest tests
-
- - **Ingest Unit Tests** (#239, #232):
- - Comprehensive unit tests for ingestion modules (file, web, and feed ingestors)
- - **Coverage**: File scanning (local/cloud S3/GCS/Azure), web ingestion (URL/sitemap/robots.txt), RSS/Atom feed parsing
- - **Testing**: 998 lines of test code with mocked external dependencies for fast, isolated execution
- - **Results**: file_ingestor (86%), web_ingestor (86%), feed_ingestor (80%) coverage
- - Covers happy paths, edge cases, and error handling
- - Contributed by @Mohammed2372
+ - **Native Datalog Reasoning Engine** (PR #371, @ZohaibHassan16) — pure-Python bottom-up semi-naive fixpoint with guaranteed termination; recursive Horn clause rules (e.g. `ancestor(X,Y) :- parent(X,Z), ancestor(Z,Y).`); O(1) delta-index lookup; `load_from_graph(ContextGraph)`; `query("pred(?X, ?Y)")` with optional `bindings=`; `DatalogReasoner`, `DatalogFact`, `DatalogRule` exported from `semantica.reasoning`.
### Fixed