Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

feat(semantic_extract): pluggable persistent backend for ExtractionCache (#1591)

Besokus5544777e59a8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 87d5065a..293480e9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Pluggable, persistent backend for `ExtractionCache`** (#1581) by @Besokus
+ - `ExtractionCache` now delegates storage to a `CacheBackend`, keeping stable SHA-256 key derivation (text + params, with `provider`/`model`/generation params; sensitive keys filtered) and the public `get`/`set`/`clear`/`get_stats` API in one place. Default behavior is unchanged — an in-memory LRU + TTL backend (`InMemoryBackend`)
+ - New `SqliteCacheBackend` (`semantica.semantic_extract`, lazy export): a persistent backend backed by the stdlib `sqlite3`, so cached extraction results **survive a process restart** — a fresh process (CI job, notebook kernel, batch worker, extraction subprocess) reuses prior results instead of re-paying every LLM call. TTL and LRU (by last access) mirror the in-memory backend. Values are serialized with a pluggable serializer (`pickle` by default; pass e.g. `json` to avoid pickle — point the DB at a trusted, local path)
+ - Selectable via config: `cache_backend` (`"memory"` | `"sqlite"`) and `cache_path`, settable through a config file, the new `SEMANTICA_CACHE_*` environment variables (`SEMANTICA_CACHE_BACKEND` / `SEMANTICA_CACHE_PATH` / `SEMANTICA_CACHE_TTL` / `SEMANTICA_CACHE_SIZE` / `SEMANTICA_CACHE_ENABLED`), or the new `configure_cache()` API at runtime. Any failure constructing the persistent backend degrades gracefully to the in-memory default, so extraction never breaks on a cache misconfiguration
+ - Persistent-cache keys are process-stable: relation/triplet extraction now folds entity/relation inputs into the key via a deterministic SHA-256 fingerprint instead of the process-randomized built-in `hash()`, so entries reliably rehit after a restart. `ttl=0` expires immediately on both backends, and corrupt/undeserializable rows are dropped rather than retained
+ - **Security / trust model:** the sqlite file is deserialized back into the process (default serializer `pickle`), so it must point at a trusted, user-private location and may hold sensitive extraction results in the clear (documented in the module usage guide). The default database lives under a per-user private directory (`$XDG_CACHE_HOME`/`~/.cache`, `0o700`); the db file is created **atomically** with `0o600` (`O_CREAT|O_EXCL`, no validate-then-open window) and an existing symlink, non-regular file, or file owned by another user is refused (falling back to in-memory). sqlite access uses a busy-timeout with bounded retry, rolls back within the same lock scope on failure, and contains all errors (reads degrade to a miss, writes to a skipped update)
+ - No new runtime dependencies (stdlib `sqlite3`). New public exports: `CacheBackend`, `InMemoryBackend`, `SqliteCacheBackend`, `configure_cache`. New `tests/semantic_extract/test_cache_backends.py`
- **Schema-guided extraction validation** (#1510) by @Besokus
- New `SchemaValidator` (`semantica.semantic_extract`, lazy export): a deterministic sibling of `ExtractionValidator` that checks extraction output for *conformance to a domain ontology* — an axis orthogonal to `ExtractionValidator`'s confidence checks. It mirrors the same interface (`validate_entities()` / `validate_relations()` returning `ValidationResult`, batch-aware), so the two compose back-to-back
- Entity labels must be concepts in the schema; relation predicates must be in the schema and satisfy their `domain` / `range`. Violations are reported in `ValidationResult.errors` with counts in `metrics` and `score` = conformance ratio; `filter_by_schema()` / `filter_relations_by_schema()` return the conforming subset (mirroring `filter_by_confidence`). No LLM required
diff --git a/semantica/semantic_extract/__init__.py b/semantica/semantic_extract/__init__.py
index d05fc0f7..e0b5b0ea 100644
--- a/semantica/semantic_extract/__init__.py
+++ b/semantica/semantic_extract/__init__.py
@@ -127,10 +127,16 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
# Config
"Config": (".config", "Config"),
"config": (".config", "config"),
+ # Cache
+ "ExtractionCache": (".cache", "ExtractionCache"),
+ "CacheBackend": (".cache", "CacheBackend"),
+ "InMemoryBackend": (".cache", "InMemoryBackend"),
+ "SqliteCacheBackend": (".cache", "SqliteCacheBackend"),
# Methods
"get_entity_method": (".methods", "get_entity_method"),
"get_relation_method": (".methods", "get_relation_method"),
"get_triplet_method": (".methods", "get_triplet_method"),
+ "configure_cache": (".methods", "configure_cache"),
}
@@ -217,6 +223,12 @@ __all__ = [
# Config
"Config",
"config",
+ # Cache
+ "ExtractionCache",
+ "CacheBackend",
+ "InMemoryBackend",
+ "SqliteCacheBackend",
+ "configure_cache",
# Methods
"get_entity_method",
"get_relation_method",
diff --git a/semantica/semantic_extract/cache.py b/semantica/semantic_extract/cache.py
index a696de85..c34d7f1b 100644
--- a/semantica/semantic_extract/cache.py
+++ b/semantica/semantic_extract/cache.py
@@ -10,26 +10,44 @@ Key Features:
- TTL Support: Expires items after a configurable duration
- Namespaced Caching: Separate caches for entities, relations, and triplets
- Hash-based Keys: Uses stable hashing for text and parameters
+ - Pluggable Backends: In-memory (default) or persistent (sqlite), selected
+ without changing the ``ExtractionCache`` public API
Classes:
- - ExtractionCache: Main cache manager
+ - ExtractionCache: Main cache manager (owns key derivation + public API)
+ - CacheBackend: Storage backend interface (owns persistence/expiry/eviction)
+ - InMemoryBackend: Thread-safe in-memory LRU + TTL backend (historical default)
+ - SqliteCacheBackend: Persistent backend backed by the stdlib ``sqlite3``
- CacheItem: Container for cached data with metadata
Author: Semantica Contributors
License: MIT
"""
+import os
+import stat
import time
import hashlib
import json
+import pickle
+import sqlite3
+import threading
+from abc import ABC, abstractmethod
from collections import OrderedDict
-from typing import Any, Dict, Optional, Union, List
+from pathlib import Path
+from typing import Any, Callable, Dict, Optional
from threading import Lock
from ..utils.logging import get_logger
+# The namespaces ExtractionCache manages. Kept as a module constant so backends
+# and get_stats() agree on the set without hard-coding it in several places.
+NAMESPACES = ("entities", "relations", "triplets")
+
+
class CacheItem:
"""Container for cached data."""
+
def __init__(self, value: Any, ttl: Optional[int] = None):
self.value = value
self.timestamp = time.time()
@@ -41,95 +59,493 @@ class CacheItem:
return False
return time.time() - self.timestamp > self.ttl
+
+class CacheBackend(ABC):
+ """Storage backend for :class:`ExtractionCache`.
+
+ A backend owns persistence, expiry and eviction for opaque
+ ``(namespace, key) -> value`` entries. ``ExtractionCache`` owns key
+ derivation (the stable SHA-256 hash over text + params) and the public
+ API, so backends never see raw text or generation parameters — only the
+ already-hashed key. This keeps the key policy in one place and lets a
+ backend be swapped (in-memory vs persistent) with no behavioral change to
+ callers.
+ """
+
+ @abstractmethod
+ def get(self, namespace: str, key: str) -> Optional[Any]:
+ """Return the cached value or ``None`` if missing/expired."""
+
+ @abstractmethod
+ def set(self, namespace: str, key: str, value: Any, ttl: Optional[int]) -> None:
+ """Store ``value`` under ``(namespace, key)`` with an optional TTL (seconds)."""
+
+ @abstractmethod
+ def clear(self, namespace: Optional[str] = None) -> None:
+ """Clear one namespace, or all namespaces when ``namespace`` is ``None``."""
+
+ @abstractmethod
+ def stats(self, namespace: str) -> Dict[str, int]:
+ """Return backend statistics for ``namespace``.
+
+ Must include at least ``size`` (stored entries) and ``max_size`` (the
+ backend's own eviction limit) so callers see the limit actually in
+ effect rather than a facade default. Implementations may add more keys.
+ """
+
+ def size(self, namespace: str) -> int:
+ """Convenience: number of stored entries in ``namespace``."""
+ return int(self.stats(namespace).get("size", 0))
+
+
+class InMemoryBackend(CacheBackend):
+ """Thread-safe in-memory LRU + TTL backend — the historical default.
+
+ Preserves the original storage layout: one ``OrderedDict`` per namespace
+ guarded by its own lock, LRU eviction at ``max_size``, and per-item TTL
+ expiry checked lazily on read.
+ """
+
+ def __init__(self, max_size: int = 1000):
+ self.max_size = max_size
+ self._caches: Dict[str, "OrderedDict[str, CacheItem]"] = {
+ ns: OrderedDict() for ns in NAMESPACES
+ }
+ self._locks: Dict[str, Lock] = {ns: Lock() for ns in NAMESPACES}
+
+ def get(self, namespace: str, key: str) -> Optional[Any]:
+ if namespace not in self._caches:
+ return None
+ with self._locks[namespace]:
+ cache = self._caches[namespace]
+ item = cache.get(key)
+ if item is None:
+ return None
+ if item.is_expired():
+ del cache[key]
+ return None
+ cache.move_to_end(key) # mark as recently used
+ return item.value
+
+ def set(self, namespace: str, key: str, value: Any, ttl: Optional[int]) -> None:
+ if namespace not in self._caches:
+ return
+ with self._locks[namespace]:
+ cache = self._caches[namespace]
+ if key in cache:
+ cache.move_to_end(key)
+ cache[key] = CacheItem(value, ttl)
+ if len(cache) > self.max_size:
+ cache.popitem(last=False) # evict least recently used
+
+ def clear(self, namespace: Optional[str] = None) -> None:
+ if namespace is not None:
+ if namespace in self._caches:
+ with self._locks[namespace]:
+ self._caches[namespace].clear()
+ else:
+ for ns in self._caches:
+ with self._locks[ns]:
+ self._caches[ns].clear()
+
+ def stats(self, namespace: str) -> Dict[str, int]:
+ cache = self._caches.get(namespace)
+ return {
+ "size": len(cache) if cache is not None else 0,
+ "max_size": self.max_size,
+ }
+
+
+class SqliteCacheBackend(CacheBackend):
+ """Persistent cache backend backed by the standard-library ``sqlite3``.
+
+ Unlike :class:`InMemoryBackend`, entries survive a process restart, so a
+ fresh process (CI job, notebook kernel, batch worker, extraction
+ subprocess) reuses previous results instead of re-paying every LLM call.
+
+ **Trust model (read before enabling).** The database file is deserialized
+ back into this process on every read, and the default ``serializer`` is
+ ``pickle`` — so a cache file an attacker can influence is a code-execution
+ vector. Therefore:
+
+ - ``db_path`` MUST point at a **trusted, per-user, private** location.
+ - The file may hold **sensitive extraction results** in the clear.
+ - The default ``pickle`` serializer must only be used with a trusted file;
+ pass ``serializer`` / ``deserializer`` (e.g. ``json``) for untrusted or
+ shared locations.
+
+ To enforce this the backend creates the file **atomically** with ``0o600``
+ (``O_CREAT | O_EXCL``, no validate-then-open window) and, when reusing an
+ existing file, rejects symlinks, non-regular files, and files owned by
+ another user — raising so the caller falls back to the in-memory backend.
+
+ Concurrency/reliability: TTL and LRU (by last access) mirror
+ :class:`InMemoryBackend`. A ``busy_timeout`` plus bounded retry handles
+ cross-process lock contention; every sqlite failure rolls back within the
+ same lock scope and is contained — reads degrade to a miss and writes to a
+ skipped update, never aborting extraction.
+
+ Cache-key invalidation is owned by :class:`ExtractionCache`: the key is a
+ SHA-256 over the input text and parameters (provider, model, generation
+ params, ...), so changing any of them yields a different key and old entries
+ are simply bypassed rather than served stale — including after a model
+ upgrade. There is no schema-version stamp; if a serialized value's shape
+ changes incompatibly, point ``cache_path`` at a fresh file (or ``clear()``).
+ """
+
+ def __init__(
+ self,
+ db_path: str,
+ max_size: int = 1000,
+ serializer: Callable[[Any], bytes] = pickle.dumps,
+ deserializer: Callable[[bytes], Any] = pickle.loads,
+ busy_timeout: float = 5.0,
+ max_retries: int = 3,
+ retry_backoff: float = 0.05,
+ ):
+ self.max_size = max_size
+ self._serialize = serializer
+ self._deserialize = deserializer
+ self._busy_timeout = busy_timeout
+ self._max_retries = max_retries
+ self._retry_backoff = retry_backoff
+ self._lock = threading.Lock()
+ self.logger = get_logger("extraction_cache.sqlite")
+
+ if db_path == ":memory:":
+ self.db_path = ":memory:"
+ else:
+ # Expand "~" and make absolute exactly once (a "~/..." path must be
+ # expanded before it reaches sqlite). Do NOT resolve() — that would
+ # follow a symlinked final component we specifically want to reject.
+ self.db_path = os.path.abspath(os.path.expanduser(db_path))
+ os.makedirs(os.path.dirname(self.db_path), exist_ok=True)
+ self._create_or_validate_secure_file(self.db_path)
+
+ self._conn = sqlite3.connect(
+ self.db_path, check_same_thread=False, timeout=self._busy_timeout
+ )
+ self._conn.execute(f"PRAGMA busy_timeout={int(self._busy_timeout * 1000)}")
+ try:
+ self._conn.execute("PRAGMA journal_mode=WAL")
+ except sqlite3.DatabaseError: # e.g. :memory: does not support WAL
+ pass
+ self._conn.execute(
+ "CREATE TABLE IF NOT EXISTS cache ("
+ " namespace TEXT NOT NULL,"
+ " key TEXT NOT NULL,"
+ " value BLOB NOT NULL,"
+ " expires_at REAL,"
+ " last_access REAL NOT NULL,"
+ " PRIMARY KEY (namespace, key)"
+ ")"
+ )
+ # Index supporting the LRU eviction scan (ORDER BY last_access per ns).
+ self._conn.execute(
+ "CREATE INDEX IF NOT EXISTS idx_cache_lru "
+ "ON cache (namespace, last_access)"
+ )
+ self._conn.commit()
+
+ def _create_or_validate_secure_file(self, path: str) -> None:
+ """Create the db file atomically with ``0o600``, or validate an existing one.
+
+ Using ``O_CREAT | O_EXCL`` closes the TOCTOU/permission window of a
+ validate-then-open-then-chmod sequence: a fresh file exists with private
+ permissions from the very first moment. An existing file is only reused
+ if it is a regular (non-symlink) file owned by the current user.
+ """
+ try:
+ fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ os.close(fd)
+ return # freshly created, private
+ except FileExistsError:
+ pass # fall through to validate the existing file
+
+ try:
+ info = os.lstat(path) # lstat: do not follow a symlinked final path
+ except OSError as exc:
+ raise PermissionError(f"Cannot stat cache database {path}: {exc}")
+
+ if stat.S_ISLNK(info.st_mode):
+ raise PermissionError(f"Refusing to use symlinked cache database: {path}")
+ if not stat.S_ISREG(info.st_mode):
+ raise PermissionError(f"Cache database is not a regular file: {path}")
+ if hasattr(os, "getuid") and info.st_uid != os.getuid(): # POSIX only
+ raise PermissionError(
+ f"Refusing to use cache database owned by another user: {path}"
+ )
+
+ def _rollback_locked(self) -> None:
+ """Roll back the current transaction. Caller MUST hold ``self._lock``
+ so a later request cannot commit over a failed one on the shared conn."""
+ try:
+ self._conn.rollback()
+ except sqlite3.Error:
+ pass
+
+ def _write_locked(self, operation: Callable[[sqlite3.Connection], None]) -> bool:
+ """Run a write ``operation`` + commit under the lock, rolling back within
+ the same lock scope on failure. Retries on 'database is locked' with
+ bounded backoff (sleeping outside the lock). Returns success."""
+ last_exc: Optional[sqlite3.Error] = None
+ for attempt in range(self._max_retries):
+ retryable = False
+ with self._lock:
+ try:
+ operation(self._conn)
+ self._conn.commit()
+ return True
+ except sqlite3.OperationalError as exc:
+ self._rollback_locked()
+ last_exc = exc
+ retryable = "locked" in str(exc).lower()
+ except sqlite3.Error as exc:
+ self._rollback_locked()
+ last_exc = exc
+ if not retryable or attempt == self._max_retries - 1:
+ break
+ time.sleep(self._retry_backoff * (2**attempt))
+ self.logger.warning(f"sqlite cache write failed (skipped): {last_exc}")
+ return False
+
+ def get(self, namespace: str, key: str) -> Optional[Any]:
+ now = time.time()
+ with self._lock:
+ try:
+ row = self._conn.execute(
+ "SELECT value, expires_at FROM cache WHERE namespace=? AND key=?",
+ (namespace, key),
+ ).fetchone()
+ except sqlite3.Error as exc:
+ self._rollback_locked()
+ self.logger.warning(f"sqlite cache read failed (miss): {exc}")
+ return None
+ if row is None:
+ return None
+ value_blob, expires_at = row
+ # ttl=0 -> expires_at == now: treat as expired (use <=), matching the
+ # in-memory backend.
+ if expires_at is not None and expires_at <= now:
+ try:
+ self._conn.execute(
+ "DELETE FROM cache WHERE namespace=? AND key=?",
+ (namespace, key),
+ )
+ self._conn.commit()
+ except sqlite3.Error as exc:
+ self._rollback_locked()
+ self.logger.warning(f"sqlite expired-entry cleanup failed: {exc}")
+ return None
+
+ # Deserialize OUTSIDE the lock but BEFORE bumping last_access: a
+ # corrupt/incompatible row must not stay "recently used" (which would
+ # keep it from being evicted), and must be dropped.
+ try:
+ value = self._deserialize(value_blob)
+ except Exception as exc:
+ self.logger.warning(
+ f"Failed to deserialize cached value; dropping row: {exc}"
+ )
+ with self._lock:
+ try:
+ self._conn.execute(
+ "DELETE FROM cache WHERE namespace=? AND key=?",
+ (namespace, key),
+ )
+ self._conn.commit()
+ except sqlite3.Error:
+ self._rollback_locked()
+ return None
+
+ with self._lock:
+ try:
+ self._conn.execute(
+ "UPDATE cache SET last_access=? WHERE namespace=? AND key=?",
+ (now, namespace, key),
+ )
+ self._conn.commit()
+ except sqlite3.Error as exc: # non-fatal: value is already in hand
+ self._rollback_locked()
+ self.logger.debug(f"sqlite last_access update failed: {exc}")
+ return value
+
+ def set(self, namespace: str, key: str, value: Any, ttl: Optional[int]) -> None:
+ now = time.time()
+ # ttl=0 must expire immediately (parity with InMemoryBackend), so guard
+ # on "is None" rather than truthiness.
+ expires_at = now + ttl if ttl is not None else None
+ try:
+ blob = self._serialize(value)
+ except Exception as exc: # unserializable value — skip caching, never crash
+ self.logger.warning(f"Failed to serialize value for caching: {exc}")
+ return
+
+ def operation(conn: sqlite3.Connection) -> None:
+ conn.execute(
+ "INSERT INTO cache (namespace, key, value, expires_at, last_access) "
+ "VALUES (?, ?, ?, ?, ?) "
+ "ON CONFLICT(namespace, key) DO UPDATE SET "
+ " value=excluded.value,"
+ " expires_at=excluded.expires_at,"
+ " last_access=excluded.last_access",
+ (namespace, key, blob, expires_at, now),
+ )
+ count = conn.execute(
+ "SELECT COUNT(*) FROM cache WHERE namespace=?", (namespace,)
+ ).fetchone()[0]
+ if count > self.max_size:
+ conn.execute(
+ "DELETE FROM cache WHERE rowid IN ("
+ " SELECT rowid FROM cache WHERE namespace=? "
+ " ORDER BY last_access ASC LIMIT ?"
+ ")",
+ (namespace, count - self.max_size),
+ )
+
+ self._write_locked(operation)
+
+ def clear(self, namespace: Optional[str] = None) -> None:
+ def operation(conn: sqlite3.Connection) -> None:
+ if namespace is None:
+ conn.execute("DELETE FROM cache")
+ else:
+ conn.execute("DELETE FROM cache WHERE namespace=?", (namespace,))
+
+ self._write_locked(operation)
+
+ def stats(self, namespace: str) -> Dict[str, int]:
+ with self._lock:
+ try:
+ row = self._conn.execute(
+ "SELECT COUNT(*) FROM cache WHERE namespace=?", (namespace,)
+ ).fetchone()
+ size = int(row[0]) if row else 0
+ except sqlite3.Error as exc:
+ self._rollback_locked()
+ self.logger.warning(f"sqlite cache stats failed: {exc}")
+ size = 0
+ return {"size": size, "max_size": self.max_size}
+
+ def close(self) -> None:
+ """Close the underlying connection (safe to call more than once)."""
+ with self._lock:
+ try:
+ self._conn.close()
+ except sqlite3.Error:
+ pass
+
+
class ExtractionCache:
"""
LRU Cache for extraction results.
- Thread-safe implementation.
+
+ Owns stable key derivation and the public ``get``/``set``/``clear`` API;
+ delegates storage to a pluggable :class:`CacheBackend`. Defaults to a
+ thread-safe in-memory backend, preserving the original behavior. Pass a
+ persistent backend (e.g. :class:`SqliteCacheBackend`) to survive restarts.
"""
- def __init__(self, max_size: int = 1000, ttl: int = 3600):
+
+ def __init__(
+ self,
+ max_size: int = 1000,
+ ttl: int = 3600,
+ backend: Optional[CacheBackend] = None,
+ ):
"""
Initialize the cache.
-
+
Args:
- max_size: Maximum number of items to store per namespace
+ max_size: Maximum number of items to store per namespace (used by
+ the default in-memory backend)
ttl: Time to live in seconds (default 1 hour)
+ backend: Storage backend. Defaults to :class:`InMemoryBackend`.
"""
self.max_size = max_size
self.ttl = ttl
- self._caches: Dict[str, OrderedDict] = {
- "entities": OrderedDict(),
- "relations": OrderedDict(),
- "triplets": OrderedDict()
- }
- self._locks: Dict[str, Lock] = {
- "entities": Lock(),
- "relations": Lock(),
- "triplets": Lock()
- }
self.logger = get_logger("extraction_cache")
self.enabled = True
+ self._backend: CacheBackend = (
+ backend if backend is not None else InMemoryBackend(max_size=max_size)
+ )
+
+ # Backward-compat: some callers/tests reach into the historical internal
+ # attributes. Proxy them to the backend when it exposes them (the default
+ # in-memory backend does); otherwise present empty mappings.
+ @property
+ def _caches(self) -> Dict[str, Any]:
+ return getattr(self._backend, "_caches", {})
+
+ @property
+ def _locks(self) -> Dict[str, Any]:
+ return getattr(self._backend, "_locks", {})
def _generate_key(self, text: str, **params) -> str:
"""
Generate a stable cache key based on text and parameters.
-
+
Note: Sensitive parameters like 'api_key' are excluded from the cache key
to prevent security risks and ensure cache sharing where appropriate.
"""
- # Filter out sensitive keys
- sensitive_keys = {'api_key', 'token', 'password', 'secret', 'auth', 'authorization'}
- filtered_params = {k: v for k, v in params.items() if k.lower() not in sensitive_keys}
+ # Filter out sensitive keys. Exact-match (not substring) so legitimate
+ # params such as "max_tokens" are never dropped.
+ sensitive_keys = {
+ "api_key",
+ "apikey",
+ "api_secret",
+ "token",
+ "access_token",
+ "refresh_token",
+ "session_token",
+ "bearer_token",
+ "password",
+ "secret",
+ "client_secret",
+ "private_key",
+ "auth",
+ "authorization",
+ "credential",
+ "credentials",
+ }
+ filtered_params = {
+ k: v for k, v in params.items() if k.lower() not in sensitive_keys
+ }
# Create a stable string representation of params
# Sort keys to ensure consistent ordering
param_str = json.dumps(filtered_params, sort_keys=True, default=str)
-
+
# Combine text and params
content = f"{text}|{param_str}"
-
+
# Return hash (SHA-256 for better security than MD5)
- return hashlib.sha256(content.encode('utf-8')).hexdigest()
+ return hashlib.sha256(content.encode("utf-8")).hexdigest()
def get(self, namespace: str, text: str, **params) -> Optional[Any]:
"""
Retrieve item from cache.
-
+
Args:
namespace: Cache namespace ("entities", "relations", "triplets")
text: Input text used for extraction
**params: Extraction parameters used
-
+
Returns:
Cached result or None if not found/expired
"""
if not self.enabled:
return None
- if namespace not in self._caches:
+ if namespace not in NAMESPACES:
return None
key = self._generate_key(text, **params)
-
- with self._locks[namespace]:
- cache = self._caches[namespace]
- if key in cache:
- item = cache[key]
-
- # Check expiration
- if item.is_expired():
- del cache[key]
- return None
-
- # Move to end (mark as recently used)
- cache.move_to_end(key)
- return item.value
-
- return None
+ return self._backend.get(namespace, key)
def set(self, namespace: str, text: str, value: Any, **params) -> None:
"""
Add item to cache.
-
+
Args:
namespace: Cache namespace
text: Input text
@@ -139,46 +555,22 @@ class ExtractionCache:
if not self.enabled:
return
- if namespace not in self._caches:
+ if namespace not in NAMESPACES:
self.logger.warning(f"Unknown cache namespace: {namespace}")
return
key = self._generate_key(text, **params)
- item = CacheItem(value, self.ttl)
-
- with self._locks[namespace]:
- cache = self._caches[namespace]
-
- # If key exists, update and move to end
- if key in cache:
- cache.move_to_end(key)
-
- cache[key] = item
-
- # Evict if full
- if len(cache) > self.max_size:
- cache.popitem(last=False) # Remove first (least recently used)
+ self._backend.set(namespace, key, value, self.ttl)
def clear(self, namespace: Optional[str] = None):
"""Clear cache(s)."""
- if namespace:
- if namespace in self._caches:
- with self._locks[namespace]:
- self._caches[namespace].clear()
- else:
- for ns in self._caches:
- with self._locks[ns]:
- self._caches[ns].clear()
+ self._backend.clear(namespace)
def get_stats(self) -> Dict[str, Dict[str, int]]:
- """Get cache statistics."""
- stats = {}
- for ns, cache in self._caches.items():
- stats[ns] = {
- "size": len(cache),
- "max_size": self.max_size
- }
- return stats
+ """Get cache statistics (delegated to the backend so the reported
+ ``max_size`` reflects the limit actually in effect)."""
+ return {ns: dict(self._backend.stats(ns)) for ns in NAMESPACES}
+
# Global cache instance
extraction_cache = ExtractionCache()
diff --git a/semantica/semantic_extract/config.py b/semantica/semantic_extract/config.py
index fb3763e8..f9f33303 100644
--- a/semantica/semantic_extract/config.py
+++ b/semantica/semantic_extract/config.py
@@ -58,6 +58,12 @@ class Config:
self._configs["optimization"] = {
"enable_cache": True,
"cache_size": 1000,
+ "cache_ttl": 3600,
+ # Cache backend: "memory" (default, in-process) or "sqlite"
+ # (persistent, survives restarts). cache_path is the sqlite file
+ # location; None lets the caller pick a default.
+ "cache_backend": "memory",
+ "cache_path": None,
"max_workers": 8,
"enable_batching": True,
"batch_size": 10,
@@ -106,10 +112,42 @@ class Config:
self._configs[provider] = {}
self._configs[provider]["api_key"] = api_key
+ # Optimization / cache settings from the environment. This lets users
+ # opt into persistent caching without a config file — e.g.
+ # SEMANTICA_CACHE_BACKEND=sqlite SEMANTICA_CACHE_PATH=~/.cache/sem.db
+ opt = self._configs.setdefault("optimization", {})
+ env_str_opts = {
+ "SEMANTICA_CACHE_BACKEND": "cache_backend",
+ "SEMANTICA_CACHE_PATH": "cache_path",
+ }
+ for env_key, opt_key in env_str_opts.items():
+ val = os.getenv(env_key)
+ if val:
+ opt[opt_key] = val
+ env_int_opts = {
+ "SEMANTICA_CACHE_TTL": "cache_ttl",
+ "SEMANTICA_CACHE_SIZE": "cache_size",
+ }
+ for env_key, opt_key in env_int_opts.items():
+ val = os.getenv(env_key)
+ if val is not None:
+ try:
+ opt[opt_key] = int(val)
+ except ValueError:
+ self.logger.warning(f"Ignoring non-integer {env_key}={val!r}")
+ enable = os.getenv("SEMANTICA_CACHE_ENABLED")
+ if enable is not None:
+ opt["enable_cache"] = enable.strip().lower() in {"1", "true", "yes", "on"}
+
def set_provider(self, name: str, **config):
"""Set provider config programmatically."""
self._configs[name] = config
+ def set_optimization(self, **options):
+ """Set optimization/cache config programmatically (e.g. cache_backend,
+ cache_path, cache_ttl, cache_size, enable_cache)."""
+ self._configs.setdefault("optimization", {}).update(options)
+
def get_provider_config(self, name: str) -> Dict:
"""Get provider configuration."""
if name in self._configs:
diff --git a/semantica/semantic_extract/methods.py b/semantica/semantic_extract/methods.py
index b9b1d4c3..fa5e5c85 100644
--- a/semantica/semantic_extract/methods.py
+++ b/semantica/semantic_extract/methods.py
@@ -106,6 +106,8 @@ Author: Semantica Contributors
License: MIT
"""
+import hashlib
+import os
import re
import difflib
import threading
@@ -118,7 +120,7 @@ from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
from .providers import HuggingFaceModelLoader, create_provider
from .registry import method_registry
-from .cache import ExtractionCache
+from .cache import ExtractionCache, InMemoryBackend
from .config import config
from .types import Entity, Relation, Triplet
@@ -136,11 +138,112 @@ except ImportError:
logger = get_logger("methods")
# Initialize global result cache
-_result_cache = ExtractionCache(
- ttl=config.get("cache_ttl", 3600)
-)
-if not config.get("cache_enabled", True):
- _result_cache.enabled = False
+def _default_cache_path() -> str:
+ """A per-user, private default location for the persistent cache.
+
+ Uses ``$XDG_CACHE_HOME`` (or ``~/.cache``) so the file is not a predictable,
+ world-writable temp path — important because the sqlite backend can
+ deserialize its rows with pickle. The directory is created ``0o700``; if that
+ is not possible, a private ``tempfile`` directory is used as a fallback.
+ """
+ import tempfile
+
+ base = os.environ.get("XDG_CACHE_HOME") or os.path.join(
+ os.path.expanduser("~"), ".cache"
+ )
+ directory = os.path.join(base, "semantica")
+ try:
+ os.makedirs(directory, mode=0o700, exist_ok=True)
+ if hasattr(os, "chmod"):
+ os.chmod(directory, 0o700)
+ except OSError:
+ directory = tempfile.mkdtemp(prefix="semantica-cache-") # 0o700, unique
+ return os.path.join(directory, "extract_cache.sqlite3")
+
+
+def _build_cache_backend():
+ """Select a cache backend from config.
+
+ Returns ``None`` (ExtractionCache's default in-memory backend) unless a
+ persistent backend is requested. Any failure to construct the persistent
+ backend degrades gracefully to the in-memory default so extraction never
+ breaks because of a cache misconfiguration.
+ """
+ backend_kind = str(config.get("cache_backend", "memory") or "memory").lower()
+ if backend_kind == "sqlite":
+ from .cache import SqliteCacheBackend
+
+ path = config.get("cache_path") or _default_cache_path()
+ try:
+ return SqliteCacheBackend(path, max_size=config.get("cache_size", 1000))
+ except Exception as exc: # defensive fallback (e.g. unsafe/owned path)
+ logger.warning(
+ f"Failed to init sqlite cache backend at {path!r} "
+ f"({exc}); falling back to in-memory cache."
+ )
+ return None
+
+
+def _make_result_cache() -> ExtractionCache:
+ cache = ExtractionCache(
+ max_size=config.get("cache_size", 1000),
+ ttl=config.get("cache_ttl", 3600),
+ backend=_build_cache_backend(),
+ )
+ if not config.get("cache_enabled", True):
+ cache.enabled = False
+ return cache
+
+
+_result_cache = _make_result_cache()
+
+
+def configure_cache(
+ backend: Optional[str] = None,
+ path: Optional[str] = None,
+ ttl: Optional[int] = None,
+ enabled: Optional[bool] = None,
+) -> ExtractionCache:
+ """Reconfigure the global extraction cache at runtime and rebuild it.
+
+ The module-level cache is created at import time, so this is the supported
+ way to switch to persistent (sqlite) storage programmatically after import
+ (the ``SEMANTICA_CACHE_*`` environment variables work at import time too).
+
+ Args:
+ backend: ``"memory"`` or ``"sqlite"``.
+ path: sqlite database path (defaults to a per-user private location).
+ ttl: time-to-live in seconds.
+ enabled: enable/disable caching.
+
+ Returns:
+ The rebuilt global :class:`ExtractionCache`.
+ """
+ updates: Dict[str, Any] = {}
+ if backend is not None:
+ updates["cache_backend"] = backend
+ if path is not None:
+ updates["cache_path"] = path
+ if ttl is not None:
+ updates["cache_ttl"] = ttl
+ if enabled is not None:
+ updates["enable_cache"] = enabled
+ if updates:
+ config.set_optimization(**updates)
+
+ # Mutate the existing cache in place (rather than rebinding the module
+ # global) so any code holding a reference to it stays valid.
+ new_max = config.get("cache_size", 1000)
+ new_backend = _build_cache_backend()
+ if hasattr(_result_cache._backend, "close"):
+ _result_cache._backend.close()
+ _result_cache.max_size = new_max
+ _result_cache.ttl = config.get("cache_ttl", 3600)
+ _result_cache._backend = (
+ new_backend if new_backend is not None else InMemoryBackend(max_size=new_max)
+ )
+ _result_cache.enabled = bool(config.get("cache_enabled", True))
+ return _result_cache
# Generation kwargs that affect provider output and must therefore be part of
# the cache key. This is the union of every generation-affecting parameter
@@ -183,6 +286,18 @@ def _generation_cache_params(kwargs: dict) -> dict:
if k in _GENERATION_CACHE_KEYS and v is not None
}
+
+def _stable_fingerprint(items) -> str:
+ """Deterministic SHA-256 fingerprint over a collection of strings.
+
+ Used to fold entity/relation inputs into a cache key. Unlike the built-in
+ ``hash()``, this is stable across interpreter processes (``hash()`` is
+ randomized per-process via ``PYTHONHASHSEED``), which the persistent cache
+ backend relies on so identical inputs map to the same key after a restart.
+ """
+ joined = "\x00".join(sorted(items))
+ return hashlib.sha256(joined.encode("utf-8")).hexdigest()
+
# Try to import spaCy
from ..utils.helpers import safe_import
@@ -1817,8 +1932,12 @@ def extract_relations_llm(
"max_retries": max_retries,
"relation_types": kwargs.get("relation_types"),
"extract_temporal_bounds": extract_temporal_bounds,
- # Include entities hash/str in cache key implicitly via **cache_params
- "entities_hash": hash(tuple(sorted([e.text for e in entities]))) if entities else 0,
+ # Deterministic fingerprint over the entity inputs (text + label) that
+ # affect the prompt. Stable across processes so the persistent cache
+ # rehits after a restart (unlike the process-randomized built-in hash()).
+ "entities_hash": _stable_fingerprint(
+ [f"{e.text}\x01{e.label}" for e in entities]
+ ) if entities else "",
**_generation_cache_params(kwargs),
}
cached_result = _result_cache.get("relations", text, **cache_params)
@@ -2486,9 +2605,15 @@ def extract_triplets_llm(
"structured_output_mode": structured_output_mode,
"max_retries": max_retries,
"triplet_types": kwargs.get("triplet_types"),
- # Include entities/relations hash in cache key implicitly via **cache_params
- "entities_hash": hash(tuple(sorted([e.text for e in entities]))) if entities else 0,
- "relations_hash": hash(tuple(sorted([str(r) for r in relations]))) if relations else 0,
+ # Deterministic fingerprints over the entity/relation inputs that affect
+ # the prompt. Stable across processes so the persistent cache rehits
+ # after a restart (unlike the process-randomized built-in hash()).
+ "entities_hash": _stable_fingerprint(
+ [f"{e.text}\x01{e.label}" for e in entities]
+ ) if entities else "",
+ "relations_hash": _stable_fingerprint(
+ [str(r) for r in relations]
+ ) if relations else "",
**_generation_cache_params(kwargs),
}
cached_result = _result_cache.get("triplets", text, **cache_params)
diff --git a/semantica/semantic_extract/semantic_extract_usage.md b/semantica/semantic_extract/semantic_extract_usage.md
index f5c2c9a0..97126f9a 100644
--- a/semantica/semantic_extract/semantic_extract_usage.md
+++ b/semantica/semantic_extract/semantic_extract_usage.md
@@ -594,3 +594,55 @@ method_registry.register("entity", "custom_method", custom_entity_extraction)
# Use custom method
from semantica.semantic_extract import NERExtractor
extractor = NERExtractor(method="custom_method")
+```
+
+## Result Caching
+
+Extraction results are cached (keyed by a stable SHA-256 over the input text and
+generation parameters — `provider`, `model`, `temperature`, `seed`, ... — with
+sensitive keys such as `api_key` excluded). The default cache is in-memory
+(LRU + TTL) and lives only for the process.
+
+### Persistent cache (sqlite)
+
+An optional persistent backend keeps results across process restarts, so a
+fresh process (CI job, notebook kernel, batch worker) reuses prior results
+instead of re-paying the LLM. Select it via config, environment, or at runtime:
+
+```python
+from semantica.semantic_extract import configure_cache
+
+# Runtime (rebuilds the global cache in place):
+configure_cache(backend="sqlite", path="~/.cache/semantica/extract.sqlite3")
+```
+
+```bash
+# Environment (applied at import time):
+export SEMANTICA_CACHE_BACKEND=sqlite
+export SEMANTICA_CACHE_PATH=~/.cache/semantica/extract.sqlite3
+# also: SEMANTICA_CACHE_TTL, SEMANTICA_CACHE_SIZE, SEMANTICA_CACHE_ENABLED
+```
+
+If the persistent backend cannot be constructed, it degrades gracefully to the
+in-memory default — extraction never fails because of a cache misconfiguration.
+
+### ⚠️ Persistent cache trust model
+
+The sqlite file is deserialized back into the process on every read, and the
+**default serializer is `pickle`**. Treat the cache file as executable input:
+
+- `cache_path` / `SEMANTICA_CACHE_PATH` **must** point at a **trusted,
+ user-private** location. The default (`$XDG_CACHE_HOME` or `~/.cache`,
+ created `0o700`; the db file created `0o600`) satisfies this.
+- The cache may contain **sensitive extraction results** in the clear.
+- The default `pickle` serializer must only be used with a trusted cache file.
+ For untrusted or shared locations, pass a non-executable serializer
+ (e.g. `json`) when constructing `SqliteCacheBackend`.
+
+For safety the backend creates the file atomically with `0o600` and refuses an
+existing path that is a symlink, a non-regular file, or owned by another user
+(falling back to the in-memory cache).
+
+Cache-key invalidation is automatic: because provider/model/generation
+parameters are part of the key, changing any of them (e.g. a model upgrade)
+produces a different key and old entries are bypassed rather than served stale.
diff --git a/tests/semantic_extract/test_cache_backends.py b/tests/semantic_extract/test_cache_backends.py
new file mode 100644
index 00000000..aa635640
--- /dev/null
+++ b/tests/semantic_extract/test_cache_backends.py
@@ -0,0 +1,367 @@
+"""Unit tests for ExtractionCache pluggable backends (in-memory + sqlite)."""
+
+from __future__ import annotations
+
+import json
+import multiprocessing as mp
+import os
+import sqlite3
+import stat
+import subprocess
+import sys
+
+import pytest
+
+from semantica.semantic_extract import (
+ CacheBackend,
+ ExtractionCache,
+ InMemoryBackend,
+ SqliteCacheBackend,
+)
+
+
+# ---------------------------------------------------------------------------
+# ExtractionCache + default in-memory backend (behavior parity)
+# ---------------------------------------------------------------------------
+
+
+def test_default_backend_is_in_memory():
+ cache = ExtractionCache()
+ assert isinstance(cache._backend, InMemoryBackend)
+ assert isinstance(cache._backend, CacheBackend)
+
+
+def test_get_set_roundtrip():
+ cache = ExtractionCache()
+ assert cache.get("entities", "hello") is None
+ cache.set("entities", "hello", [{"name": "X"}])
+ assert cache.get("entities", "hello") == [{"name": "X"}]
+
+
+def test_sensitive_params_excluded_from_key():
+ cache = ExtractionCache()
+ cache.set("entities", "hi", ["v"], provider="openai", api_key="secret-1")
+ # A different api_key/token must not change the key -> still a hit.
+ assert cache.get("entities", "hi", provider="openai", api_key="secret-2") == ["v"]
+
+
+def test_different_params_produce_different_keys():
+ cache = ExtractionCache()
+ cache.set("entities", "hi", ["a"], model="gpt-4")
+ assert cache.get("entities", "hi", model="gpt-4o") is None
+ assert cache.get("entities", "hi", model="gpt-4") == ["a"]
+
+
+def test_disabled_cache_is_noop():
+ cache = ExtractionCache()
+ cache.enabled = False
+ cache.set("entities", "hi", ["a"])
+ assert cache.get("entities", "hi") is None
+
+
+def test_unknown_namespace_is_ignored():
+ cache = ExtractionCache()
+ cache.set("nope", "hi", ["a"]) # warns, stores nothing
+ assert cache.get("nope", "hi") is None
+
+
+def test_get_stats_structure():
+ cache = ExtractionCache(max_size=42)
+ cache.set("entities", "a", [1])
+ stats = cache.get_stats()
+ assert set(stats) == {"entities", "relations", "triplets"}
+ assert stats["entities"] == {"size": 1, "max_size": 42}
+
+
+def test_get_stats_reports_injected_backend_limit(tmp_path):
+ # get_stats() must reflect the injected backend's real eviction limit,
+ # not the facade default.
+ cache = ExtractionCache(backend=SqliteCacheBackend(_db(tmp_path), max_size=2))
+ assert cache.get_stats()["entities"]["max_size"] == 2
+ cache._backend.close()
+
+
+def test_backward_compat_caches_proxy():
+ # Existing callers/tests reach into ._caches / ._locks; keep that working.
+ cache = ExtractionCache()
+ cache.set("entities", "a", [1])
+ assert "entities" in cache._caches
+ assert len(cache._caches["entities"]) == 1
+ assert "entities" in cache._locks
+ cache._caches["entities"].clear()
+ assert cache.get_stats()["entities"]["size"] == 0
+
+
+# ---------------------------------------------------------------------------
+# InMemoryBackend directly
+# ---------------------------------------------------------------------------
+
+
+def test_in_memory_ttl_expiry():
+ backend = InMemoryBackend()
+ backend.set("entities", "k", "v", ttl=-1) # already expired
+ assert backend.get("entities", "k") is None
+
+
+def test_in_memory_ttl_none_never_expires():
+ backend = InMemoryBackend()
+ backend.set("entities", "k", "v", ttl=None)
+ assert backend.get("entities", "k") == "v"
+
+
+def test_in_memory_lru_eviction():
+ backend = InMemoryBackend(max_size=2)
+ backend.set("entities", "a", 1, ttl=None)
+ backend.set("entities", "b", 2, ttl=None)
+ backend.get("entities", "a") # 'a' now most-recently-used
+ backend.set("entities", "c", 3, ttl=None) # evicts LRU -> 'b'
+ assert backend.size("entities") == 2
+ assert backend.get("entities", "b") is None
+ assert backend.get("entities", "a") == 1
+ assert backend.get("entities", "c") == 3
+
+
+def test_zero_ttl_expires_immediately_on_both_backends(tmp_path):
+ mem = InMemoryBackend()
+ mem.set("entities", "k", "v", ttl=0)
+ assert mem.get("entities", "k") is None
+
+ sq = SqliteCacheBackend(_db(tmp_path))
+ sq.set("entities", "k", "v", ttl=0)
+ assert sq.get("entities", "k") is None
+ sq.close()
+
+
+# ---------------------------------------------------------------------------
+# SqliteCacheBackend
+# ---------------------------------------------------------------------------
+
+
+def _db(tmp_path):
+ return str(tmp_path / "cache.sqlite3")
+
+
+def test_sqlite_roundtrip_and_size(tmp_path):
+ backend = SqliteCacheBackend(_db(tmp_path))
+ assert backend.get("entities", "k") is None
+ backend.set("entities", "k", {"x": 1}, ttl=None)
+ assert backend.get("entities", "k") == {"x": 1}
+ assert backend.size("entities") == 1
+ backend.close()
+
+
+def test_sqlite_persists_across_restart(tmp_path):
+ path = _db(tmp_path)
+ cache = ExtractionCache(backend=SqliteCacheBackend(path))
+ cache.set("entities", "hello", [{"name": "X"}], provider="openai", model="gpt-4")
+ cache._backend.close()
+
+ # Simulate a fresh process: new backend + new cache over the same file.
+ reopened = ExtractionCache(backend=SqliteCacheBackend(path))
+ assert reopened.get("entities", "hello", provider="openai", model="gpt-4") == [
+ {"name": "X"}
+ ]
+ reopened._backend.close()
+
+
+def test_sqlite_ttl_expiry(tmp_path):
+ backend = SqliteCacheBackend(_db(tmp_path))
+ backend.set("entities", "k", "v", ttl=-1) # already expired
+ assert backend.get("entities", "k") is None
+ backend.close()
+
+
+def test_sqlite_lru_eviction_bounds_size(tmp_path):
+ backend = SqliteCacheBackend(_db(tmp_path), max_size=2)
+ for k in ("a", "b", "c"):
+ backend.set("entities", k, k, ttl=None)
+ assert backend.size("entities") == 2
+ backend.close()
+
+
+def test_sqlite_clear(tmp_path):
+ backend = SqliteCacheBackend(_db(tmp_path))
+ backend.set("entities", "a", 1, ttl=None)
+ backend.set("relations", "b", 2, ttl=None)
+ backend.clear("entities")
+ assert backend.size("entities") == 0
+ assert backend.size("relations") == 1
+ backend.clear()
+ assert backend.size("relations") == 0
+ backend.close()
+
+
+def test_sqlite_custom_json_serializer(tmp_path):
+ backend = SqliteCacheBackend(
+ _db(tmp_path),
+ serializer=lambda v: json.dumps(v).encode("utf-8"),
+ deserializer=lambda b: json.loads(b.decode("utf-8")),
+ )
+ backend.set("entities", "k", {"a": [1, 2, 3]}, ttl=None)
+ assert backend.get("entities", "k") == {"a": [1, 2, 3]}
+ backend.close()
+
+
+def test_sqlite_unserializable_value_is_skipped(tmp_path):
+ backend = SqliteCacheBackend(
+ _db(tmp_path),
+ serializer=lambda v: json.dumps(v).encode("utf-8"), # can't encode a set
+ deserializer=lambda b: json.loads(b.decode("utf-8")),
+ )
+ backend.set("entities", "k", {1, 2, 3}, ttl=None) # not JSON-serializable
+ assert backend.get("entities", "k") is None # skipped, no crash
+ backend.close()
+
+
+def test_sqlite_corrupt_payload_is_dropped(tmp_path):
+ # serializer writes bytes the default pickle deserializer can't load.
+ backend = SqliteCacheBackend(_db(tmp_path), serializer=lambda v: b"not-a-pickle")
+ backend.set("entities", "k", "v", ttl=None)
+ assert backend.get("entities", "k") is None
+ # A corrupt row must be evicted, not left occupying capacity.
+ assert backend.size("entities") == 0
+ backend.close()
+
+
+def test_sqlite_home_relative_path_is_expanded(tmp_path, monkeypatch):
+ monkeypatch.setenv("HOME", str(tmp_path))
+ monkeypatch.setenv("USERPROFILE", str(tmp_path)) # Windows expanduser
+ backend = SqliteCacheBackend("~/nested/cache.sqlite3")
+ backend.set("entities", "k", "v", ttl=None)
+ assert backend.get("entities", "k") == "v"
+ assert (tmp_path / "nested" / "cache.sqlite3").exists()
+ backend.close()
+
+
+def test_stable_fingerprint_is_process_independent():
+ # Finding: relation/triplet cache keys must not depend on the built-in
+ # process-randomized hash(). The deterministic fingerprint must be equal
+ # across interpreters started with different PYTHONHASHSEED values.
+ code = (
+ "from semantica.semantic_extract.methods import _stable_fingerprint;"
+ "print(_stable_fingerprint(['b', 'a', 'c']))"
+ )
+ out0 = subprocess.check_output(
+ [sys.executable, "-c", code], env={**os.environ, "PYTHONHASHSEED": "0"}
+ ).strip()
+ out1 = subprocess.check_output(
+ [sys.executable, "-c", code], env={**os.environ, "PYTHONHASHSEED": "1"}
+ ).strip()
+ assert out0 and out0 == out1
+
+
+def _mp_writer(db_path, namespace, count):
+ """Top-level worker (picklable) for the concurrency test."""
+ from semantica.semantic_extract import SqliteCacheBackend as _Backend
+
+ backend = _Backend(db_path)
+ for i in range(count):
+ backend.set(namespace, f"k{i}", {"v": i}, ttl=None)
+ backend.close()
+
+
+def test_sqlite_concurrent_workers_share_one_file(tmp_path):
+ # Overlapping workers write distinct namespaces to the same file; the DB
+ # must stay valid and every entry reusable afterwards.
+ path = _db(tmp_path)
+ ctx = mp.get_context("spawn")
+ procs = [
+ ctx.Process(target=_mp_writer, args=(path, ns, 20))
+ for ns in ("entities", "relations", "triplets")
+ ]
+ for p in procs:
+ p.start()
+ for p in procs:
+ p.join(60)
+ for p in procs:
+ assert p.exitcode == 0
+
+ reader = SqliteCacheBackend(path)
+ for ns in ("entities", "relations", "triplets"):
+ assert reader.size(ns) == 20
+ assert reader.get(ns, "k5") == {"v": 5}
+ reader.close()
+
+
+# ---------------------------------------------------------------------------
+# Security & reliability of the persistent file
+# ---------------------------------------------------------------------------
+
+
[email protected](not hasattr(os, "getuid"), reason="POSIX permissions only")
+def test_sqlite_new_db_created_private(tmp_path):
+ path = _db(tmp_path)
+ backend = SqliteCacheBackend(path)
+ mode = stat.S_IMODE(os.stat(path).st_mode)
+ assert mode == 0o600 # created 0o600 atomically, no widen-then-chmod window
+ backend.close()
+
+
[email protected](not hasattr(os, "symlink"), reason="symlinks unavailable")
+def test_sqlite_rejects_symlinked_db(tmp_path):
+ target = tmp_path / "target.sqlite3"
+ target.write_bytes(b"")
+ link = tmp_path / "link.sqlite3"
+ os.symlink(target, link)
+ with pytest.raises(PermissionError):
+ SqliteCacheBackend(str(link))
+
+
+def test_sqlite_write_error_is_contained(tmp_path):
+ # A sqlite failure on write must be rolled back and swallowed (skipped
+ # update), never propagated into the caller.
+ backend = SqliteCacheBackend(_db(tmp_path))
+
+ class _BoomConn:
+ def __init__(self):
+ self.rolled_back = False
+
+ def execute(self, *a, **k):
+ raise sqlite3.OperationalError("boom")
+
+ def commit(self):
+ raise AssertionError("commit must not run after a failed op")
+
+ def rollback(self):
+ self.rolled_back = True
+
+ boom = _BoomConn()
+ backend._conn = boom
+ backend.set("entities", "k", "v", ttl=None) # must not raise
+ assert boom.rolled_back # rollback happened (inside the lock scope)
+ assert backend.get("entities", "k") is None # read failure -> miss
+
+
+def test_sqlite_read_error_is_contained(tmp_path):
+ backend = SqliteCacheBackend(_db(tmp_path))
+
+ class _BoomConn:
+ def execute(self, *a, **k):
+ raise sqlite3.OperationalError("boom")
+
+ def rollback(self):
+ pass
+
+ backend._conn = _BoomConn()
+ assert backend.get("entities", "k") is None # miss, no crash
+
+
+# ---------------------------------------------------------------------------
+# Config-driven selection (methods.configure_cache)
+# ---------------------------------------------------------------------------
+
+
+def test_configure_cache_switches_backend(tmp_path):
+ from semantica.semantic_extract import methods
+
+ try:
+ cache = methods.configure_cache(backend="sqlite", path=_db(tmp_path))
+ assert isinstance(cache._backend, SqliteCacheBackend)
+ assert methods._result_cache is cache
+ cache.set("entities", "hi", ["v"])
+ assert cache.get("entities", "hi") == ["v"]
+ finally:
+ # Restore the default in-memory global cache for other tests
+ # (configure_cache closes the sqlite backend and mutates in place).
+ methods.config.set_optimization(cache_path=None)
+ methods.configure_cache(backend="memory")