Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(memory): depth 9.2 — semantic recall with provenance and conflict review (#1309)
Beenu Aroraa83174411570
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 84f8a2a4a..31cae7ca8 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -701,6 +701,18 @@ jobs:run: python3 scripts/check_cloud_native_collection.py --self-test- name: Cloud-native collection gaterun: python3 scripts/check_cloud_native_collection.py+ # Depth 9.2. The probe set behind the recall measurement has to stay+ # usable: every probe must name an item that is in the corpus, the+ # probes must stay low-overlap with their answers (a probe written in+ # the document's own words hands the lexical baseline a score it could+ # not earn on a real question), and the baseline must still be a real+ # retriever rather than a stub. The scorer itself runs too: with no+ # embedding backend on a runner it prints "not measured" and exits 0,+ # which is the honest row and also proves the refusal path works.+ - name: Memory recall scorer — self-test+ run: python3 scripts/score_memory_recall.py --self-test+ - name: Memory recall scorer+ run: python3 scripts/score_memory_recall.py# One layer in from the name: the profile gate asks whether a connector# reaches a normalization path, this asks which OCSF class it reaches# when it gets there. The class uid divided by a thousand is thediff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.mdindex a75a88df4..52b94ffa9 100644--- a/DEPTH_PROGRESS.md+++ b/DEPTH_PROGRESS.md@@ -269,6 +269,50 @@ the one that finishes it. `test_a_batch_delivered_out_of_arrival_order_still_firis the regression test for that, and reverting the event-time read makes itand two others fail.+### D11 -- Semantic recall measured *worse* than the full-text baseline on every embedding backend reachable from here++Parity 6.5 asks for the accuracy delta. It is measured, by+`scripts/score_memory_recall.py` over 12 hand-authored probes against a+12-item corpus, and the delta is negative:++| metric | lexical | semantic (`qwen2.5:0.5b`) | delta |+|---|---|---|---|+| recall@1 | 0.667 | 0.333 | **-0.333** |+| recall@3 | 0.750 | 0.417 | **-0.333** |+| recall@5 | 0.750 | 0.667 | -0.083 |+| MRR | 0.708 | 0.465 | **-0.243** |++The honest reading is not "semantic recall does not work". It is that+**no embedding model is reachable from this project without a funded key**.+The two models served locally are *generative* models used as embedders,+which is a known-poor substitute; `nomic-embed-text` was attempted and the+pull did not complete in ten minutes. Hosted embedding rows are blocked by+the same thing as D3 (maintainer item M2).++Three things follow, and all three are in the tree rather than in this note:++1. **The default did not move.** `POST /kb/query` stays full-text; semantic+ recall is `recall_memory: false` and strictly additive, so a caller that+ does not ask for it gets byte-identical behaviour. Putting a measurably+ worse retriever in front of the ranker would have made analyst search+ worse on exactly the deployments that cannot pay for an embedding model.+ The field's docstring carries the number and says to move the default when+ the measurement does.+2. **The measurement found a real defect in the first implementation.** The+ 3,072-dimension model initially scored 0.167 at recall@1 — and that was+ measuring `_fit()` *truncating* the vector to the collection's 1,536, not+ measuring the model. Padding a short vector is harmless under cosine;+ truncating a long one discards most of what the model said. Truncation is+ now refused with a message naming both dimensions, which is a+ configuration an operator can fix rather than a silently worse retriever+ they cannot see.+3. **What is still not measured, stated plainly:** the downstream effect of+ memory recall on *verdict* quality, which is what the Phase 1 scorecard+ grades. That needs an agent run per corpus item against a model worth+ grading, and no such model is funded or locally servable here. The+ retrieval delta above is the part that can be measured honestly today,+ and it is the part that is actually about what 9.2 builds.+### D8 -- No upstream Sigma correlation rule exists in this tree to import`detections/sigma-imports/` holds 3,132 rule documents and **not one carries@@ -418,7 +462,30 @@ invented fields on the first run of the gate.## Phase 9: Analyst surfaces- [ ] **9.1** MCP over streamable HTTP (parity 6.6)-- [ ] **9.2** Semantic memory (parity 6.5)+- [x] **9.2** Semantic memory (parity 6.5). All three bullets of parity 6.5+ closed, and the third one — "replay shows the accuracy delta" — produced a+ result that changed the design rather than confirming it. `aisoc_memory_items`+ (migration `105`) is the store of record for past cases, runbooks and+ institutional memory; Qdrant is an index over it, reached through its REST+ API so this service gains no dependency and its lockfile does not move.+ Every item carries an author, a source, a scope, an expiry and a state, and+ the author is the **authenticated principal**, never the body — provenance a+ caller can set is not provenance. Two claims about one subject suspend+ **both** sides and recall returns neither until an analyst settles it;+ preferring the newer one silently restores a bad entry the next time the job+ that wrote it runs, which is the case the feature exists for. Recall is+ tenant-scoped in the Qdrant filter itself and `search()` **raises** rather+ than offering an unscoped branch. With no embedding backend recall is 503+ with the reason, never an empty list. `kind='case'` is written automatically+ by the case-closure path, so the half of the "exists unwired" family that+ reachability analysis cannot see — a table every surface reads and nothing+ writes — does not apply. Measured by+ `scripts/score_memory_recall.py` (12 hand-authored low-overlap probes,+ mean lexical overlap 0.166, `is_synthetic: true`); its self-test is wired+ into `ci.yml :: python-lint`. 17 new tests, api suite +17. Live end to end+ against real Postgres 16, real Qdrant v1.7.0 and a real embedding backend,+ recorded in the PR. **The measurement is published as it came out, and it is+ unflattering:** see D11.- [ ] **9.3** Case depth (parity 5.6 remainder)- [ ] **9.4** War rooms- [ ] **9.5** Hunt to report@@ -433,6 +500,7 @@ invented fields on the first run of the gate.| Date | Item | What happened ||---|---|---|+| 2026-10-11 | 9.2 | Reproduced: `knowledge_base.py` searched with `to_tsvector`/`plainto_tsquery` and nothing else; `grep -rn qdrant services/api/app/` found only the migration runner and the tenant-deletion sweep, no recall path; and no memory row anywhere carried an author, a source, a scope, an expiry or a state. Built the store of record, the Qdrant index over it, the conflict state machine, the case-closure writer and the recall scorer. Two negative controls recorded in the PR: restoring last-write-wins on a contradiction fails three tests, and restoring vector truncation fails one. The measurement is negative and is published as such — see D11, which also records that it caught the truncation defect. || 2026-10-07 | 0.1 | This file created at base commit `1b8bc2d4`. || 2026-10-07 | 0.2 | Every figure re-derived. Two matched exactly (detections, unreachable families); executor arms measured 74 against a captured 73; the cloud/identity/SaaS/code figure measured 395 against a captured 461 on a grouping the plan does not pin, recorded above. Two measurement caveats found: executable and quarantined overlap by 1,724 rules, and the "69% Windows" figure does not reproduce from the index. || 2026-10-09 | 4.3 | Reproduced: `check_retention_window.py` on the unmodified tree reported six problems — the lake deleting at 90 days against a 3,650-day cap, no `MODIFY TTL` migration, tiering deleting at 90, and the purge worker calling neither `alerts_under_legal_hold` nor `may_purge` and reporting nothing withheld. Fixed all but the cold-query bullet. Four negative controls recorded in the PR, including the positive control: a purge stuck closed withholds every tenant and fails four of the eight hold tests, which is the only thing that separates a working control from commendable caution. One pre-existing test (`test_storage_tiering.py`) pinned the 90-day tiering DELETE and now reads the shared constant; `test_retention_worker.py`'s session double could not answer the new holds query, so the worker failed closed and five of its cases failed — the double was taught the query rather than the fail-closed behaviour relaxed. |diff --git a/docs/openapi.yaml b/docs/openapi.yamlindex 36056e700..0ccdaa92f 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -7035,6 +7035,246 @@ paths:application/json:schema:$ref: '#/components/schemas/HTTPValidationError'+ /api/v1/memory/items:+ post:+ tags:+ - semantic-memory+ summary: Ingest Item+ operationId: ingest_item_api_v1_memory_items_post+ security:+ - HTTPBearer: []+ requestBody:+ required: true+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/MemoryItemIn'+ responses:+ '201':+ description: Successful Response+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/MemoryIngestResult'+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ get:+ tags:+ - semantic-memory+ summary: List Items+ operationId: list_items_api_v1_memory_items_get+ security:+ - HTTPBearer: []+ parameters:+ - name: kind+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ title: Kind+ - name: state+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ title: State+ - name: limit+ in: query+ required: false+ schema:+ type: integer+ maximum: 500+ minimum: 1+ default: 100+ title: Limit+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ type: array+ items:+ $ref: '#/components/schemas/MemoryItemOut'+ title: Response List Items Api V1 Memory Items Get+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ /api/v1/memory/items/{item_id}:+ delete:+ tags:+ - semantic-memory+ summary: Retire Item+ description: Retire, never hard-delete. 'What did the agent know' must stay+ answerable.+ operationId: retire_item_api_v1_memory_items__item_id__delete+ security:+ - HTTPBearer: []+ parameters:+ - name: item_id+ in: path+ required: true+ schema:+ type: string+ format: uuid+ title: Item Id+ - name: reason+ in: query+ required: false+ schema:+ type: string+ default: ''+ title: Reason+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/MemoryItemOut'+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ /api/v1/memory/recall:+ post:+ tags:+ - semantic-memory+ summary: Recall+ description: 'Vector recall over this tenant''s memory. Every hit carries its+ provenance.+++ ``knowledge_base:read`` is the right permission and it already exists:++ ``POST /kb/query`` uses it for the full-text half of the same corpus,++ and every role that triages holds it. Reusing it rather than minting a++ ``memory:read`` keeps one answer to "may this principal search what the++ organisation knows" — two permissions over one corpus is two answers++ that can disagree, and the disagreement surfaces on the day somebody++ narrows a role.'+ operationId: recall_api_v1_memory_recall_post+ requestBody:+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/MemoryRecallIn'+ required: true+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ items:+ $ref: '#/components/schemas/MemoryRecallHit'+ type: array+ title: Response Recall Api V1 Memory Recall Post+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ security:+ - HTTPBearer: []+ /api/v1/memory/conflicts:+ get:+ tags:+ - semantic-memory+ summary: List Conflicts+ operationId: list_conflicts_api_v1_memory_conflicts_get+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ items:+ $ref: '#/components/schemas/MemoryConflictOut'+ type: array+ title: Response List Conflicts Api V1 Memory Conflicts Get+ security:+ - HTTPBearer: []+ /api/v1/memory/conflicts/{item_id}/resolve:+ post:+ tags:+ - semantic-memory+ summary: Resolve Conflict+ description: Settle a conflict by naming which side survives. The other is retired.+ operationId: resolve_conflict_api_v1_memory_conflicts__item_id__resolve_post+ security:+ - HTTPBearer: []+ parameters:+ - name: item_id+ in: path+ required: true+ schema:+ type: string+ format: uuid+ title: Item Id+ requestBody:+ required: true+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/MemoryConflictResolution'+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/MemoryConflictOut'+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ /api/v1/memory/status:+ get:+ tags:+ - semantic-memory+ summary: Recall Status+ description: 'Can this deployment recall at all, and over how much.+++ Every count here is computed from rows this request counted, not read++ from a cached figure, and an unknown stays ``null`` rather than becoming++ a confident zero.'+ operationId: recall_status_api_v1_memory_status_get+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ additionalProperties: true+ type: object+ title: Response Recall Status Api V1 Memory Status Get+ security:+ - HTTPBearer: []/api/v1/tenant-skills:get:tags:@@ -29750,6 +29990,269 @@ components:required:- user_idtitle: MemberUpsert+ MemoryConflictOut:+ properties:+ left:+ $ref: '#/components/schemas/MemoryItemOut'+ right:+ anyOf:+ - $ref: '#/components/schemas/MemoryItemOut'+ - type: 'null'+ type: object+ required:+ - left+ - right+ title: MemoryConflictOut+ MemoryConflictResolution:+ properties:+ keep:+ type: string+ title: Keep+ note:+ type: string+ maxLength: 500+ title: Note+ default: ''+ type: object+ required:+ - keep+ title: MemoryConflictResolution+ MemoryIngestResult:+ properties:+ item:+ $ref: '#/components/schemas/MemoryItemOut'+ outcome:+ type: string+ title: Outcome+ indexed:+ type: boolean+ title: Indexed+ index_error:+ anyOf:+ - type: string+ - type: 'null'+ title: Index Error+ type: object+ required:+ - item+ - outcome+ - indexed+ title: MemoryIngestResult+ MemoryItemIn:+ properties:+ kind:+ type: string+ title: Kind+ description: case | runbook | memory+ subject:+ type: string+ maxLength: 400+ minLength: 1+ title: Subject+ content:+ type: string+ maxLength: 200000+ minLength: 1+ title: Content+ title:+ type: string+ maxLength: 400+ title: Title+ default: ''+ source_ref:+ type: string+ maxLength: 400+ title: Source Ref+ default: ''+ source:+ type: string+ maxLength: 60+ title: Source+ default: analyst+ scope:+ type: string+ title: Scope+ default: tenant+ expires_at:+ anyOf:+ - type: string+ format: date-time+ - type: 'null'+ title: Expires At+ type: object+ required:+ - kind+ - subject+ - content+ title: MemoryItemIn+ MemoryItemOut:+ properties:+ id:+ type: string+ title: Id+ kind:+ type: string+ title: Kind+ subject:+ type: string+ title: Subject+ title:+ type: string+ title: Title+ content:+ type: string+ title: Content+ source_ref:+ type: string+ title: Source Ref+ author:+ type: string+ title: Author+ source:+ type: string+ title: Source+ scope:+ type: string+ title: Scope+ state:+ type: string+ title: State+ state_reason:+ type: string+ title: State Reason+ conflicts_with:+ anyOf:+ - type: string+ - type: 'null'+ title: Conflicts With+ expires_at:+ anyOf:+ - type: string+ format: date-time+ - type: 'null'+ title: Expires At+ embedding_model:+ anyOf:+ - type: string+ - type: 'null'+ title: Embedding Model+ embedded_at:+ anyOf:+ - type: string+ format: date-time+ - type: 'null'+ title: Embedded At+ created_at:+ type: string+ format: date-time+ title: Created At+ updated_at:+ type: string+ format: date-time+ title: Updated At+ type: object+ required:+ - id+ - kind+ - subject+ - title+ - content+ - source_ref+ - author+ - source+ - scope+ - state+ - state_reason+ - conflicts_with+ - expires_at+ - embedding_model+ - embedded_at+ - created_at+ - updated_at+ title: MemoryItemOut+ MemoryRecallChunk:+ properties:+ item_id:+ type: string+ format: uuid+ title: Item Id+ kind:+ type: string+ title: Kind+ subject:+ type: string+ title: Subject+ title:+ type: string+ title: Title+ content:+ type: string+ title: Content+ score:+ type: number+ title: Score+ author:+ type: string+ title: Author+ source:+ type: string+ title: Source+ type: object+ required:+ - item_id+ - kind+ - subject+ - title+ - content+ - score+ - author+ - source+ title: MemoryRecallChunk+ description: 'One semantically recalled memory item, with the provenance to+ weigh it.+++ A separate type from ``KBChunk`` on purpose. These come from a different++ store, by a different retrieval, with an author and a source attached,++ and flattening them into the same list would let a reader take a++ similarity hit for a full-text match.'+ MemoryRecallHit:+ properties:+ item:+ $ref: '#/components/schemas/MemoryItemOut'+ score:+ type: number+ title: Score+ type: object+ required:+ - item+ - score+ title: MemoryRecallHit+ MemoryRecallIn:+ properties:+ query:+ type: string+ maxLength: 4000+ minLength: 1+ title: Query+ kinds:+ items:+ type: string+ type: array+ title: Kinds+ limit:+ type: integer+ maximum: 50.0+ minimum: 1.0+ title: Limit+ default: 5+ type: object+ required:+ - query+ title: MemoryRecallInMetricsOut:properties:tenant_id:@@ -32457,6 +32960,12 @@ components:title: Synthesisedescription: Use LLM to synthesise an answer from retrieved chunks.default: true+ recall_memory:+ type: boolean+ title: Recall Memory+ description: Append semantically recalled memory items to the result. Never+ reorders the full-text chunks. Off by default; see scripts/score_memory_recall.py.+ default: falsetype: objectrequired:- question@@ -32481,6 +32990,16 @@ components:type: stringtype: arraytitle: Sources+ memory:+ items:+ $ref: '#/components/schemas/MemoryRecallChunk'+ type: array+ title: Memory+ memory_unavailable:+ anyOf:+ - type: string+ - type: 'null'+ title: Memory Unavailabletype: objectrequired:- questiondiff --git a/scripts/score_memory_recall.py b/scripts/score_memory_recall.pynew file mode 100644index 000000000..d43e9e7df--- /dev/null+++ b/scripts/score_memory_recall.py@@ -0,0 +1,344 @@+#!/usr/bin/env python3+"""Measure what semantic recall adds over the full-text search it sits beside.++Depth 9.2 (parity 6.5), the half that stops the rest being an assertion. The+parity item asks for "the accuracy delta"; this is the delta that can be+measured without a funded model, and the one that is actually about the thing+9.2 builds.++What is measured+----------------+Retrieval, not verdicts. Each probe is an analyst's question and the one+memory item that answers it. Two retrievers run over the same corpus:++``lexical``+ The baseline the knowledge base has always used: Postgres+ ``to_tsvector``/``plainto_tsquery`` ranking, reproduced here as an+ English-stemmed bag-of-words with ``ts_rank``-shaped scoring so the+ comparison needs no database. The reproduction is approximate and the+ report says so; what it has to be is *the same kind of retriever*, and a+ stemmed term-overlap ranker is exactly that.++``semantic``+ Cosine over embeddings from whichever backend this deployment has+ configured, through ``app.services.semantic_memory``.++Reported: recall@1, recall@3, recall@5 and mean reciprocal rank for each, the+delta, and the measured lexical overlap between each probe and its answer.++Why the overlap figure is published beside the scores+-----------------------------------------------------+A probe set written in the documents' own words would hand the lexical+baseline a score it could not earn on a real analyst's question, and the+delta would look small for a reason that has nothing to do with the+retriever. Publishing the overlap lets a reader check the probes really are+phrased differently rather than take the design note on trust.++Honest unavailability+---------------------+With no embedding backend the semantic rows read **"not measured"**, never+``0.000``. A reader who sees a zero concludes the retriever was graded and+failed. This is the same rule ``run_model_matrix.py`` follows for an+unfunded model row.++Usage:+ python3 scripts/score_memory_recall.py+ python3 scripts/score_memory_recall.py --json+ python3 scripts/score_memory_recall.py --self-test++Exit codes: 0 scored (or an announced skip), 1 the probe set is unusable.+"""++from __future__ import annotations++import argparse+import asyncio+import json+import math+import re+import sys+from collections import Counter+from pathlib import Path+from typing import Any++sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import SELF_TEST_FLAG, repo_root, self_test_main # noqa: E402++PROBES_REL = Path("services/api/tests/eval_data/memory_recall_probes_v1.json")++#: Rank positions reported. @1 is what a prompt with room for one runbook+#: gets; @5 is what a context bundle gets.+AT_K = (1, 3, 5)++#: Snowball-ish suffix stripping. Not a real stemmer: a dependency for a+#: scoring script is a dependency in every image that installs this service's+#: requirements, and the baseline only has to be the *same kind* of retriever+#: as Postgres's, not a bit-exact reimplementation of it. The report says so.+_SUFFIXES = ("ingly", "edly", "ing", "ed", "ly", "es", "s")++#: Postgres's english stop-word list, trimmed to the words these probes use.+#: Present so the baseline is not scored down for matching "the".+_STOPWORDS = frozenset(+ """a an and are as at be been before but by do does for from had has have how i if in into is it+ its of on onto or should so than that the their them then there these they this to was were what+ when where which who will with you your do not no any every""".split()+)++_WORD_RE = re.compile(r"[a-z0-9][a-z0-9._/-]*")+++def stem(word: str) -> str:+ for suffix in _SUFFIXES:+ if len(word) > len(suffix) + 2 and word.endswith(suffix):+ return word[: -len(suffix)]+ return word+++def tokens(text: str) -> list[str]:+ return [stem(w) for w in _WORD_RE.findall(text.lower()) if w not in _STOPWORDS and len(w) > 1]+++def lexical_rank(query: str, corpus: list[dict[str, Any]]) -> list[str]:+ """Item ids ordered the way a ``ts_rank`` search would order them.++ ``ts_rank`` weights by term frequency and normalises by document length;+ this does the same with a cosine over term-frequency vectors, which has+ the property that matters for the comparison: a document only ranks if it+ *contains the words*, which is precisely what the baseline cannot do and+ the measurement is about.+ """+ q = Counter(tokens(query))+ if not q:+ return []+ scored: list[tuple[float, str]] = []+ for item in corpus:+ d = Counter(tokens(f"{item.get('title', '')} {item['content']}"))+ shared = set(q) & set(d)+ if not shared:+ continue+ dot = sum(q[t] * d[t] for t in shared)+ norm = math.sqrt(sum(v * v for v in q.values())) * math.sqrt(sum(v * v for v in d.values()))+ scored.append((dot / norm if norm else 0.0, item["id"]))+ scored.sort(key=lambda pair: (-pair[0], pair[1]))+ return [item_id for _, item_id in scored]+++def overlap(query: str, item: dict[str, Any]) -> float:+ """Share of the probe's content words that appear in the answering item."""+ q = set(tokens(query))+ if not q:+ return 0.0+ d = set(tokens(f"{item.get('title', '')} {item['content']}"))+ return len(q & d) / len(q)+++def _cosine(a: list[float], b: list[float]) -> float:+ dot = sum(x * y for x, y in zip(a, b, strict=False))+ na = math.sqrt(sum(x * x for x in a))+ nb = math.sqrt(sum(y * y for y in b))+ return dot / (na * nb) if na and nb else 0.0+++async def semantic_rank_all(probes: list[dict[str, Any]], corpus: list[dict[str, Any]]) -> dict[str, list[str]] | str:+ """Ranked item ids per probe, or a sentence saying why it could not run.++ Returns a string rather than raising so the caller can print+ "not measured" with the reason, which is the honest row.+ """+ sys.path.insert(0, str(repo_root() / "services" / "api"))+ try:+ from app.services.semantic_memory import EmbeddingsUnavailable, embed # noqa: PLC0415+ except Exception as exc: # noqa: BLE001+ return f"the API service could not be imported ({type(exc).__name__}: {exc})"++ doc_texts = [f"{item.get('title', '')}\n\n{item['content']}".strip() for item in corpus]+ try:+ doc_vectors, model = await embed(doc_texts)+ query_vectors, _ = await embed([p["query"] for p in probes])+ except EmbeddingsUnavailable as exc:+ return str(exc)+ except Exception as exc: # noqa: BLE001+ return f"the embedding backend failed ({type(exc).__name__}: {exc})"++ out: dict[str, list[str]] = {}+ for index, probe in enumerate(probes):+ scored = [(_cosine(query_vectors[index], doc_vectors[j]), corpus[j]["id"]) for j in range(len(corpus))]+ scored.sort(key=lambda pair: (-pair[0], pair[1]))+ out[probe["id"]] = [item_id for _, item_id in scored]+ out["__model__"] = [model]+ return out+++def score(ranked: dict[str, list[str]], probes: list[dict[str, Any]]) -> dict[str, Any]:+ hits = dict.fromkeys(AT_K, 0)+ reciprocal = 0.0+ misses: list[str] = []+ for probe in probes:+ order = ranked.get(probe["id"], [])+ try:+ position = order.index(probe["expected"]) + 1+ except ValueError:+ misses.append(probe["id"])+ continue+ reciprocal += 1.0 / position+ for k in AT_K:+ if position <= k:+ hits[k] += 1+ total = len(probes)+ return {+ **{f"recall_at_{k}": hits[k] / total for k in AT_K},+ "mrr": reciprocal / total,+ "probes": total,+ "not_retrieved_at_all": misses,+ }+++def load_probes(root: Path) -> dict[str, Any]:+ return json.loads((root / PROBES_REL).read_text(encoding="utf-8"))+++def _fmt(value: float) -> str:+ return f"{value:.3f}"+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--json", action="store_true")+ args = parser.parse_args(argv)++ root = repo_root()+ try:+ data = load_probes(root)+ except (OSError, json.JSONDecodeError) as exc:+ # An unreadable probe set is a refusal with a sentence, not a+ # traceback: a scorer that cannot find its corpus must not look like+ # a crashed scorer that might have measured something.+ print(f"score_memory_recall: cannot read {PROBES_REL} ({exc}) — nothing to score", file=sys.stderr)+ return 1+ corpus = data["corpus"]+ probes = data["probes"]++ by_id = {item["id"]: item for item in corpus}+ unknown = [p["id"] for p in probes if p["expected"] not in by_id]+ if unknown:+ print(f"probe set names items that are not in the corpus: {unknown}", file=sys.stderr)+ return 1+ if len(probes) < 10:+ # Small-corpus refusal, the same reasoning `aisoc_benchmark.replay`+ # applies to headline accuracy: a rate over a handful of probes is+ # misleading in the direction that sells.+ print(f"only {len(probes)} probe(s); refusing to publish a rate", file=sys.stderr)+ return 1++ lexical = {p["id"]: lexical_rank(p["query"], corpus) for p in probes}+ lexical_score = score(lexical, probes)++ semantic_raw = asyncio.run(semantic_rank_all(probes, corpus))+ semantic_score: dict[str, Any] | None = None+ semantic_reason: str | None = None+ semantic_model: str | None = None+ if isinstance(semantic_raw, str):+ semantic_reason = semantic_raw+ else:+ semantic_model = (semantic_raw.pop("__model__", [""]) or [""])[0]+ semantic_score = score(semantic_raw, probes)++ overlaps = [overlap(p["query"], by_id[p["expected"]]) for p in probes]+ mean_overlap = sum(overlaps) / len(overlaps)++ result = {+ "probe_set": str(PROBES_REL),+ "is_synthetic": bool(data.get("is_synthetic")),+ "corpus_items": len(corpus),+ "probes": len(probes),+ "mean_lexical_overlap": mean_overlap,+ "lexical": lexical_score,+ "semantic": semantic_score,+ "semantic_model": semantic_model,+ "semantic_not_measured_because": semantic_reason,+ }++ if args.json:+ print(json.dumps(result, indent=2))+ return 0++ print("Memory recall — lexical baseline against semantic recall")+ print(f" probe set {PROBES_REL} (synthetic: {result['is_synthetic']})")+ print(f" corpus {len(corpus)} items, {len(probes)} probes")+ print(f" mean lexical overlap {_fmt(mean_overlap)} (share of a probe's words present in its answer)")+ print()+ header = f" {'metric':<18}{'lexical':>12}{'semantic':>14}{'delta':>12}"+ print(header)+ print(" " + "-" * (len(header) - 2))+ for key in [f"recall_at_{k}" for k in AT_K] + ["mrr"]:+ lex = lexical_score[key]+ if semantic_score is None:+ print(f" {key:<18}{_fmt(lex):>12}{'not measured':>14}{'—':>12}")+ else:+ sem = semantic_score[key]+ print(f" {key:<18}{_fmt(lex):>12}{_fmt(sem):>14}{sem - lex:>+12.3f}")+ print()+ if semantic_score is None:+ # Never `0.000`. A reader who sees a zero concludes the retriever was+ # graded and failed.+ print(f" semantic recall NOT MEASURED: {semantic_reason}")+ else:+ print(f" semantic backend model: {semantic_model}")+ if lexical_score["not_retrieved_at_all"]:+ print(f" lexical returned nothing at all for: {', '.join(lexical_score['not_retrieved_at_all'])}")+ if semantic_score["not_retrieved_at_all"]:+ print(f" semantic returned nothing at all for: {', '.join(semantic_score['not_retrieved_at_all'])}")+ print()+ print(" The lexical row reproduces Postgres ts_rank's *kind* of retrieval (stemmed term")+ print(" overlap, length-normalised), not its exact scoring. The comparison is between a")+ print(" retriever that needs the words and one that does not.")+ return 0+++# ---------------------------------------------------------------------------+# Self-test+# ---------------------------------------------------------------------------+++def _rule_self_test() -> list[tuple[str, bool]]:+ root = repo_root()+ data = load_probes(root)+ corpus = data["corpus"]+ probes = data["probes"]+ by_id = {item["id"]: item for item in corpus}++ perfect = {p["id"]: [p["expected"]] for p in probes}+ perfect_score = score(perfect, probes)+ empty_score = score({p["id"]: [] for p in probes}, probes)+ # Rank the right answer second everywhere.+ second = {p["id"]: ["m-backup-runbook" if p["expected"] != "m-backup-runbook" else "m-travel", p["expected"]] for p in probes}+ second_score = score(second, probes)++ return [+ ("every probe names an item in the corpus", all(p["expected"] in by_id for p in probes)),+ ("a perfect ranking scores 1.0 at every k", all(perfect_score[f"recall_at_{k}"] == 1.0 for k in AT_K)),+ (+ "an empty ranking scores 0.0, and names every miss",+ empty_score["mrr"] == 0.0 and len(empty_score["not_retrieved_at_all"]) == len(probes),+ ),+ (+ "the right answer in second place scores 0 at k=1 and 1 at k=3",+ second_score["recall_at_1"] == 0.0 and second_score["recall_at_3"] == 1.0,+ ),+ ("MRR for a second-place answer is 0.5", abs(second_score["mrr"] - 0.5) < 1e-9),+ (+ "the probes really are low-overlap with their answers (mean < 0.40)",+ (sum(overlap(p["query"], by_id[p["expected"]]) for p in probes) / len(probes)) < 0.40,+ ),+ (+ "the lexical baseline is a real retriever, not a stub: it finds at least one answer",+ any(p["expected"] in lexical_rank(p["query"], corpus)[:3] for p in probes),+ ),+ ]+++if __name__ == "__main__":+ if SELF_TEST_FLAG in sys.argv[1:]:+ sys.exit(self_test_main(Path(__file__).name, extra=_rule_self_test()))+ sys.exit(main())diff --git a/services/api/app/api/v1/endpoints/cases.py b/services/api/app/api/v1/endpoints/cases.pyindex fb57e53c4..0ed024a78 100644--- a/services/api/app/api/v1/endpoints/cases.py+++ b/services/api/app/api/v1/endpoints/cases.py@@ -62,6 +62,7 @@ from app.services.case_postmortem import build_case_postmortemfrom app.services.case_postmortem_html import render_case_postmortem_htmlfrom app.services.case_summary import build_case_summaryfrom app.services.case_summary_html import render_case_summary_html+from app.services.semantic_memory import remember_closed_caselogger = logging.getLogger(__name__)@@ -832,6 +833,31 @@ async def update_case(response = _row_to_case(row)+ # Depth 9.2. A closed case is the first of the three things parity 6.5+ # names as semantic memory, and nothing else writes `kind='case'` -- so+ # without this call that half of the store would be permanently empty on+ # every deployment where nobody ingests by hand, which is the half of the+ # "exists unwired" family that reachability analysis cannot see.+ #+ # After the commit and fail-soft on purpose: closing a case must not 503+ # because a vector store is down. The row is written in its own+ # transaction, and `/memory/status` reports how many rows are waiting to+ # be embedded so a degraded store is visible rather than silent.+ if body.status == "closed" and body.status != existing.status:+ try:+ await remember_closed_case(+ db,+ tenant_id=user.tenant_id,+ case_id=cid,+ title=response.title,+ summary=response.description or "",+ author=getattr(user, "email", "") or "",+ )+ await db.commit()+ except Exception: # noqa: BLE001 - see above; never fails the close+ await db.rollback()+ logger.warning("semantic memory: could not record closed case %s", cid, exc_info=True)+# Workstream 8 — if the operator changed status, project the# transition onto every connector this case is already linked to.# ``fanout_status_change`` no-ops when the case has nodiff --git a/services/api/app/api/v1/endpoints/knowledge_base.py b/services/api/app/api/v1/endpoints/knowledge_base.pyindex 9ef069f40..966d7a3f4 100644--- a/services/api/app/api/v1/endpoints/knowledge_base.py+++ b/services/api/app/api/v1/endpoints/knowledge_base.py@@ -48,15 +48,18 @@ from typing import Annotated, Any, Literalfrom fastapi import APIRouter, Depends, Header, HTTPException, Query, statusfrom pydantic import BaseModel, Field-from sqlalchemy import text+from sqlalchemy import select, textfrom app.api.v1.deps import AuthUser, DBSession, require_permissionfrom app.api.v1.endpoints.alert_writeback import service_token_validfrom app.core.airgap import AirgapViolation, enforce_airgap_for_urlfrom app.db.rls import set_rls_context+from app.models.semantic_memory import MemoryItemfrom app.services.kb_chunking import chunk_textfrom app.services.llm_safety import LLMContractViolation, safe_chat_completions_requestfrom app.services.model_aliases import chat_completions_url, resolve_api_key, resolve_model_alias+from app.services.semantic_memory import EmbeddingsUnavailable, VectorStoreUnavailable+from app.services.semantic_memory import search as memory_searchlogger = logging.getLogger(__name__)@@ -95,6 +98,26 @@ class QueryRequest(BaseModel):doc_kinds: list[DocKind] = Field(default_factory=list)top_k: int = Field(5, ge=1, le=20)synthesise: bool = Field(True, description="Use LLM to synthesise an answer from retrieved chunks.")+ #: Depth 9.2. Also recall semantically from `aisoc_memory_items` -- past+ #: cases, runbooks and institutional memory -- and append the hits.+ #:+ #: Off by default, and that default is a measured decision rather than+ #: caution. `scripts/score_memory_recall.py` grades both retrievers over+ #: the same labelled probe set, and on the only embedding backend this+ #: project can reach without a funded key (a small generative model+ #: served by Ollama) semantic recall scored **worse** than the lexical+ #: baseline: recall@1 0.333 against 0.667, MRR 0.465 against 0.708.+ #: Putting it in front of the ranker by default would have made analyst+ #: search worse on exactly the deployments that cannot pay for a hosted+ #: embedding model. Re-run that script after configuring a real+ #: embedding model and move this default when the number says to.+ recall_memory: bool = Field(+ False,+ description=(+ "Append semantically recalled memory items to the result. Never reorders the "+ "full-text chunks. Off by default; see scripts/score_memory_recall.py."+ ),+ )class KBChunk(BaseModel):@@ -105,11 +128,37 @@ class KBChunk(BaseModel):score: float | None = None+class MemoryRecallChunk(BaseModel):+ """One semantically recalled memory item, with the provenance to weigh it.++ A separate type from ``KBChunk`` on purpose. These come from a different+ store, by a different retrieval, with an author and a source attached,+ and flattening them into the same list would let a reader take a+ similarity hit for a full-text match.+ """++ item_id: uuid.UUID+ kind: str+ subject: str+ title: str+ content: str+ score: float+ author: str+ source: str++class QueryResponse(BaseModel):question: strchunks: list[KBChunk]answer: str | None # None when synthesise=False or no LLM keysources: list[str]+ #: Populated only when the caller asked for `recall_memory`.+ memory: list[MemoryRecallChunk] = Field(default_factory=list)+ #: Why `memory` is empty when it was asked for. ``None`` means it was+ #: either not asked for or genuinely returned nothing -- which the caller+ #: can tell apart because it knows what it asked. An empty list under a+ #: silent failure is the honest-state defect this field exists to avoid.+ memory_unavailable: str | None = None# ────────────────────────────────────────────────────────────────────────────@@ -329,7 +378,58 @@ async def query_kb(answer = Nonesources = list(dict.fromkeys(c.title for c in chunks))- return QueryResponse(question=body.question, chunks=chunks, answer=answer, sources=sources)++ # Depth 9.2. Strictly additive: the full-text chunks above are not+ # reordered, filtered or replaced, so a caller that does not ask for this+ # gets byte-identical behaviour.+ memory: list[MemoryRecallChunk] = []+ memory_unavailable: str | None = None+ if body.recall_memory:+ try:+ hits = await memory_search(body.question, tenant_id=user.tenant_id, limit=body.top_k)+ if hits:+ scores = dict(hits)+ rows = (+ (+ await db.execute(+ select(MemoryItem).where(+ MemoryItem.tenant_id == user.tenant_id,+ MemoryItem.id.in_(list(scores)),+ MemoryItem.state == "in_use",+ )+ )+ )+ .scalars()+ .all()+ )+ rows.sort(key=lambda r: scores.get(r.id, 0.0), reverse=True)+ memory = [+ MemoryRecallChunk(+ item_id=r.id,+ kind=r.kind,+ subject=r.subject,+ title=r.title,+ content=r.content,+ score=scores.get(r.id, 0.0),+ author=r.author,+ source=r.source,+ )+ for r in rows+ ]+ except (EmbeddingsUnavailable, VectorStoreUnavailable) as exc:+ # Say so. An empty list where the caller asked for recall reads as+ # "nothing matched", which is a different fact from "I could not+ # look", and the full-text half of the answer is still good.+ memory_unavailable = str(exc)++ return QueryResponse(+ question=body.question,+ chunks=chunks,+ answer=answer,+ sources=sources,+ memory=memory,+ memory_unavailable=memory_unavailable,+ )# ────────────────────────────────────────────────────────────────────────────diff --git a/services/api/app/api/v1/endpoints/semantic_memory.py b/services/api/app/api/v1/endpoints/semantic_memory.pynew file mode 100644index 000000000..0a1bd2e4f--- /dev/null+++ b/services/api/app/api/v1/endpoints/semantic_memory.py@@ -0,0 +1,588 @@+"""Semantic memory: recall with provenance, and the conflicts before it.++Depth plan 9.2 (parity 6.5).++Routes:++* ``POST /memory/items`` ingest or correct one item+* ``GET /memory/items`` list, filtered by kind and state+* ``POST /memory/recall`` vector recall, every hit with provenance+* ``GET /memory/conflicts`` what an analyst has to settle+* ``POST /memory/conflicts/{id}/resolve`` settle one+* ``DELETE /memory/items/{id}`` retire one (never a hard delete)+* ``GET /memory/status`` can this deployment recall at all++Why ingest can *create* a conflict rather than overwrite+---------------------------------------------------------+Two items with the same ``(kind, subject)`` and different content are two+claims about the same thing. The cheap behaviour is last-write-wins, and it+is wrong in the case the feature exists for: an analyst corrects an entry an+agent got wrong, and an hour later the job that produced the wrong one runs+again. Last-write-wins silently restores the error, and the agent then cites+it. So a contradiction suspends **both** sides and asks a human.++An ingest that carries the same content hash is not a contradiction -- it is+the same claim arriving twice -- and is idempotent.++Why retirement is not deletion+------------------------------+A memory item is something the agent was told, so "what did it know when it+decided that" has to stay answerable after somebody decides the item was+wrong. Retiring sets ``state='not_in_use'`` and drops the vector; the row+and its provenance stay.++Honest unavailability+---------------------+Recall needs an embedding backend and a reachable Qdrant. Where either is+missing the route answers **503 with the reason**, and ``/memory/status``+says which. It does not fall back to full-text and present the result as+semantic recall, and it does not return an empty list -- "we searched and+found nothing" and "we could not search" are different facts and the console+renders them differently.+"""++from __future__ import annotations++import uuid+from datetime import UTC, datetime+from typing import Annotated, Any++import structlog+from fastapi import APIRouter, Depends, HTTPException, Query, Request, status+from pydantic import BaseModel, Field+from sqlalchemy import select++from app.api.v1.deps import AuthUser, require_permission+from app.db.rls import TenantDBSession+from app.models.semantic_memory import MEMORY_KINDS, MemoryItem+from app.services.audit import emit_audit+from app.services.semantic_memory import (+ EmbeddingsUnavailable,+ VectorStoreUnavailable,+ content_hash,+ forget_item,+ index_item,+)+from app.services.semantic_memory import search as vector_search+from app.services.semantic_memory import status as memory_status++logger = structlog.get_logger()++router = APIRouter(prefix="/memory", tags=["semantic-memory"])+++# ---------------------------------------------------------------------------+# Wire models+# ---------------------------------------------------------------------------+++class MemoryItemIn(BaseModel):+ kind: str = Field(description="case | runbook | memory")+ subject: str = Field(min_length=1, max_length=400)+ content: str = Field(min_length=1, max_length=200_000)+ title: str = Field(default="", max_length=400)+ source_ref: str = Field(default="", max_length=400)+ #: Provenance the caller can state. ``author`` defaults to the+ #: authenticated principal below rather than being trusted from the body:+ #: an author a caller can set is not provenance.+ source: str = Field(default="analyst", max_length=60)+ scope: str = Field(default="tenant")+ expires_at: datetime | None = None+++class MemoryItemOut(BaseModel):+ id: str+ kind: str+ subject: str+ title: str+ content: str+ source_ref: str+ author: str+ source: str+ scope: str+ state: str+ state_reason: str+ conflicts_with: str | None+ expires_at: datetime | None+ embedding_model: str | None+ embedded_at: datetime | None+ created_at: datetime+ updated_at: datetime+++class MemoryIngestResult(BaseModel):+ item: MemoryItemOut+ #: ``created`` | ``unchanged`` | ``conflict``. Named rather than inferred+ #: from the state, because "I stored it" and "I suspended it and the one+ #: it disagrees with" need different words in a console.+ outcome: str+ indexed: bool+ #: Why the vector was not written, when it was not. Never silent.+ index_error: str | None = None+++class MemoryRecallIn(BaseModel):+ query: str = Field(min_length=1, max_length=4_000)+ kinds: list[str] = Field(default_factory=list)+ limit: int = Field(default=5, ge=1, le=50)+++class MemoryRecallHit(BaseModel):+ item: MemoryItemOut+ score: float+++class MemoryConflictOut(BaseModel):+ left: MemoryItemOut+ right: MemoryItemOut | None+++class MemoryConflictResolution(BaseModel):+ #: Which of the two to keep. The other is retired.+ keep: str+ note: str = Field(default="", max_length=500)+++def _out(row: MemoryItem) -> MemoryItemOut:+ return MemoryItemOut(+ id=str(row.id),+ kind=row.kind,+ subject=row.subject,+ title=row.title,+ content=row.content,+ source_ref=row.source_ref,+ author=row.author,+ source=row.source,+ scope=row.scope,+ state=row.state,+ state_reason=row.state_reason,+ conflicts_with=str(row.conflicts_with) if row.conflicts_with else None,+ expires_at=row.expires_at,+ embedding_model=row.embedding_model,+ embedded_at=row.embedded_at,+ created_at=row.created_at,+ updated_at=row.updated_at,+ )+++# ---------------------------------------------------------------------------+# Ingest+# ---------------------------------------------------------------------------++[email protected]("/items", response_model=MemoryIngestResult, status_code=status.HTTP_201_CREATED)+async def ingest_item(+ body: MemoryItemIn,+ request: Request,+ db: TenantDBSession,+ current_user: AuthUser,+ _: Annotated[None, Depends(require_permission("settings:write"))],+) -> MemoryIngestResult:+ if body.kind not in MEMORY_KINDS:+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail=f"kind must be one of {list(MEMORY_KINDS)}",+ )+ if body.scope not in ("tenant", "global"):+ raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="scope must be tenant or global")++ digest = content_hash(body.content)+ # The author is the authenticated principal, never the body. Provenance a+ # caller can set is not provenance.+ author = current_user.api_key_prefix or current_user.email++ existing = (+ await db.execute(+ select(MemoryItem).where(+ MemoryItem.tenant_id == current_user.tenant_id,+ MemoryItem.kind == body.kind,+ MemoryItem.subject == body.subject,+ MemoryItem.state == "in_use",+ )+ )+ ).scalar_one_or_none()++ if existing is not None and existing.content_hash == digest:+ # The same claim arriving twice. Idempotent, and deliberately does not+ # re-embed: that would spend an embedding call per re-ingest on a+ # vector that cannot change.+ return MemoryIngestResult(item=_out(existing), outcome="unchanged", indexed=True)++ row = MemoryItem(+ tenant_id=current_user.tenant_id,+ kind=body.kind,+ subject=body.subject,+ source_ref=body.source_ref,+ title=body.title or body.subject,+ content=body.content,+ content_hash=digest,+ author=author,+ source=body.source,+ scope=body.scope,+ expires_at=body.expires_at,+ state="in_use",+ )++ outcome = "created"+ if existing is not None:+ # A contradiction. Both sides are suspended; see the module header for+ # why the newer one does not simply win.+ outcome = "conflict"+ row.state = "conflict"+ row.state_reason = f"Contradicts {existing.id}, which makes a different claim about {body.subject!r}."+ existing.state = "conflict"+ existing.state_reason = row.state_reason.replace(str(existing.id), "the newly ingested item")+ existing.updated_at = datetime.now(UTC)++ db.add(row)+ await db.flush()+ if existing is not None:+ row.conflicts_with = existing.id+ existing.conflicts_with = row.id+ # The suspended item must stop being recalled immediately, so its+ # point goes now rather than on a sweep.+ await forget_item(existing.id)++ indexed = False+ index_error: str | None = None+ if row.state == "in_use":+ try:+ model = await index_item(+ tenant_id=current_user.tenant_id,+ item_id=row.id,+ kind=row.kind,+ subject=row.subject,+ title=row.title,+ content=row.content,+ state=row.state,+ )+ row.embedding_model = model+ row.embedded_at = datetime.now(UTC)+ indexed = True+ except (EmbeddingsUnavailable, VectorStoreUnavailable) as exc:+ # Stored, not indexed, and the response says so. Silently+ # storing an unindexed row produces a memory that exists and is+ # never recalled, which is worse than a visible failure.+ index_error = str(exc)+ logger.warning("semantic_memory.index_failed", item_id=str(row.id), error=str(exc))++ await emit_audit(+ db=db,+ tenant_id=current_user.tenant_id,+ actor_id=current_user.user_id,+ actor_email=current_user.email,+ api_key_prefix=getattr(current_user, "api_key_prefix", None),+ action="memory:ingest",+ resource="memory_item",+ resource_id=str(row.id),+ changes={"kind": row.kind, "subject": row.subject, "outcome": outcome, "indexed": indexed},+ request=request,+ )+ await db.commit()+ await db.refresh(row)+ return MemoryIngestResult(item=_out(row), outcome=outcome, indexed=indexed, index_error=index_error)++[email protected]("/items", response_model=list[MemoryItemOut])+async def list_items(+ db: TenantDBSession,+ current_user: AuthUser,+ _: Annotated[None, Depends(require_permission("settings:read"))],+ kind: str | None = Query(default=None),+ state: str | None = Query(default=None),+ limit: int = Query(default=100, ge=1, le=500),+) -> list[MemoryItemOut]:+ stmt = select(MemoryItem).where(MemoryItem.tenant_id == current_user.tenant_id).order_by(MemoryItem.updated_at.desc()).limit(limit)+ if kind:+ stmt = stmt.where(MemoryItem.kind == kind)+ if state:+ stmt = stmt.where(MemoryItem.state == state)+ return [_out(row) for row in (await db.execute(stmt)).scalars().all()]++[email protected]("/items/{item_id}", response_model=MemoryItemOut)+async def retire_item(+ item_id: uuid.UUID,+ request: Request,+ db: TenantDBSession,+ current_user: AuthUser,+ _: Annotated[None, Depends(require_permission("settings:write"))],+ reason: str = Query(default=""),+) -> MemoryItemOut:+ """Retire, never hard-delete. 'What did the agent know' must stay answerable."""+ row = (+ await db.execute(+ select(MemoryItem).where(+ MemoryItem.id == item_id,+ MemoryItem.tenant_id == current_user.tenant_id,+ )+ )+ ).scalar_one_or_none()+ if row is None:+ raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="No such memory item")+ row.state = "not_in_use"+ row.state_reason = reason or "retired"+ row.updated_at = datetime.now(UTC)+ await forget_item(row.id)+ await emit_audit(+ db=db,+ tenant_id=current_user.tenant_id,+ actor_id=current_user.user_id,+ actor_email=current_user.email,+ api_key_prefix=getattr(current_user, "api_key_prefix", None),+ action="memory:retire",+ resource="memory_item",+ resource_id=str(row.id),+ changes={"reason": row.state_reason},+ request=request,+ )+ await db.commit()+ await db.refresh(row)+ return _out(row)+++# ---------------------------------------------------------------------------+# Recall+# ---------------------------------------------------------------------------++[email protected]("/recall", response_model=list[MemoryRecallHit])+async def recall(+ body: MemoryRecallIn,+ db: TenantDBSession,+ current_user: AuthUser,+ _: Annotated[None, Depends(require_permission("knowledge_base:read"))],+) -> list[MemoryRecallHit]:+ """Vector recall over this tenant's memory. Every hit carries its provenance.++ ``knowledge_base:read`` is the right permission and it already exists:+ ``POST /kb/query`` uses it for the full-text half of the same corpus,+ and every role that triages holds it. Reusing it rather than minting a+ ``memory:read`` keeps one answer to "may this principal search what the+ organisation knows" — two permissions over one corpus is two answers+ that can disagree, and the disagreement surfaces on the day somebody+ narrows a role.+ """+ unknown = [k for k in body.kinds if k not in MEMORY_KINDS]+ if unknown:+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail=f"unknown kind(s): {unknown}. Expected a subset of {list(MEMORY_KINDS)}.",+ )+ try:+ hits = await vector_search(+ body.query,+ tenant_id=current_user.tenant_id,+ kinds=body.kinds or None,+ limit=body.limit,+ )+ except (EmbeddingsUnavailable, VectorStoreUnavailable) as exc:+ # 503 with the reason, never an empty list. "Found nothing" and+ # "could not search" must not render the same.+ raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail=str(exc)) from exc++ if not hits:+ return []++ scores = dict(hits)+ rows = (+ (+ await db.execute(+ select(MemoryItem).where(+ MemoryItem.tenant_id == current_user.tenant_id,+ MemoryItem.id.in_(list(scores)),+ MemoryItem.state == "in_use",+ )+ )+ )+ .scalars()+ .all()+ )+ now = datetime.now(UTC)++ # Expiry is applied at read, not by a sweep: a worker being down must not+ # resurrect an expired claim.+ #+ # `_aware` exists because the column is TIMESTAMPTZ on Postgres and+ # returns an aware value, while some drivers hand back a naive one — and+ # comparing the two raises `TypeError` inside the handler, which would+ # turn an expired item from "withheld" into a 500 on the whole recall.+ # Treating a naive timestamp as UTC is right here: every writer in this+ # service stores UTC.+ def _aware(value: datetime) -> datetime:+ return value if value.tzinfo is not None else value.replace(tzinfo=UTC)++ live = [r for r in rows if r.expires_at is None or _aware(r.expires_at) > now]+ live.sort(key=lambda r: scores.get(r.id, 0.0), reverse=True)+ return [MemoryRecallHit(item=_out(r), score=scores.get(r.id, 0.0)) for r in live]+++# ---------------------------------------------------------------------------+# Conflicts+# ---------------------------------------------------------------------------++[email protected]("/conflicts", response_model=list[MemoryConflictOut])+async def list_conflicts(+ db: TenantDBSession,+ current_user: AuthUser,+ _: Annotated[None, Depends(require_permission("settings:read"))],+) -> list[MemoryConflictOut]:+ rows = (+ (+ await db.execute(+ select(MemoryItem)+ .where(+ MemoryItem.tenant_id == current_user.tenant_id,+ MemoryItem.state == "conflict",+ )+ .order_by(MemoryItem.updated_at.desc())+ )+ )+ .scalars()+ .all()+ )+ by_id = {row.id: row for row in rows}+ seen: set[uuid.UUID] = set()+ out: list[MemoryConflictOut] = []+ for row in rows:+ if row.id in seen:+ continue+ seen.add(row.id)+ other = by_id.get(row.conflicts_with) if row.conflicts_with else None+ if other is not None:+ seen.add(other.id)+ out.append(MemoryConflictOut(left=_out(row), right=_out(other) if other else None))+ return out++[email protected]("/conflicts/{item_id}/resolve", response_model=MemoryConflictOut)+async def resolve_conflict(+ item_id: uuid.UUID,+ body: MemoryConflictResolution,+ request: Request,+ db: TenantDBSession,+ current_user: AuthUser,+ _: Annotated[None, Depends(require_permission("settings:write"))],+) -> MemoryConflictOut:+ """Settle a conflict by naming which side survives. The other is retired."""+ try:+ keep_id = uuid.UUID(body.keep)+ except ValueError as exc:+ raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="keep must be an item id") from exc++ row = (+ await db.execute(+ select(MemoryItem).where(+ MemoryItem.id == item_id,+ MemoryItem.tenant_id == current_user.tenant_id,+ )+ )+ ).scalar_one_or_none()+ if row is None or row.state != "conflict":+ raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="No such conflict")+ other = None+ if row.conflicts_with:+ other = (+ await db.execute(+ select(MemoryItem).where(+ MemoryItem.id == row.conflicts_with,+ MemoryItem.tenant_id == current_user.tenant_id,+ )+ )+ ).scalar_one_or_none()++ pair = [r for r in (row, other) if r is not None]+ if keep_id not in {r.id for r in pair}:+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail="keep must name one of the two items in this conflict",+ )++ actor = current_user.api_key_prefix or current_user.email+ for item in pair:+ item.conflicts_with = None+ item.updated_at = datetime.now(UTC)+ if item.id == keep_id:+ item.state = "in_use"+ item.state_reason = body.note or f"kept by {actor}"+ else:+ item.state = "not_in_use"+ item.state_reason = body.note or f"superseded by {keep_id}, decided by {actor}"+ await forget_item(item.id)++ kept = next(r for r in pair if r.id == keep_id)+ try:+ model = await index_item(+ tenant_id=current_user.tenant_id,+ item_id=kept.id,+ kind=kept.kind,+ subject=kept.subject,+ title=kept.title,+ content=kept.content,+ state="in_use",+ )+ kept.embedding_model = model+ kept.embedded_at = datetime.now(UTC)+ except (EmbeddingsUnavailable, VectorStoreUnavailable) as exc:+ logger.warning("semantic_memory.reindex_failed", item_id=str(kept.id), error=str(exc))++ await emit_audit(+ db=db,+ tenant_id=current_user.tenant_id,+ actor_id=current_user.user_id,+ actor_email=current_user.email,+ api_key_prefix=getattr(current_user, "api_key_prefix", None),+ action="memory:conflict_resolved",+ resource="memory_item",+ resource_id=str(keep_id),+ changes={"kept": str(keep_id), "retired": [str(r.id) for r in pair if r.id != keep_id]},+ request=request,+ )+ await db.commit()+ for item in pair:+ await db.refresh(item)+ return MemoryConflictOut(left=_out(row), right=_out(other) if other else None)+++# ---------------------------------------------------------------------------+# Status+# ---------------------------------------------------------------------------++[email protected]("/status")+async def recall_status(+ db: TenantDBSession,+ current_user: AuthUser,+) -> dict[str, Any]:+ """Can this deployment recall at all, and over how much.++ Every count here is computed from rows this request counted, not read+ from a cached figure, and an unknown stays ``null`` rather than becoming+ a confident zero.+ """+ out = await memory_status()+ rows = (+ (await db.execute(select(MemoryItem.state, MemoryItem.embedding_model).where(MemoryItem.tenant_id == current_user.tenant_id)))+ .tuples()+ .all()+ )+ by_state: dict[str, int] = {}+ unembedded = 0+ stale_model = 0+ configured = out.get("embedding_model")+ for state_value, model in rows:+ by_state[state_value] = by_state.get(state_value, 0) + 1+ if state_value != "in_use":+ continue+ if model is None:+ unembedded += 1+ elif configured and model != configured:+ # Embedded by a model nobody is running any more. Distinct from+ # never embedded, and the two call for different actions.+ stale_model += 1+ out["items_by_state"] = by_state+ out["in_use_never_embedded"] = unembedded+ out["in_use_embedded_by_another_model"] = stale_model+ out["recall_available"] = bool(out.get("embedding_backend")) and bool(out.get("vector_store_reachable"))+ return outdiff --git a/services/api/app/api/v1/router.py b/services/api/app/api/v1/router.pyindex 5cadb3d76..b1a0c26c3 100644--- a/services/api/app/api/v1/router.py+++ b/services/api/app/api/v1/router.py@@ -86,6 +86,7 @@ from app.api.v1.endpoints import (saved_hunts,saved_views,scim_tokens,+ semantic_memory,sla,sso_connections,stix_taxii,@@ -162,6 +163,12 @@ api_router.include_router(plugins.router)api_router.include_router(community.router)api_router.include_router(marketplace.router)api_router.include_router(mcp_servers.router)+# Depth 9.2 (parity 6.5): vector recall over past cases, runbooks and+# institutional memory, every hit carrying an author, a source, a scope,+# an expiry and a state. `knowledge_base` above is the full-text half and+# stays: the two answer differently-shaped questions and the recall score+# in `scripts/score_memory_recall.py` measures exactly that difference.+api_router.include_router(semantic_memory.router)# Gap-closure Phase 6.1 and 6.2: tenant-authored investigation skills:# authored in YAML, validated against the tools this tenant's agent can# actually call, backtested through the Phase 1 replay with and without thediff --git a/services/api/app/models/semantic_memory.py b/services/api/app/models/semantic_memory.pynew file mode 100644index 000000000..a435ec566--- /dev/null+++ b/services/api/app/models/semantic_memory.py@@ -0,0 +1,77 @@+"""ORM model for semantic memory items.++Depth plan 9.2 (parity 6.5); migration ``105_semantic_memory.sql`` carries+the reasoning for each column and for the conflict state machine.++The short version of why this table exists beside Qdrant: a vector payload+cannot hold a foreign key, a CHECK or a row-level policy, and all three are+load-bearing here. The vectors are derivable by re-embedding from these rows,+so this is the store of record and Qdrant is an index over it.+"""++from __future__ import annotations++import uuid+from datetime import UTC, datetime++from sqlalchemy import CheckConstraint, DateTime, ForeignKey, Text+from sqlalchemy.dialects.postgresql import UUID+from sqlalchemy.orm import Mapped, mapped_column++from app.db.database import Base++#: What a memory item can be about.+MEMORY_KINDS: tuple[str, ...] = ("case", "runbook", "memory")++#: Recall returns ``in_use`` only. ``conflict`` withholds *both* sides of a+#: disagreement rather than preferring the newer one, because resolving by+#: timestamp is wrong in exactly the case this feature exists for: an analyst+#: corrects a bad entry and a job re-ingests the bad one an hour later.+MEMORY_STATES: tuple[str, ...] = ("in_use", "conflict", "not_in_use")+++class MemoryItem(Base):+ """One attributable claim the agent may be told, and why it may be trusted."""++ __tablename__ = "aisoc_memory_items"+ __table_args__ = (+ CheckConstraint("kind IN ('case', 'runbook', 'memory')", name="aisoc_memory_items_kind_check"),+ CheckConstraint("scope IN ('tenant', 'global')", name="aisoc_memory_items_scope_check"),+ CheckConstraint(+ "state IN ('in_use', 'conflict', 'not_in_use')",+ name="aisoc_memory_items_state_check",+ ),+ )++ id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)+ tenant_id: Mapped[uuid.UUID] = mapped_column(+ UUID(as_uuid=True), ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True+ )+ kind: Mapped[str] = mapped_column(Text, nullable=False)+ #: What the item is about. Two items with the same subject make claims+ #: about the same thing, which is what makes a contradiction detectable.+ subject: Mapped[str] = mapped_column(Text, nullable=False)+ source_ref: Mapped[str] = mapped_column(Text, nullable=False, default="")+ title: Mapped[str] = mapped_column(Text, nullable=False, default="")+ content: Mapped[str] = mapped_column(Text, nullable=False)+ content_hash: Mapped[str] = mapped_column(Text, nullable=False)++ author: Mapped[str] = mapped_column(Text, nullable=False, default="")+ source: Mapped[str] = mapped_column(Text, nullable=False, default="analyst")+ scope: Mapped[str] = mapped_column(Text, nullable=False, default="tenant")+ expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)++ state: Mapped[str] = mapped_column(Text, nullable=False, default="in_use")+ conflicts_with: Mapped[uuid.UUID | None] = mapped_column(+ UUID(as_uuid=True), ForeignKey("aisoc_memory_items.id", ondelete="SET NULL"), nullable=True+ )+ state_reason: Mapped[str] = mapped_column(Text, nullable=False, default="")++ #: Which model produced the vector, and when. Both nullable, and the two+ #: NULL cases are different: never embedded, versus embedded by a model+ #: nobody is running any more. The status route renders them apart.+ embedding_model: Mapped[str | None] = mapped_column(Text, nullable=True)+ embedded_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)++ created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=lambda: datetime.now(UTC))+ updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=lambda: datetime.now(UTC))diff --git a/services/api/app/services/semantic_memory.py b/services/api/app/services/semantic_memory.pynew file mode 100644index 000000000..655536908--- /dev/null+++ b/services/api/app/services/semantic_memory.py@@ -0,0 +1,541 @@+"""Vector recall over past cases, runbooks and institutional memory.++Depth plan 9.2 (parity 6.5). ``knowledge_base.py`` searched with+``to_tsvector('english', content) @@ plainto_tsquery(...)``, which answers a+question phrased in the words the runbook happens to use and answers nothing+otherwise. Mid-incident that is most of the time: "someone is brute-forcing a+VPN account" shares no stem with "credential stuffing against the+remote-access gateway".++Three deliberate choices, each of which has a cheaper wrong version.++**Qdrant over its REST API, not the Python client.**+``qdrant-client`` is a dependency of ``services/threatintel`` and not of this+service. Adding it here means regenerating this service's lockfile, and the+determinism work in this tree is explicit that a dependency added for one+module changes what every image resolves. Qdrant's REST surface is four+endpoints and ``httpx`` is already a dependency, so there is nothing to add.++**The tenant filter is mandatory and the function refuses without one.**+``services/threatintel``'s store treats ``tenant_id=None`` as a privileged+global read, and documents that callers must gate it. That is a correct+design for feed intel, which is genuinely shared. It is the wrong one here:+nothing in this store is shared, so an unscoped read has no legitimate+caller, and a default that can be reached by forgetting an argument is not a+control. :func:`recall` raises rather than searching.++**No embedding backend means "unavailable", never a zero vector.**+``QdrantStore._embed`` falls back to zero vectors when ``fastembed`` is+missing, so every similarity is identical and the search returns an+arbitrary ``limit`` rows that look like results. A caller cannot tell that+from a working retriever. This module raises+:class:`EmbeddingsUnavailable`, the route reports it, and the console shows+an honest empty state -- the same rule as "a column nobody writes renders as+'the measurement does not exist', never as a confident zero".++Dimension immutability+----------------------+A Qdrant collection's vector size cannot be altered, so changing the+embedding model is a re-embed into a new collection behind an alias, not a+settings change. ``app/db/vector_migrations.py`` already models that and+exposes :func:`pending_rebuilds`; :func:`status` surfaces it, together with+the per-row ``embedding_model`` this store records, so "which rows were+embedded by a model nobody is running any more" is answerable before a+recall silently degrades rather than after.+"""++from __future__ import annotations++import hashlib+import logging+import os+import uuid+from dataclasses import dataclass+from datetime import UTC, datetime+from typing import Any++import httpx++from app.core.config import settings+from app.services.model_aliases import resolve_api_key, resolve_base_url++logger = logging.getLogger("aisoc.semantic_memory")++#: One collection for all three kinds. ``kind`` is a payload field and a+#: filter, not a collection: three collections would mean three searches for+#: "what do we know about this", and the ranking across them would have to be+#: invented rather than measured.+COLLECTION = "aisoc_memory"++#: Dimension of the collection. Fixed at creation and immutable thereafter --+#: see the module header. 1536 matches the threat-intel collections and the+#: common hosted embedding models, so a deployment running both does not need+#: two answers to "what size are our vectors".+EMBEDDING_DIM = 1536++#: Default embedding model asked of whichever backend is configured.+DEFAULT_EMBEDDING_MODEL = os.environ.get("AISOC_EMBEDDING_MODEL", "").strip() or "text-embedding-3-small"++_TIMEOUT = httpx.Timeout(20.0, connect=5.0)+++class EmbeddingsUnavailable(RuntimeError):+ """No embedding backend answered. Recall is unavailable, not empty.++ Separated from a generic error because the two render differently and+ must: "we searched and found nothing" and "we could not search" are+ different facts, and collapsing them is how a console shows a confident+ empty state over a broken retriever.+ """+++class VectorStoreUnavailable(RuntimeError):+ """Qdrant did not answer. Same reasoning as above."""+++@dataclass(frozen=True)+class RecallHit:+ """One recalled item, with everything a reader needs to weigh it."""++ item_id: uuid.UUID+ kind: str+ subject: str+ title: str+ content: str+ score: float+ author: str+ source: str+ scope: str+ state: str+ embedding_model: str | None+++def content_hash(content: str) -> str:+ """Stable digest of an item's content.++ Whitespace-normalised, because a runbook re-ingested with different line+ wrapping is the same claim and should not read as a contradiction of+ itself.+ """+ normalised = " ".join(content.split())+ return hashlib.sha256(normalised.encode("utf-8")).hexdigest()+++def point_id(item_id: uuid.UUID) -> str:+ """Qdrant point id for a memory item. The row's own id, as a string.++ Qdrant accepts unsigned integers or UUIDs. Using the row id rather than a+ hash means a delete in Postgres has an obvious counterpart in Qdrant and+ neither store needs a mapping table to find the other.+ """+ return str(item_id)+++# ---------------------------------------------------------------------------+# Embeddings+# ---------------------------------------------------------------------------+++def embedding_backend() -> tuple[str, str, str | None] | None:+ """``(kind, url, api_key)`` for the configured embedding backend, or None.++ Two backends, in order:++ * the LLM gateway this deployment already routes chat through, at+ ``/v1/embeddings`` -- which is LiteLLM in CORE and whatever an operator+ pointed ``OPENAI_BASE_URL`` or ``LLM_BASE_URL`` at otherwise;+ * Ollama at ``OLLAMA_BASE_URL``, which CORE also ships, for a deployment+ with no gateway key at all.++ Returns ``None`` rather than guessing. A guessed backend produces a+ connection error at recall time, which reads to an operator as "the+ feature is broken" rather than "nothing is configured".+ """+ base = resolve_base_url(DEFAULT_EMBEDDING_MODEL)+ if base:+ return ("gateway", f"{str(base).rstrip('/')}/embeddings", resolve_api_key(DEFAULT_EMBEDDING_MODEL))+ ollama = os.environ.get("OLLAMA_BASE_URL", "").strip()+ if ollama:+ return ("ollama", f"{ollama.rstrip('/')}/api/embeddings", None)+ return None+++def _fit(vector: list[float]) -> list[float]:+ """Pad a short vector to the collection's dimension. Refuse to truncate.++ The two directions are not symmetric and treating them as one cost a+ measurement. Padding with zeros is harmless under cosine: the appended+ components contribute nothing to either the dot product or to a norm+ computed over the same padded space, so a 768-dimension model ranks+ exactly as it would in its own space.++ Truncating is destructive. Cutting a 3072-component embedding to 1536+ throws away half of what the model said, and the distances that survive+ are not the model's distances. The first version of this function+ truncated, and the recall score for ``llama3.2:3b`` came out at 0.167+ against a 0.667 lexical baseline — a number that was measuring the+ truncation, not the model. So an oversized vector is a refusal naming the+ two dimensions, which is a configuration an operator can fix, rather than+ a silently worse retriever they cannot see.+ """+ if len(vector) < EMBEDDING_DIM:+ return vector + [0.0] * (EMBEDDING_DIM - len(vector))+ if len(vector) > EMBEDDING_DIM:+ raise EmbeddingsUnavailable(+ f"{DEFAULT_EMBEDDING_MODEL} returns {len(vector)}-dimension vectors and the "+ f"{COLLECTION} collection holds {EMBEDDING_DIM}. Truncating would discard most of "+ "what the model said and the resulting distances would not be the model's, so this "+ "is refused. Choose an embedding model of at most "+ f"{EMBEDDING_DIM} dimensions, or re-embed into a wider collection through "+ "app/db/vector_migrations.py — a collection's dimension is immutable."+ )+ return vector+++async def embed(texts: list[str], *, client: httpx.AsyncClient | None = None) -> tuple[list[list[float]], str]:+ """Embed ``texts``. Raises :class:`EmbeddingsUnavailable` rather than faking.++ Returns the vectors and the model that produced them, so the caller can+ record it on the row.+ """+ if not texts:+ return [], DEFAULT_EMBEDDING_MODEL+ backend = embedding_backend()+ if backend is None:+ raise EmbeddingsUnavailable(+ "No embedding backend is configured. Set OPENAI_BASE_URL or LLM_BASE_URL (or run "+ "the bundled gateway, which sets LLM_GATEWAY_URL), or OLLAMA_BASE_URL. Semantic "+ "recall is unavailable until then; full-text search still works."+ )+ kind, url, api_key = backend+ headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}++ owns_client = client is None+ http = client or httpx.AsyncClient(timeout=_TIMEOUT)+ try:+ if kind == "gateway":+ res = await http.post(url, json={"model": DEFAULT_EMBEDDING_MODEL, "input": texts}, headers=headers)+ if res.status_code >= 400:+ raise EmbeddingsUnavailable(f"The embedding backend answered HTTP {res.status_code}: {res.text[:200]}")+ payload = res.json()+ rows = payload.get("data") or []+ if len(rows) != len(texts):+ raise EmbeddingsUnavailable(f"The embedding backend returned {len(rows)} vector(s) for {len(texts)} input(s).")+ return [_fit([float(x) for x in row["embedding"]]) for row in rows], DEFAULT_EMBEDDING_MODEL+ # Ollama embeds one prompt per call.+ vectors: list[list[float]] = []+ for text_value in texts:+ res = await http.post(url, json={"model": DEFAULT_EMBEDDING_MODEL, "prompt": text_value})+ if res.status_code >= 400:+ raise EmbeddingsUnavailable(f"Ollama answered HTTP {res.status_code}: {res.text[:200]}")+ vectors.append(_fit([float(x) for x in res.json().get("embedding", [])]))+ return vectors, DEFAULT_EMBEDDING_MODEL+ except httpx.HTTPError as exc:+ raise EmbeddingsUnavailable(f"The embedding backend was unreachable: {exc}") from exc+ finally:+ if owns_client:+ await http.aclose()+++# ---------------------------------------------------------------------------+# Qdrant, over its REST API+# ---------------------------------------------------------------------------+++def _qdrant_url(path: str) -> str:+ return f"{settings.QDRANT_URL.rstrip('/')}{path}"+++async def ensure_collection(*, client: httpx.AsyncClient | None = None) -> None:+ """Create the collection if it is absent. Never alters an existing one.++ Creation is the only safe idempotent operation: a collection's vector+ size and distance are immutable, so "converge it to what we want" is not+ something this can do. A dimension change goes through+ ``app/db/vector_migrations.py`` as a re-embed.+ """+ owns = client is None+ http = client or httpx.AsyncClient(timeout=_TIMEOUT)+ try:+ existing = await http.get(_qdrant_url(f"/collections/{COLLECTION}"))+ if existing.status_code == 200:+ return+ res = await http.put(+ _qdrant_url(f"/collections/{COLLECTION}"),+ json={"vectors": {"size": EMBEDDING_DIM, "distance": "Cosine"}},+ )+ if res.status_code >= 400:+ raise VectorStoreUnavailable(f"Qdrant refused to create {COLLECTION}: HTTP {res.status_code} {res.text[:200]}")+ except httpx.HTTPError as exc:+ raise VectorStoreUnavailable(f"Qdrant was unreachable at {settings.QDRANT_URL}: {exc}") from exc+ finally:+ if owns:+ await http.aclose()+++async def index_item(+ *,+ tenant_id: uuid.UUID,+ item_id: uuid.UUID,+ kind: str,+ subject: str,+ title: str,+ content: str,+ state: str,+ client: httpx.AsyncClient | None = None,+) -> str:+ """Embed one item and upsert its point. Returns the embedding model used.++ ``tenant_id`` goes into the payload as a string because that is what the+ filter below matches on, and a filter that has to coerce is a filter that+ can silently match nothing.+ """+ owns = client is None+ http = client or httpx.AsyncClient(timeout=_TIMEOUT)+ try:+ vectors, model = await embed([f"{title}\n\n{content}".strip()], client=http)+ await ensure_collection(client=http)+ res = await http.put(+ _qdrant_url(f"/collections/{COLLECTION}/points?wait=true"),+ json={+ "points": [+ {+ "id": point_id(item_id),+ "vector": vectors[0],+ "payload": {+ "tenant_id": str(tenant_id),+ "item_id": str(item_id),+ "kind": kind,+ "subject": subject,+ "state": state,+ },+ }+ ]+ },+ )+ if res.status_code >= 400:+ raise VectorStoreUnavailable(f"Qdrant refused the upsert: HTTP {res.status_code} {res.text[:200]}")+ return model+ except httpx.HTTPError as exc:+ raise VectorStoreUnavailable(f"Qdrant was unreachable at {settings.QDRANT_URL}: {exc}") from exc+ finally:+ if owns:+ await http.aclose()+++async def forget_item(item_id: uuid.UUID, *, client: httpx.AsyncClient | None = None) -> None:+ """Drop one point. Called when an item is retired or deleted."""+ owns = client is None+ http = client or httpx.AsyncClient(timeout=_TIMEOUT)+ try:+ await http.post(+ _qdrant_url(f"/collections/{COLLECTION}/points/delete?wait=true"),+ json={"points": [point_id(item_id)]},+ )+ except httpx.HTTPError as exc:+ # A point left behind is recalled and then filtered out by the+ # Postgres join below, so this degrades to a slower search rather+ # than to a leak. Logged at warning because it does accumulate.+ logger.warning("semantic_memory: could not delete point %s: %s", item_id, exc)+ finally:+ if owns:+ await http.aclose()+++def tenant_filter(tenant_id: uuid.UUID, kinds: list[str] | None = None) -> dict[str, Any]:+ """The Qdrant filter every search must carry.++ A separate function so there is exactly one spelling of it and a test can+ assert that :func:`recall` uses it. ``state`` is matched here as well as+ in Postgres: belt and braces, because a point whose row was retired but+ whose delete failed is otherwise recalled and discarded late.+ """+ must: list[dict[str, Any]] = [+ {"key": "tenant_id", "match": {"value": str(tenant_id)}},+ {"key": "state", "match": {"value": "in_use"}},+ ]+ if kinds:+ must.append({"key": "kind", "match": {"any": list(kinds)}})+ return {"must": must}+++async def search(+ query: str,+ *,+ tenant_id: uuid.UUID,+ kinds: list[str] | None = None,+ limit: int = 10,+ client: httpx.AsyncClient | None = None,+) -> list[tuple[uuid.UUID, float]]:+ """Vector search, scoped to one tenant. Returns ``(item_id, score)``.++ ``tenant_id`` is a required keyword with no default and no ``None``+ branch. The threat-intel store permits an unscoped read for shared feed+ intel; nothing here is shared, so an unscoped read has no caller and a+ default reachable by forgetting an argument is not a control.+ """+ if not isinstance(tenant_id, uuid.UUID):+ raise ValueError("recall must be scoped to a tenant")+ owns = client is None+ http = client or httpx.AsyncClient(timeout=_TIMEOUT)+ try:+ vectors, _ = await embed([query], client=http)+ res = await http.post(+ _qdrant_url(f"/collections/{COLLECTION}/points/search"),+ json={+ "vector": vectors[0],+ "limit": max(1, min(limit, 100)),+ "filter": tenant_filter(tenant_id, kinds),+ "with_payload": True,+ },+ )+ if res.status_code == 404:+ # The collection has never been created, which means nothing has+ # been indexed. Empty is the honest answer, not an error.+ return []+ if res.status_code >= 400:+ raise VectorStoreUnavailable(f"Qdrant refused the search: HTTP {res.status_code} {res.text[:200]}")+ out: list[tuple[uuid.UUID, float]] = []+ for hit in res.json().get("result", []):+ payload = hit.get("payload") or {}+ raw = payload.get("item_id") or hit.get("id")+ try:+ out.append((uuid.UUID(str(raw)), float(hit.get("score", 0.0))))+ except (TypeError, ValueError):+ continue+ return out+ except httpx.HTTPError as exc:+ raise VectorStoreUnavailable(f"Qdrant was unreachable at {settings.QDRANT_URL}: {exc}") from exc+ finally:+ if owns:+ await http.aclose()+++async def remember_closed_case(+ db: Any,+ *,+ tenant_id: uuid.UUID,+ case_id: Any,+ title: str,+ summary: str,+ author: str,+) -> uuid.UUID | None:+ """Write a closed case into memory. The automatic writer for ``kind='case'``.++ This exists because of the half of the "exists unwired" family that+ reachability analysis cannot see: a table every surface reads and no code+ path writes. ``POST /memory/items`` alone would have left ``kind='case'``+ permanently empty on every deployment where nobody ingests by hand, and+ "past cases" is the first of the three things parity 6.5 names.++ Fails soft and returns ``None``. Closing a case must not 503 because a+ vector store is down; the row is written either way and+ ``/memory/status`` reports how many rows are waiting to be embedded, so+ the degradation is visible rather than silent.+ """+ from sqlalchemy import select # noqa: PLC0415++ from app.models.semantic_memory import MemoryItem # noqa: PLC0415 (circular at module scope)++ content = summary.strip()+ if not content:+ # A case closed with no narrative teaches nothing. Writing it would+ # pad recall with rows whose content is a title, which is the kind of+ # filler that makes a retrieval score look busy and answer worse.+ return None++ subject = f"case:{case_id}"+ digest = content_hash(content)+ existing = (+ await db.execute(+ select(MemoryItem).where(+ MemoryItem.tenant_id == tenant_id,+ MemoryItem.kind == "case",+ MemoryItem.subject == subject,+ MemoryItem.state == "in_use",+ )+ )+ ).scalar_one_or_none()+ if existing is not None and existing.content_hash == digest:+ return existing.id+ if existing is not None:+ # A case reopened and re-closed with a different outcome supersedes+ # its own earlier entry. Not a conflict: it is the same author+ # correcting the same subject, which is the one case where+ # last-write-wins is right.+ existing.state = "not_in_use"+ existing.state_reason = "superseded when the case was closed again"+ await forget_item(existing.id)++ row = MemoryItem(+ tenant_id=tenant_id,+ kind="case",+ subject=subject,+ source_ref=str(case_id),+ title=title or subject,+ content=content,+ content_hash=digest,+ author=author,+ source="case_closure",+ scope="tenant",+ state="in_use",+ )+ db.add(row)+ await db.flush()+ try:+ model = await index_item(+ tenant_id=tenant_id,+ item_id=row.id,+ kind="case",+ subject=subject,+ title=row.title,+ content=content,+ state="in_use",+ )+ row.embedding_model = model+ row.embedded_at = datetime.now(UTC)+ except (EmbeddingsUnavailable, VectorStoreUnavailable) as exc:+ logger.warning("semantic_memory: closed case %s stored but not embedded: %s", case_id, exc)+ return row.id+++async def status(*, client: httpx.AsyncClient | None = None) -> dict[str, Any]:+ """What a reader needs to know before trusting a recall.++ Reports the configured backend, the collection's actual dimension as+ Qdrant holds it (not as this module wishes it were), and whether a+ re-embed is pending. Every field can be ``None`` and the route renders+ that as "not known", because the alternative -- a confident zero or a+ confident "ok" -- is the failure this repository keeps finding.+ """+ backend = embedding_backend()+ out: dict[str, Any] = {+ "embedding_backend": backend[0] if backend else None,+ "embedding_model": DEFAULT_EMBEDDING_MODEL if backend else None,+ "expected_dimension": EMBEDDING_DIM,+ "collection": COLLECTION,+ "collection_dimension": None,+ "points": None,+ "vector_store_reachable": False,+ "pending_rebuilds": None,+ }+ owns = client is None+ http = client or httpx.AsyncClient(timeout=_TIMEOUT)+ try:+ res = await http.get(_qdrant_url(f"/collections/{COLLECTION}"))+ if res.status_code == 200:+ result = res.json().get("result") or {}+ params = ((result.get("config") or {}).get("params") or {}).get("vectors") or {}+ out["collection_dimension"] = params.get("size")+ out["points"] = result.get("points_count")+ out["vector_store_reachable"] = True+ elif res.status_code == 404:+ # Reachable, nothing indexed yet. Distinct from unreachable, and+ # the console must not render them the same way.+ out["vector_store_reachable"] = True+ except httpx.HTTPError:+ out["vector_store_reachable"] = False+ finally:+ if owns:+ await http.aclose()+ return outdiff --git a/services/api/migrations/105_semantic_memory.sql b/services/api/migrations/105_semantic_memory.sqlnew file mode 100644index 000000000..a57fe2a40--- /dev/null+++ b/services/api/migrations/105_semantic_memory.sql@@ -0,0 +1,170 @@+-- 105_semantic_memory.sql — memory items with provenance, and the conflicts+-- an analyst has to settle before the agent reads either side.+--+-- Depth plan 9.2 (parity 6.5). The knowledge base was full-text only:+-- `to_tsvector('english', content) @@ plainto_tsquery(...)`. That answers a+-- question phrased in the words the runbook happens to use and answers+-- nothing otherwise, which for an analyst mid-incident is most of the time —+-- "someone is brute-forcing a VPN account" does not lexically overlap+-- "credential stuffing against the remote-access gateway".+--+-- Why a table *and* a vector store+-- --------------------------------+-- Qdrant holds the vectors and nothing else that matters. Everything an+-- analyst or an auditor needs to decide whether to trust a recalled item+-- lives here, in Postgres, under RLS, because:+--+-- * a vector payload is not a place to enforce a foreign key, a CHECK or a+-- row-level policy, and every one of those is load-bearing for this data;+-- * the vectors are derivable — re-embedding rebuilds them from this table+-- — and Qdrant's collections are immutable in their dimension, so an+-- embedding-model change is a re-embed. A store you may have to rebuild+-- is not the store of record;+-- * the conflict workflow below is a state machine, and a state machine+-- living in a payload nobody can transactionally update is not one.+--+-- Provenance, which is the whole point of 6.5+-- -------------------------------------------+-- Every item carries an author, a source, a scope, an expiry and a state.+-- Those five exist because an agent that cites a memory item is making an+-- argument from authority, and the reader has to be able to ask whose+-- authority, from where, over what, until when, and whether anybody has+-- disputed it. A recalled item with none of that is an unattributed+-- assertion in the prompt that decides what happens to somebody's estate.+--+-- `state` is the enforcement point:+--+-- in_use recall returns it+-- conflict recall does NOT return it, and neither side of the conflict,+-- until an analyst settles which one is right+-- not_in_use retired; kept for the audit trail, never recalled+--+-- Withholding *both* sides of a conflict is deliberate and is the opposite of+-- the obvious design. Returning the newer one silently resolves a+-- disagreement by timestamp, and the case where that is wrong -- an analyst+-- correcting a bad entry, then the bad entry being re-ingested by a job -- is+-- exactly the case the feature exists for.+--+-- `expires_at` is nullable because some memory is genuinely permanent ("this+-- subnet is the payment estate"), but an expired item is `not_in_use` by+-- query rather than by a sweep job, so a worker being down cannot resurrect+-- one.+--+-- `content_hash` is what makes re-ingestion idempotent and what detects a+-- conflict: same `subject`, different hash, both `in_use` is a contradiction+-- by definition.+--+-- `embedding_model` and `embedded_at` travel with the row so a re-embed can+-- find the stale ones, and so a reader can tell an item that has never been+-- embedded from one embedded by a model that is no longer configured. Those+-- two render differently and a NULL rendered as "fine" is the failure this+-- repository keeps finding.+--+-- Reversible: DROP TABLE IF EXISTS aisoc_memory_items.++CREATE TABLE IF NOT EXISTS aisoc_memory_items (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,++ -- What kind of thing this is. Closed set: a free-text kind is one nobody+ -- can filter a recall by, and "runbook" spelled three ways is how a+ -- retrieval quietly stops covering a third of the corpus.+ kind TEXT NOT NULL+ CHECK (kind IN ('case', 'runbook', 'memory')),++ -- What the item is *about*. Two items with the same subject are making+ -- claims about the same thing, which is what makes a contradiction+ -- detectable at all. For a closed case this is its id; for institutional+ -- memory, the memory key; for a runbook, its title.+ subject TEXT NOT NULL,++ -- Where the original lives, when there is one. Free text because it is a+ -- case id, a document id or a memory key depending on `kind`.+ source_ref TEXT NOT NULL DEFAULT '',++ title TEXT NOT NULL DEFAULT '',+ content TEXT NOT NULL,+ -- TEXT, not CHAR(64). `char_length` on a `bpchar` strips trailing+ -- blanks, so the padding a CHAR(64) adds does not satisfy the length+ -- check below -- the two would have been fighting each other, and the+ -- type would have bought nothing the check does not already give.+ content_hash TEXT NOT NULL+ CHECK (char_length(content_hash) = 64),++ -- ── Provenance ────────────────────────────────────────────────────────+ -- Who said it. An email, an API key prefix, or a worker's name.+ author TEXT NOT NULL DEFAULT '',+ -- Where it came from: 'analyst', 'agent', 'case_closure', 'import', …+ -- Not a CHECK: the set grows with every writer, and refusing an unknown+ -- source at the database would mean a new writer silently dropping rows.+ source TEXT NOT NULL DEFAULT 'analyst',+ -- 'tenant' (private) or 'global' (shipped content, visible to everyone).+ -- A row is still tenant_id-stamped either way; 'global' is a statement+ -- about who authored it, never a licence to cross tenants on read.+ scope TEXT NOT NULL DEFAULT 'tenant'+ CHECK (scope IN ('tenant', 'global')),+ expires_at TIMESTAMPTZ,++ state TEXT NOT NULL DEFAULT 'in_use'+ CHECK (state IN ('in_use', 'conflict', 'not_in_use')),+ -- The other item this one contradicts. Set on both rows when a conflict+ -- is raised, cleared on both when it is settled.+ conflicts_with UUID REFERENCES aisoc_memory_items(id) ON DELETE SET NULL,+ -- Why it is not in use, or how the conflict was settled. Shown to the+ -- analyst and carried into the audit row.+ state_reason TEXT NOT NULL DEFAULT '',++ -- ── Embedding bookkeeping ─────────────────────────────────────────────+ embedding_model TEXT,+ embedded_at TIMESTAMPTZ,++ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),++ -- One live item per subject per kind. The partial unique index means a+ -- retired or conflicted row does not block a replacement, which is what+ -- makes "correct a bad entry" a one-statement operation rather than a+ -- delete followed by an insert with a window in between.+ CONSTRAINT aisoc_memory_items_content_hash_len CHECK (char_length(content_hash) = 64)+);++CREATE UNIQUE INDEX IF NOT EXISTS aisoc_memory_items_live_subject_idx+ ON aisoc_memory_items (tenant_id, kind, subject)+ WHERE state = 'in_use';++CREATE INDEX IF NOT EXISTS aisoc_memory_items_tenant_state_idx+ ON aisoc_memory_items (tenant_id, state, updated_at DESC);++-- The re-embed worker's query: rows whose vector was made by a model that is+-- no longer the configured one, plus rows that have never been embedded.+CREATE INDEX IF NOT EXISTS aisoc_memory_items_embedding_idx+ ON aisoc_memory_items (tenant_id, embedding_model);++COMMENT ON TABLE aisoc_memory_items IS+ 'Store of record for semantic memory. Qdrant holds the vectors and nothing else that '+ 'matters: the vectors are derivable by re-embedding from here, and a vector payload '+ 'cannot carry a foreign key, a CHECK or a row-level policy.';++COMMENT ON COLUMN aisoc_memory_items.state IS+ 'in_use | conflict | not_in_use. Recall returns in_use only, and withholds BOTH sides '+ 'of a conflict rather than preferring the newer one -- resolving a disagreement by '+ 'timestamp is wrong in exactly the case this feature exists for.';++-- Written out rather than looped with format(): a loop makes the policy+-- invisible to grep and to scripts/check_tenant_query_predicates.py, which+-- reads migrations to learn which tables carry one.+DO $$+BEGIN+ EXECUTE 'ALTER TABLE aisoc_memory_items ENABLE ROW LEVEL SECURITY';+ EXECUTE 'ALTER TABLE aisoc_memory_items FORCE ROW LEVEL SECURITY';+ EXECUTE 'DROP POLICY IF EXISTS tenant_isolation ON aisoc_memory_items';+ EXECUTE+ 'CREATE POLICY tenant_isolation ON aisoc_memory_items '+ 'USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL) '+ 'WITH CHECK (tenant_id = current_tenant_id())';++ IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'aisoc_app') THEN+ EXECUTE 'GRANT SELECT, INSERT, UPDATE, DELETE ON aisoc_memory_items TO aisoc_app';+ END IF;+END+$$;diff --git a/services/api/tests/eval_data/memory_recall_probes_v1.json b/services/api/tests/eval_data/memory_recall_probes_v1.jsonnew file mode 100644index 000000000..603f1202f--- /dev/null+++ b/services/api/tests/eval_data/memory_recall_probes_v1.json@@ -0,0 +1,159 @@+{+ "version": 1,+ "description": "Labelled recall probes for depth 9.2 (parity 6.5). Each probe is an analyst's question and the one memory item that answers it. The corpus and the probes are hand-authored under this repository's own licence and are labelled is_synthetic: true, the same rule the verdict corpus follows.",+ "is_synthetic": true,+ "authored_by": "AiSOC contributors",+ "design": [+ "Every probe is phrased the way an analyst speaks mid-incident, and deliberately shares as little vocabulary as possible with the item that answers it. That is the whole measurement: full-text retrieval answers a question asked in the document's own words and answers nothing otherwise, so a probe set that reuses the document's words would score the baseline far too well and prove nothing.",+ "lexical_overlap records the share of the probe's content words that appear as a word in the answering item. It is computed by the scorer, not authored, and it is reported with the result so a reader can see the probes really are low-overlap rather than take the claim on trust.",+ "There are distractor items that are topically adjacent to a probe but do not answer it. Without them a retriever can score well by returning the only item about identity, and the measurement would be of corpus size rather than of retrieval."+ ],+ "corpus": [+ {+ "id": "m-vpn-stuffing",+ "kind": "runbook",+ "subject": "Credential stuffing against the remote-access gateway",+ "title": "Credential stuffing against the remote-access gateway",+ "content": "When the remote-access gateway reports a surge of authentication failures spread across many distinct accounts from a small number of source addresses, treat it as credential stuffing rather than as a lockout storm. Confirm by checking whether the failing accounts share nothing but the source. Block the source ranges at the gateway, force a password reset for any account that subsequently succeeded, and raise a case tagged T1110.004."+ },+ {+ "id": "m-vpn-lockout",+ "kind": "runbook",+ "subject": "Account lockout storm after a password policy change",+ "title": "Account lockout storm after a password policy change",+ "content": "A sudden rise in authentication failures confined to a single account, repeating from one workstation, is almost always a stale credential in a scheduled task or a mapped drive after a password policy change. Do not block the source. Find the service holding the old secret and update it."+ },+ {+ "id": "m-s3-public",+ "kind": "runbook",+ "subject": "Object storage bucket exposed to the internet",+ "title": "Object storage bucket exposed to the internet",+ "content": "A bucket policy granting read to every principal is an exposure whatever the contents. Snapshot the access log first, because removing the policy also removes the evidence of who used it. Then remove the grant, enable the account-level public access block, and check the object inventory for anything carrying regulated data."+ },+ {+ "id": "m-s3-replication",+ "kind": "runbook",+ "subject": "Cross-region replication failing on an object store",+ "title": "Cross-region replication failing on an object store",+ "content": "Replication failures on an object store are an availability problem, not a security one, unless the destination account is outside the organisation. Check the replication role's trust policy before escalating."+ },+ {+ "id": "m-token-theft",+ "kind": "memory",+ "subject": "Session token replay from an unusual geography",+ "title": "Session token replay from an unusual geography",+ "content": "This estate's workforce is entirely in one country, so a session presented from elsewhere with no corresponding interactive sign-in is token replay until proven otherwise. The correct first action is to revoke the session rather than to reset the password, because a reset does not invalidate a stolen cookie."+ },+ {+ "id": "m-travel",+ "kind": "memory",+ "subject": "Approved overseas travel for the field sales team",+ "title": "Approved overseas travel for the field sales team",+ "content": "The field sales team travels continuously and signs in from many countries. Geographic improbability alone is not a finding for members of that team; look for a second signal before raising a case."+ },+ {+ "id": "m-payment-subnet",+ "kind": "memory",+ "subject": "Business context for the 10.40.0.0/16 range",+ "title": "Business context for the 10.40.0.0/16 range",+ "content": "Everything in 10.40.0.0/16 is the card payment estate and is in scope for PCI DSS. Any containment action there needs the on-call payments lead on the call before it is taken, and any case touching it is reportable."+ },+ {+ "id": "m-lab-subnet",+ "kind": "memory",+ "subject": "Business context for the 10.90.0.0/16 range",+ "title": "Business context for the 10.90.0.0/16 range",+ "content": "10.90.0.0/16 is the security team's own malware laboratory. Detonations there are expected and are not findings. Hosts in it are rebuilt nightly."+ },+ {+ "id": "m-ransom-case",+ "kind": "case",+ "subject": "case:2026-0431",+ "title": "File server encrypted overnight after a vendor appliance compromise",+ "content": "Closed as a true positive. The intruder reached the file server through an unpatched management interface on a network appliance that the vendor exposes on a non-standard port. The lesson recorded at closure was that the asset inventory did not list the appliance at all, so no vulnerability scan had ever covered it."+ },+ {+ "id": "m-phish-case",+ "kind": "case",+ "subject": "case:2026-0502",+ "title": "Finance invoice lure delivered through a compromised supplier mailbox",+ "content": "Closed as a true positive with no impact. The message came from a genuine supplier whose mailbox had been taken over, so sender reputation and authentication both passed. What caught it was the reply-to pointing at a domain registered eleven days earlier."+ },+ {+ "id": "m-benign-scan-case",+ "kind": "case",+ "subject": "case:2026-0388",+ "title": "Quarterly external assessment mistaken for reconnaissance",+ "content": "Closed as benign. The source was the firm contracted for the quarterly external assessment, and the engagement window had been recorded in the change calendar but not in any system the detection pipeline reads. The action recorded at closure was to put the window into an allowlist ahead of the next engagement."+ },+ {+ "id": "m-backup-runbook",+ "kind": "runbook",+ "subject": "Restoring from the offline backup copy",+ "title": "Restoring from the offline backup copy",+ "content": "The offline copy is written weekly and is not reachable from the production network by design. Restoring from it needs two people, takes about six hours for the file estate, and must be preceded by confirming the restore target is clean, because restoring onto a host the intruder still holds simply re-encrypts the data."+ }+ ],+ "probes": [+ {+ "id": "p-stuffing",+ "query": "lots of failed logins on the VPN from a handful of IPs, hitting loads of different usernames",+ "expected": "m-vpn-stuffing"+ },+ {+ "id": "p-lockout",+ "query": "one user keeps getting locked out all day and it is always the same desktop",+ "expected": "m-vpn-lockout"+ },+ {+ "id": "p-bucket",+ "query": "someone made a storage container readable by anyone on the internet, what do I do first",+ "expected": "m-s3-public"+ },+ {+ "id": "p-replay",+ "query": "a cookie from this account is being used abroad but there was no login from there",+ "expected": "m-token-theft"+ },+ {+ "id": "p-travel",+ "query": "is a sign-in from another country on its own enough to open a ticket for a salesperson",+ "expected": "m-travel"+ },+ {+ "id": "p-payments",+ "query": "do I need anyone's sign-off before I isolate a machine on 10.40.12.7",+ "expected": "m-payment-subnet"+ },+ {+ "id": "p-lab",+ "query": "a host at 10.90.3.4 keeps detonating samples, should I be worried",+ "expected": "m-lab-subnet"+ },+ {+ "id": "p-appliance",+ "query": "have we ever been broken into through kit that was not in the asset register",+ "expected": "m-ransom-case"+ },+ {+ "id": "p-supplier",+ "query": "an email passed every authentication check but still turned out to be a lure, how was it spotted",+ "expected": "m-phish-case"+ },+ {+ "id": "p-pentest",+ "query": "we are seeing port sweeps from an external firm, was this ever something we booked",+ "expected": "m-benign-scan-case"+ },+ {+ "id": "p-restore",+ "query": "how long does it take to get the data back from the copy that is kept disconnected",+ "expected": "m-backup-runbook"+ },+ {+ "id": "p-reencrypt",+ "query": "what do I have to check before putting data back onto a machine that was hit",+ "expected": "m-backup-runbook"+ }+ ]+}diff --git a/services/api/tests/test_semantic_memory.py b/services/api/tests/test_semantic_memory.pynew file mode 100644index 000000000..f89fd8323--- /dev/null+++ b/services/api/tests/test_semantic_memory.py@@ -0,0 +1,471 @@+"""Semantic memory: provenance, conflict review, and refusing to fake a recall.++Depth plan 9.2 (parity 6.5).++The properties, each asserted by name below:++* an ingest records the **authenticated** principal as the author, not+ whatever the body claimed -- provenance a caller can set is not provenance;+* two claims about the same subject suspend **both** sides rather than+ last-write-wins, and recall returns neither until a human settles it;+* re-ingesting the identical content is idempotent and does not re-embed;+* retirement is not deletion: the row and its provenance survive;+* recall is **tenant-scoped in the filter itself**, and the function refuses+ to run unscoped rather than defaulting;+* with no embedding backend, recall is **503 with the reason** -- never an+ empty list, because "found nothing" and "could not look" are different+ facts;+* an oversized embedding is refused rather than truncated, because+ truncating measured as a worse retriever that nobody could see.++The database is real, on the SQLite-compiles-Postgres-types basis+``test_mcp_registry.py`` uses. Qdrant and the embedding backend are stubbed+at the HTTP boundary with a real ``httpx`` transport rather than by patching+the functions under test, so what runs is the request this service really+makes.+"""++from __future__ import annotations++import uuid+from datetime import UTC, datetime, timedelta++import httpx+import pytest+import pytest_asyncio+from app.api.v1.deps import CurrentUser, get_current_user+from app.api.v1.endpoints.semantic_memory import router as memory_router+from app.db.database import Base, get_db+from app.db.rls import get_tenant_db+from app.models.audit import AuditLog+from app.models.semantic_memory import MemoryItem+from app.services import semantic_memory as svc+from fastapi import FastAPI+from fastapi.testclient import TestClient+from sqlalchemy import select+from sqlalchemy.dialects.postgresql import INET, JSONB+from sqlalchemy.dialects.postgresql import UUID as PgUUID+from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine+from sqlalchemy.ext.compiler import compiles++TENANT = uuid.UUID("aaaaaaaa-0000-0000-0000-0000000000b1")+OTHER_TENANT = uuid.UUID("bbbbbbbb-0000-0000-0000-0000000000b2")+ANALYST = uuid.UUID("33333333-3333-3333-3333-333333333333")+++@compiles(JSONB, "sqlite")+def _jsonb_sqlite(_type_, _compiler_, **_kw_):+ return "TEXT"+++@compiles(PgUUID, "sqlite")+def _uuid_sqlite(_type_, _compiler_, **_kw_):+ return "CHAR(36)"+++@compiles(INET, "sqlite")+def _inet_sqlite(_type_, _compiler_, **_kw_):+ return "VARCHAR(45)"+++class FakeVectorBackend:+ """Qdrant and an embedding backend, behind a real httpx transport.++ A deterministic embedding: the vector is a bag of character counts, so+ two similar strings land near each other and the search below ranks for+ a reason rather than by insertion order. Crude, and that is the point --+ the test is about the plumbing and the filter, not about retrieval+ quality, which ``scripts/score_memory_recall.py`` measures properly.+ """++ def __init__(self, *, embed_dim: int = 1536) -> None:+ self.embed_dim = embed_dim+ self.points: dict[str, dict] = {}+ self.searches: list[dict] = []+ self.embed_calls = 0+ self.collection_created = False++ def _vector(self, text: str) -> list[float]:+ vec = [0.0] * self.embed_dim+ for index, ch in enumerate(text.lower()[:400]):+ vec[(ord(ch) * 7 + index) % self.embed_dim] += 1.0+ return vec++ def handler(self, request: httpx.Request) -> httpx.Response:+ path = request.url.path+ if path.endswith("/embeddings"):+ self.embed_calls += 1+ body = request.read().decode()+ import json as _json++ inputs = _json.loads(body)["input"]+ return httpx.Response(200, json={"data": [{"embedding": self._vector(t)} for t in inputs]})+ if path == f"/collections/{svc.COLLECTION}" and request.method == "GET":+ if not self.collection_created:+ return httpx.Response(404, json={})+ return httpx.Response(+ 200,+ json={+ "result": {+ "config": {"params": {"vectors": {"size": self.embed_dim}}},+ "points_count": len(self.points),+ }+ },+ )+ if path == f"/collections/{svc.COLLECTION}" and request.method == "PUT":+ self.collection_created = True+ return httpx.Response(200, json={"result": True})+ if path == f"/collections/{svc.COLLECTION}/points":+ import json as _json++ for point in _json.loads(request.read().decode())["points"]:+ self.points[str(point["id"])] = point+ return httpx.Response(200, json={"result": {"status": "completed"}})+ if path == f"/collections/{svc.COLLECTION}/points/delete":+ import json as _json++ for pid in _json.loads(request.read().decode())["points"]:+ self.points.pop(str(pid), None)+ return httpx.Response(200, json={"result": {"status": "completed"}})+ if path == f"/collections/{svc.COLLECTION}/points/search":+ import json as _json++ body = _json.loads(request.read().decode())+ self.searches.append(body)+ wanted = {c["key"]: c["match"]["value"] for c in body["filter"]["must"] if "value" in c.get("match", {})}+ results = []+ for point in self.points.values():+ payload = point["payload"]+ if any(payload.get(k) != v for k, v in wanted.items()):+ continue+ score = sum(a * b for a, b in zip(point["vector"], body["vector"], strict=False))+ results.append({"id": point["id"], "score": score, "payload": payload})+ results.sort(key=lambda r: -r["score"])+ return httpx.Response(200, json={"result": results[: body["limit"]]})+ return httpx.Response(404, json={})+++def backend(monkeypatch):+ """Point the service at the fake, through a real transport."""+ fake = FakeVectorBackend()+ transport = httpx.MockTransport(fake.handler)+ original = httpx.AsyncClient++ def _client(*args, **kwargs):+ kwargs["transport"] = transport+ return original(*args, **kwargs)++ monkeypatch.setattr(svc.httpx, "AsyncClient", _client)+ monkeypatch.setenv("OPENAI_BASE_URL", "http://embeddings.test/v1")+ monkeypatch.setattr(svc, "DEFAULT_EMBEDDING_MODEL", "test-embed")+ monkeypatch.setattr(svc.settings, "QDRANT_URL", "http://qdrant.test", raising=False)+ return fake+++@pytest_asyncio.fixture+async def session_factory():+ engine = create_async_engine("sqlite+aiosqlite:///:memory:")+ async with engine.begin() as conn:+ await conn.run_sync(Base.metadata.create_all, tables=[MemoryItem.__table__, AuditLog.__table__])+ yield async_sessionmaker(engine, expire_on_commit=False)+ await engine.dispose()+++def _principal(tenant: uuid.UUID = TENANT, key_prefix: str = "aisoc_analyst") -> CurrentUser:+ return CurrentUser(+ user_id=ANALYST,+ tenant_id=tenant,+ role="analyst",+ email="[email protected]",+ scopes=["*"],+ api_key_prefix=key_prefix,+ )+++@pytest_asyncio.fixture+async def client(session_factory):+ app = FastAPI()+ app.include_router(memory_router, prefix="/api/v1")+ state = {"user": _principal()}++ async def _db():+ async with session_factory() as session:+ yield session++ app.dependency_overrides[get_db] = _db+ app.dependency_overrides[get_tenant_db] = _db+ app.dependency_overrides[get_current_user] = lambda: state["user"]+ with TestClient(app) as c:+ c.state_holder = state # type: ignore[attr-defined]+ yield c+++def _ingest(client, subject: str, content: str, **kw):+ return client.post(+ "/api/v1/memory/items",+ json={"kind": kw.pop("kind", "memory"), "subject": subject, "content": content, **kw},+ )+++# ---------------------------------------------------------------------------+# Provenance+# ---------------------------------------------------------------------------+++async def test_the_author_is_the_principal_not_the_body(client, backend) -> None:+ res = _ingest(client, "10.40.0.0/16", "The card payment estate.", source="import")+ assert res.status_code == 201, res.text+ item = res.json()["item"]+ # The caller chose `source`, which is a fact about where the content came+ # from. It did not get to choose `author`, which is a claim about who is+ # standing behind it.+ assert item["source"] == "import"+ assert item["author"] == "aisoc_analyst"+ assert item["scope"] == "tenant"+ assert item["state"] == "in_use"+ assert res.json()["indexed"] is True+ assert item["embedding_model"] == "test-embed"+ assert item["embedded_at"] is not None+++async def test_re_ingesting_the_same_content_is_idempotent_and_does_not_re_embed(client, backend) -> None:+ _ingest(client, "10.40.0.0/16", "The card payment estate.")+ calls_after_first = backend.embed_calls+ again = _ingest(client, "10.40.0.0/16", "The card payment estate.")+ assert again.json()["outcome"] == "unchanged"+ # Re-embedding a vector that cannot have changed spends a model call per+ # re-ingest, and a nightly re-sync would spend one per item per night.+ assert backend.embed_calls == calls_after_first+++# ---------------------------------------------------------------------------+# Conflict review+# ---------------------------------------------------------------------------+++async def test_a_contradiction_suspends_both_sides_and_recall_returns_neither(client, backend) -> None:+ first = _ingest(client, "10.40.0.0/16", "The card payment estate, PCI in scope.").json()["item"]+ second = _ingest(client, "10.40.0.0/16", "Decommissioned in March; nothing lives there.")+ assert second.json()["outcome"] == "conflict"++ conflicts = client.get("/api/v1/memory/conflicts").json()+ assert len(conflicts) == 1+ pair = {conflicts[0]["left"]["id"], conflicts[0]["right"]["id"]}+ assert first["id"] in pair++ # Neither side is recalled. Preferring the newer one would silently undo+ # an analyst's correction the next time the job that wrote the bad entry+ # ran again, which is the case the feature exists for.+ hits = client.post("/api/v1/memory/recall", json={"query": "card payment estate"}).json()+ assert hits == []+++async def test_resolving_a_conflict_restores_exactly_one_side(client, backend) -> None:+ first = _ingest(client, "10.40.0.0/16", "The card payment estate, PCI in scope.").json()["item"]+ second = _ingest(client, "10.40.0.0/16", "Decommissioned in March.").json()["item"]++ resolved = client.post(+ f"/api/v1/memory/conflicts/{second['id']}/resolve",+ json={"keep": first["id"], "note": "the decommission was a different range"},+ )+ assert resolved.status_code == 200, resolved.text++ states = {row["id"]: row["state"] for row in client.get("/api/v1/memory/items").json()}+ assert states[first["id"]] == "in_use"+ assert states[second["id"]] == "not_in_use"+ assert client.get("/api/v1/memory/conflicts").json() == []++ hits = client.post("/api/v1/memory/recall", json={"query": "card payment estate PCI"}).json()+ assert [h["item"]["id"] for h in hits] == [first["id"]]+++async def test_resolving_must_name_one_of_the_two(client, backend) -> None:+ first = _ingest(client, "subject-a", "one").json()["item"]+ second = _ingest(client, "subject-a", "two").json()["item"]+ unrelated = _ingest(client, "subject-b", "three").json()["item"]+ res = client.post(f"/api/v1/memory/conflicts/{second['id']}/resolve", json={"keep": unrelated["id"]})+ assert res.status_code == 422+ assert first["id"] not in res.text or "one of the two" in res.text+++# ---------------------------------------------------------------------------+# Retirement, expiry, isolation+# ---------------------------------------------------------------------------+++async def test_retirement_keeps_the_row_and_its_provenance(client, backend, session_factory) -> None:+ item = _ingest(client, "subject-a", "a claim").json()["item"]+ res = client.delete(f"/api/v1/memory/items/{item['id']}?reason=wrong")+ assert res.status_code == 200+ assert res.json()["state"] == "not_in_use"+ assert res.json()["author"] == "aisoc_analyst"++ async with session_factory() as session:+ row = (await session.execute(select(MemoryItem).where(MemoryItem.id == uuid.UUID(item["id"])))).scalar_one()+ # "What did the agent know when it decided that" has to stay+ # answerable after somebody decides the item was wrong.+ assert row.content == "a claim"+ assert row.state_reason == "wrong"+++async def test_an_expired_item_is_not_recalled_without_a_sweep_job(client, backend, session_factory) -> None:+ item = _ingest(client, "subject-a", "temporary maintenance window").json()["item"]+ async with session_factory() as session:+ row = (await session.execute(select(MemoryItem).where(MemoryItem.id == uuid.UUID(item["id"])))).scalar_one()+ row.expires_at = datetime.now(UTC) - timedelta(minutes=1)+ await session.commit()+ # Expiry is applied at read, so a worker being down cannot resurrect it.+ assert client.post("/api/v1/memory/recall", json={"query": "maintenance window"}).json() == []+++async def test_recall_is_scoped_in_the_filter_itself(client, backend) -> None:+ _ingest(client, "subject-a", "tenant one's claim")+ client.post("/api/v1/memory/recall", json={"query": "claim"})+ assert backend.searches, "no search reached the vector store"+ must = backend.searches[-1]["filter"]["must"]+ assert {"key": "tenant_id", "match": {"value": str(TENANT)}} in must+ # `state` is matched in the store as well as in Postgres: a point whose+ # row was retired but whose delete failed would otherwise be recalled and+ # discarded late.+ assert {"key": "state", "match": {"value": "in_use"}} in must+++async def test_another_tenant_sees_none_of_it(client, backend) -> None:+ _ingest(client, "subject-a", "tenant one's claim")+ client.state_holder["user"] = _principal(OTHER_TENANT, "aisoc_other") # type: ignore[attr-defined]+ assert client.post("/api/v1/memory/recall", json={"query": "claim"}).json() == []+ assert client.get("/api/v1/memory/items").json() == []+++async def test_search_refuses_to_run_unscoped() -> None:+ """No `tenant_id=None` branch. Nothing here is shared, so there is no caller."""+ with pytest.raises(ValueError, match="scoped to a tenant"):+ await svc.search("anything", tenant_id=None) # type: ignore[arg-type]+++# ---------------------------------------------------------------------------+# Honest unavailability+# ---------------------------------------------------------------------------+++async def test_no_embedding_backend_is_503_with_a_reason_not_an_empty_list(client, monkeypatch) -> None:+ monkeypatch.delenv("OPENAI_BASE_URL", raising=False)+ monkeypatch.delenv("LLM_BASE_URL", raising=False)+ monkeypatch.delenv("LLM_GATEWAY_URL", raising=False)+ monkeypatch.delenv("OLLAMA_BASE_URL", raising=False)+ monkeypatch.setattr(svc, "embedding_backend", lambda: None)+ res = client.post("/api/v1/memory/recall", json={"query": "anything"})+ # An empty list here would render as "we searched your memory and found+ # nothing", which is a confident claim about a search that never ran.+ assert res.status_code == 503+ assert "No embedding backend is configured" in res.json()["detail"]+++async def test_an_item_stored_but_not_indexed_says_so(client, monkeypatch) -> None:+ monkeypatch.setattr(svc, "embedding_backend", lambda: None)+ res = _ingest(client, "subject-a", "a claim nobody can recall yet")+ assert res.status_code == 201+ body = res.json()+ assert body["indexed"] is False+ assert "No embedding backend" in body["index_error"]+ assert body["item"]["embedding_model"] is None+++def test_an_oversized_embedding_is_refused_rather_than_truncated() -> None:+ """Truncating measured as a worse retriever that nobody could see.++ Scored over the probe set with the 3072-dimension model available+ locally, truncation put recall@1 at 0.167 against a 0.667 lexical+ baseline -- a number measuring the truncation, not the model.+ """+ with pytest.raises(svc.EmbeddingsUnavailable, match="3072-dimension"):+ svc._fit([0.1] * 3072)+ # Padding is harmless under cosine and stays allowed.+ assert len(svc._fit([0.1] * 768)) == svc.EMBEDDING_DIM+++async def test_status_distinguishes_never_embedded_from_embedded_by_another_model(client, backend, session_factory) -> None:+ item = _ingest(client, "subject-a", "a claim").json()["item"]+ _ingest(client, "subject-b", "another claim")+ async with session_factory() as session:+ row = (await session.execute(select(MemoryItem).where(MemoryItem.id == uuid.UUID(item["id"])))).scalar_one()+ row.embedding_model = "a-model-nobody-runs-now"+ await session.commit()++ status = client.get("/api/v1/memory/status").json()+ # Two different states calling for two different actions. Collapsing+ # them into one "needs attention" number is how an operator ends up+ # re-embedding rows that were never embedded in the first place.+ assert status["in_use_embedded_by_another_model"] == 1+ assert status["in_use_never_embedded"] == 0+ assert status["items_by_state"]["in_use"] == 2+ assert status["recall_available"] is True+++async def test_status_says_recall_is_unavailable_rather_than_reporting_zeroes(client, monkeypatch) -> None:+ monkeypatch.setattr(svc, "embedding_backend", lambda: None)+ status = client.get("/api/v1/memory/status").json()+ assert status["recall_available"] is False+ assert status["embedding_backend"] is None+ # Not 0 — a dimension nobody could ask about is unknown, not zero.+ assert status["collection_dimension"] is None+++# ---------------------------------------------------------------------------+# The automatic writer+# ---------------------------------------------------------------------------+++async def test_closing_a_case_writes_a_memory_item(backend, session_factory) -> None:+ """``kind='case'`` would be permanently empty without this caller."""+ case_id = uuid.uuid4()+ async with session_factory() as session:+ item_id = await svc.remember_closed_case(+ session,+ tenant_id=TENANT,+ case_id=case_id,+ title="Ransomware on the file server",+ summary="Closed as a true positive; the appliance was not in the asset register.",+ author="[email protected]",+ )+ await session.commit()+ assert item_id is not None+ async with session_factory() as session:+ row = (await session.execute(select(MemoryItem).where(MemoryItem.id == item_id))).scalar_one()+ assert row.kind == "case"+ assert row.subject == f"case:{case_id}"+ assert row.source == "case_closure"+ assert row.embedding_model == "test-embed"+++async def test_a_case_closed_with_no_narrative_writes_nothing(backend, session_factory) -> None:+ """Padding recall with rows whose content is a title makes it answer worse."""+ async with session_factory() as session:+ assert (+ await svc.remember_closed_case(+ session,+ tenant_id=TENANT,+ case_id=uuid.uuid4(),+ title="A case",+ summary=" ",+ author="[email protected]",+ )+ is None+ )diff --git a/tests/isolation/test_postgres_rls.py b/tests/isolation/test_postgres_rls.pyindex 54f5a7cba..fba423cc2 100644--- a/tests/isolation/test_postgres_rls.py+++ b/tests/isolation/test_postgres_rls.py@@ -135,6 +135,7 @@ def _literal_for(nonce: str,allowed: list[str] | None = None,max_len: int | None = None,+ exact_len: int | None = None,) -> str:"""A syntactically valid value for a NOT NULL column with no default.@@ -152,9 +153,20 @@ def _literal_for(``identity_nodes.node_type`` and four others accept a fixed vocabulary.* ``max_len`` respects ``varchar(n)``; ``aisoc_attack_chains`` has a``varchar(8)``.+ * ``exact_len`` carries a ``char_length(col) = n`` check, which a hex+ digest column has and which the short filler cannot satisfy. Taught+ to the seeder rather than relaxed in the schema: a constraint+ loosened so a prober can insert is a constraint that stops being one."""if allowed:return f"'{allowed[0]}'"+ if exact_len is not None and data_type in ("character varying", "text", "character"):+ # A column whose CHECK pins an exact length -- a hex digest, a+ # country code -- cannot be seeded with the short filler below. The+ # seeder is taught the constraint rather than the constraint being+ # relaxed for the seeder's benefit, which is the direction that+ # keeps the invariant.+ return "'" + ("a" * exact_len) + "'"if data_type == "uuid":return f"'{uuid.uuid4()}'::uuid"if data_type in ("character varying", "text", "character"):@@ -208,6 +220,35 @@ async def _protected_tables(conn) -> list[str]:return [r["t"] for r in rows]+#: `char_length(col) = n`, with or without spaces. A column pinned to an+#: exact length -- a hex digest, an ISO country code -- cannot be seeded with+#: the short filler, and teaching the seeder is the direction that keeps the+#: invariant; relaxing the schema so a prober can insert would make the+#: constraint stop being one.+_EXACT_LEN = re.compile(r"char_length\s*\(\s*\(?\s*[\w\".]+\s*\)?\s*\)\s*=\s*(\d+)")+++async def _check_exact_lengths(conn, table: str) -> dict[str, int]:+ """column → the exact character length its CHECK constraint demands."""+ rows = await conn.fetch(+ """+ SELECT a.attname AS col, pg_get_constraintdef(c.oid) AS def+ FROM pg_constraint c+ JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = c.conkey[1]+ WHERE c.contype = 'c'+ AND c.conrelid = to_regclass('public.' || quote_ident($1))+ AND array_length(c.conkey, 1) = 1+ """,+ table,+ )+ out: dict[str, int] = {}+ for row in rows:+ match = _EXACT_LEN.search(row["def"])+ if match:+ out[row["col"]] = int(match.group(1))+ return out++async def _check_vocabularies(conn, table: str) -> dict[str, list[str]]:"""column → literals its CHECK constraint permits.@@ -251,6 +292,7 @@ async def _seed_row(conn, table: str, tenant: uuid.UUID, enum_labels: dict[str,table,)vocab = await _check_vocabularies(conn, table)+ exact = await _check_exact_lengths(conn, table)tenant_type = await conn.fetchval("""SELECT format_type(a.atttypid, a.atttypmod)@@ -272,6 +314,7 @@ async def _seed_row(conn, table: str, tenant: uuid.UUID, enum_labels: dict[str,nonce=nonce,allowed=vocab.get(col["column_name"]),max_len=col["character_maximum_length"],+ exact_len=exact.get(col["column_name"]),)if literal == "DEFAULT":continue