Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat: add graph retrieval and memory maintenance
AkitaOnRails43be7a92f95e
diff --git a/README.md b/README.mdindex fdf273d6..2d22a185 100644--- a/README.md+++ b/README.md@@ -71,7 +71,7 @@ priors are at the [bottom](#influences-and-prior-art).terminal) - FTS5 over the wiki. Pages are LLM-consolidated, sothe hit is a coherent decision page, not a raw chat log.- **"This new project has months of history before ai-memory."**- `cd ~/Projects/<that-project> && ai-memory bootstrap` collects+ `cd /path/to/my-project && ai-memory bootstrap` collects`git log`, README, `docs/`, module headers, project rules andone-shot-summarises them into seed wiki pages. Future sessionsbuild on top.@@ -186,6 +186,13 @@ wired up alone. Set `AI_MEMORY_NO_VERSION_CHECK=1` to silence thedaily check, or `AI_MEMORY_WRAPPER_URL=<url>` to pin the self-upgradesource (e.g. a fork or a tagged release).+After the upgraded server starts, ai-memory applies SQLite schema+migrations automatically (`Store::open`) and runs pending wiki-structure+migrations before the watcher starts. No manual database reset or wiki+rewrite is required for normal upgrades; derived indexes such as FTS+tables are rebuilt/backfilled by migrations or the watcher reconciliation+path.+> **If your server runs on a different host** (scenarios C/D), `ai-memory upgrade`> only refreshes the local wrapper, your local image, and your hook scripts. You> still need to redeploy the server separately — run `bin/deploy` on the homelab@@ -476,7 +483,7 @@ snippet is installed (see "Nudge the agent" below):| You say… | Agent calls… | Effect ||---|---|---|-| "Have we discussed X?" / "search memory for Y" | `memory_query` | FTS5 over consolidated wiki pages; returns ranked snippets with `<mark>`-highlighted hits. |+| "Have we discussed X?" / "search memory for Y" | `memory_query` | FTS5 + link-neighbour RRF over compiled wiki pages, optional vector RRF, and bounded raw observation fallback when pages miss. || (before proposing architecture) implicit | `memory_query` | The routing snippet tells the agent to check prior decisions / gotchas before proposing anything, so you don't get contradicted-by-its-own-history suggestions. || "Catch me up" / "I've been away" | `memory_explore` | Prose digest. Verbosity auto-scales to time-since-last-activity: < 1 h → one line, > 30 days → full catchup. || "Where did we leave off?" | (handoff already prepended) | SessionStart hook already prepended the handoff before your first prompt; the agent just reads from that block. |@@ -656,8 +663,8 @@ OPENAI_API_KEY=ollama-local```Skipping the embedding provider entirely is fine --`memory_query` falls back to pure FTS5 (BM25) and still-works; you just lose vector re-ranking.+`memory_query` still uses FTS5, link-neighbour expansion, and raw+fallback; you just lose vector re-ranking.Per-tier feature breakdown + the openai-compat / Ollama setupis in [`docs/install.md`](docs/install.md#llm-provider-tiers).@@ -682,10 +689,12 @@ ingress. The server queues writes through a single SQLite writer(no `database is locked`). On session end an optional LLM-driven passrewrites pages atomically with supersession (`is_latest=false` +`supersedes` chain) and opens a typed handoff for the next agent.-The retention sweep decays unused episodic content while semantic-concept pages compound forever; pinned pages are exempt. Retrieval-is FTS5 by default; when an embedder is configured, hybrid RRF over-`page_embeddings` joins the FTS5 ranks.+The scheduled retention sweep decays unused episodic content while+semantic concept pages compound forever; pinned pages and `_slots/`+memory slots are exempt. Retrieval is FTS5 + link-neighbour RRF by+default; when an embedder is configured, vector RRF over+`page_embeddings` joins the ranks. If compiled wiki search misses,+bounded raw observation FTS provides exact-detail fallback.See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the canonicaldata-flow diagram + crate breakdown + cross-cutting invariants.diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex 78de7c54..47494ce9 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -2,12 +2,12 @@use std::sync::Arc;-use ai_memory_consolidate::Consolidator;+use ai_memory_consolidate::{Consolidator, run_lint, run_sweep};use ai_memory_core::{ActiveProject, ProjectId, Sanitizer, WorkspaceId};use ai_memory_hooks::{HookState, hook_router};use ai_memory_llm::{Embedder, LlmProvider, build_embedder, build_provider};use ai_memory_mcp::{AdminState, AiMemoryServer, admin_router};-use ai_memory_store::{ReaderPool, Store};+use ai_memory_store::{ReaderPool, Store, WriterHandle, f32_vec_to_bytes};use ai_memory_web;use ai_memory_wiki::{WatcherHandle, Wiki, migrations, run_wiki_migrations};use anyhow::{Context, Result};@@ -26,7 +26,7 @@ use tracing::info;use crate::auth::{AuthState, require_bearer};use crate::cli::{ServeArgs, TransportKind};-use crate::config::Config;+use crate::config::{Config, MaintenanceSettings};/// 10 MB cap on inbound HTTP bodies. The /hook ingress accepts the/// agent's raw payload which can include a tool output excerpt@@ -101,6 +101,17 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {let server = consolidator_setup.server;let consolidator = consolidator_setup.consolidator;let admin_llm = consolidator_setup.admin_llm;+ let _maintenance_tasks = start_maintenance_scheduler(+ config.maintenance.clone(),+ store.reader.clone(),+ store.writer.clone(),+ wiki.clone(),+ embedder.clone(),+ admin_llm.clone(),+ ws,+ proj,+ config.decay,+ );match args.transport {TransportKind::Stdio => {@@ -211,6 +222,183 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {Ok(())}+#[allow(clippy::too_many_arguments)]+fn start_maintenance_scheduler(+ settings: MaintenanceSettings,+ reader: ReaderPool,+ writer: WriterHandle,+ wiki: Wiki,+ embedder: Option<Arc<dyn Embedder>>,+ llm: Option<Arc<dyn LlmProvider>>,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ decay: ai_memory_store::DecayParams,+) -> Vec<tokio::task::JoinHandle<()>> {+ if !settings.enabled {+ info!("scheduled maintenance disabled");+ return Vec::new();+ }++ let forget_sweep_interval_secs = settings.forget_sweep_interval_secs;+ let lint_interval_secs = settings.lint_interval_secs;+ let embedding_backfill_interval_secs = settings.embedding_backfill_interval_secs;++ let mut tasks = Vec::new();+ if forget_sweep_interval_secs > 0 {+ let reader = reader.clone();+ let writer = writer.clone();+ tasks.push(tokio::spawn(async move {+ let interval = std::time::Duration::from_secs(forget_sweep_interval_secs);+ loop {+ tokio::time::sleep(interval).await;+ match run_sweep(&reader, &writer, workspace_id, project_id, &decay, false).await {+ Ok(report) => info!(+ evicted = report.evicted.len(),+ hard_deleted = report.hard_deleted,+ "scheduled forget sweep completed"+ ),+ Err(e) => tracing::warn!(error = %e, "scheduled forget sweep failed"),+ }+ }+ }));+ }++ if lint_interval_secs > 0 {+ let reader = reader.clone();+ let wiki = wiki.clone();+ let llm = llm.clone();+ tasks.push(tokio::spawn(async move {+ let interval = std::time::Duration::from_secs(lint_interval_secs);+ loop {+ tokio::time::sleep(interval).await;+ match run_lint(+ &reader,+ &wiki,+ llm.as_ref(),+ workspace_id,+ project_id,+ false,+ false,+ )+ .await+ {+ Ok(report) => info!(+ findings = report.findings.len(),+ "scheduled rule-based lint completed"+ ),+ Err(e) => tracing::warn!(error = %e, "scheduled lint failed"),+ }+ }+ }));+ }++ if embedding_backfill_interval_secs > 0 {+ if let Some(embedder) = embedder {+ let reader = reader.clone();+ let writer = writer.clone();+ let wiki = wiki.clone();+ tasks.push(tokio::spawn(async move {+ let interval = std::time::Duration::from_secs(embedding_backfill_interval_secs);+ loop {+ tokio::time::sleep(interval).await;+ match run_embedding_backfill(+ &reader,+ &writer,+ &wiki,+ &embedder,+ workspace_id,+ project_id,+ )+ .await+ {+ Ok((embedded, failed)) => {+ info!(embedded, failed, "scheduled embedding backfill completed")+ }+ Err(e) => tracing::warn!(error = %e, "scheduled embedding backfill failed"),+ }+ }+ }));+ } else {+ tracing::warn!(+ "maintenance.embedding_backfill_interval_secs is set but no embedder is configured"+ );+ }+ }++ if tasks.is_empty() {+ info!("scheduled maintenance enabled but all intervals are disabled");+ } else {+ info!(jobs = tasks.len(), "scheduled maintenance started");+ }+ tasks+}++async fn run_embedding_backfill(+ reader: &ReaderPool,+ writer: &WriterHandle,+ wiki: &Wiki,+ embedder: &Arc<dyn Embedder>,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+) -> Result<(usize, usize)> {+ let provider = embedder.provider().to_string();+ let model = embedder.model().to_string();+ let dim = embedder.dim();+ let candidates = reader.decay_candidates(workspace_id, project_id).await?;+ let already: std::collections::HashSet<_> = reader+ .load_embeddings(+ workspace_id,+ project_id,+ provider.clone(),+ model.clone(),+ dim,+ )+ .await?+ .into_iter()+ .map(|s| s.id)+ .collect();++ let mut embedded = 0usize;+ let mut failed = 0usize;+ for cand in candidates {+ if already.contains(&cand.id) {+ continue;+ }+ let md = match wiki.read_page(workspace_id, project_id, &cand.path) {+ Ok(md) => md,+ Err(e) => {+ failed += 1;+ tracing::warn!(path = %cand.path, error = %e, "scheduled embed: unreadable page");+ continue;+ }+ };+ let vec = match embedder.embed(&md.body).await {+ Ok(vec) => vec,+ Err(e) => {+ failed += 1;+ tracing::warn!(path = %cand.path, error = %e, "scheduled embed: provider failed");+ continue;+ }+ };+ if let Err(e) = writer+ .store_embedding(+ cand.id,+ f32_vec_to_bytes(&vec),+ provider.clone(),+ model.clone(),+ dim,+ )+ .await+ {+ failed += 1;+ tracing::warn!(path = %cand.path, error = %e, "scheduled embed: store failed");+ continue;+ }+ embedded += 1;+ }+ Ok((embedded, failed))+}+async fn configure_embedder(config: &Config,store: &Store,diff --git a/crates/ai-memory-cli/src/commands/status.rs b/crates/ai-memory-cli/src/commands/status.rsindex f7a16522..7a49eec2 100644--- a/crates/ai-memory-cli/src/commands/status.rs+++ b/crates/ai-memory-cli/src/commands/status.rs@@ -5,14 +5,14 @@//! the store directly — the server is the source of truth.use anyhow::Result;-use serde::Deserialize;+use serde::{Deserialize, Serialize};use crate::cli::StatusArgs;use crate::config::Config;use crate::http_client::{ServerEndpoint, get_json};/// Server-shaped response. Mirrors `ai_memory_mcp::admin::StatusReport`.-#[derive(Debug, Deserialize)]+#[derive(Debug, Deserialize, Serialize)]struct Report {/// Server binary version.version: String,@@ -24,9 +24,12 @@ struct Report {db_path: String,/// Lifetime counts.counts: Counts,+ /// Derived-index diagnostics.+ #[serde(default)]+ derived: Derived,}-#[derive(Debug, Deserialize)]+#[derive(Debug, Deserialize, Serialize)]struct Counts {pages_latest: u64,pages_all: u64,@@ -34,6 +37,28 @@ struct Counts {observations: u64,}+#[derive(Debug, Default, Deserialize, Serialize)]+struct Derived {+ pages_rows: u64,+ pages_fts_rows: u64,+ observations_rows: u64,+ observations_fts_rows: u64,+ latest_pages_missing_embeddings: u64,+ embedding_rows: u64,+ embedding_triples: Vec<EmbeddingTriple>,+ links_from_latest_pages: u64,+ unresolved_links_from_latest_pages: u64,+ stale_links_from_latest_pages: u64,+}++#[derive(Debug, Deserialize, Serialize)]+struct EmbeddingTriple {+ provider: String,+ model: String,+ dim: u32,+ count: u64,+}+/// Run the `status` subcommand.////// # Errors@@ -57,6 +82,7 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {"sessions": report.counts.sessions,"observations": report.counts.observations,},+ "derived": report.derived,"client": { "server_url": ep.url, "auth": ep.auth_token.is_some() },}))?);@@ -72,6 +98,23 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {);println!(" sessions: {}", report.counts.sessions);println!(" observations: {}", report.counts.observations);+ println!(+ " fts: pages {}/{}; observations {}/{}",+ report.derived.pages_fts_rows,+ report.derived.pages_rows,+ report.derived.observations_fts_rows,+ report.derived.observations_rows+ );+ println!(+ " embeddings: {} rows; {} latest pages missing",+ report.derived.embedding_rows, report.derived.latest_pages_missing_embeddings+ );+ println!(+ " links: {} latest-page links (unresolved: {}, stale: {})",+ report.derived.links_from_latest_pages,+ report.derived.unresolved_links_from_latest_pages,+ report.derived.stale_links_from_latest_pages+ );}Ok(())}diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex b97b631a..4b71dd34 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -72,6 +72,8 @@ pub struct Config {/// hard-deletion. Tune `decay.lambda` down to slow decay or/// `decay.cold_threshold` to evict more / less aggressively.pub decay: ai_memory_store::DecayParams,+ /// Server-side scheduled maintenance. Jobs run outside hook latency.+ pub maintenance: MaintenanceSettings,/// Privacy-strip tuning. Built-in patterns always run; this section/// lets the operator extend or punch holes in them.pub sanitize: ai_memory_core::SanitizeConfig,@@ -190,6 +192,7 @@ impl Default for Config {embedding_dim: None,embedding_base_url: None,decay: ai_memory_store::DecayParams::default(),+ maintenance: MaintenanceSettings::default(),sanitize: ai_memory_core::SanitizeConfig::default(),auth: AuthSettings::default(),allowed_hosts: vec!["localhost".into(), "127.0.0.1".into(), "::1".into()],@@ -198,6 +201,32 @@ impl Default for Config {}}+/// `[maintenance]` scheduled server jobs.+#[derive(Debug, Clone, Serialize, Deserialize)]+#[serde(default)]+pub struct MaintenanceSettings {+ /// Master switch for scheduled jobs.+ pub enabled: bool,+ /// Interval for the retention forget sweep. `0` disables this job.+ pub forget_sweep_interval_secs: u64,+ /// Interval for rule-based wiki lint. `0` disables this job.+ pub lint_interval_secs: u64,+ /// Interval for embedding backfill. `0` disables this job.+ /// Defaults to off because it may call a paid provider.+ pub embedding_backfill_interval_secs: u64,+}++impl Default for MaintenanceSettings {+ fn default() -> Self {+ Self {+ enabled: true,+ forget_sweep_interval_secs: 86_400,+ lint_interval_secs: 86_400,+ embedding_backfill_interval_secs: 0,+ }+ }+}+impl Config {/// Load the merged configuration: defaults → file → env → CLI.///@@ -421,6 +450,10 @@ mod tests {assert_eq!(cfg.bind, DEFAULT_BIND);assert_eq!(cfg.server_url, DEFAULT_SERVER_URL);assert_eq!(cfg.log_level, "info");+ assert!(cfg.maintenance.enabled);+ assert_eq!(cfg.maintenance.forget_sweep_interval_secs, 86_400);+ assert_eq!(cfg.maintenance.lint_interval_secs, 86_400);+ assert_eq!(cfg.maintenance.embedding_backfill_interval_secs, 0);}#[test]@@ -449,6 +482,10 @@ mod tests {r#"bind = "0.0.0.0:9999"log_level = "debug"++ [maintenance]+ enabled = false+ lint_interval_secs = 3600"#,).unwrap();@@ -458,5 +495,7 @@ mod tests {let cfg = Config::load(Some(&cfg_path), Some(tmp.path().to_path_buf())).unwrap();assert_eq!(cfg.bind, "0.0.0.0:9999");assert_eq!(cfg.log_level, "debug");+ assert!(!cfg.maintenance.enabled);+ assert_eq!(cfg.maintenance.lint_interval_secs, 3600);}}diff --git a/crates/ai-memory-consolidate/tests/recall_eval.rs b/crates/ai-memory-consolidate/tests/recall_eval.rsindex 9ad526bb..229940c2 100644--- a/crates/ai-memory-consolidate/tests/recall_eval.rs+++ b/crates/ai-memory-consolidate/tests/recall_eval.rs@@ -1,8 +1,9 @@//! Recall@5 eval harness.//!//! Loads a small hand-crafted corpus + a probe set, measures recall@5-//! against both the pure-FTS5 path and the hybrid (FTS5 + RRF + vector)-//! path, and asserts a baseline. The point is *the framework*: once the+//! against both the pure-FTS5 path and the hybrid (FTS5 + graph RRF ++//! vector) path, and asserts a baseline. It also pins graph-neighbor+//! expansion and raw observation fallback. The point is *the framework*: once the//! harness is in CI, anyone can drop in real embeddings (set the//! `AI_MEMORY_EMBEDDING_PROVIDER` env vars + plug in a real model) and//! watch the numbers move. The synthetic embedder shipped for tests is@@ -15,7 +16,9 @@use std::sync::Arc;-use ai_memory_core::{PagePath, Tier};+use ai_memory_core::{+ AgentKind, NewObservation, NewSession, ObservationKind, PagePath, SessionId, Tier,+};use ai_memory_llm::{Embedder, SyntheticEmbedder};use ai_memory_store::Store;use ai_memory_wiki::{Wiki, WritePageRequest};@@ -161,6 +164,139 @@ async fn recall_at_5_baseline() {);}+#[tokio::test]+async fn graph_neighbor_expansion_recovers_linked_page() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("open store");+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .expect("ws");+ let proj = store+ .writer+ .get_or_create_project(ws, "eval", None)+ .await+ .expect("proj");+ let wiki = Wiki::new(tmp.path(), store.writer.clone()).expect("wiki");++ wiki.write_page(WritePageRequest {+ workspace_id: ws,+ project_id: proj,+ path: PagePath::new("notes/hidden_target.md").expect("path"),+ frontmatter: serde_json::json!({"title": "Hidden target"}),+ body: "This page contains implementation detail but not the seed term.".into(),+ tier: Tier::Semantic,+ pinned: false,+ title: None,+ })+ .await+ .expect("write target");+ wiki.write_page(WritePageRequest {+ workspace_id: ws,+ project_id: proj,+ path: PagePath::new("notes/source.md").expect("path"),+ frontmatter: serde_json::json!({"title": "Source"}),+ body: "graphseed points to [[notes/hidden_target]].".into(),+ tier: Tier::Semantic,+ pinned: false,+ title: None,+ })+ .await+ .expect("write source");++ let fts_hits = store+ .reader+ .search_pages_for_project(ws, proj, "graphseed".into(), 5)+ .await+ .expect("fts search");+ assert!(+ !fts_hits+ .iter()+ .any(|hit| hit.path.as_str() == "notes/hidden_target.md"),+ "pure FTS should not find the target without the seed term"+ );++ let hybrid_hits = store+ .reader+ .hybrid_search(+ ws,+ proj,+ "graphseed".into(),+ None,+ String::new(),+ String::new(),+ 0,+ 5,+ )+ .await+ .expect("hybrid search");+ assert!(+ hybrid_hits+ .iter()+ .any(|hit| hit.path.as_str() == "notes/hidden_target.md"),+ "graph expansion should include the linked target"+ );+}++#[tokio::test]+async fn raw_observation_fallback_recovers_detail_when_wiki_misses() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("open store");+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .expect("ws");+ let proj = store+ .writer+ .get_or_create_project(ws, "eval", None)+ .await+ .expect("proj");+ let session_id = SessionId::new();++ store+ .writer+ .begin_session(NewSession {+ id: session_id,+ workspace_id: ws,+ project_id: proj,+ agent_kind: AgentKind::OpenCode,+ cwd: None,+ })+ .await+ .expect("begin session");+ store+ .writer+ .insert_observation(NewObservation {+ session_id,+ workspace_id: ws,+ project_id: proj,+ kind: ObservationKind::UserPrompt,+ title: "raw prompt".into(),+ body: "raw fallback detail mentions capybara exactly once".into(),+ importance: 5,+ })+ .await+ .expect("insert observation");++ let page_hits = store+ .reader+ .search_pages_for_project(ws, proj, "capybara".into(), 5)+ .await+ .expect("page search");+ assert!(page_hits.is_empty(), "compiled wiki should miss");++ let raw_hits = store+ .reader+ .search_observations_for_project(ws, proj, "capybara".into(), 5)+ .await+ .expect("raw search");+ assert_eq!(raw_hits.len(), 1);+ assert_eq!(raw_hits[0].session_id, session_id);+ assert!(raw_hits[0].snippet.contains("<mark>capybara</mark>"));+}+async fn measure_recall(store: &Store,ws: ai_memory_core::WorkspaceId,diff --git a/crates/ai-memory-core/src/page.rs b/crates/ai-memory-core/src/page.rsindex deb0d755..e0559f41 100644--- a/crates/ai-memory-core/src/page.rs+++ b/crates/ai-memory-core/src/page.rs@@ -50,6 +50,11 @@ pub struct NewPage {pub frontmatter_json: serde_json::Value,/// `true` to mark the page as user-pinned (never decay).pub pinned: bool,+ /// Outgoing links discovered in the page body.+ ///+ /// The store resolves these against the latest page rows in the same+ /// project and keeps unresolved forward links as `to_page_id = NULL`.+ pub links: Vec<PagePath>,}/// Materialised view of a page row.diff --git a/crates/ai-memory-hooks/src/synth.rs b/crates/ai-memory-hooks/src/synth.rsindex 7e335f35..66d09575 100644--- a/crates/ai-memory-hooks/src/synth.rs+++ b/crates/ai-memory-hooks/src/synth.rs@@ -41,6 +41,7 @@ pub fn synthesize_session_page("tier": "episodic",}),pinned: false,+ links: Vec::new(),}}diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rsindex 9b7d6d19..2cbdae77 100644--- a/crates/ai-memory-mcp/src/admin.rs+++ b/crates/ai-memory-mcp/src/admin.rs@@ -235,23 +235,32 @@ pub struct StatusReport {pub db_path: String,/// Lifetime counts: pages_latest, pages_all, sessions, observations.pub counts: ai_memory_store::StatusCounts,+ /// Derived-index and retrieval-readiness diagnostics.+ pub derived: ai_memory_store::DerivedIndexStatus,}async fn handle_status(State(state): State<Arc<AdminState>>) -> impl IntoResponse {match state.reader.status_counts().await {- Ok(counts) => {- let report = StatusReport {- version: env!("CARGO_PKG_VERSION").to_string(),- data_dir: state.data_dir.display().to_string(),- bind: state.bind.clone(),- db_path: state.db_path.display().to_string(),- counts,- };- (- StatusCode::OK,- Json(serde_json::to_value(&report).unwrap_or_else(|_| serde_json::json!({}))),- )- }+ Ok(counts) => match state.reader.derived_index_status().await {+ Ok(derived) => {+ let report = StatusReport {+ version: env!("CARGO_PKG_VERSION").to_string(),+ data_dir: state.data_dir.display().to_string(),+ bind: state.bind.clone(),+ db_path: state.db_path.display().to_string(),+ counts,+ derived,+ };+ (+ StatusCode::OK,+ Json(serde_json::to_value(&report).unwrap_or_else(|_| serde_json::json!({}))),+ )+ }+ Err(e) => (+ StatusCode::INTERNAL_SERVER_ERROR,+ Json(serde_json::json!({ "error": e.to_string() })),+ ),+ },Err(e) => (StatusCode::INTERNAL_SERVER_ERROR,Json(serde_json::json!({ "error": e.to_string() })),diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rsindex 5702e02c..334b0f33 100644--- a/crates/ai-memory-mcp/src/server.rs+++ b/crates/ai-memory-mcp/src/server.rs@@ -153,6 +153,13 @@ struct QueryResponse<T: Serialize> {hits: Vec<T>,}+#[derive(Debug, Serialize)]+struct MemoryQueryResponse {+ hits: Vec<ai_memory_store::PageHit>,+ #[serde(skip_serializing_if = "Vec::is_empty")]+ raw_hits: Vec<ai_memory_store::ObservationHit>,+}+#[derive(Debug, Serialize)]struct StatusResponse {counts: ai_memory_store::StatusCounts,@@ -371,22 +378,25 @@ impl AiMemoryServer {self}- /// Full-text search the wiki via FTS5. Returns up to `limit` hits with- /// HTML-marked snippets and a rank score.+ /// Search the compiled wiki via FTS5/vector/graph retrieval. Falls back+ /// to bounded raw observation search when no compiled page matches.#[tool(description = "Search the project's long-term memory wiki — \prior sessions, decisions, gotchas, architecture notes captured \by ai-memory across earlier runs. Call this BEFORE proposing \designs, BEFORE answering 'why does X work this way', and \whenever the user references prior work you don't recognise. \- FTS5 + (when configured) hybrid RRF re-ranking. Returns up to \- `limit` pages with HTML-marked snippets and a rank score \- (lower rank = better match). Only latest page versions.")]+ FTS5 + graph RRF + (when configured) vector RRF re-ranking. \+ Returns up to `limit` pages with HTML-marked snippets and a rank \+ score (lower rank = better match). Only latest page versions. \+ If compiled wiki search misses, `raw_hits` contains bounded raw \+ observation fallback matches.")]async fn memory_query(&self,Parameters(args): Parameters<QueryArgs>,) -> Result<CallToolResult, McpError> {let limit = args.limit.unwrap_or(self.default_limit).clamp(1, 100);let (ws, proj) = self.effective_ids(args.project.as_deref()).await;+ let query = args.query.clone();// Hybrid path: try to embed the query; if it fails (provider// outage, cold-start timeout, billing, etc.) degrade to BM25// instead of erroring the whole query — losing semantic@@ -398,7 +408,7 @@ impl AiMemoryServer {.hybrid_search(ws,proj,- args.query.clone(),+ query.clone(),Some(qv),embedder.provider().to_string(),embedder.model().to_string(),@@ -415,18 +425,44 @@ impl AiMemoryServer {"embedder failed; degrading memory_query to BM25-only");self.reader- .search_pages_for_project(ws, proj, args.query, limit)+ .hybrid_search(+ ws,+ proj,+ query.clone(),+ None,+ String::new(),+ String::new(),+ 0,+ limit,+ ).await}}} else {self.reader- .search_pages_for_project(ws, proj, args.query, limit)+ .hybrid_search(+ ws,+ proj,+ query.clone(),+ None,+ String::new(),+ String::new(),+ 0,+ limit,+ ).await};let hits = hits.map_err(|e| McpError::internal_error(e.to_string(), None))?;self.spawn_access_bump(hits.iter().map(|h| h.id).collect());- let response = QueryResponse { hits };+ let raw_hits = if hits.is_empty() {+ self.reader+ .search_observations_for_project(ws, proj, query, limit)+ .await+ .map_err(|e| McpError::internal_error(e.to_string(), None))?+ } else {+ Vec::new()+ };+ let response = MemoryQueryResponse { hits, raw_hits };ok_json(&response)}@@ -920,7 +956,7 @@ const EXPLORE_SYSTEM_PROMPT: &str = include_str!("../prompts/explore_system.md")#[cfg(test)]mod tests {use super::*;- use ai_memory_core::{NewPage, PagePath, Tier};+ use ai_memory_core::{NewObservation, NewPage, NewSession, ObservationKind, PagePath, Tier};use ai_memory_store::Store;use tempfile::TempDir;@@ -948,6 +984,7 @@ mod tests {tier: Tier::Semantic,frontmatter_json: serde_json::json!({}),pinned: false,+ links: Vec::new(),}).await.unwrap();@@ -1019,6 +1056,58 @@ mod tests {assert!(text.contains("foo.md"), "expected hit; got {text}");}+ #[tokio::test]+ async fn memory_query_returns_raw_hits_when_pages_miss() {+ let (_tmp, store, server, ws, proj) = setup_server().await;+ let session_id = SessionId::new();+ store+ .writer+ .begin_session(NewSession {+ id: session_id,+ workspace_id: ws,+ project_id: proj,+ agent_kind: AgentKind::OpenCode,+ cwd: None,+ })+ .await+ .unwrap();+ store+ .writer+ .insert_observation(NewObservation {+ session_id,+ workspace_id: ws,+ project_id: proj,+ kind: ObservationKind::UserPrompt,+ title: "raw prompt".into(),+ body: "raw fallback contains quokka only detail".into(),+ importance: 5,+ })+ .await+ .unwrap();++ let result = server+ .memory_query(Parameters(QueryArgs {+ query: "quokka".into(),+ limit: Some(5),+ project: None,+ }))+ .await+ .unwrap();+ let text = match result.content.first().and_then(|c| c.as_text()) {+ Some(t) => t.text.clone(),+ None => panic!("expected text content"),+ };+ assert!(+ text.contains("\"hits\": []"),+ "expected no page hits; got {text}"+ );+ assert!(+ text.contains("raw_hits"),+ "expected raw fallback; got {text}"+ );+ assert!(text.contains("quokka"), "expected raw snippet; got {text}");+ }+#[tokio::test]async fn memory_status_returns_counts() {let (_tmp, _store, server, _ws, _pj) = setup_server().await;@@ -1060,12 +1149,13 @@ mod tests {"\"last_observation_at\":","\"pending_handoff_count\":","\"rules\":",+ "\"slots\":","\"recent_pages\":",] {assert!(text.contains(key), "missing {key} in briefing:\n{text}");}// setup_server inserts one page, no sessions/observations,- // no rules. The activity windows therefore observe zero.+ // no rules/slots. The activity windows therefore observe zero.assert!(text.contains("\"sessions\": 0"),"expected lifetime sessions: 0\n{text}"diff --git a/crates/ai-memory-mcp/tests/admin_phase3.rs b/crates/ai-memory-mcp/tests/admin_phase3.rsindex 790f3f03..01a97b0e 100644--- a/crates/ai-memory-mcp/tests/admin_phase3.rs+++ b/crates/ai-memory-mcp/tests/admin_phase3.rs@@ -225,6 +225,7 @@ async fn lint_dry_run_returns_lint_report_shape() {tier: Tier::Semantic,frontmatter_json: serde_json::json!({}),pinned: false,+ links: Vec::new(),}).await.unwrap();diff --git a/crates/ai-memory-mcp/tests/admin_status_search.rs b/crates/ai-memory-mcp/tests/admin_status_search.rsindex c5108b03..758cbaa6 100644--- a/crates/ai-memory-mcp/tests/admin_status_search.rs+++ b/crates/ai-memory-mcp/tests/admin_status_search.rs@@ -52,6 +52,7 @@ async fn seed_page(store: &Store, title: &str, path: &str, body: &str) {tier: Tier::Semantic,frontmatter_json: serde_json::json!({}),pinned: false,+ links: Vec::new(),};store.writer.upsert_page(page).await.unwrap();}@@ -102,6 +103,14 @@ async fn status_returns_counts_and_paths() {assert!(body["db_path"].as_str().unwrap().ends_with(".sqlite"));assert_eq!(body["counts"]["pages_latest"].as_u64().unwrap(), 1);assert_eq!(body["counts"]["pages_all"].as_u64().unwrap(), 1);+ assert_eq!(body["derived"]["pages_rows"].as_u64().unwrap(), 1);+ assert_eq!(body["derived"]["pages_fts_rows"].as_u64().unwrap(), 1);+ assert_eq!(+ body["derived"]["latest_pages_missing_embeddings"]+ .as_u64()+ .unwrap(),+ 1+ );}#[tokio::test]diff --git a/crates/ai-memory-store/migrations/V07__observations_fts_and_link_index.sql b/crates/ai-memory-store/migrations/V07__observations_fts_and_link_index.sqlnew file mode 100644index 00000000..ca8b2968--- /dev/null+++ b/crates/ai-memory-store/migrations/V07__observations_fts_and_link_index.sql@@ -0,0 +1,39 @@+-- Raw observation fallback search + unresolved-link lookup support.+--+-- Additive migration only: existing observations are backfilled into an+-- FTS5 index, and future inserts/updates/deletes stay synchronized by+-- triggers. No existing pages, observations, or wiki paths are rewritten.++CREATE VIRTUAL TABLE observations_fts USING fts5(+ title, body,+ content='observations',+ content_rowid='rowid',+ tokenize="unicode61 tokenchars '/_-'"+);++INSERT INTO observations_fts(rowid, title, body)+ SELECT rowid, title, body FROM observations;++CREATE TRIGGER observations_fts_ai AFTER INSERT ON observations BEGIN+ INSERT INTO observations_fts(rowid, title, body)+ VALUES (new.rowid, new.title, new.body);+END;++CREATE TRIGGER observations_fts_ad AFTER DELETE ON observations BEGIN+ INSERT INTO observations_fts(observations_fts, rowid, title, body)+ VALUES ('delete', old.rowid, old.title, old.body);+END;++CREATE TRIGGER observations_fts_au AFTER UPDATE ON observations BEGIN+ INSERT INTO observations_fts(observations_fts, rowid, title, body)+ VALUES ('delete', old.rowid, old.title, old.body);+ INSERT INTO observations_fts(rowid, title, body)+ VALUES (new.rowid, new.title, new.body);+END;++-- Link parsing existed as schema but not as a populated retrieval signal.+-- This index makes repeated unresolved-link resolution cheap once writers+-- start filling links(to_path, to_page_id=NULL).+CREATE INDEX IF NOT EXISTS idx_links_unresolved_path+ ON links(to_path)+ WHERE to_page_id IS NULL;diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex 3ffc0beb..889eb6dc 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -23,8 +23,9 @@ pub use decay::{DecayParams, retention_score};pub use error::{StoreError, StoreResult};pub use ops::{PurgeSummary, ReorgSummary};pub use reader::{- ActivityWindow, BriefingPage, BriefingSnapshot, DecayCandidate, PageHit, PageMeta, PageSummary,- ProjectSummary, ReaderPool, StatusCounts, StoredEmbedding, f32_vec_to_bytes,+ ActivityWindow, BriefingPage, BriefingSnapshot, DecayCandidate, DerivedIndexStatus,+ EmbeddingTripleCount, ObservationHit, PageHit, PageMeta, PageSummary, ProjectSummary,+ ReaderPool, StatusCounts, StoredEmbedding, f32_vec_to_bytes,};pub use writer::WriterHandle;@@ -83,7 +84,10 @@ impl Store {#[cfg(test)]mod tests {use super::*;- use ai_memory_core::{NewPage, PagePath, ProjectId, Tier, WorkspaceId};+ use ai_memory_core::{+ AgentKind, NewObservation, NewPage, NewSession, ObservationKind, PagePath, ProjectId,+ SessionId, Tier, WorkspaceId,+ };use tempfile::TempDir;fn sample_page(ws: WorkspaceId, proj: ProjectId, path: &str, body: &str) -> NewPage {@@ -96,6 +100,7 @@ mod tests {tier: Tier::Semantic,frontmatter_json: serde_json::json!({}),pinned: false,+ links: Vec::new(),}}@@ -305,6 +310,137 @@ mod tests {);}+ #[tokio::test]+ async fn search_quotes_hyphenated_tokens_for_fts5() {+ let tmp = TempDir::new().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "ai-memory", None)+ .await+ .unwrap();++ store+ .writer+ .upsert_page(sample_page(+ ws,+ proj,+ "hyphen.md",+ "the ai-memory token should be searchable",+ ))+ .await+ .unwrap();++ let hits = store+ .reader+ .search_pages_for_project(ws, proj, "ai-memory".into(), 10)+ .await+ .unwrap();+ assert_eq!(hits.len(), 1);+ assert_eq!(hits[0].path.as_str(), "hyphen.md");+ }++ #[tokio::test]+ async fn hybrid_search_includes_linked_neighbors() {+ let tmp = TempDir::new().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "ai-memory", None)+ .await+ .unwrap();++ store+ .writer+ .upsert_page(sample_page(ws, proj, "target.md", "neighbor-only content"))+ .await+ .unwrap();+ let mut source = sample_page(ws, proj, "source.md", "needle source content");+ source.links = vec![PagePath::new("target.md").unwrap()];+ store.writer.upsert_page(source).await.unwrap();++ let hits = store+ .reader+ .hybrid_search(+ ws,+ proj,+ "needle".into(),+ None,+ String::new(),+ String::new(),+ 0,+ 10,+ )+ .await+ .unwrap();+ let paths: Vec<&str> = hits.iter().map(|h| h.path.as_str()).collect();+ assert!(paths.contains(&"source.md"));+ assert!(+ paths.contains(&"target.md"),+ "linked neighbor should be included"+ );+ }++ #[tokio::test]+ async fn observation_fts_finds_raw_fallback_hits() {+ let tmp = TempDir::new().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "ai-memory", None)+ .await+ .unwrap();+ let session_id = SessionId::new();+ store+ .writer+ .begin_session(NewSession {+ id: session_id,+ workspace_id: ws,+ project_id: proj,+ agent_kind: AgentKind::OpenCode,+ cwd: None,+ })+ .await+ .unwrap();+ store+ .writer+ .insert_observation(NewObservation {+ session_id,+ workspace_id: ws,+ project_id: proj,+ kind: ObservationKind::UserPrompt,+ title: "prompt".into(),+ body: "the raw-only zebra detail lives here".into(),+ importance: 5,+ })+ .await+ .unwrap();++ let hits = store+ .reader+ .search_observations_for_project(ws, proj, "zebra".into(), 5)+ .await+ .unwrap();+ assert_eq!(hits.len(), 1);+ assert_eq!(hits[0].session_id, session_id);+ assert!(hits[0].snippet.contains("<mark>zebra</mark>"));+ }+#[tokio::test]async fn list_projects_with_stats_returns_aggregates() {let tmp = TempDir::new().unwrap();@@ -400,6 +536,7 @@ mod tests {tier: Tier::Semantic,frontmatter_json: serde_json::json!({"kind": "decision"}),pinned: true,+ links: Vec::new(),};store.writer.upsert_page(page).await.unwrap();diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex 20468ccb..a610c6a1 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -4,9 +4,11 @@//! the writer thread would violate the single-writer invariant (see//! [`crate::writer`]).+use std::collections::BTreeSet;+use ai_memory_core::{AgentKind, HandoffId, NewHandoff, NewObservation, NewPage, NewSession, ObservationId,- ObservationKind, PageId, ProjectId, SessionId, WorkspaceId,+ ObservationKind, PageId, PagePath, ProjectId, SessionId, WorkspaceId,};/// Summary returned by [`reorg_sessions`] and exposed via@@ -175,7 +177,10 @@ fn upsert_page_in_tx("UPDATE pages SET updated_at = ?1 WHERE id = ?2",params![now, existing_id],)?;- return PageId::from_slice(&existing_id).map_err(StoreError::from);+ let existing_page_id = PageId::from_slice(&existing_id).map_err(StoreError::from)?;+ replace_links_in_tx(tx, &existing_page_id, page)?;+ refresh_incoming_links_for_path(tx, page, &existing_page_id)?;+ return Ok(existing_page_id);}let new_id = PageId::new();tx.execute(@@ -202,6 +207,8 @@ fn upsert_page_in_tx(now,],)?;+ replace_links_in_tx(tx, &new_id, page)?;+ refresh_incoming_links_for_path(tx, page, &new_id)?;audit(tx,"supersede_page",@@ -232,6 +239,8 @@ fn upsert_page_in_tx(now,],)?;+ replace_links_in_tx(tx, &new_id, page)?;+ refresh_incoming_links_for_path(tx, page, &new_id)?;audit(tx,"create_page",@@ -243,6 +252,79 @@ fn upsert_page_in_tx(Ok(new_id)}+fn replace_links_in_tx(+ tx: &rusqlite::Transaction<'_>,+ from_page_id: &PageId,+ page: &NewPage,+) -> StoreResult<()> {+ tx.execute(+ "DELETE FROM links WHERE from_page_id = ?1",+ params![from_page_id.as_bytes()],+ )?;++ let mut seen = BTreeSet::new();+ for to_path in &page.links {+ if !seen.insert(to_path.as_str().to_string()) {+ continue;+ }+ let to_page_id = latest_page_id_for_path(tx, page, to_path)?;+ let to_page_blob = to_page_id.as_ref().map(|id| &id.as_bytes()[..]);+ tx.execute(+ "INSERT INTO links (from_page_id, to_page_id, to_path, link_type) \+ VALUES (?1, ?2, ?3, 'references')",+ params![from_page_id.as_bytes(), to_page_blob, to_path.as_str()],+ )?;+ }+ Ok(())+}++fn latest_page_id_for_path(+ tx: &rusqlite::Transaction<'_>,+ page: &NewPage,+ to_path: &PagePath,+) -> StoreResult<Option<PageId>> {+ let bytes: Option<Vec<u8>> = tx+ .query_row(+ "SELECT id FROM pages \+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 AND is_latest = 1",+ params![+ page.workspace_id.as_bytes(),+ page.project_id.as_bytes(),+ to_path.as_str(),+ ],+ |row| row.get(0),+ )+ .optional()?;+ bytes+ .map(|bytes| PageId::from_slice(&bytes).map_err(StoreError::from))+ .transpose()+}++fn refresh_incoming_links_for_path(+ tx: &rusqlite::Transaction<'_>,+ page: &NewPage,+ latest_page_id: &PageId,+) -> StoreResult<()> {+ tx.execute(+ "UPDATE links \+ SET to_page_id = ?1 \+ WHERE to_path = ?2 \+ AND EXISTS ( \+ SELECT 1 FROM pages from_page \+ WHERE from_page.id = links.from_page_id \+ AND from_page.workspace_id = ?3 \+ AND from_page.project_id = ?4 \+ )",+ params![+ latest_page_id.as_bytes(),+ page.path.as_str(),+ page.workspace_id.as_bytes(),+ page.project_id.as_bytes(),+ ],+ )?;+ Ok(())+}+/// Begin (or re-affirm) a session row keyed on the caller-supplied id./// Idempotent: a second call with the same id leaves the row untouched.pub fn begin_session(conn: &mut Connection, session: &NewSession) -> StoreResult<()> {@@ -749,6 +831,7 @@ mod tests {tier: Tier::Semantic,frontmatter_json: serde_json::json!({}),pinned: false,+ links: Vec::new(),}}@@ -816,6 +899,39 @@ mod tests {assert_eq!(total, 1, "no duplicate row for unchanged content");}+ #[test]+ fn upsert_page_persists_and_resolves_links() {+ let (_tmp, mut conn, ws, proj) = fresh_db();+ let mut source = page(ws, proj, "concepts/source.md", "see target");+ source.links = vec![PagePath::new("decisions/target.md").unwrap()];+ let source_id = upsert_page(&mut conn, &source).unwrap();++ let unresolved: i64 = conn+ .query_row(+ "SELECT COUNT(*) FROM links \+ WHERE from_page_id = ?1 AND to_path = ?2 AND to_page_id IS NULL",+ params![source_id.as_bytes(), "decisions/target.md"],+ |r| r.get(0),+ )+ .unwrap();+ assert_eq!(unresolved, 1, "forward link should persist unresolved");++ let target_id = upsert_page(+ &mut conn,+ &page(ws, proj, "decisions/target.md", "target body"),+ )+ .unwrap();++ let resolved: Option<Vec<u8>> = conn+ .query_row(+ "SELECT to_page_id FROM links WHERE from_page_id = ?1 AND to_path = ?2",+ params![source_id.as_bytes(), "decisions/target.md"],+ |r| r.get(0),+ )+ .unwrap();+ assert_eq!(resolved.as_deref(), Some(&target_id.as_bytes()[..]));+ }+/// Handoff state machine: insert → Open; accept_handoff → Accepted/// with accepted_by stamped. Calling accept again must be safe/// (idempotent at the DB level) because hooks fire-and-forget.diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rsindex 823738f9..88930c1f 100644--- a/crates/ai-memory-store/src/reader.rs+++ b/crates/ai-memory-store/src/reader.rs@@ -37,6 +37,25 @@ pub struct PageHit {pub rank: f64,}+/// One raw observation fallback hit returned when compiled wiki pages miss.+#[derive(Debug, Clone, Serialize)]+pub struct ObservationHit {+ /// Stable observation identifier.+ pub id: ObservationId,+ /// Owning session identifier.+ pub session_id: SessionId,+ /// Observation kind as stored on the lifecycle row.+ pub kind: String,+ /// Observation title.+ pub title: String,+ /// FTS5 snippet of the raw observation body around the matched terms.+ pub snippet: String,+ /// FTS5 rank score (lower is better).+ pub rank: f64,+ /// ISO-8601 creation timestamp.+ pub created_at: String,+}+/// Aggregate counts surfaced by [`ReaderPool::status_counts`] and consumed/// by `ai-memory status`.#[derive(Debug, Clone, Default, Serialize)]@@ -51,6 +70,44 @@ pub struct StatusCounts {pub observations: u64,}+/// Derived-index health counters surfaced by admin status.+#[derive(Debug, Clone, Default, Serialize)]+pub struct DerivedIndexStatus {+ /// All page rows in the source table. `pages_fts_rows` should match this.+ pub pages_rows: u64,+ /// Rows currently present in the page FTS5 index.+ pub pages_fts_rows: u64,+ /// All observation rows. `observations_fts_rows` should match this.+ pub observations_rows: u64,+ /// Rows currently present in the observation FTS5 index.+ pub observations_fts_rows: u64,+ /// Latest pages without any embedding row.+ pub latest_pages_missing_embeddings: u64,+ /// Stored embedding rows, regardless of provider/model/dim.+ pub embedding_rows: u64,+ /// Stored embedding triples and row counts.+ pub embedding_triples: Vec<EmbeddingTripleCount>,+ /// Outgoing links whose source page is latest.+ pub links_from_latest_pages: u64,+ /// Latest-page outgoing links whose target path has not resolved yet.+ pub unresolved_links_from_latest_pages: u64,+ /// Latest-page outgoing links pointing at a non-latest target row.+ pub stale_links_from_latest_pages: u64,+}++/// Count of embedding rows sharing one `(provider, model, dim)` triple.+#[derive(Debug, Clone, Serialize)]+pub struct EmbeddingTripleCount {+ /// Embedding provider name.+ pub provider: String,+ /// Embedding model name.+ pub model: String,+ /// Vector dimension.+ pub dim: u32,+ /// Rows using this triple.+ pub count: u64,+}+/// Rolling activity counters over a fixed time window. Surfaced by/// [`ReaderPool::briefing`] so the caller (or an LLM-driven `memory_explore`)/// can calibrate verbosity against how busy the project's been.@@ -89,6 +146,9 @@ pub struct BriefingSnapshot {/// All pages currently under `_rules/` — small, surfaced verbatim/// because they're the highest-signal type of memory.pub rules: Vec<BriefingPage>,+ /// Small pinned pages under `_slots/` for active project context,+ /// preferences, current focus, and pending items.+ pub slots: Vec<BriefingPage>,/// Top-N most-recently-updated `is_latest = 1` pages.pub recent_pages: Vec<BriefingPage>,}@@ -228,6 +288,7 @@ impl ReaderPool {/// # Errors/// Propagates any SQL or pool error.pub async fn search_pages(&self, query: String, limit: usize) -> StoreResult<Vec<PageHit>> {+ let query = normalize_fts_query(&query);self.with_conn(move |conn| {let mut stmt = conn.prepare("SELECT pages.id, pages.path, pages.title, \@@ -276,6 +337,7 @@ impl ReaderPool {query: String,limit: usize,) -> StoreResult<Vec<PageHit>> {+ let query = normalize_fts_query(&query);self.with_conn(move |conn| {let mut stmt = conn.prepare("SELECT pages.id, pages.path, pages.title, \@@ -324,6 +386,81 @@ impl ReaderPool {.await}+ /// Run a full-text search against raw observations scoped to one project.+ ///+ /// # Errors+ /// Propagates any SQL or pool error.+ pub async fn search_observations_for_project(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ query: String,+ limit: usize,+ ) -> StoreResult<Vec<ObservationHit>> {+ let query = normalize_fts_query(&query);+ self.with_conn(move |conn| {+ let mut stmt = conn.prepare(+ "SELECT observations.id, observations.session_id, observations.kind, \+ observations.title, \+ snippet(observations_fts, 1, '<mark>', '</mark>', '…', 24) AS snip, \+ observations_fts.rank, observations.created_at \+ FROM observations_fts \+ JOIN observations ON observations.rowid = observations_fts.rowid \+ WHERE observations_fts MATCH ?1 \+ AND observations.workspace_id = ?2 \+ AND observations.project_id = ?3 \+ ORDER BY observations_fts.rank \+ LIMIT ?4",+ )?;+ #[allow(clippy::cast_possible_wrap)]+ let rows = stmt.query_map(+ params![+ query,+ workspace_id.as_bytes(),+ project_id.as_bytes(),+ limit as i64,+ ],+ |row| {+ let id_bytes: Vec<u8> = row.get(0)?;+ let session_bytes: Vec<u8> = row.get(1)?;+ let kind: String = row.get(2)?;+ let title: String = row.get(3)?;+ let snippet: String = row.get(4)?;+ let rank: f64 = row.get(5)?;+ let created_us: i64 = row.get(6)?;+ Ok((+ id_bytes,+ session_bytes,+ kind,+ title,+ snippet,+ rank,+ created_us,+ ))+ },+ )?;++ let mut hits = Vec::new();+ for row in rows {+ let (id_bytes, session_bytes, kind, title, snippet, rank, created_us) = row?;+ let created_at = jiff::Timestamp::from_microsecond(created_us)+ .map(|ts| ts.to_string())+ .unwrap_or_default();+ hits.push(ObservationHit {+ id: ObservationId::from_slice(&id_bytes)?,+ session_id: SessionId::from_slice(&session_bytes)?,+ kind,+ title,+ snippet,+ rank,+ created_at,+ });+ }+ Ok(hits)+ })+ .await+ }+/// Return the N most-recently-updated `is_latest = 1` pages.////// # Errors@@ -604,13 +741,124 @@ impl ReaderPool {.await}+ /// Return pages linked to or from the seed pages, scoped to latest pages+ /// in the same project.+ ///+ /// # Errors+ /// Propagates any SQL or pool error.+ pub async fn graph_neighbors_for_project(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ seed_ids: Vec<PageId>,+ limit: usize,+ ) -> StoreResult<Vec<PageHit>> {+ if seed_ids.is_empty() || limit == 0 {+ return Ok(Vec::new());+ }+ self.with_conn(move |conn| {+ let seed_set: std::collections::HashSet<PageId> = seed_ids.iter().copied().collect();+ let mut seen = std::collections::HashSet::new();+ let mut out = Vec::new();++ let mut outgoing = conn.prepare(+ "SELECT tp.id, tp.path, tp.title, substr(tp.body, 1, 240) \+ FROM links l \+ JOIN pages tp ON tp.id = l.to_page_id \+ WHERE l.from_page_id = ?1 \+ AND tp.workspace_id = ?2 \+ AND tp.project_id = ?3 \+ AND tp.is_latest = 1 \+ ORDER BY tp.updated_at DESC",+ )?;+ let mut incoming = conn.prepare(+ "SELECT fp.id, fp.path, fp.title, substr(fp.body, 1, 240) \+ FROM links l \+ JOIN pages fp ON fp.id = l.from_page_id \+ WHERE l.to_page_id = ?1 \+ AND fp.workspace_id = ?2 \+ AND fp.project_id = ?3 \+ AND fp.is_latest = 1 \+ ORDER BY fp.updated_at DESC",+ )?;++ for seed_id in &seed_ids {+ let rows = outgoing.query_map(+ params![+ seed_id.as_bytes(),+ workspace_id.as_bytes(),+ project_id.as_bytes()+ ],+ |row| {+ let id_bytes: Vec<u8> = row.get(0)?;+ let path: String = row.get(1)?;+ let title: String = row.get(2)?;+ let snippet: String = row.get(3)?;+ Ok((id_bytes, path, title, snippet))+ },+ )?;+ for row in rows {+ let (id_bytes, path, title, snippet) = row?;+ let id = PageId::from_slice(&id_bytes)?;+ if seed_set.contains(&id) || !seen.insert(id) {+ continue;+ }+ out.push(PageHit {+ id,+ path: PagePath::new(path)?,+ title,+ snippet,+ rank: 0.0,+ });+ if out.len() >= limit {+ return Ok(out);+ }+ }++ let rows = incoming.query_map(+ params![+ seed_id.as_bytes(),+ workspace_id.as_bytes(),+ project_id.as_bytes()+ ],+ |row| {+ let id_bytes: Vec<u8> = row.get(0)?;+ let path: String = row.get(1)?;+ let title: String = row.get(2)?;+ let snippet: String = row.get(3)?;+ Ok((id_bytes, path, title, snippet))+ },+ )?;+ for row in rows {+ let (id_bytes, path, title, snippet) = row?;+ let id = PageId::from_slice(&id_bytes)?;+ if seed_set.contains(&id) || !seen.insert(id) {+ continue;+ }+ out.push(PageHit {+ id,+ path: PagePath::new(path)?,+ title,+ snippet,+ rank: 0.0,+ });+ if out.len() >= limit {+ return Ok(out);+ }+ }+ }+ Ok(out)+ })+ .await+ }+/// Hybrid search: RRF-fuse FTS5 results with cosine-similarity/// over the stored embeddings of the matching `(provider, model,- /// dim)`. Returns the top-`limit` pages by fused score.+ /// dim)`, then add link-neighbour expansion as a third RRF stream.+ /// Returns the top-`limit` pages by fused score.///- /// When `query_vec` is `None`, falls back to pure FTS5- /// (equivalent to [`Self::search_pages`]). When no embeddings- /// exist for the configured triple, also falls back to pure FTS5.+ /// When `query_vec` is `None`, the vector stream is skipped but graph+ /// expansion still runs from the FTS seeds.////// k=60 is the canonical RRF constant.///@@ -632,30 +880,37 @@ impl ReaderPool {let fts_hits = self.search_pages_for_project(workspace_id, project_id, query, limit * 2).await?;- let Some(qv) = query_vec else {- // No query vector → pure FTS5.- let mut out = fts_hits;- out.truncate(limit);- return Ok(out);- };- // Compute cosines against all stored embeddings.- let stored = self- .load_embeddings(workspace_id, project_id, provider, model, dim)- .await?;- if stored.is_empty() {- let mut out = fts_hits;- out.truncate(limit);- return Ok(out);+ let mut vec_hits: Vec<(PageId, PagePath, f32)> = Vec::new();+ if let Some(qv) = query_vec {+ // Compute cosines against all stored embeddings.+ let stored = self+ .load_embeddings(workspace_id, project_id, provider, model, dim)+ .await?;+ vec_hits = stored+ .iter()+ .map(|s| {+ let score = dot(&qv, &s.vector);+ (s.id, s.path.clone(), score)+ })+ .collect();+ vec_hits.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap_or(std::cmp::Ordering::Equal));+ vec_hits.truncate(limit * 2);}- let mut vec_hits: Vec<(PageId, PagePath, f32)> = stored++ let mut seed_seen = std::collections::HashSet::new();+ let mut seed_ids = Vec::new();+ for id in fts_hits.iter()- .map(|s| {- let score = dot(&qv, &s.vector);- (s.id, s.path.clone(), score)- })- .collect();- vec_hits.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap_or(std::cmp::Ordering::Equal));- vec_hits.truncate(limit * 2);+ .map(|h| h.id)+ .chain(vec_hits.iter().map(|(id, _, _)| *id))+ {+ if seed_seen.insert(id) {+ seed_ids.push(id);+ }+ }+ let graph_hits = self+ .graph_neighbors_for_project(workspace_id, project_id, seed_ids, limit * 2)+ .await?;// RRF fuse: score(d) = Σ 1/(k + rank_i(d)) over rankers.let k = 60.0_f64;@@ -682,6 +937,19 @@ impl ReaderPool {.and_modify(|entry| entry.3 += contrib).or_insert((path.clone(), String::new(), String::new(), contrib, 0.0));}+ for (rank, h) in graph_hits.iter().enumerate() {+ let contrib = 1.0 / (k + (rank + 1) as f64);+ fused+ .entry(h.id)+ .and_modify(|entry| entry.3 += contrib)+ .or_insert((+ h.path.clone(),+ h.title.clone(),+ h.snippet.clone(),+ contrib,+ h.rank,+ ));+ }let mut out: Vec<PageHit> = fused.into_iter()@@ -845,6 +1113,20 @@ impl ReaderPool {.into_iter().collect::<Result<Vec<_>, _>>()?;+ let mut slots_stmt = conn.prepare(+ "SELECT path, title, \+ COALESCE(json_extract(frontmatter_json, '$.kind'), 'slot') AS kind, \+ updated_at \+ FROM pages \+ WHERE is_latest = 1 AND path LIKE '_slots/%' \+ ORDER BY path ASC",+ )?;+ let slots: Vec<BriefingPage> = slots_stmt+ .query_map([], briefing_page_from_row)?+ .collect::<Result<Vec<_>, _>>()?+ .into_iter()+ .collect::<Result<Vec<_>, _>>()?;+let mut recent_stmt = conn.prepare("SELECT path, title, \COALESCE(json_extract(frontmatter_json, '$.kind'), 'fact') AS kind, \@@ -867,6 +1149,7 @@ impl ReaderPool {last_observation_at,pending_handoff_count,rules,+ slots,recent_pages,})})@@ -954,6 +1237,20 @@ impl ReaderPool {.into_iter().collect::<Result<Vec<_>, _>>()?;+ let mut slots_stmt = conn.prepare(+ "SELECT path, title, \+ COALESCE(json_extract(frontmatter_json, '$.kind'), 'slot') AS kind, \+ updated_at \+ FROM pages \+ WHERE workspace_id = ?1 AND project_id = ?2 AND is_latest = 1 AND path LIKE '_slots/%' \+ ORDER BY path ASC",+ )?;+ let slots: Vec<BriefingPage> = slots_stmt+ .query_map(params![workspace_id.as_bytes(), project_id.as_bytes()], briefing_page_from_row)?+ .collect::<Result<Vec<_>, _>>()?+ .into_iter()+ .collect::<Result<Vec<_>, _>>()?;+let mut recent_stmt = conn.prepare("SELECT path, title, \COALESCE(json_extract(frontmatter_json, '$.kind'), 'fact') AS kind, \@@ -976,6 +1273,7 @@ impl ReaderPool {last_observation_at,pending_handoff_count,rules,+ slots,recent_pages,})})@@ -1246,6 +1544,80 @@ impl ReaderPool {.await}+ /// Return health counters for derived indexes and link/embedding state.+ ///+ /// These checks are intentionally read-only and derived-index-safe: they+ /// report drift but do not repair it. Rebuild/backfill paths stay behind+ /// explicit admin operations.+ ///+ /// # Errors+ /// Propagates any SQL or pool error.+ pub async fn derived_index_status(&self) -> StoreResult<DerivedIndexStatus> {+ self.with_conn(|conn| {+ let mut triples_stmt = conn.prepare(+ "SELECT provider, model, dim, COUNT(*) \+ FROM page_embeddings \+ GROUP BY provider, model, dim \+ ORDER BY COUNT(*) DESC, provider, model, dim",+ )?;+ let embedding_triples = triples_stmt+ .query_map([], |row| {+ let provider: String = row.get(0)?;+ let model: String = row.get(1)?;+ let dim: i64 = row.get(2)?;+ let count: i64 = row.get(3)?;+ Ok(EmbeddingTripleCount {+ provider,+ model,+ dim: u32::try_from(dim.max(0)).unwrap_or(0),+ count: u64::try_from(count.max(0)).unwrap_or(0),+ })+ })?+ .collect::<Result<Vec<_>, _>>()?;++ Ok(DerivedIndexStatus {+ pages_rows: count(conn, "SELECT COUNT(*) FROM pages")?,+ pages_fts_rows: count(conn, "SELECT COUNT(*) FROM pages_fts")?,+ observations_rows: count(conn, "SELECT COUNT(*) FROM observations")?,+ observations_fts_rows: count(conn, "SELECT COUNT(*) FROM observations_fts")?,+ latest_pages_missing_embeddings: count(+ conn,+ "SELECT COUNT(*) \+ FROM pages pg \+ LEFT JOIN page_embeddings pe ON pe.page_id = pg.id \+ WHERE pg.is_latest = 1 AND pe.page_id IS NULL",+ )?,+ embedding_rows: count(conn, "SELECT COUNT(*) FROM page_embeddings")?,+ embedding_triples,+ links_from_latest_pages: count(+ conn,+ "SELECT COUNT(*) \+ FROM links l \+ JOIN pages fp ON fp.id = l.from_page_id \+ WHERE fp.is_latest = 1",+ )?,+ unresolved_links_from_latest_pages: count(+ conn,+ "SELECT COUNT(*) \+ FROM links l \+ JOIN pages fp ON fp.id = l.from_page_id \+ WHERE fp.is_latest = 1 AND l.to_page_id IS NULL",+ )?,+ stale_links_from_latest_pages: count(+ conn,+ "SELECT COUNT(*) \+ FROM links l \+ JOIN pages fp ON fp.id = l.from_page_id \+ LEFT JOIN pages tp ON tp.id = l.to_page_id \+ WHERE fp.is_latest = 1 \+ AND l.to_page_id IS NOT NULL \+ AND COALESCE(tp.is_latest, 0) != 1",+ )?,+ })+ })+ .await+ }+/// Return aggregate counts scoped to one project.////// # Errors@@ -1649,6 +2021,24 @@ fn count_project(Ok(u64::try_from(n.unwrap_or(0)).unwrap_or(0))}+fn normalize_fts_query(query: &str) -> String {+ query+ .split_whitespace()+ .map(|token| {+ if should_quote_fts_token(token) {+ format!("\"{}\"", token.replace('"', "\"\""))+ } else {+ token.to_string()+ }+ })+ .collect::<Vec<_>>()+ .join(" ")+}++fn should_quote_fts_token(token: &str) -> bool {+ token.contains('-') && !(token.starts_with('"') && token.ends_with('"'))+}+/// Count rows in a time-bounded window. Used by [`ReaderPool::briefing`]/// to compute "last 7 days" / "last 30 days" activity slices.fn window_activity(conn: &Connection, days: u32, cutoff_us: i64) -> StoreResult<ActivityWindow> {diff --git a/crates/ai-memory-web/tests/routes.rs b/crates/ai-memory-web/tests/routes.rsindex f302dc78..3deccf46 100644--- a/crates/ai-memory-web/tests/routes.rs+++ b/crates/ai-memory-web/tests/routes.rs@@ -35,6 +35,7 @@ fn new_page(tier: Tier::Semantic,frontmatter_json: serde_json::json!({"kind": "fact"}),pinned: false,+ links: Vec::new(),}}diff --git a/crates/ai-memory-wiki/src/markdown.rs b/crates/ai-memory-wiki/src/markdown.rsindex d544793a..61068994 100644--- a/crates/ai-memory-wiki/src/markdown.rs+++ b/crates/ai-memory-wiki/src/markdown.rs@@ -6,6 +6,8 @@//! basic-memory hit (#528). Going through `serde_yaml` directly keeps the//! round-trip predictable.+use std::collections::BTreeSet;+use ai_memory_core::PagePath;use serde::{Deserialize, Serialize};@@ -100,6 +102,150 @@ pub fn derive_title(frontmatter: &serde_json::Value, body: &str, path: &PagePathstem.strip_suffix(".md").unwrap_or(stem).to_string()}+/// Extract internal wiki links from a markdown body.+///+/// Supports `[[wiki links]]`, `[[wiki links|labels]]`, and ordinary+/// markdown links such as `[label](../decisions/foo.md#anchor)`. External+/// URLs, anchors, images, and non-markdown assets are ignored. Returned+/// paths are normalised to wiki-root-relative [`PagePath`] values.+#[must_use]+pub fn extract_links(body: &str, page_path: &PagePath) -> Vec<PagePath> {+ let mut out = BTreeSet::new();+ let mut in_fence = false;++ for line in body.lines() {+ let trimmed = line.trim_start();+ if trimmed.starts_with("```") || trimmed.starts_with("~~~") {+ in_fence = !in_fence;+ continue;+ }+ if in_fence {+ continue;+ }+ extract_wikilinks(line, page_path, &mut out);+ extract_markdown_links(line, page_path, &mut out);+ }++ out.into_iter()+ .filter_map(|path| PagePath::new(path).ok())+ .collect()+}++fn extract_wikilinks(line: &str, page_path: &PagePath, out: &mut BTreeSet<String>) {+ let mut rest = line;+ while let Some(start) = rest.find("[[") {+ let after_start = &rest[start + 2..];+ let Some(end) = after_start.find("]]") else {+ break;+ };+ let raw = &after_start[..end];+ if let Some(path) = normalize_link_target(raw, page_path, true) {+ out.insert(path);+ }+ rest = &after_start[end + 2..];+ }+}++fn extract_markdown_links(line: &str, page_path: &PagePath, out: &mut BTreeSet<String>) {+ let mut start_at = 0;+ while let Some(rel_start) = line[start_at..].find('[') {+ let start = start_at + rel_start;+ if start > 0 && line.as_bytes()[start - 1] == b'!' {+ start_at = start + 1;+ continue;+ }+ let after_start = start + 1;+ let Some(rel_close) = line[after_start..].find(']') else {+ break;+ };+ let close = after_start + rel_close;+ if !line[close + 1..].starts_with('(') {+ start_at = close + 1;+ continue;+ }+ let target_start = close + 2;+ let Some(rel_end) = line[target_start..].find(')') else {+ break;+ };+ let target_end = target_start + rel_end;+ let raw = &line[target_start..target_end];+ if let Some(path) = normalize_link_target(raw, page_path, false) {+ out.insert(path);+ }+ start_at = target_end + 1;+ }+}++fn normalize_link_target(raw: &str, page_path: &PagePath, wikilink: bool) -> Option<String> {+ let target = raw+ .split_once('|')+ .map_or(raw, |(path, _)| path)+ .trim()+ .trim_matches('<')+ .trim_matches('>');+ if target.is_empty() || target.starts_with('#') || target.contains("://") {+ return None;+ }+ let lower = target.to_ascii_lowercase();+ if lower.starts_with("mailto:")+ || lower.starts_with("data:")+ || lower.starts_with("javascript:")+ || lower.starts_with("tel:")+ {+ return None;+ }++ let target = target.split_once('#').map_or(target, |(path, _)| path);+ let target = target+ .split_once('?')+ .map_or(target, |(path, _)| path)+ .trim();+ if target.is_empty() || target.contains('\\') {+ return None;+ }++ let mut target = target.to_string();+ let last_segment = target.rsplit_once('/').map_or(target.as_str(), |(_, s)| s);+ if last_segment.contains('.') {+ if !target.ends_with(".md") {+ return None;+ }+ } else if wikilink || !last_segment.is_empty() {+ target.push_str(".md");+ }++ resolve_relative(page_path, &target, wikilink)+}++fn resolve_relative(page_path: &PagePath, target: &str, root_relative: bool) -> Option<String> {+ let mut parts: Vec<&str> = if root_relative || target.starts_with('/') {+ Vec::new()+ } else {+ page_path+ .as_str()+ .rsplit_once('/')+ .map_or_else(Vec::new, |(dir, _)| {+ dir.split('/').filter(|part| !part.is_empty()).collect()+ })+ };++ for part in target.trim_start_matches('/').split('/') {+ match part {+ "" | "." => {}+ ".." => {+ parts.pop()?;+ }+ part => parts.push(part),+ }+ }++ if parts.is_empty() {+ None+ } else {+ Some(parts.join("/"))+ }+}+#[cfg(test)]mod tests {use super::*;@@ -181,4 +327,27 @@ mod tests {"foo");}++ #[test]+ fn extracts_internal_wiki_and_markdown_links() {+ let path = PagePath::new("concepts/current.md").unwrap();+ let body = "See [[decisions/0001-single-sqlite-file|SQLite]] and \+ [gotcha](../gotchas/hooks.md#details). Also \+ [external](https://example.com) and .";+ let links = extract_links(body, &path);+ let paths: Vec<&str> = links.iter().map(PagePath::as_str).collect();+ assert_eq!(+ paths,+ vec!["decisions/0001-single-sqlite-file.md", "gotchas/hooks.md"]+ );+ }++ #[test]+ fn extract_links_ignores_fenced_code_blocks() {+ let path = PagePath::new("notes/a.md").unwrap();+ let body = "```\n[[notes/ignored]]\n```\n[[notes/kept]]\n";+ let links = extract_links(body, &path);+ let paths: Vec<&str> = links.iter().map(PagePath::as_str).collect();+ assert_eq!(paths, vec!["notes/kept.md"]);+ }}diff --git a/crates/ai-memory-wiki/src/wiki.rs b/crates/ai-memory-wiki/src/wiki.rsindex 70c43394..5dbe4932 100644--- a/crates/ai-memory-wiki/src/wiki.rs+++ b/crates/ai-memory-wiki/src/wiki.rs@@ -10,7 +10,7 @@ use ai_memory_store::{WriterHandle, f32_vec_to_bytes};use crate::atomic;use crate::error::WikiResult;use crate::git::GitAdapter;-use crate::markdown::{Markdown, derive_title, emit, parse};+use crate::markdown::{Markdown, derive_title, emit, extract_links, parse};/// Wiki filesystem handle.///@@ -67,10 +67,12 @@ impl Wiki {self}- /// Attach an embedder. When set, every successful `write_page` /- /// `apply_batch` also computes + stores an embedding for the new- /// version. Without this, vector search is unavailable and- /// `ReaderPool::hybrid_search` falls back to pure FTS5.+ /// Attach an embedder. When set, `write_page` computes + stores an+ /// embedding for the new version synchronously. `apply_batch` keeps+ /// the SQL/file fan-out atomic and leaves vector completeness to+ /// admin or scheduled embedding backfill. Without an embedder,+ /// vector search is skipped and `ReaderPool::hybrid_search` uses+ /// FTS5 + graph expansion.#[must_use]pub fn with_embedder(mut self, embedder: Arc<dyn Embedder>) -> Self {self.embedder = Some(embedder);@@ -189,6 +191,8 @@ impl Wiki {) -> WikiResult<PageId> {let md = self.read_page(workspace_id, project_id, &path)?;let title = derive_title(&md.frontmatter, &md.body, &path);+ let links = extract_links(&md.body, &path);+ let pinned = is_slot_path(&path);let id = self.writer.upsert_page(NewPage {@@ -199,7 +203,8 @@ impl Wiki {body: md.body,tier: Tier::Semantic,frontmatter_json: md.frontmatter,- pinned: false,+ pinned,+ links,}).await?;Ok(id)@@ -263,7 +268,8 @@ impl Wiki {body: req.body.clone(),tier: req.tier,frontmatter_json: req.frontmatter.clone(),- pinned: req.pinned,+ pinned: req.pinned || is_slot_path(&req.path),+ links: extract_links(&req.body, &req.path),}).collect();@@ -310,6 +316,8 @@ impl Wiki {.clone().map(|t| self.sanitizer.scrub(&t)).unwrap_or_else(|| derive_title(&frontmatter, &body, &path));+ let links = extract_links(&body, &path);+ let pinned = pinned || is_slot_path(&path);let markdown = Markdown {frontmatter: frontmatter.clone(),body: body.clone(),@@ -332,6 +340,7 @@ impl Wiki {tier,frontmatter_json: frontmatter,pinned,+ links,}).await?;// Embed if configured. We do this on the caller's task so the@@ -389,6 +398,10 @@ fn ai_memory_wiki_error(msg: &str) -> crate::WikiError {crate::WikiError::Io(std::io::Error::other(msg.to_string()))}+fn is_slot_path(path: &PagePath) -> bool {+ path.as_str().starts_with("_slots/")+}+#[cfg(test)]mod tests {use super::*;@@ -550,6 +563,37 @@ mod tests {assert!(!hits[0].snippet.contains("hunter2"));}+ #[tokio::test]+ async fn slot_pages_are_pinned_automatically() {+ let tmp = TempDir::new().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "scratch", None)+ .await+ .unwrap();+ let wiki = Wiki::new(tmp.path(), store.writer.clone()).unwrap();++ wiki.write_page(req(+ ws,+ proj,+ "_slots/current_focus.md",+ "Keep this tiny and durable.",+ serde_json::json!({ "title": "Current focus", "kind": "slot" }),+ ))+ .await+ .unwrap();++ let candidates = store.reader.decay_candidates(ws, proj).await.unwrap();+ assert_eq!(candidates.len(), 1);+ assert!(candidates[0].pinned, "slot pages should be decay-immune");+ }+#[tokio::test]async fn apply_batch_persists_all_pages_in_one_transaction() {let tmp = TempDir::new().unwrap();diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 2ce99121..614cf065 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -85,14 +85,15 @@ markdown stays the source of truth.4. When `AI_MEMORY_LLM_PROVIDER` is set, `memory_consolidate` rewritesthat summary into a richer durable page or fans out into amulti-page batch under `concepts/`, `decisions/`, `gotchas/`.-5. `memory_query` answers via FTS5; when an embedder is configured,- via RRF fusion of FTS5 + cosine over `page_embeddings`. Every- query hit bumps `access_count` + `last_accessed_at` on the page -- the M8 reinforcement term.-6. The forget sweep runs on demand (or on a future schedule): pages- with `retention < cold_threshold` are soft-deleted; soft-deletions- older than `hard_delete_after_days` with no subsequent access get- purged. Semantic / pinned / freshly-touched pages survive.+5. `memory_query` answers via FTS5 + link-neighbour RRF; when an+ embedder is configured, vector cosine over `page_embeddings` joins+ the same RRF. If compiled wiki pages miss entirely, bounded raw+ observation FTS returns fallback `raw_hits`. Page hits bump+ `access_count` + `last_accessed_at` - the M8 reinforcement term.+6. The forget sweep runs on demand and on the server's `[maintenance]`+ schedule: pages with `retention < cold_threshold` are soft-deleted;+ soft-deletions older than `hard_delete_after_days` with no subsequent+ access get purged. Semantic / pinned / freshly-touched pages survive.7. Backups: `ai-memory backup --to <tarball>` uses SQLite's onlinebackup API so the source stays writable; `ai-memory restore`reverses. Or: `git push` the wiki dir + `rsync` the data dir.@@ -108,7 +109,8 @@ markdown stays the source of truth.* `<data_dir>/db/memory.sqlite` - derived index. WAL mode. Onewriter actor owns the writer `Connection`; reads go through acloneable read-only pool.-* `<data_dir>/raw/` - reserved for raw session log archives (future).+* `<data_dir>/raw/` - reserved for raw session log archives; current raw+ fallback recall searches the durable `observations` table via FTS5.* `<data_dir>/logs/` - rolling daily `tracing` output.* `<data_dir>/models/` - reserved for bundled embedding models(M9.5+, when local `ort` lands).@@ -121,9 +123,10 @@ markdown stays the source of truth.| `pages` | Versioned wiki pages with `is_latest` + `supersedes` chain. M8 columns: `last_accessed_at`, `access_count`, `superseded_at`. M9 cols: `embedding_provider`, `embedding_model`, `embedding_dim`. || `pages_fts` | FTS5 virtual table over `(title, body)`, auto-synced by triggers. || `sessions`, `observations` | Hook capture, full audit log. |-| `links` | Wiki cross-references with `to_page_id` nullable for unresolved. |+| `observations_fts` | FTS5 virtual table over raw observation `(title, body)`, used only as bounded fallback. |+| `links` | Wikilink / markdown cross-references with `to_page_id` nullable for unresolved forward links. || `handoffs` | Typed cross-agent handoff records (open / accepted / expired). |-| `page_embeddings` | One vector per latest page, with `(provider, model, dim)` denormalised for refuse-on-mismatch checks. |+| `page_embeddings` | Optional vector rows for latest pages, with `(provider, model, dim)` denormalised for refuse-on-mismatch checks and missing-embedding diagnostics. || `audit_log` | Every mutation, addressable by `at DESC`. |**Memory tiers (M8 policy):**@@ -136,7 +139,8 @@ markdown stays the source of truth.| Procedural | Indefinite | Frequency-decay if not re-observed |Pinned pages (`pinned: true` in frontmatter) are exempt from all-decay paths.+decay paths. Pages under `_slots/` are pinned automatically and surfaced+in briefing/explore snapshots as tiny editable memory slots.## Crate layout@@ -156,18 +160,21 @@ Each crate has a single responsibility and exposes a typed API. Nocircular deps. Inter-crate boundaries enforce the cross-cuttinginvariants below.-## MCP tool surface (10 tools)+## MCP tool surface (11 tools)| Tool | Hint | Purpose ||---|---|---|-| `memory_query` | read-only | FTS5 or hybrid RRF search. Bumps access counters. |+| `memory_query` | read-only | FTS5 + graph RRF + optional vector RRF search, with raw fallback. Bumps access counters for page hits. || `memory_recent` | read-only | Most-recently-updated `is_latest=1` pages. || `memory_status` | read-only | Counts, paths, version. |+| `memory_briefing` | read-only | Structured counts/activity/rules/slots/recent snapshot. |+| `memory_explore` | read-only | LLM prose digest over the briefing snapshot, degrading to JSON without a provider. || `memory_handoff_begin` | destructive | Open a handoff for the next agent. || `memory_handoff_accept` | destructive | Fetch + ack the latest open handoff (auto-cwd-matched). || `memory_consolidate` | destructive | LLM-driven page rewrite. `multi_page=true` for atomic fan-out. || `memory_forget_sweep` | destructive | M8 retention pass. `dry_run=true` for preview. || `memory_lint` | destructive | Rule-based + LLM contradiction findings → `wiki/_lint/`. |+| `memory_install_self_routing` | read-only | Return the canonical routing snippet for CLAUDE.md / AGENTS.md. |All MCP tools carry `readOnlyHint` / `destructiveHint` annotations sothe calling agent can pick safely. Parameter aliases (`query|q|search`,diff --git a/docs/design-decisions.md b/docs/design-decisions.mdindex 0d267ba8..f1c34926 100644--- a/docs/design-decisions.md+++ b/docs/design-decisions.md@@ -57,7 +57,7 @@ Three options surveyed:## 4. Database choice - single SQLite file-**Decision: one SQLite file with FTS5 + `sqlite-vec` extension + JSON columns for graph edges.**+**Decision: one SQLite file with FTS5, packed-vector embeddings, and SQL tables for graph edges.**Why not Postgres/pgvector? Cognee's #2717 and basic-memory's #830/#831 show Postgres is a real-deployment-only pain. v1 ships embedded.@@ -66,13 +66,13 @@ Why not LanceDB/Qdrant/Kuzu/CozoDB/SurrealDB?- Kuzu / Ladybug: cognee #2098/#2768 (upstream archived, fork-risk realized).- CozoDB: small bus factor.- SurrealDB: heavy, multi-mode storage; we'd inherit a lot of surface we don't need.-- Embedded `sqlite-vec` is well-maintained, single dependency, fits in one file with FTS5 + relational tables.+- Packed vectors in SQLite keep v1 dependency-light; `sqlite-vec` remains the scale-up path once brute-force cosine stops being enough.**The graph is just SQL tables.** A `wiki_pages` table, a `wiki_links (from_id, to_id, link_type)` table, optional `wiki_concepts (page_id, concept)`. Graph queries are recursive CTEs in SQLite. Petgraph in-memory for batch traversals. Avoids the entire "embedded graph DB" footgun cognee fell into.**Crates** (research-backed picks):-- `rusqlite` + `rusqlite-extension` for sqlite-vec loading. `bundled-sqlcipher` if we want encryption later.-- `sqlx` for migrations (`sqlx::migrate!`). Async, type-checked.+- `rusqlite` for embedded SQLite access. `bundled-sqlcipher` if we want encryption later.+- `refinery` for SQL migrations.- `tantivy` *not* used initially - sqlite FTS5 is sufficient at the corpus sizes we expect (hundreds to low-thousands of pages per project). Revisit only if FTS5 ranking proves inadequate.- `petgraph` for in-memory graph algorithms during consolidation.diff --git a/docs/install.md b/docs/install.mdindex 337acf96..a9f3ffec 100644--- a/docs/install.md+++ b/docs/install.md@@ -344,11 +344,11 @@ docker run --rm akitaonrails/ai-memory:latest --help # full subcommand tree| Subcommand | Pattern | What it does ||---|---|---|| `serve` | `docker compose up -d` (already done) | Run the HTTP MCP server |-| `status` | `docker exec` | Counts + paths + provider config |-| `search "<query>"` | `docker exec` | FTS5 query against the wiki |+| `status` | `docker exec` | Counts, paths, and derived-index diagnostics |+| `search "<query>"` | `docker exec` | Wiki search with FTS5 + graph/vector RRF || `write-page` | `docker exec` | Manual page write (atomic + indexed) || `backup --to` / `restore --from` | `docker exec` | Snapshot or restore the data dir |-| `forget-sweep` / `lint` / `embed` | `docker exec` | Periodic maintenance |+| `forget-sweep` / `lint` / `embed` | `docker exec` | Manual maintenance; sweep + lint also run on the server schedule by default || `commit -m "…"` | `docker exec` | Stage + commit the wiki tree || `reset --confirm` | `docker exec` | Wipe data (refuses while siblings alive) || `generate-auth-token` | `docker run --rm` | Print a random hex bearer token |@@ -360,6 +360,12 @@ docker run --rm akitaonrails/ai-memory:latest --help # full subcommand treeData dir inside the container is `/data` (mounted via the composevolume). Outside docker, override with `AI_MEMORY_DATA_DIR=/path`.+Scheduled maintenance is configured in `[maintenance]` in `config.toml`.+By default, rule-based lint and forget sweep run daily outside hook+latency. Embedding backfill is supported but defaults to off because it+can call a paid provider; enable it with+`embedding_backfill_interval_secs` after configuring an embedder.+---## Bootstrap mid-project {#bootstrap-mid-project}diff --git a/docs/issues-mempalace.md b/docs/issues-mempalace.mdindex b2ea100c..2f7bb4a6 100644--- a/docs/issues-mempalace.md+++ b/docs/issues-mempalace.md@@ -1,9 +1,9 @@# MemPalace - Issue & Architecture Synthesis-> Source: GitHub `MemPalace/mempalace` + local checkout at `~/Projects/mempalace`.+> Source: GitHub `MemPalace/mempalace`.> Captured 2026-05-24. Repo: 52.7k stars, MIT, Python, created 2026-04-05,-> v3.3.5, 241 open issues, ~1600 PRs in ~7 weeks. Extremely active, unusually-> transparent maintainers (see `docs/HISTORY.md` — they publicly retract+> v3.3.5, 241 open issues, ~1600 PRs in ~7 weeks. The maintainers publish+> unusually candid history notes (see `docs/HISTORY.md` — they publicly retract> overclaims). Dominant themes: **ChromaDB/HNSW/FTS5 corruption under> concurrent writes**, **silent persistence failures**, **repair tooling that> destroys data**, **embedding-model drift**.diff --git a/docs/lifecycle-ops.md b/docs/lifecycle-ops.mdindex f923c162..29f683cb 100644--- a/docs/lifecycle-ops.md+++ b/docs/lifecycle-ops.md@@ -237,11 +237,11 @@ ai-memory purge-project --project experimental --confirm# Sibling projects (ai-memory, distrobox-gaming, …) untouched.```-### "Rename a project after `mv ~/Projects/old ~/Projects/new`"+### "Rename a project after moving its directory"```bashai-memory rename-project --from old --to new-# Future sessions in ~/Projects/new will append to the same project+# Future sessions in /path/to/new will append to the same project# (the hook router stamps by basename(cwd) = "new"); past# observations stay under that project too because the project_id# is stable.diff --git a/docs/mcp-install.md b/docs/mcp-install.mdindex 5162d3e1..e6e8206c 100644--- a/docs/mcp-install.md+++ b/docs/mcp-install.md@@ -235,8 +235,9 @@ You: List the MCP tools you can call. Use one of them to checkai-memory's status.Model (any client): I can call: memory_query, memory_recent,- memory_status, memory_handoff_accept, memory_handoff_begin,- memory_consolidate, memory_lint, memory_forget_sweep.+ memory_status, memory_briefing, memory_explore,+ memory_handoff_accept, memory_handoff_begin, memory_consolidate,+ memory_lint, memory_forget_sweep, memory_install_self_routing.memory_status reports: 0 pages, 0 observations, 0 sessions.```diff --git a/docs/prior-art-implementation-findings.md b/docs/prior-art-implementation-findings.mdnew file mode 100644index 00000000..419734fb--- /dev/null+++ b/docs/prior-art-implementation-findings.md@@ -0,0 +1,338 @@+# Prior-Art Implementation Findings++> Scope: compare the checked-in prior-art analyses for agentmemory,+> basic-memory, cognee, and MemPalace against the current ai-memory+> implementation. This is an analysis document only; no code changes+> are implied by this file.++## Executive Summary++ai-memory already captured the most important prior-art lessons:+automatic capture, a narrow MCP surface, markdown-in-git as the human+source of truth, SQLite as the derived index/state store, single-writer+durability, opt-in LLM consolidation, structured JSON outputs, typed+handoffs, versioned supersession, per-project UUID isolation, and an+agentmemory-style retention formula.++The remaining high-value improvements were not a rewrite. They were+mostly lifecycle and retrieval layers on top of the solid base: small+editable memory slots, scheduled maintenance, first-class graph/link+retrieval, bounded raw/verbatim fallback recall, diagnostics, and real+retrieval checks. The most important caution remains keeping the current+substrate simple. Do not copy agentmemory's sidecar surface, cognee's+multi-store orchestration, basic-memory's manual write-note workflow, or+MemPalace's verbatim-everything storage model.++## Implementation Status++The follow-up implementation landed the pragmatic subset of this roadmap:++| Area | Status |+|---|---|+| Slots | `_slots/` pages are pinned automatically and surfaced in briefing/explore snapshots. |+| Scheduled maintenance | Server-side schedule runs lint and forget sweep daily by default; embedding backfill is opt-in. |+| Vector contract | Status reports missing embeddings and stored triples; batch writes leave vector completeness to backfill. |+| Links / graph | Markdown and wikilinks are parsed into `links`; unresolved forward links resolve when targets appear; query uses graph-neighbor RRF. |+| Raw fallback | `observations_fts` provides bounded raw fallback when compiled wiki search misses. |+| Diagnostics | `status` includes FTS row counts, missing embeddings, embedding triples, and unresolved/stale link counts. |+| Retrieval evals | Recall harness now covers FTS/vector, graph expansion, and raw fallback. |++## Prior Art Summary++| Project | Main strengths | Main weaknesses | ai-memory stance |+|---|---|---|---|+| agentmemory | Automatic hook capture, four memory tiers, versioned supersession, retention scoring, hybrid BM25/vector/graph retrieval, slots, handoff patterns, snapshots, diagnostics. | iii-engine sidecar, JSON-in-KV storage, in-memory indexes, XML parsing, too many tools/endpoints, early default-on LLM costs, hook blocking/config drift bugs. | ai-memory follows the concepts more than any other project, but replaces the substrate with one Rust binary, SQLite, markdown, structured JSON, and a narrow tool surface. |+| basic-memory | Markdown files as source of truth, derived SQL index, human-editable notes, MCP hints/aliases, `memory://` style navigation, unresolved links, multi-project routing. | Manual `write_note` ceremony, append-only memory, fragile file watcher/sync, markdown grammar burden on the LLM, weak lifecycle/ranking. | ai-memory borrows markdown/source-of-truth and human editability, but rejects manual capture and adds lifecycle, decay, supersession, and hook capture. |+| cognee | Task-list pipeline shape, structured graph extraction, provenance stamping, triplet embeddings, multi-collection retrieval, feedback-weighted `improve()`. | Heavy Python dependency stack, LiteLLM/Instructor brittleness, multi-store graph/vector/relational sync bugs, SQLite deadlocks, retrieval filter propagation regressions. | ai-memory should borrow only the pipeline and retrieval ideas, not the multi-store architecture or generic LLM gateway. |+| MemPalace | Strong verbatim/raw recall story, deterministic IDs, resume-safe mining, transparent benchmark culture, temporal KG ideas. | Chroma/HNSW/FTS corruption under concurrent writers, deferred persistence, destructive repair paths, no decay causing bloat, embedding metadata drift. | ai-memory already avoids the durability failures through a single writer, one SQLite file, provider/model/dim metadata, and decay. The useful feature to steal is bounded raw fallback recall. |++## What ai-memory Already Implements Well++### Storage And Durability++ai-memory's store matches the strongest lesson from the issue research:+keep durable storage and mutation ordering boring. `Store::open` uses+SQLite in WAL mode with foreign keys and migrations. `WriterHandle`+serializes all mutations through one writer thread. Page writes go+through `Wiki::write_page` or `Wiki::apply_batch`, which also update the+store rows and FTS triggers.++This directly avoids the major failure classes in cognee and MemPalace:+parallel SQLite deadlocks, uncoordinated writer processes, Chroma/HNSW+corruption, and write-success-before-durable-index bugs.++### Source Of Truth++The markdown wiki is organized as+`wiki/<workspace_id>/<project_id>/<page-path>`, so project renames do+not move files and two projects can use the same page path without+collision. That preserves basic-memory's best property, human-editable+plain files, while avoiding the permalink/workspace retrofit problems+documented in basic-memory's tracker.++### Capture And Handoff++The hook router accepts lifecycle events over HTTP, returns `202` fast,+and does the writer work outside the agent's critical path. `SessionEnd`+creates a deterministic session page, ends the session, opens a typed+handoff, and commits the wiki. `PreCompact` checkpoints state, using the+LLM consolidator if configured and deterministic synthesis otherwise.++The explicit `memory_handoff_begin` / `memory_handoff_accept` model is a+cleaner version of agentmemory's informal handoff behavior.++### Consolidation Model++ai-memory implements versioned page supersession (`is_latest`,+`supersedes`) and structured JSON-schema consolidation. Multi-page+consolidation can fan out into `sessions/`, `concepts/`, `decisions/`,+`gotchas/`, and `_rules/`. Rule routing is a useful extension beyond the+prior art: durable project rules are surfaced as `_rules/` pages and+lint suggests moving them into `CLAUDE.md` / `AGENTS.md` where agents see+them every turn.++### Retrieval And Decay++The current retrieval path is FTS5 by default, with graph-neighbor RRF+and optional RRF fusion against stored page embeddings. Raw observation+FTS is used as a bounded fallback when compiled wiki pages miss. The+retention sweep implements the agentmemory-style decay/reinforcement+formula for episodic pages and exempts semantic, procedural, pinned, and+slot pages. Query and recent-page reads bump access counters, which feed+the reinforcement term.++### Operability++The CLI is mostly a thin HTTP client to the server. State-changing admin+routes cover backup, bootstrap, reorg, lint, forget sweep, embed,+commit, purge, rename, and write-page. This preserves the single source+of truth: the running server owns state.++## Gaps And Improvement Opportunities++### P0: Add Bounded Editable Memory Slots++agentmemory's most useful missing idea is the slot system: small,+human-editable, durable memory blocks for project context, user+preferences, current focus, and pending items. ai-memory now implements+this as pinned `_slots/` markdown pages.++Recommended shape:++| Slot | Purpose | Injection policy |+|---|---|---|+| `_slots/project_context.md` | Stable project summary and active architecture constraints. | Small budget, eligible for session-start digest. |+| `_slots/user_preferences.md` | User preferences that are not project rules. | Small budget, never auto-expanded. |+| `_slots/current_focus.md` | What the next session should know before searching. | Expires or is overwritten frequently. |+| `_slots/pending_items.md` | Short actionable queue distinct from handoffs. | Prefer bullets, capped length. |++The important constraint is budget. Do not copy agentmemory's broad+always-injected context problem. Slots should be tiny, explicit, and+auditable.++### P0: Make Maintenance Scheduled, Not Only Manual++ai-memory already has the machinery for lint, forget sweep, embeddings,+and consolidation. Most of it is still on-demand. A memory system should+quietly maintain itself.++Recommended next step: a server-side scheduler with clear intervals,+logs, and dry-run-equivalent summaries for:++| Job | Why |+|---|---|+| forget sweep | Make the retention formula real without relying on the user to ask. |+| lint | Surface stale pages, duplicate titles, rule suggestions, and contradictions before they accumulate. |+| embedding backfill | Keep hybrid retrieval complete after pages are created without embeddings or after provider recovery. |+| optional consolidation queue | Keep session summaries compiled without blocking hooks. |++This should stay outside hook latency. Agentmemory's hook-blocking+incidents are the warning label.++### P0: Clarify And Fix The Vector Indexing Contract++There was a concrete mismatch in the implementation notes: `Wiki`'s+comment implied `write_page` / `apply_batch` both computed embeddings+when an embedder was attached, but the code only embedded in+`write_page`. The comment now states the contract explicitly:+`apply_batch` keeps SQL/file fan-out atomic and vector completeness is+owned by admin or scheduled backfill.++Also, `write_page` logs and continues if the embedder fails. That is a+reasonable availability tradeoff, but docs and invariants should be+precise: FTS5/page persistence is transactional; vector completeness is+best-effort unless a backfill/status mechanism says otherwise.++Recommended options:++| Option | Tradeoff |+|---|---|+| Embed synchronously in `apply_batch` too | Stronger completeness, slower consolidation, external provider failures can delay writes. |+| Add `embedding_status` and scheduled backfill | Better availability and observability, slightly more schema/state. |+| Document vector indexing as optional best-effort | Minimal change, but less trustworthy without visible status. |++The best pragmatic path is status plus scheduled backfill. It preserves+the durable FTS path while making vector gaps visible.++### P1: Make Graph/Link Retrieval First-Class++The schema has a `links` table with nullable `to_page_id`. Markdown+parsing now extracts wikilinks and markdown links into that table, and+`memory_query` uses graph-neighbor expansion alongside FTS5 and optional+page embeddings.+This leaves two prior-art strengths untapped:++| Source | Idea to adopt |+|---|---|+| basic-memory | unresolved forward links and graph context building. |+| agentmemory | graph as a third retrieval stream in RRF. |+| cognee | triplet-like searchable relationship text, later. |++Recommended sequence:++1. Parse `[[wiki links]]` and ordinary markdown links from page bodies.+2. Store unresolved links with `to_page_id = NULL` and resolve them when+ the target appears.+3. Add a graph-neighbor expansion mode for pages already retrieved by+ FTS/vector.+4. Fold graph hits into RRF as the third signal.++Avoid a separate graph database. SQLite tables and recursive CTEs are+enough for the expected corpus size.++### P1: Add Bounded Raw/Verbatim Fallback Recall++MemPalace's strongest useful idea is not its storage stack. It is the+evidence that raw text recall can recover details that compiled summaries+drop. ai-memory reserves `raw/`; the implemented fallback searches the+durable `observations` table through `observations_fts`.++Recommended shape:++| Layer | Role |+|---|---|+| compiled wiki | Default retrieval, high signal, human-readable. |+| raw observation fallback | Only searched when compiled wiki misses or the user asks for raw/session detail. |+| retention budget | Prevent MemPalace-style bloat. |+| privacy boundary | Use the same sanitizer before raw text becomes durable/searchable. |++This gives ai-memory the best of both philosophies: compile first, but+keep a bounded escape hatch for exact details.++### P1: Add Diagnostics And Safe Heal Paths++The prior-art trackers show that users forgive complexity less than they+forgive uncertainty. ai-memory has a simple architecture, but it should+still expose health checks.++Useful diagnostics:++| Check | Why |+|---|---|+| wiki file exists for latest page row | Detect file/db drift. |+| FTS row count matches latest pages | Detect derived-index corruption. |+| embedding triples are homogeneous | Already checked at startup, also useful on demand. |+| pages missing embeddings | Supports vector-status/backfill contract. |+| unresolved/broken links | Once link parsing exists. |+| orphan sessions/observations/handoffs | Confirms FK/cascade expectations. |+| watcher degraded streak | Make reconciliation failures visible in status. |++Safe heal should rebuild derived indexes from markdown and backfill+embeddings. It should not delete source files without a graveyard step.++### P1: Add Real Retrieval Evaluation++The current recall@5 test is useful as a harness, but it is synthetic.+Do not make LongMemEval-style claims until they are backed by a real+held-out dataset.++Recommended benchmark matrix:++| Variant | Purpose |+|---|---|+| FTS5 only | Baseline, zero-LLM/zero-embedding mode. |+| vector only | Measures semantic signal independent of keywords. |+| FTS5 + vector RRF | Current hybrid path. |+| FTS5 + vector + graph RRF | Future graph path. |+| compiled wiki + raw fallback | Future MemPalace-inspired path. |++Borrow MemPalace's transparency, not its headline-chasing. Public claims+should have tests or should not be claims.++### P2: Consider Feedback/Reinforcement Beyond Access Counts++ai-memory already tracks access count and last access time. cognee's+useful idea is finer-grained feedback: which memory items were used in a+successful answer or handoff. This can wait until retrieval paths are+more mature.++Possible future signals:++| Signal | Use |+|---|---|+| page included in handoff | Boost because it supported continuity. |+| page included in explore digest | Boost if repeatedly surfaced. |+| user marks finding stale/wrong | Lower confidence or route to lint. |+| query hit ignored repeatedly | Decay ranking without deleting. |++### P2: Temporal Triples Later++MemPalace's temporal triples are useful for facts that change over time,+but ai-memory should not add them before basic link extraction and graph+retrieval exist. A future lightweight table could model+`subject/predicate/object/valid_from/valid_to/source_page_id`, but adding+that now would be speculative.++## Ideas Not To Copy++| Temptation | Why not |+|---|---|+| agentmemory's iii-engine sidecar | Install and ops fragility were the largest pain cluster. |+| JSON blobs as primary state with in-memory indexes | Scaling, rebuild, and durability issues. |+| XML or regex-parsed LLM output | Fragile extraction; ai-memory's JSON-schema path is better. |+| 50+ MCP tools and 100+ REST endpoints | Burns context and confuses agents. |+| Broad automatic context injection | Token costs and stale context outweigh convenience. |+| basic-memory's manual `write_note` workflow | Ambient capture is ai-memory's core advantage. |+| LLM-authored markdown grammar as storage API | Forces models to generate syntax instead of meaning. |+| cognee's LiteLLM/Instructor gateway | Provider drift and silent kwarg drops are documented failure modes. |+| cognee's three-store sync | Correctness bugs concentrate at graph/vector/relational seams. |+| MemPalace's Chroma/HNSW surface | Concurrent-writer corruption and native-binding failures. |+| Verbatim-everything with no decay | Leads to bloat and later destructive cleanup pressure. |+| Destructive repair tooling | Any repair/rebuild must be graveyard-first or derived-index-only. |++## Documentation Mismatches To Fix++These are not necessarily product bugs, but docs should match the code.++| Area | Current status |+|---|---|+| MCP tool count/list | Fixed: architecture and install docs list the current 11-tool surface. |+| Lint scope | Some descriptions mention broken cross-reference auditing, but current lint implements stale episodic pages, duplicate titles, rule suggestions, and optional LLM contradictions. Link checking should wait for link parsing. |+| sqlite-vec status | Some design language frames sqlite-vec as the chosen/current vector index. Current implementation stores packed vectors in SQLite and does brute-force cosine. |+| raw archive status | Fixed: docs distinguish reserved `raw/` files from implemented observation-FTS fallback. |+| scheduled maintenance | Fixed: docs describe default scheduled lint/sweep and opt-in embedding backfill. |+| procedural tier | The type exists, but procedural extraction/frequency decay is not a developed lifecycle path yet. |+| SessionEnd consolidation | SessionEnd currently writes a deterministic session page and handoff. LLM consolidation is opt-in and available through PreCompact/manual consolidation; docs should avoid implying all SessionEnd pages are LLM-fanned-out automatically. |+| batch embeddings | Fixed: code comments and status diagnostics now describe batch vector completeness as backfill-owned. |++## Implemented Roadmap Slice++1. **Tighten docs and vector status first.** Missing embeddings are now+ visible in status diagnostics.+2. **Add scheduled jobs.** Periodic lint and forget sweep run by default;+ embedding backfill is available but opt-in.+3. **Add slots.** `_slots/` pages are pinned and surfaced through briefing.+4. **Add link extraction and graph RRF.** The existing SQLite `links`+ table powers graph-neighbor expansion.+5. **Add bounded raw fallback recall.** Observation FTS handles exact-detail+ fallback when compiled pages miss.+6. **Make retrieval claims test-backed.** The recall harness covers the new+ graph and raw fallback paths.++## Bottom Line++ai-memory followed agentmemory most closely at the idea level and made+the right substrate choices to avoid agentmemory's worst operational+costs. The implemented follow-up kept that discipline: slots, scheduled+decay/lint, graph-aware retrieval, raw fallback, and diagnostics landed+without adding a sidecar, broad MCP surface, or multi-store sync layer.diff --git a/docs/research-agentmemory.md b/docs/research-agentmemory.mdindex 2535c9a9..90adb071 100644--- a/docs/research-agentmemory.md+++ b/docs/research-agentmemory.md@@ -1,8 +1,8 @@# agentmemory - Research Report-> Source project: `~/Projects/agentmemory` (TypeScript, MCP server built on `iii-engine`).-> This is the user's own existing project. The Rust effort in this repo is a-> spiritual successor: keep the *ideas*, replace the *substrate*.+> Source project: `rohitg00/agentmemory` (TypeScript, MCP server built on `iii-engine`).+> This repo builds on that earlier TypeScript project: keep the *ideas*, replace+> the *substrate*.## 1. Purpose & Scopediff --git a/docs/research-basic-memory.md b/docs/research-basic-memory.mdindex a400b2e6..a6fcc05b 100644--- a/docs/research-basic-memory.md+++ b/docs/research-basic-memory.md@@ -1,6 +1,6 @@# basic-memory - Research Report-> Source project: `~/Projects/basic-memory` (Python, MCP-native, markdown-on-disk).+> Source project: `basicmachines-co/basic-memory` (Python, MCP-native, markdown-on-disk).> Studied as inspiration *and* as the manual-write-note model we explicitly> diverge from.diff --git a/docs/research-cognee.md b/docs/research-cognee.mdindex 9275a602..12b84462 100644--- a/docs/research-cognee.md+++ b/docs/research-cognee.md@@ -1,6 +1,6 @@# cognee - Research Report-> Source project: `~/Projects/cognee` (Python, knowledge-graph + vector + relational, MCP server).+> Source project: `topoteretes/cognee` (Python, knowledge-graph + vector + relational, MCP server).## 1. Purpose & Scope