Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat: add graph retrieval and memory maintenance

AkitaOnRails43be7a92f95e
diff --git a/README.md b/README.md
index fdf273d6..2d22a185 100644
--- a/README.md
+++ b/README.md
@@ -71,7 +71,7 @@ priors are at the [bottom](#influences-and-prior-art).
terminal) - FTS5 over the wiki. Pages are LLM-consolidated, so
the hit is a coherent decision page, not a raw chat log.
- **"This new project has months of history before ai-memory."**
- `cd ~/Projects/<that-project> && ai-memory bootstrap` collects
+ `cd /path/to/my-project && ai-memory bootstrap` collects
`git log`, README, `docs/`, module headers, project rules and
one-shot-summarises them into seed wiki pages. Future sessions
build on top.
@@ -186,6 +186,13 @@ wired up alone. Set `AI_MEMORY_NO_VERSION_CHECK=1` to silence the
daily check, or `AI_MEMORY_WRAPPER_URL=<url>` to pin the self-upgrade
source (e.g. a fork or a tagged release).
+After the upgraded server starts, ai-memory applies SQLite schema
+migrations automatically (`Store::open`) and runs pending wiki-structure
+migrations before the watcher starts. No manual database reset or wiki
+rewrite is required for normal upgrades; derived indexes such as FTS
+tables are rebuilt/backfilled by migrations or the watcher reconciliation
+path.
+
> **If your server runs on a different host** (scenarios C/D), `ai-memory upgrade`
> only refreshes the local wrapper, your local image, and your hook scripts. You
> still need to redeploy the server separately — run `bin/deploy` on the homelab
@@ -476,7 +483,7 @@ snippet is installed (see "Nudge the agent" below):
| You say… | Agent calls… | Effect |
|---|---|---|
-| "Have we discussed X?" / "search memory for Y" | `memory_query` | FTS5 over consolidated wiki pages; returns ranked snippets with `<mark>`-highlighted hits. |
+| "Have we discussed X?" / "search memory for Y" | `memory_query` | FTS5 + link-neighbour RRF over compiled wiki pages, optional vector RRF, and bounded raw observation fallback when pages miss. |
| (before proposing architecture) implicit | `memory_query` | The routing snippet tells the agent to check prior decisions / gotchas before proposing anything, so you don't get contradicted-by-its-own-history suggestions. |
| "Catch me up" / "I've been away" | `memory_explore` | Prose digest. Verbosity auto-scales to time-since-last-activity: < 1 h → one line, > 30 days → full catchup. |
| "Where did we leave off?" | (handoff already prepended) | SessionStart hook already prepended the handoff before your first prompt; the agent just reads from that block. |
@@ -656,8 +663,8 @@ OPENAI_API_KEY=ollama-local
```
Skipping the embedding provider entirely is fine -
-`memory_query` falls back to pure FTS5 (BM25) and still
-works; you just lose vector re-ranking.
+`memory_query` still uses FTS5, link-neighbour expansion, and raw
+fallback; you just lose vector re-ranking.
Per-tier feature breakdown + the openai-compat / Ollama setup
is in [`docs/install.md`](docs/install.md#llm-provider-tiers).
@@ -682,10 +689,12 @@ ingress. The server queues writes through a single SQLite writer
(no `database is locked`). On session end an optional LLM-driven pass
rewrites pages atomically with supersession (`is_latest=false` +
`supersedes` chain) and opens a typed handoff for the next agent.
-The retention sweep decays unused episodic content while semantic
-concept pages compound forever; pinned pages are exempt. Retrieval
-is FTS5 by default; when an embedder is configured, hybrid RRF over
-`page_embeddings` joins the FTS5 ranks.
+The scheduled retention sweep decays unused episodic content while
+semantic concept pages compound forever; pinned pages and `_slots/`
+memory slots are exempt. Retrieval is FTS5 + link-neighbour RRF by
+default; when an embedder is configured, vector RRF over
+`page_embeddings` joins the ranks. If compiled wiki search misses,
+bounded raw observation FTS provides exact-detail fallback.
See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the canonical
data-flow diagram + crate breakdown + cross-cutting invariants.
diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rs
index 78de7c54..47494ce9 100644
--- a/crates/ai-memory-cli/src/commands/serve.rs
+++ b/crates/ai-memory-cli/src/commands/serve.rs
@@ -2,12 +2,12 @@
use std::sync::Arc;
-use ai_memory_consolidate::Consolidator;
+use ai_memory_consolidate::{Consolidator, run_lint, run_sweep};
use ai_memory_core::{ActiveProject, ProjectId, Sanitizer, WorkspaceId};
use ai_memory_hooks::{HookState, hook_router};
use ai_memory_llm::{Embedder, LlmProvider, build_embedder, build_provider};
use ai_memory_mcp::{AdminState, AiMemoryServer, admin_router};
-use ai_memory_store::{ReaderPool, Store};
+use ai_memory_store::{ReaderPool, Store, WriterHandle, f32_vec_to_bytes};
use ai_memory_web;
use ai_memory_wiki::{WatcherHandle, Wiki, migrations, run_wiki_migrations};
use anyhow::{Context, Result};
@@ -26,7 +26,7 @@ use tracing::info;
use crate::auth::{AuthState, require_bearer};
use crate::cli::{ServeArgs, TransportKind};
-use crate::config::Config;
+use crate::config::{Config, MaintenanceSettings};
/// 10 MB cap on inbound HTTP bodies. The /hook ingress accepts the
/// agent's raw payload which can include a tool output excerpt
@@ -101,6 +101,17 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {
let server = consolidator_setup.server;
let consolidator = consolidator_setup.consolidator;
let admin_llm = consolidator_setup.admin_llm;
+ let _maintenance_tasks = start_maintenance_scheduler(
+ config.maintenance.clone(),
+ store.reader.clone(),
+ store.writer.clone(),
+ wiki.clone(),
+ embedder.clone(),
+ admin_llm.clone(),
+ ws,
+ proj,
+ config.decay,
+ );
match args.transport {
TransportKind::Stdio => {
@@ -211,6 +222,183 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {
Ok(())
}
+#[allow(clippy::too_many_arguments)]
+fn start_maintenance_scheduler(
+ settings: MaintenanceSettings,
+ reader: ReaderPool,
+ writer: WriterHandle,
+ wiki: Wiki,
+ embedder: Option<Arc<dyn Embedder>>,
+ llm: Option<Arc<dyn LlmProvider>>,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ decay: ai_memory_store::DecayParams,
+) -> Vec<tokio::task::JoinHandle<()>> {
+ if !settings.enabled {
+ info!("scheduled maintenance disabled");
+ return Vec::new();
+ }
+
+ let forget_sweep_interval_secs = settings.forget_sweep_interval_secs;
+ let lint_interval_secs = settings.lint_interval_secs;
+ let embedding_backfill_interval_secs = settings.embedding_backfill_interval_secs;
+
+ let mut tasks = Vec::new();
+ if forget_sweep_interval_secs > 0 {
+ let reader = reader.clone();
+ let writer = writer.clone();
+ tasks.push(tokio::spawn(async move {
+ let interval = std::time::Duration::from_secs(forget_sweep_interval_secs);
+ loop {
+ tokio::time::sleep(interval).await;
+ match run_sweep(&reader, &writer, workspace_id, project_id, &decay, false).await {
+ Ok(report) => info!(
+ evicted = report.evicted.len(),
+ hard_deleted = report.hard_deleted,
+ "scheduled forget sweep completed"
+ ),
+ Err(e) => tracing::warn!(error = %e, "scheduled forget sweep failed"),
+ }
+ }
+ }));
+ }
+
+ if lint_interval_secs > 0 {
+ let reader = reader.clone();
+ let wiki = wiki.clone();
+ let llm = llm.clone();
+ tasks.push(tokio::spawn(async move {
+ let interval = std::time::Duration::from_secs(lint_interval_secs);
+ loop {
+ tokio::time::sleep(interval).await;
+ match run_lint(
+ &reader,
+ &wiki,
+ llm.as_ref(),
+ workspace_id,
+ project_id,
+ false,
+ false,
+ )
+ .await
+ {
+ Ok(report) => info!(
+ findings = report.findings.len(),
+ "scheduled rule-based lint completed"
+ ),
+ Err(e) => tracing::warn!(error = %e, "scheduled lint failed"),
+ }
+ }
+ }));
+ }
+
+ if embedding_backfill_interval_secs > 0 {
+ if let Some(embedder) = embedder {
+ let reader = reader.clone();
+ let writer = writer.clone();
+ let wiki = wiki.clone();
+ tasks.push(tokio::spawn(async move {
+ let interval = std::time::Duration::from_secs(embedding_backfill_interval_secs);
+ loop {
+ tokio::time::sleep(interval).await;
+ match run_embedding_backfill(
+ &reader,
+ &writer,
+ &wiki,
+ &embedder,
+ workspace_id,
+ project_id,
+ )
+ .await
+ {
+ Ok((embedded, failed)) => {
+ info!(embedded, failed, "scheduled embedding backfill completed")
+ }
+ Err(e) => tracing::warn!(error = %e, "scheduled embedding backfill failed"),
+ }
+ }
+ }));
+ } else {
+ tracing::warn!(
+ "maintenance.embedding_backfill_interval_secs is set but no embedder is configured"
+ );
+ }
+ }
+
+ if tasks.is_empty() {
+ info!("scheduled maintenance enabled but all intervals are disabled");
+ } else {
+ info!(jobs = tasks.len(), "scheduled maintenance started");
+ }
+ tasks
+}
+
+async fn run_embedding_backfill(
+ reader: &ReaderPool,
+ writer: &WriterHandle,
+ wiki: &Wiki,
+ embedder: &Arc<dyn Embedder>,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+) -> Result<(usize, usize)> {
+ let provider = embedder.provider().to_string();
+ let model = embedder.model().to_string();
+ let dim = embedder.dim();
+ let candidates = reader.decay_candidates(workspace_id, project_id).await?;
+ let already: std::collections::HashSet<_> = reader
+ .load_embeddings(
+ workspace_id,
+ project_id,
+ provider.clone(),
+ model.clone(),
+ dim,
+ )
+ .await?
+ .into_iter()
+ .map(|s| s.id)
+ .collect();
+
+ let mut embedded = 0usize;
+ let mut failed = 0usize;
+ for cand in candidates {
+ if already.contains(&cand.id) {
+ continue;
+ }
+ let md = match wiki.read_page(workspace_id, project_id, &cand.path) {
+ Ok(md) => md,
+ Err(e) => {
+ failed += 1;
+ tracing::warn!(path = %cand.path, error = %e, "scheduled embed: unreadable page");
+ continue;
+ }
+ };
+ let vec = match embedder.embed(&md.body).await {
+ Ok(vec) => vec,
+ Err(e) => {
+ failed += 1;
+ tracing::warn!(path = %cand.path, error = %e, "scheduled embed: provider failed");
+ continue;
+ }
+ };
+ if let Err(e) = writer
+ .store_embedding(
+ cand.id,
+ f32_vec_to_bytes(&vec),
+ provider.clone(),
+ model.clone(),
+ dim,
+ )
+ .await
+ {
+ failed += 1;
+ tracing::warn!(path = %cand.path, error = %e, "scheduled embed: store failed");
+ continue;
+ }
+ embedded += 1;
+ }
+ Ok((embedded, failed))
+}
+
async fn configure_embedder(
config: &Config,
store: &Store,
diff --git a/crates/ai-memory-cli/src/commands/status.rs b/crates/ai-memory-cli/src/commands/status.rs
index f7a16522..7a49eec2 100644
--- a/crates/ai-memory-cli/src/commands/status.rs
+++ b/crates/ai-memory-cli/src/commands/status.rs
@@ -5,14 +5,14 @@
//! the store directly — the server is the source of truth.
use anyhow::Result;
-use serde::Deserialize;
+use serde::{Deserialize, Serialize};
use crate::cli::StatusArgs;
use crate::config::Config;
use crate::http_client::{ServerEndpoint, get_json};
/// Server-shaped response. Mirrors `ai_memory_mcp::admin::StatusReport`.
-#[derive(Debug, Deserialize)]
+#[derive(Debug, Deserialize, Serialize)]
struct Report {
/// Server binary version.
version: String,
@@ -24,9 +24,12 @@ struct Report {
db_path: String,
/// Lifetime counts.
counts: Counts,
+ /// Derived-index diagnostics.
+ #[serde(default)]
+ derived: Derived,
}
-#[derive(Debug, Deserialize)]
+#[derive(Debug, Deserialize, Serialize)]
struct Counts {
pages_latest: u64,
pages_all: u64,
@@ -34,6 +37,28 @@ struct Counts {
observations: u64,
}
+#[derive(Debug, Default, Deserialize, Serialize)]
+struct Derived {
+ pages_rows: u64,
+ pages_fts_rows: u64,
+ observations_rows: u64,
+ observations_fts_rows: u64,
+ latest_pages_missing_embeddings: u64,
+ embedding_rows: u64,
+ embedding_triples: Vec<EmbeddingTriple>,
+ links_from_latest_pages: u64,
+ unresolved_links_from_latest_pages: u64,
+ stale_links_from_latest_pages: u64,
+}
+
+#[derive(Debug, Deserialize, Serialize)]
+struct EmbeddingTriple {
+ provider: String,
+ model: String,
+ dim: u32,
+ count: u64,
+}
+
/// Run the `status` subcommand.
///
/// # Errors
@@ -57,6 +82,7 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {
"sessions": report.counts.sessions,
"observations": report.counts.observations,
},
+ "derived": report.derived,
"client": { "server_url": ep.url, "auth": ep.auth_token.is_some() },
}))?
);
@@ -72,6 +98,23 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {
);
println!(" sessions: {}", report.counts.sessions);
println!(" observations: {}", report.counts.observations);
+ println!(
+ " fts: pages {}/{}; observations {}/{}",
+ report.derived.pages_fts_rows,
+ report.derived.pages_rows,
+ report.derived.observations_fts_rows,
+ report.derived.observations_rows
+ );
+ println!(
+ " embeddings: {} rows; {} latest pages missing",
+ report.derived.embedding_rows, report.derived.latest_pages_missing_embeddings
+ );
+ println!(
+ " links: {} latest-page links (unresolved: {}, stale: {})",
+ report.derived.links_from_latest_pages,
+ report.derived.unresolved_links_from_latest_pages,
+ report.derived.stale_links_from_latest_pages
+ );
}
Ok(())
}
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index b97b631a..4b71dd34 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -72,6 +72,8 @@ pub struct Config {
/// hard-deletion. Tune `decay.lambda` down to slow decay or
/// `decay.cold_threshold` to evict more / less aggressively.
pub decay: ai_memory_store::DecayParams,
+ /// Server-side scheduled maintenance. Jobs run outside hook latency.
+ pub maintenance: MaintenanceSettings,
/// Privacy-strip tuning. Built-in patterns always run; this section
/// lets the operator extend or punch holes in them.
pub sanitize: ai_memory_core::SanitizeConfig,
@@ -190,6 +192,7 @@ impl Default for Config {
embedding_dim: None,
embedding_base_url: None,
decay: ai_memory_store::DecayParams::default(),
+ maintenance: MaintenanceSettings::default(),
sanitize: ai_memory_core::SanitizeConfig::default(),
auth: AuthSettings::default(),
allowed_hosts: vec!["localhost".into(), "127.0.0.1".into(), "::1".into()],
@@ -198,6 +201,32 @@ impl Default for Config {
}
}
+/// `[maintenance]` scheduled server jobs.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(default)]
+pub struct MaintenanceSettings {
+ /// Master switch for scheduled jobs.
+ pub enabled: bool,
+ /// Interval for the retention forget sweep. `0` disables this job.
+ pub forget_sweep_interval_secs: u64,
+ /// Interval for rule-based wiki lint. `0` disables this job.
+ pub lint_interval_secs: u64,
+ /// Interval for embedding backfill. `0` disables this job.
+ /// Defaults to off because it may call a paid provider.
+ pub embedding_backfill_interval_secs: u64,
+}
+
+impl Default for MaintenanceSettings {
+ fn default() -> Self {
+ Self {
+ enabled: true,
+ forget_sweep_interval_secs: 86_400,
+ lint_interval_secs: 86_400,
+ embedding_backfill_interval_secs: 0,
+ }
+ }
+}
+
impl Config {
/// Load the merged configuration: defaults → file → env → CLI.
///
@@ -421,6 +450,10 @@ mod tests {
assert_eq!(cfg.bind, DEFAULT_BIND);
assert_eq!(cfg.server_url, DEFAULT_SERVER_URL);
assert_eq!(cfg.log_level, "info");
+ assert!(cfg.maintenance.enabled);
+ assert_eq!(cfg.maintenance.forget_sweep_interval_secs, 86_400);
+ assert_eq!(cfg.maintenance.lint_interval_secs, 86_400);
+ assert_eq!(cfg.maintenance.embedding_backfill_interval_secs, 0);
}
#[test]
@@ -449,6 +482,10 @@ mod tests {
r#"
bind = "0.0.0.0:9999"
log_level = "debug"
+
+ [maintenance]
+ enabled = false
+ lint_interval_secs = 3600
"#,
)
.unwrap();
@@ -458,5 +495,7 @@ mod tests {
let cfg = Config::load(Some(&cfg_path), Some(tmp.path().to_path_buf())).unwrap();
assert_eq!(cfg.bind, "0.0.0.0:9999");
assert_eq!(cfg.log_level, "debug");
+ assert!(!cfg.maintenance.enabled);
+ assert_eq!(cfg.maintenance.lint_interval_secs, 3600);
}
}
diff --git a/crates/ai-memory-consolidate/tests/recall_eval.rs b/crates/ai-memory-consolidate/tests/recall_eval.rs
index 9ad526bb..229940c2 100644
--- a/crates/ai-memory-consolidate/tests/recall_eval.rs
+++ b/crates/ai-memory-consolidate/tests/recall_eval.rs
@@ -1,8 +1,9 @@
//! Recall@5 eval harness.
//!
//! Loads a small hand-crafted corpus + a probe set, measures recall@5
-//! against both the pure-FTS5 path and the hybrid (FTS5 + RRF + vector)
-//! path, and asserts a baseline. The point is *the framework*: once the
+//! against both the pure-FTS5 path and the hybrid (FTS5 + graph RRF +
+//! vector) path, and asserts a baseline. It also pins graph-neighbor
+//! expansion and raw observation fallback. The point is *the framework*: once the
//! harness is in CI, anyone can drop in real embeddings (set the
//! `AI_MEMORY_EMBEDDING_PROVIDER` env vars + plug in a real model) and
//! watch the numbers move. The synthetic embedder shipped for tests is
@@ -15,7 +16,9 @@
use std::sync::Arc;
-use ai_memory_core::{PagePath, Tier};
+use ai_memory_core::{
+ AgentKind, NewObservation, NewSession, ObservationKind, PagePath, SessionId, Tier,
+};
use ai_memory_llm::{Embedder, SyntheticEmbedder};
use ai_memory_store::Store;
use ai_memory_wiki::{Wiki, WritePageRequest};
@@ -161,6 +164,139 @@ async fn recall_at_5_baseline() {
);
}
+#[tokio::test]
+async fn graph_neighbor_expansion_recovers_linked_page() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("open store");
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .expect("ws");
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "eval", None)
+ .await
+ .expect("proj");
+ let wiki = Wiki::new(tmp.path(), store.writer.clone()).expect("wiki");
+
+ wiki.write_page(WritePageRequest {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new("notes/hidden_target.md").expect("path"),
+ frontmatter: serde_json::json!({"title": "Hidden target"}),
+ body: "This page contains implementation detail but not the seed term.".into(),
+ tier: Tier::Semantic,
+ pinned: false,
+ title: None,
+ })
+ .await
+ .expect("write target");
+ wiki.write_page(WritePageRequest {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new("notes/source.md").expect("path"),
+ frontmatter: serde_json::json!({"title": "Source"}),
+ body: "graphseed points to [[notes/hidden_target]].".into(),
+ tier: Tier::Semantic,
+ pinned: false,
+ title: None,
+ })
+ .await
+ .expect("write source");
+
+ let fts_hits = store
+ .reader
+ .search_pages_for_project(ws, proj, "graphseed".into(), 5)
+ .await
+ .expect("fts search");
+ assert!(
+ !fts_hits
+ .iter()
+ .any(|hit| hit.path.as_str() == "notes/hidden_target.md"),
+ "pure FTS should not find the target without the seed term"
+ );
+
+ let hybrid_hits = store
+ .reader
+ .hybrid_search(
+ ws,
+ proj,
+ "graphseed".into(),
+ None,
+ String::new(),
+ String::new(),
+ 0,
+ 5,
+ )
+ .await
+ .expect("hybrid search");
+ assert!(
+ hybrid_hits
+ .iter()
+ .any(|hit| hit.path.as_str() == "notes/hidden_target.md"),
+ "graph expansion should include the linked target"
+ );
+}
+
+#[tokio::test]
+async fn raw_observation_fallback_recovers_detail_when_wiki_misses() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("open store");
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .expect("ws");
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "eval", None)
+ .await
+ .expect("proj");
+ let session_id = SessionId::new();
+
+ store
+ .writer
+ .begin_session(NewSession {
+ id: session_id,
+ workspace_id: ws,
+ project_id: proj,
+ agent_kind: AgentKind::OpenCode,
+ cwd: None,
+ })
+ .await
+ .expect("begin session");
+ store
+ .writer
+ .insert_observation(NewObservation {
+ session_id,
+ workspace_id: ws,
+ project_id: proj,
+ kind: ObservationKind::UserPrompt,
+ title: "raw prompt".into(),
+ body: "raw fallback detail mentions capybara exactly once".into(),
+ importance: 5,
+ })
+ .await
+ .expect("insert observation");
+
+ let page_hits = store
+ .reader
+ .search_pages_for_project(ws, proj, "capybara".into(), 5)
+ .await
+ .expect("page search");
+ assert!(page_hits.is_empty(), "compiled wiki should miss");
+
+ let raw_hits = store
+ .reader
+ .search_observations_for_project(ws, proj, "capybara".into(), 5)
+ .await
+ .expect("raw search");
+ assert_eq!(raw_hits.len(), 1);
+ assert_eq!(raw_hits[0].session_id, session_id);
+ assert!(raw_hits[0].snippet.contains("<mark>capybara</mark>"));
+}
+
async fn measure_recall(
store: &Store,
ws: ai_memory_core::WorkspaceId,
diff --git a/crates/ai-memory-core/src/page.rs b/crates/ai-memory-core/src/page.rs
index deb0d755..e0559f41 100644
--- a/crates/ai-memory-core/src/page.rs
+++ b/crates/ai-memory-core/src/page.rs
@@ -50,6 +50,11 @@ pub struct NewPage {
pub frontmatter_json: serde_json::Value,
/// `true` to mark the page as user-pinned (never decay).
pub pinned: bool,
+ /// Outgoing links discovered in the page body.
+ ///
+ /// The store resolves these against the latest page rows in the same
+ /// project and keeps unresolved forward links as `to_page_id = NULL`.
+ pub links: Vec<PagePath>,
}
/// Materialised view of a page row.
diff --git a/crates/ai-memory-hooks/src/synth.rs b/crates/ai-memory-hooks/src/synth.rs
index 7e335f35..66d09575 100644
--- a/crates/ai-memory-hooks/src/synth.rs
+++ b/crates/ai-memory-hooks/src/synth.rs
@@ -41,6 +41,7 @@ pub fn synthesize_session_page(
"tier": "episodic",
}),
pinned: false,
+ links: Vec::new(),
}
}
diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rs
index 9b7d6d19..2cbdae77 100644
--- a/crates/ai-memory-mcp/src/admin.rs
+++ b/crates/ai-memory-mcp/src/admin.rs
@@ -235,23 +235,32 @@ pub struct StatusReport {
pub db_path: String,
/// Lifetime counts: pages_latest, pages_all, sessions, observations.
pub counts: ai_memory_store::StatusCounts,
+ /// Derived-index and retrieval-readiness diagnostics.
+ pub derived: ai_memory_store::DerivedIndexStatus,
}
async fn handle_status(State(state): State<Arc<AdminState>>) -> impl IntoResponse {
match state.reader.status_counts().await {
- Ok(counts) => {
- let report = StatusReport {
- version: env!("CARGO_PKG_VERSION").to_string(),
- data_dir: state.data_dir.display().to_string(),
- bind: state.bind.clone(),
- db_path: state.db_path.display().to_string(),
- counts,
- };
- (
- StatusCode::OK,
- Json(serde_json::to_value(&report).unwrap_or_else(|_| serde_json::json!({}))),
- )
- }
+ Ok(counts) => match state.reader.derived_index_status().await {
+ Ok(derived) => {
+ let report = StatusReport {
+ version: env!("CARGO_PKG_VERSION").to_string(),
+ data_dir: state.data_dir.display().to_string(),
+ bind: state.bind.clone(),
+ db_path: state.db_path.display().to_string(),
+ counts,
+ derived,
+ };
+ (
+ StatusCode::OK,
+ Json(serde_json::to_value(&report).unwrap_or_else(|_| serde_json::json!({}))),
+ )
+ }
+ Err(e) => (
+ StatusCode::INTERNAL_SERVER_ERROR,
+ Json(serde_json::json!({ "error": e.to_string() })),
+ ),
+ },
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({ "error": e.to_string() })),
diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rs
index 5702e02c..334b0f33 100644
--- a/crates/ai-memory-mcp/src/server.rs
+++ b/crates/ai-memory-mcp/src/server.rs
@@ -153,6 +153,13 @@ struct QueryResponse<T: Serialize> {
hits: Vec<T>,
}
+#[derive(Debug, Serialize)]
+struct MemoryQueryResponse {
+ hits: Vec<ai_memory_store::PageHit>,
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ raw_hits: Vec<ai_memory_store::ObservationHit>,
+}
+
#[derive(Debug, Serialize)]
struct StatusResponse {
counts: ai_memory_store::StatusCounts,
@@ -371,22 +378,25 @@ impl AiMemoryServer {
self
}
- /// Full-text search the wiki via FTS5. Returns up to `limit` hits with
- /// HTML-marked snippets and a rank score.
+ /// Search the compiled wiki via FTS5/vector/graph retrieval. Falls back
+ /// to bounded raw observation search when no compiled page matches.
#[tool(description = "Search the project's long-term memory wiki — \
prior sessions, decisions, gotchas, architecture notes captured \
by ai-memory across earlier runs. Call this BEFORE proposing \
designs, BEFORE answering 'why does X work this way', and \
whenever the user references prior work you don't recognise. \
- FTS5 + (when configured) hybrid RRF re-ranking. Returns up to \
- `limit` pages with HTML-marked snippets and a rank score \
- (lower rank = better match). Only latest page versions.")]
+ FTS5 + graph RRF + (when configured) vector RRF re-ranking. \
+ Returns up to `limit` pages with HTML-marked snippets and a rank \
+ score (lower rank = better match). Only latest page versions. \
+ If compiled wiki search misses, `raw_hits` contains bounded raw \
+ observation fallback matches.")]
async fn memory_query(
&self,
Parameters(args): Parameters<QueryArgs>,
) -> Result<CallToolResult, McpError> {
let limit = args.limit.unwrap_or(self.default_limit).clamp(1, 100);
let (ws, proj) = self.effective_ids(args.project.as_deref()).await;
+ let query = args.query.clone();
// Hybrid path: try to embed the query; if it fails (provider
// outage, cold-start timeout, billing, etc.) degrade to BM25
// instead of erroring the whole query — losing semantic
@@ -398,7 +408,7 @@ impl AiMemoryServer {
.hybrid_search(
ws,
proj,
- args.query.clone(),
+ query.clone(),
Some(qv),
embedder.provider().to_string(),
embedder.model().to_string(),
@@ -415,18 +425,44 @@ impl AiMemoryServer {
"embedder failed; degrading memory_query to BM25-only"
);
self.reader
- .search_pages_for_project(ws, proj, args.query, limit)
+ .hybrid_search(
+ ws,
+ proj,
+ query.clone(),
+ None,
+ String::new(),
+ String::new(),
+ 0,
+ limit,
+ )
.await
}
}
} else {
self.reader
- .search_pages_for_project(ws, proj, args.query, limit)
+ .hybrid_search(
+ ws,
+ proj,
+ query.clone(),
+ None,
+ String::new(),
+ String::new(),
+ 0,
+ limit,
+ )
.await
};
let hits = hits.map_err(|e| McpError::internal_error(e.to_string(), None))?;
self.spawn_access_bump(hits.iter().map(|h| h.id).collect());
- let response = QueryResponse { hits };
+ let raw_hits = if hits.is_empty() {
+ self.reader
+ .search_observations_for_project(ws, proj, query, limit)
+ .await
+ .map_err(|e| McpError::internal_error(e.to_string(), None))?
+ } else {
+ Vec::new()
+ };
+ let response = MemoryQueryResponse { hits, raw_hits };
ok_json(&response)
}
@@ -920,7 +956,7 @@ const EXPLORE_SYSTEM_PROMPT: &str = include_str!("../prompts/explore_system.md")
#[cfg(test)]
mod tests {
use super::*;
- use ai_memory_core::{NewPage, PagePath, Tier};
+ use ai_memory_core::{NewObservation, NewPage, NewSession, ObservationKind, PagePath, Tier};
use ai_memory_store::Store;
use tempfile::TempDir;
@@ -948,6 +984,7 @@ mod tests {
tier: Tier::Semantic,
frontmatter_json: serde_json::json!({}),
pinned: false,
+ links: Vec::new(),
})
.await
.unwrap();
@@ -1019,6 +1056,58 @@ mod tests {
assert!(text.contains("foo.md"), "expected hit; got {text}");
}
+ #[tokio::test]
+ async fn memory_query_returns_raw_hits_when_pages_miss() {
+ let (_tmp, store, server, ws, proj) = setup_server().await;
+ let session_id = SessionId::new();
+ store
+ .writer
+ .begin_session(NewSession {
+ id: session_id,
+ workspace_id: ws,
+ project_id: proj,
+ agent_kind: AgentKind::OpenCode,
+ cwd: None,
+ })
+ .await
+ .unwrap();
+ store
+ .writer
+ .insert_observation(NewObservation {
+ session_id,
+ workspace_id: ws,
+ project_id: proj,
+ kind: ObservationKind::UserPrompt,
+ title: "raw prompt".into(),
+ body: "raw fallback contains quokka only detail".into(),
+ importance: 5,
+ })
+ .await
+ .unwrap();
+
+ let result = server
+ .memory_query(Parameters(QueryArgs {
+ query: "quokka".into(),
+ limit: Some(5),
+ project: None,
+ }))
+ .await
+ .unwrap();
+ let text = match result.content.first().and_then(|c| c.as_text()) {
+ Some(t) => t.text.clone(),
+ None => panic!("expected text content"),
+ };
+ assert!(
+ text.contains("\"hits\": []"),
+ "expected no page hits; got {text}"
+ );
+ assert!(
+ text.contains("raw_hits"),
+ "expected raw fallback; got {text}"
+ );
+ assert!(text.contains("quokka"), "expected raw snippet; got {text}");
+ }
+
#[tokio::test]
async fn memory_status_returns_counts() {
let (_tmp, _store, server, _ws, _pj) = setup_server().await;
@@ -1060,12 +1149,13 @@ mod tests {
"\"last_observation_at\":",
"\"pending_handoff_count\":",
"\"rules\":",
+ "\"slots\":",
"\"recent_pages\":",
] {
assert!(text.contains(key), "missing {key} in briefing:\n{text}");
}
// setup_server inserts one page, no sessions/observations,
- // no rules. The activity windows therefore observe zero.
+ // no rules/slots. The activity windows therefore observe zero.
assert!(
text.contains("\"sessions\": 0"),
"expected lifetime sessions: 0\n{text}"
diff --git a/crates/ai-memory-mcp/tests/admin_phase3.rs b/crates/ai-memory-mcp/tests/admin_phase3.rs
index 790f3f03..01a97b0e 100644
--- a/crates/ai-memory-mcp/tests/admin_phase3.rs
+++ b/crates/ai-memory-mcp/tests/admin_phase3.rs
@@ -225,6 +225,7 @@ async fn lint_dry_run_returns_lint_report_shape() {
tier: Tier::Semantic,
frontmatter_json: serde_json::json!({}),
pinned: false,
+ links: Vec::new(),
})
.await
.unwrap();
diff --git a/crates/ai-memory-mcp/tests/admin_status_search.rs b/crates/ai-memory-mcp/tests/admin_status_search.rs
index c5108b03..758cbaa6 100644
--- a/crates/ai-memory-mcp/tests/admin_status_search.rs
+++ b/crates/ai-memory-mcp/tests/admin_status_search.rs
@@ -52,6 +52,7 @@ async fn seed_page(store: &Store, title: &str, path: &str, body: &str) {
tier: Tier::Semantic,
frontmatter_json: serde_json::json!({}),
pinned: false,
+ links: Vec::new(),
};
store.writer.upsert_page(page).await.unwrap();
}
@@ -102,6 +103,14 @@ async fn status_returns_counts_and_paths() {
assert!(body["db_path"].as_str().unwrap().ends_with(".sqlite"));
assert_eq!(body["counts"]["pages_latest"].as_u64().unwrap(), 1);
assert_eq!(body["counts"]["pages_all"].as_u64().unwrap(), 1);
+ assert_eq!(body["derived"]["pages_rows"].as_u64().unwrap(), 1);
+ assert_eq!(body["derived"]["pages_fts_rows"].as_u64().unwrap(), 1);
+ assert_eq!(
+ body["derived"]["latest_pages_missing_embeddings"]
+ .as_u64()
+ .unwrap(),
+ 1
+ );
}
#[tokio::test]
diff --git a/crates/ai-memory-store/migrations/V07__observations_fts_and_link_index.sql b/crates/ai-memory-store/migrations/V07__observations_fts_and_link_index.sql
new file mode 100644
index 00000000..ca8b2968
--- /dev/null
+++ b/crates/ai-memory-store/migrations/V07__observations_fts_and_link_index.sql
@@ -0,0 +1,39 @@
+-- Raw observation fallback search + unresolved-link lookup support.
+--
+-- Additive migration only: existing observations are backfilled into an
+-- FTS5 index, and future inserts/updates/deletes stay synchronized by
+-- triggers. No existing pages, observations, or wiki paths are rewritten.
+
+CREATE VIRTUAL TABLE observations_fts USING fts5(
+ title, body,
+ content='observations',
+ content_rowid='rowid',
+ tokenize="unicode61 tokenchars '/_-'"
+);
+
+INSERT INTO observations_fts(rowid, title, body)
+ SELECT rowid, title, body FROM observations;
+
+CREATE TRIGGER observations_fts_ai AFTER INSERT ON observations BEGIN
+ INSERT INTO observations_fts(rowid, title, body)
+ VALUES (new.rowid, new.title, new.body);
+END;
+
+CREATE TRIGGER observations_fts_ad AFTER DELETE ON observations BEGIN
+ INSERT INTO observations_fts(observations_fts, rowid, title, body)
+ VALUES ('delete', old.rowid, old.title, old.body);
+END;
+
+CREATE TRIGGER observations_fts_au AFTER UPDATE ON observations BEGIN
+ INSERT INTO observations_fts(observations_fts, rowid, title, body)
+ VALUES ('delete', old.rowid, old.title, old.body);
+ INSERT INTO observations_fts(rowid, title, body)
+ VALUES (new.rowid, new.title, new.body);
+END;
+
+-- Link parsing existed as schema but not as a populated retrieval signal.
+-- This index makes repeated unresolved-link resolution cheap once writers
+-- start filling links(to_path, to_page_id=NULL).
+CREATE INDEX IF NOT EXISTS idx_links_unresolved_path
+ ON links(to_path)
+ WHERE to_page_id IS NULL;
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index 3ffc0beb..889eb6dc 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -23,8 +23,9 @@ pub use decay::{DecayParams, retention_score};
pub use error::{StoreError, StoreResult};
pub use ops::{PurgeSummary, ReorgSummary};
pub use reader::{
- ActivityWindow, BriefingPage, BriefingSnapshot, DecayCandidate, PageHit, PageMeta, PageSummary,
- ProjectSummary, ReaderPool, StatusCounts, StoredEmbedding, f32_vec_to_bytes,
+ ActivityWindow, BriefingPage, BriefingSnapshot, DecayCandidate, DerivedIndexStatus,
+ EmbeddingTripleCount, ObservationHit, PageHit, PageMeta, PageSummary, ProjectSummary,
+ ReaderPool, StatusCounts, StoredEmbedding, f32_vec_to_bytes,
};
pub use writer::WriterHandle;
@@ -83,7 +84,10 @@ impl Store {
#[cfg(test)]
mod tests {
use super::*;
- use ai_memory_core::{NewPage, PagePath, ProjectId, Tier, WorkspaceId};
+ use ai_memory_core::{
+ AgentKind, NewObservation, NewPage, NewSession, ObservationKind, PagePath, ProjectId,
+ SessionId, Tier, WorkspaceId,
+ };
use tempfile::TempDir;
fn sample_page(ws: WorkspaceId, proj: ProjectId, path: &str, body: &str) -> NewPage {
@@ -96,6 +100,7 @@ mod tests {
tier: Tier::Semantic,
frontmatter_json: serde_json::json!({}),
pinned: false,
+ links: Vec::new(),
}
}
@@ -305,6 +310,137 @@ mod tests {
);
}
+ #[tokio::test]
+ async fn search_quotes_hyphenated_tokens_for_fts5() {
+ let tmp = TempDir::new().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "ai-memory", None)
+ .await
+ .unwrap();
+
+ store
+ .writer
+ .upsert_page(sample_page(
+ ws,
+ proj,
+ "hyphen.md",
+ "the ai-memory token should be searchable",
+ ))
+ .await
+ .unwrap();
+
+ let hits = store
+ .reader
+ .search_pages_for_project(ws, proj, "ai-memory".into(), 10)
+ .await
+ .unwrap();
+ assert_eq!(hits.len(), 1);
+ assert_eq!(hits[0].path.as_str(), "hyphen.md");
+ }
+
+ #[tokio::test]
+ async fn hybrid_search_includes_linked_neighbors() {
+ let tmp = TempDir::new().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "ai-memory", None)
+ .await
+ .unwrap();
+
+ store
+ .writer
+ .upsert_page(sample_page(ws, proj, "target.md", "neighbor-only content"))
+ .await
+ .unwrap();
+ let mut source = sample_page(ws, proj, "source.md", "needle source content");
+ source.links = vec![PagePath::new("target.md").unwrap()];
+ store.writer.upsert_page(source).await.unwrap();
+
+ let hits = store
+ .reader
+ .hybrid_search(
+ ws,
+ proj,
+ "needle".into(),
+ None,
+ String::new(),
+ String::new(),
+ 0,
+ 10,
+ )
+ .await
+ .unwrap();
+ let paths: Vec<&str> = hits.iter().map(|h| h.path.as_str()).collect();
+ assert!(paths.contains(&"source.md"));
+ assert!(
+ paths.contains(&"target.md"),
+ "linked neighbor should be included"
+ );
+ }
+
+ #[tokio::test]
+ async fn observation_fts_finds_raw_fallback_hits() {
+ let tmp = TempDir::new().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "ai-memory", None)
+ .await
+ .unwrap();
+ let session_id = SessionId::new();
+ store
+ .writer
+ .begin_session(NewSession {
+ id: session_id,
+ workspace_id: ws,
+ project_id: proj,
+ agent_kind: AgentKind::OpenCode,
+ cwd: None,
+ })
+ .await
+ .unwrap();
+ store
+ .writer
+ .insert_observation(NewObservation {
+ session_id,
+ workspace_id: ws,
+ project_id: proj,
+ kind: ObservationKind::UserPrompt,
+ title: "prompt".into(),
+ body: "the raw-only zebra detail lives here".into(),
+ importance: 5,
+ })
+ .await
+ .unwrap();
+
+ let hits = store
+ .reader
+ .search_observations_for_project(ws, proj, "zebra".into(), 5)
+ .await
+ .unwrap();
+ assert_eq!(hits.len(), 1);
+ assert_eq!(hits[0].session_id, session_id);
+ assert!(hits[0].snippet.contains("<mark>zebra</mark>"));
+ }
+
#[tokio::test]
async fn list_projects_with_stats_returns_aggregates() {
let tmp = TempDir::new().unwrap();
@@ -400,6 +536,7 @@ mod tests {
tier: Tier::Semantic,
frontmatter_json: serde_json::json!({"kind": "decision"}),
pinned: true,
+ links: Vec::new(),
};
store.writer.upsert_page(page).await.unwrap();
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index 20468ccb..a610c6a1 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -4,9 +4,11 @@
//! the writer thread would violate the single-writer invariant (see
//! [`crate::writer`]).
+use std::collections::BTreeSet;
+
use ai_memory_core::{
AgentKind, HandoffId, NewHandoff, NewObservation, NewPage, NewSession, ObservationId,
- ObservationKind, PageId, ProjectId, SessionId, WorkspaceId,
+ ObservationKind, PageId, PagePath, ProjectId, SessionId, WorkspaceId,
};
/// Summary returned by [`reorg_sessions`] and exposed via
@@ -175,7 +177,10 @@ fn upsert_page_in_tx(
"UPDATE pages SET updated_at = ?1 WHERE id = ?2",
params![now, existing_id],
)?;
- return PageId::from_slice(&existing_id).map_err(StoreError::from);
+ let existing_page_id = PageId::from_slice(&existing_id).map_err(StoreError::from)?;
+ replace_links_in_tx(tx, &existing_page_id, page)?;
+ refresh_incoming_links_for_path(tx, page, &existing_page_id)?;
+ return Ok(existing_page_id);
}
let new_id = PageId::new();
tx.execute(
@@ -202,6 +207,8 @@ fn upsert_page_in_tx(
now,
],
)?;
+ replace_links_in_tx(tx, &new_id, page)?;
+ refresh_incoming_links_for_path(tx, page, &new_id)?;
audit(
tx,
"supersede_page",
@@ -232,6 +239,8 @@ fn upsert_page_in_tx(
now,
],
)?;
+ replace_links_in_tx(tx, &new_id, page)?;
+ refresh_incoming_links_for_path(tx, page, &new_id)?;
audit(
tx,
"create_page",
@@ -243,6 +252,79 @@ fn upsert_page_in_tx(
Ok(new_id)
}
+fn replace_links_in_tx(
+ tx: &rusqlite::Transaction<'_>,
+ from_page_id: &PageId,
+ page: &NewPage,
+) -> StoreResult<()> {
+ tx.execute(
+ "DELETE FROM links WHERE from_page_id = ?1",
+ params![from_page_id.as_bytes()],
+ )?;
+
+ let mut seen = BTreeSet::new();
+ for to_path in &page.links {
+ if !seen.insert(to_path.as_str().to_string()) {
+ continue;
+ }
+ let to_page_id = latest_page_id_for_path(tx, page, to_path)?;
+ let to_page_blob = to_page_id.as_ref().map(|id| &id.as_bytes()[..]);
+ tx.execute(
+ "INSERT INTO links (from_page_id, to_page_id, to_path, link_type) \
+ VALUES (?1, ?2, ?3, 'references')",
+ params![from_page_id.as_bytes(), to_page_blob, to_path.as_str()],
+ )?;
+ }
+ Ok(())
+}
+
+fn latest_page_id_for_path(
+ tx: &rusqlite::Transaction<'_>,
+ page: &NewPage,
+ to_path: &PagePath,
+) -> StoreResult<Option<PageId>> {
+ let bytes: Option<Vec<u8>> = tx
+ .query_row(
+ "SELECT id FROM pages \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 AND is_latest = 1",
+ params![
+ page.workspace_id.as_bytes(),
+ page.project_id.as_bytes(),
+ to_path.as_str(),
+ ],
+ |row| row.get(0),
+ )
+ .optional()?;
+ bytes
+ .map(|bytes| PageId::from_slice(&bytes).map_err(StoreError::from))
+ .transpose()
+}
+
+fn refresh_incoming_links_for_path(
+ tx: &rusqlite::Transaction<'_>,
+ page: &NewPage,
+ latest_page_id: &PageId,
+) -> StoreResult<()> {
+ tx.execute(
+ "UPDATE links \
+ SET to_page_id = ?1 \
+ WHERE to_path = ?2 \
+ AND EXISTS ( \
+ SELECT 1 FROM pages from_page \
+ WHERE from_page.id = links.from_page_id \
+ AND from_page.workspace_id = ?3 \
+ AND from_page.project_id = ?4 \
+ )",
+ params![
+ latest_page_id.as_bytes(),
+ page.path.as_str(),
+ page.workspace_id.as_bytes(),
+ page.project_id.as_bytes(),
+ ],
+ )?;
+ Ok(())
+}
+
/// Begin (or re-affirm) a session row keyed on the caller-supplied id.
/// Idempotent: a second call with the same id leaves the row untouched.
pub fn begin_session(conn: &mut Connection, session: &NewSession) -> StoreResult<()> {
@@ -749,6 +831,7 @@ mod tests {
tier: Tier::Semantic,
frontmatter_json: serde_json::json!({}),
pinned: false,
+ links: Vec::new(),
}
}
@@ -816,6 +899,39 @@ mod tests {
assert_eq!(total, 1, "no duplicate row for unchanged content");
}
+ #[test]
+ fn upsert_page_persists_and_resolves_links() {
+ let (_tmp, mut conn, ws, proj) = fresh_db();
+ let mut source = page(ws, proj, "concepts/source.md", "see target");
+ source.links = vec![PagePath::new("decisions/target.md").unwrap()];
+ let source_id = upsert_page(&mut conn, &source).unwrap();
+
+ let unresolved: i64 = conn
+ .query_row(
+ "SELECT COUNT(*) FROM links \
+ WHERE from_page_id = ?1 AND to_path = ?2 AND to_page_id IS NULL",
+ params![source_id.as_bytes(), "decisions/target.md"],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(unresolved, 1, "forward link should persist unresolved");
+
+ let target_id = upsert_page(
+ &mut conn,
+ &page(ws, proj, "decisions/target.md", "target body"),
+ )
+ .unwrap();
+
+ let resolved: Option<Vec<u8>> = conn
+ .query_row(
+ "SELECT to_page_id FROM links WHERE from_page_id = ?1 AND to_path = ?2",
+ params![source_id.as_bytes(), "decisions/target.md"],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(resolved.as_deref(), Some(&target_id.as_bytes()[..]));
+ }
+
/// Handoff state machine: insert → Open; accept_handoff → Accepted
/// with accepted_by stamped. Calling accept again must be safe
/// (idempotent at the DB level) because hooks fire-and-forget.
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index 823738f9..88930c1f 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -37,6 +37,25 @@ pub struct PageHit {
pub rank: f64,
}
+/// One raw observation fallback hit returned when compiled wiki pages miss.
+#[derive(Debug, Clone, Serialize)]
+pub struct ObservationHit {
+ /// Stable observation identifier.
+ pub id: ObservationId,
+ /// Owning session identifier.
+ pub session_id: SessionId,
+ /// Observation kind as stored on the lifecycle row.
+ pub kind: String,
+ /// Observation title.
+ pub title: String,
+ /// FTS5 snippet of the raw observation body around the matched terms.
+ pub snippet: String,
+ /// FTS5 rank score (lower is better).
+ pub rank: f64,
+ /// ISO-8601 creation timestamp.
+ pub created_at: String,
+}
+
/// Aggregate counts surfaced by [`ReaderPool::status_counts`] and consumed
/// by `ai-memory status`.
#[derive(Debug, Clone, Default, Serialize)]
@@ -51,6 +70,44 @@ pub struct StatusCounts {
pub observations: u64,
}
+/// Derived-index health counters surfaced by admin status.
+#[derive(Debug, Clone, Default, Serialize)]
+pub struct DerivedIndexStatus {
+ /// All page rows in the source table. `pages_fts_rows` should match this.
+ pub pages_rows: u64,
+ /// Rows currently present in the page FTS5 index.
+ pub pages_fts_rows: u64,
+ /// All observation rows. `observations_fts_rows` should match this.
+ pub observations_rows: u64,
+ /// Rows currently present in the observation FTS5 index.
+ pub observations_fts_rows: u64,
+ /// Latest pages without any embedding row.
+ pub latest_pages_missing_embeddings: u64,
+ /// Stored embedding rows, regardless of provider/model/dim.
+ pub embedding_rows: u64,
+ /// Stored embedding triples and row counts.
+ pub embedding_triples: Vec<EmbeddingTripleCount>,
+ /// Outgoing links whose source page is latest.
+ pub links_from_latest_pages: u64,
+ /// Latest-page outgoing links whose target path has not resolved yet.
+ pub unresolved_links_from_latest_pages: u64,
+ /// Latest-page outgoing links pointing at a non-latest target row.
+ pub stale_links_from_latest_pages: u64,
+}
+
+/// Count of embedding rows sharing one `(provider, model, dim)` triple.
+#[derive(Debug, Clone, Serialize)]
+pub struct EmbeddingTripleCount {
+ /// Embedding provider name.
+ pub provider: String,
+ /// Embedding model name.
+ pub model: String,
+ /// Vector dimension.
+ pub dim: u32,
+ /// Rows using this triple.
+ pub count: u64,
+}
+
/// Rolling activity counters over a fixed time window. Surfaced by
/// [`ReaderPool::briefing`] so the caller (or an LLM-driven `memory_explore`)
/// can calibrate verbosity against how busy the project's been.
@@ -89,6 +146,9 @@ pub struct BriefingSnapshot {
/// All pages currently under `_rules/` — small, surfaced verbatim
/// because they're the highest-signal type of memory.
pub rules: Vec<BriefingPage>,
+ /// Small pinned pages under `_slots/` for active project context,
+ /// preferences, current focus, and pending items.
+ pub slots: Vec<BriefingPage>,
/// Top-N most-recently-updated `is_latest = 1` pages.
pub recent_pages: Vec<BriefingPage>,
}
@@ -228,6 +288,7 @@ impl ReaderPool {
/// # Errors
/// Propagates any SQL or pool error.
pub async fn search_pages(&self, query: String, limit: usize) -> StoreResult<Vec<PageHit>> {
+ let query = normalize_fts_query(&query);
self.with_conn(move |conn| {
let mut stmt = conn.prepare(
"SELECT pages.id, pages.path, pages.title, \
@@ -276,6 +337,7 @@ impl ReaderPool {
query: String,
limit: usize,
) -> StoreResult<Vec<PageHit>> {
+ let query = normalize_fts_query(&query);
self.with_conn(move |conn| {
let mut stmt = conn.prepare(
"SELECT pages.id, pages.path, pages.title, \
@@ -324,6 +386,81 @@ impl ReaderPool {
.await
}
+ /// Run a full-text search against raw observations scoped to one project.
+ ///
+ /// # Errors
+ /// Propagates any SQL or pool error.
+ pub async fn search_observations_for_project(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ query: String,
+ limit: usize,
+ ) -> StoreResult<Vec<ObservationHit>> {
+ let query = normalize_fts_query(&query);
+ self.with_conn(move |conn| {
+ let mut stmt = conn.prepare(
+ "SELECT observations.id, observations.session_id, observations.kind, \
+ observations.title, \
+ snippet(observations_fts, 1, '<mark>', '</mark>', '…', 24) AS snip, \
+ observations_fts.rank, observations.created_at \
+ FROM observations_fts \
+ JOIN observations ON observations.rowid = observations_fts.rowid \
+ WHERE observations_fts MATCH ?1 \
+ AND observations.workspace_id = ?2 \
+ AND observations.project_id = ?3 \
+ ORDER BY observations_fts.rank \
+ LIMIT ?4",
+ )?;
+ #[allow(clippy::cast_possible_wrap)]
+ let rows = stmt.query_map(
+ params![
+ query,
+ workspace_id.as_bytes(),
+ project_id.as_bytes(),
+ limit as i64,
+ ],
+ |row| {
+ let id_bytes: Vec<u8> = row.get(0)?;
+ let session_bytes: Vec<u8> = row.get(1)?;
+ let kind: String = row.get(2)?;
+ let title: String = row.get(3)?;
+ let snippet: String = row.get(4)?;
+ let rank: f64 = row.get(5)?;
+ let created_us: i64 = row.get(6)?;
+ Ok((
+ id_bytes,
+ session_bytes,
+ kind,
+ title,
+ snippet,
+ rank,
+ created_us,
+ ))
+ },
+ )?;
+
+ let mut hits = Vec::new();
+ for row in rows {
+ let (id_bytes, session_bytes, kind, title, snippet, rank, created_us) = row?;
+ let created_at = jiff::Timestamp::from_microsecond(created_us)
+ .map(|ts| ts.to_string())
+ .unwrap_or_default();
+ hits.push(ObservationHit {
+ id: ObservationId::from_slice(&id_bytes)?,
+ session_id: SessionId::from_slice(&session_bytes)?,
+ kind,
+ title,
+ snippet,
+ rank,
+ created_at,
+ });
+ }
+ Ok(hits)
+ })
+ .await
+ }
+
/// Return the N most-recently-updated `is_latest = 1` pages.
///
/// # Errors
@@ -604,13 +741,124 @@ impl ReaderPool {
.await
}
+ /// Return pages linked to or from the seed pages, scoped to latest pages
+ /// in the same project.
+ ///
+ /// # Errors
+ /// Propagates any SQL or pool error.
+ pub async fn graph_neighbors_for_project(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ seed_ids: Vec<PageId>,
+ limit: usize,
+ ) -> StoreResult<Vec<PageHit>> {
+ if seed_ids.is_empty() || limit == 0 {
+ return Ok(Vec::new());
+ }
+ self.with_conn(move |conn| {
+ let seed_set: std::collections::HashSet<PageId> = seed_ids.iter().copied().collect();
+ let mut seen = std::collections::HashSet::new();
+ let mut out = Vec::new();
+
+ let mut outgoing = conn.prepare(
+ "SELECT tp.id, tp.path, tp.title, substr(tp.body, 1, 240) \
+ FROM links l \
+ JOIN pages tp ON tp.id = l.to_page_id \
+ WHERE l.from_page_id = ?1 \
+ AND tp.workspace_id = ?2 \
+ AND tp.project_id = ?3 \
+ AND tp.is_latest = 1 \
+ ORDER BY tp.updated_at DESC",
+ )?;
+ let mut incoming = conn.prepare(
+ "SELECT fp.id, fp.path, fp.title, substr(fp.body, 1, 240) \
+ FROM links l \
+ JOIN pages fp ON fp.id = l.from_page_id \
+ WHERE l.to_page_id = ?1 \
+ AND fp.workspace_id = ?2 \
+ AND fp.project_id = ?3 \
+ AND fp.is_latest = 1 \
+ ORDER BY fp.updated_at DESC",
+ )?;
+
+ for seed_id in &seed_ids {
+ let rows = outgoing.query_map(
+ params![
+ seed_id.as_bytes(),
+ workspace_id.as_bytes(),
+ project_id.as_bytes()
+ ],
+ |row| {
+ let id_bytes: Vec<u8> = row.get(0)?;
+ let path: String = row.get(1)?;
+ let title: String = row.get(2)?;
+ let snippet: String = row.get(3)?;
+ Ok((id_bytes, path, title, snippet))
+ },
+ )?;
+ for row in rows {
+ let (id_bytes, path, title, snippet) = row?;
+ let id = PageId::from_slice(&id_bytes)?;
+ if seed_set.contains(&id) || !seen.insert(id) {
+ continue;
+ }
+ out.push(PageHit {
+ id,
+ path: PagePath::new(path)?,
+ title,
+ snippet,
+ rank: 0.0,
+ });
+ if out.len() >= limit {
+ return Ok(out);
+ }
+ }
+
+ let rows = incoming.query_map(
+ params![
+ seed_id.as_bytes(),
+ workspace_id.as_bytes(),
+ project_id.as_bytes()
+ ],
+ |row| {
+ let id_bytes: Vec<u8> = row.get(0)?;
+ let path: String = row.get(1)?;
+ let title: String = row.get(2)?;
+ let snippet: String = row.get(3)?;
+ Ok((id_bytes, path, title, snippet))
+ },
+ )?;
+ for row in rows {
+ let (id_bytes, path, title, snippet) = row?;
+ let id = PageId::from_slice(&id_bytes)?;
+ if seed_set.contains(&id) || !seen.insert(id) {
+ continue;
+ }
+ out.push(PageHit {
+ id,
+ path: PagePath::new(path)?,
+ title,
+ snippet,
+ rank: 0.0,
+ });
+ if out.len() >= limit {
+ return Ok(out);
+ }
+ }
+ }
+ Ok(out)
+ })
+ .await
+ }
+
/// Hybrid search: RRF-fuse FTS5 results with cosine-similarity
/// over the stored embeddings of the matching `(provider, model,
- /// dim)`. Returns the top-`limit` pages by fused score.
+ /// dim)`, then add link-neighbour expansion as a third RRF stream.
+ /// Returns the top-`limit` pages by fused score.
///
- /// When `query_vec` is `None`, falls back to pure FTS5
- /// (equivalent to [`Self::search_pages`]). When no embeddings
- /// exist for the configured triple, also falls back to pure FTS5.
+ /// When `query_vec` is `None`, the vector stream is skipped but graph
+ /// expansion still runs from the FTS seeds.
///
/// k=60 is the canonical RRF constant.
///
@@ -632,30 +880,37 @@ impl ReaderPool {
let fts_hits = self
.search_pages_for_project(workspace_id, project_id, query, limit * 2)
.await?;
- let Some(qv) = query_vec else {
- // No query vector → pure FTS5.
- let mut out = fts_hits;
- out.truncate(limit);
- return Ok(out);
- };
- // Compute cosines against all stored embeddings.
- let stored = self
- .load_embeddings(workspace_id, project_id, provider, model, dim)
- .await?;
- if stored.is_empty() {
- let mut out = fts_hits;
- out.truncate(limit);
- return Ok(out);
+ let mut vec_hits: Vec<(PageId, PagePath, f32)> = Vec::new();
+ if let Some(qv) = query_vec {
+ // Compute cosines against all stored embeddings.
+ let stored = self
+ .load_embeddings(workspace_id, project_id, provider, model, dim)
+ .await?;
+ vec_hits = stored
+ .iter()
+ .map(|s| {
+ let score = dot(&qv, &s.vector);
+ (s.id, s.path.clone(), score)
+ })
+ .collect();
+ vec_hits.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap_or(std::cmp::Ordering::Equal));
+ vec_hits.truncate(limit * 2);
}
- let mut vec_hits: Vec<(PageId, PagePath, f32)> = stored
+
+ let mut seed_seen = std::collections::HashSet::new();
+ let mut seed_ids = Vec::new();
+ for id in fts_hits
.iter()
- .map(|s| {
- let score = dot(&qv, &s.vector);
- (s.id, s.path.clone(), score)
- })
- .collect();
- vec_hits.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap_or(std::cmp::Ordering::Equal));
- vec_hits.truncate(limit * 2);
+ .map(|h| h.id)
+ .chain(vec_hits.iter().map(|(id, _, _)| *id))
+ {
+ if seed_seen.insert(id) {
+ seed_ids.push(id);
+ }
+ }
+ let graph_hits = self
+ .graph_neighbors_for_project(workspace_id, project_id, seed_ids, limit * 2)
+ .await?;
// RRF fuse: score(d) = Σ 1/(k + rank_i(d)) over rankers.
let k = 60.0_f64;
@@ -682,6 +937,19 @@ impl ReaderPool {
.and_modify(|entry| entry.3 += contrib)
.or_insert((path.clone(), String::new(), String::new(), contrib, 0.0));
}
+ for (rank, h) in graph_hits.iter().enumerate() {
+ let contrib = 1.0 / (k + (rank + 1) as f64);
+ fused
+ .entry(h.id)
+ .and_modify(|entry| entry.3 += contrib)
+ .or_insert((
+ h.path.clone(),
+ h.title.clone(),
+ h.snippet.clone(),
+ contrib,
+ h.rank,
+ ));
+ }
let mut out: Vec<PageHit> = fused
.into_iter()
@@ -845,6 +1113,20 @@ impl ReaderPool {
.into_iter()
.collect::<Result<Vec<_>, _>>()?;
+ let mut slots_stmt = conn.prepare(
+ "SELECT path, title, \
+ COALESCE(json_extract(frontmatter_json, '$.kind'), 'slot') AS kind, \
+ updated_at \
+ FROM pages \
+ WHERE is_latest = 1 AND path LIKE '_slots/%' \
+ ORDER BY path ASC",
+ )?;
+ let slots: Vec<BriefingPage> = slots_stmt
+ .query_map([], briefing_page_from_row)?
+ .collect::<Result<Vec<_>, _>>()?
+ .into_iter()
+ .collect::<Result<Vec<_>, _>>()?;
+
let mut recent_stmt = conn.prepare(
"SELECT path, title, \
COALESCE(json_extract(frontmatter_json, '$.kind'), 'fact') AS kind, \
@@ -867,6 +1149,7 @@ impl ReaderPool {
last_observation_at,
pending_handoff_count,
rules,
+ slots,
recent_pages,
})
})
@@ -954,6 +1237,20 @@ impl ReaderPool {
.into_iter()
.collect::<Result<Vec<_>, _>>()?;
+ let mut slots_stmt = conn.prepare(
+ "SELECT path, title, \
+ COALESCE(json_extract(frontmatter_json, '$.kind'), 'slot') AS kind, \
+ updated_at \
+ FROM pages \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND is_latest = 1 AND path LIKE '_slots/%' \
+ ORDER BY path ASC",
+ )?;
+ let slots: Vec<BriefingPage> = slots_stmt
+ .query_map(params![workspace_id.as_bytes(), project_id.as_bytes()], briefing_page_from_row)?
+ .collect::<Result<Vec<_>, _>>()?
+ .into_iter()
+ .collect::<Result<Vec<_>, _>>()?;
+
let mut recent_stmt = conn.prepare(
"SELECT path, title, \
COALESCE(json_extract(frontmatter_json, '$.kind'), 'fact') AS kind, \
@@ -976,6 +1273,7 @@ impl ReaderPool {
last_observation_at,
pending_handoff_count,
rules,
+ slots,
recent_pages,
})
})
@@ -1246,6 +1544,80 @@ impl ReaderPool {
.await
}
+ /// Return health counters for derived indexes and link/embedding state.
+ ///
+ /// These checks are intentionally read-only and derived-index-safe: they
+ /// report drift but do not repair it. Rebuild/backfill paths stay behind
+ /// explicit admin operations.
+ ///
+ /// # Errors
+ /// Propagates any SQL or pool error.
+ pub async fn derived_index_status(&self) -> StoreResult<DerivedIndexStatus> {
+ self.with_conn(|conn| {
+ let mut triples_stmt = conn.prepare(
+ "SELECT provider, model, dim, COUNT(*) \
+ FROM page_embeddings \
+ GROUP BY provider, model, dim \
+ ORDER BY COUNT(*) DESC, provider, model, dim",
+ )?;
+ let embedding_triples = triples_stmt
+ .query_map([], |row| {
+ let provider: String = row.get(0)?;
+ let model: String = row.get(1)?;
+ let dim: i64 = row.get(2)?;
+ let count: i64 = row.get(3)?;
+ Ok(EmbeddingTripleCount {
+ provider,
+ model,
+ dim: u32::try_from(dim.max(0)).unwrap_or(0),
+ count: u64::try_from(count.max(0)).unwrap_or(0),
+ })
+ })?
+ .collect::<Result<Vec<_>, _>>()?;
+
+ Ok(DerivedIndexStatus {
+ pages_rows: count(conn, "SELECT COUNT(*) FROM pages")?,
+ pages_fts_rows: count(conn, "SELECT COUNT(*) FROM pages_fts")?,
+ observations_rows: count(conn, "SELECT COUNT(*) FROM observations")?,
+ observations_fts_rows: count(conn, "SELECT COUNT(*) FROM observations_fts")?,
+ latest_pages_missing_embeddings: count(
+ conn,
+ "SELECT COUNT(*) \
+ FROM pages pg \
+ LEFT JOIN page_embeddings pe ON pe.page_id = pg.id \
+ WHERE pg.is_latest = 1 AND pe.page_id IS NULL",
+ )?,
+ embedding_rows: count(conn, "SELECT COUNT(*) FROM page_embeddings")?,
+ embedding_triples,
+ links_from_latest_pages: count(
+ conn,
+ "SELECT COUNT(*) \
+ FROM links l \
+ JOIN pages fp ON fp.id = l.from_page_id \
+ WHERE fp.is_latest = 1",
+ )?,
+ unresolved_links_from_latest_pages: count(
+ conn,
+ "SELECT COUNT(*) \
+ FROM links l \
+ JOIN pages fp ON fp.id = l.from_page_id \
+ WHERE fp.is_latest = 1 AND l.to_page_id IS NULL",
+ )?,
+ stale_links_from_latest_pages: count(
+ conn,
+ "SELECT COUNT(*) \
+ FROM links l \
+ JOIN pages fp ON fp.id = l.from_page_id \
+ LEFT JOIN pages tp ON tp.id = l.to_page_id \
+ WHERE fp.is_latest = 1 \
+ AND l.to_page_id IS NOT NULL \
+ AND COALESCE(tp.is_latest, 0) != 1",
+ )?,
+ })
+ })
+ .await
+ }
+
/// Return aggregate counts scoped to one project.
///
/// # Errors
@@ -1649,6 +2021,24 @@ fn count_project(
Ok(u64::try_from(n.unwrap_or(0)).unwrap_or(0))
}
+fn normalize_fts_query(query: &str) -> String {
+ query
+ .split_whitespace()
+ .map(|token| {
+ if should_quote_fts_token(token) {
+ format!("\"{}\"", token.replace('"', "\"\""))
+ } else {
+ token.to_string()
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(" ")
+}
+
+fn should_quote_fts_token(token: &str) -> bool {
+ token.contains('-') && !(token.starts_with('"') && token.ends_with('"'))
+}
+
/// Count rows in a time-bounded window. Used by [`ReaderPool::briefing`]
/// to compute "last 7 days" / "last 30 days" activity slices.
fn window_activity(conn: &Connection, days: u32, cutoff_us: i64) -> StoreResult<ActivityWindow> {
diff --git a/crates/ai-memory-web/tests/routes.rs b/crates/ai-memory-web/tests/routes.rs
index f302dc78..3deccf46 100644
--- a/crates/ai-memory-web/tests/routes.rs
+++ b/crates/ai-memory-web/tests/routes.rs
@@ -35,6 +35,7 @@ fn new_page(
tier: Tier::Semantic,
frontmatter_json: serde_json::json!({"kind": "fact"}),
pinned: false,
+ links: Vec::new(),
}
}
diff --git a/crates/ai-memory-wiki/src/markdown.rs b/crates/ai-memory-wiki/src/markdown.rs
index d544793a..61068994 100644
--- a/crates/ai-memory-wiki/src/markdown.rs
+++ b/crates/ai-memory-wiki/src/markdown.rs
@@ -6,6 +6,8 @@
//! basic-memory hit (#528). Going through `serde_yaml` directly keeps the
//! round-trip predictable.
+use std::collections::BTreeSet;
+
use ai_memory_core::PagePath;
use serde::{Deserialize, Serialize};
@@ -100,6 +102,150 @@ pub fn derive_title(frontmatter: &serde_json::Value, body: &str, path: &PagePath
stem.strip_suffix(".md").unwrap_or(stem).to_string()
}
+/// Extract internal wiki links from a markdown body.
+///
+/// Supports `[[wiki links]]`, `[[wiki links|labels]]`, and ordinary
+/// markdown links such as `[label](../decisions/foo.md#anchor)`. External
+/// URLs, anchors, images, and non-markdown assets are ignored. Returned
+/// paths are normalised to wiki-root-relative [`PagePath`] values.
+#[must_use]
+pub fn extract_links(body: &str, page_path: &PagePath) -> Vec<PagePath> {
+ let mut out = BTreeSet::new();
+ let mut in_fence = false;
+
+ for line in body.lines() {
+ let trimmed = line.trim_start();
+ if trimmed.starts_with("```") || trimmed.starts_with("~~~") {
+ in_fence = !in_fence;
+ continue;
+ }
+ if in_fence {
+ continue;
+ }
+ extract_wikilinks(line, page_path, &mut out);
+ extract_markdown_links(line, page_path, &mut out);
+ }
+
+ out.into_iter()
+ .filter_map(|path| PagePath::new(path).ok())
+ .collect()
+}
+
+fn extract_wikilinks(line: &str, page_path: &PagePath, out: &mut BTreeSet<String>) {
+ let mut rest = line;
+ while let Some(start) = rest.find("[[") {
+ let after_start = &rest[start + 2..];
+ let Some(end) = after_start.find("]]") else {
+ break;
+ };
+ let raw = &after_start[..end];
+ if let Some(path) = normalize_link_target(raw, page_path, true) {
+ out.insert(path);
+ }
+ rest = &after_start[end + 2..];
+ }
+}
+
+fn extract_markdown_links(line: &str, page_path: &PagePath, out: &mut BTreeSet<String>) {
+ let mut start_at = 0;
+ while let Some(rel_start) = line[start_at..].find('[') {
+ let start = start_at + rel_start;
+ if start > 0 && line.as_bytes()[start - 1] == b'!' {
+ start_at = start + 1;
+ continue;
+ }
+ let after_start = start + 1;
+ let Some(rel_close) = line[after_start..].find(']') else {
+ break;
+ };
+ let close = after_start + rel_close;
+ if !line[close + 1..].starts_with('(') {
+ start_at = close + 1;
+ continue;
+ }
+ let target_start = close + 2;
+ let Some(rel_end) = line[target_start..].find(')') else {
+ break;
+ };
+ let target_end = target_start + rel_end;
+ let raw = &line[target_start..target_end];
+ if let Some(path) = normalize_link_target(raw, page_path, false) {
+ out.insert(path);
+ }
+ start_at = target_end + 1;
+ }
+}
+
+fn normalize_link_target(raw: &str, page_path: &PagePath, wikilink: bool) -> Option<String> {
+ let target = raw
+ .split_once('|')
+ .map_or(raw, |(path, _)| path)
+ .trim()
+ .trim_matches('<')
+ .trim_matches('>');
+ if target.is_empty() || target.starts_with('#') || target.contains("://") {
+ return None;
+ }
+ let lower = target.to_ascii_lowercase();
+ if lower.starts_with("mailto:")
+ || lower.starts_with("data:")
+ || lower.starts_with("javascript:")
+ || lower.starts_with("tel:")
+ {
+ return None;
+ }
+
+ let target = target.split_once('#').map_or(target, |(path, _)| path);
+ let target = target
+ .split_once('?')
+ .map_or(target, |(path, _)| path)
+ .trim();
+ if target.is_empty() || target.contains('\\') {
+ return None;
+ }
+
+ let mut target = target.to_string();
+ let last_segment = target.rsplit_once('/').map_or(target.as_str(), |(_, s)| s);
+ if last_segment.contains('.') {
+ if !target.ends_with(".md") {
+ return None;
+ }
+ } else if wikilink || !last_segment.is_empty() {
+ target.push_str(".md");
+ }
+
+ resolve_relative(page_path, &target, wikilink)
+}
+
+fn resolve_relative(page_path: &PagePath, target: &str, root_relative: bool) -> Option<String> {
+ let mut parts: Vec<&str> = if root_relative || target.starts_with('/') {
+ Vec::new()
+ } else {
+ page_path
+ .as_str()
+ .rsplit_once('/')
+ .map_or_else(Vec::new, |(dir, _)| {
+ dir.split('/').filter(|part| !part.is_empty()).collect()
+ })
+ };
+
+ for part in target.trim_start_matches('/').split('/') {
+ match part {
+ "" | "." => {}
+ ".." => {
+ parts.pop()?;
+ }
+ part => parts.push(part),
+ }
+ }
+
+ if parts.is_empty() {
+ None
+ } else {
+ Some(parts.join("/"))
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -181,4 +327,27 @@ mod tests {
"foo"
);
}
+
+ #[test]
+ fn extracts_internal_wiki_and_markdown_links() {
+ let path = PagePath::new("concepts/current.md").unwrap();
+ let body = "See [[decisions/0001-single-sqlite-file|SQLite]] and \
+ [gotcha](../gotchas/hooks.md#details). Also \
+ [external](https://example.com) and ![image](../img/logo.png).";
+ let links = extract_links(body, &path);
+ let paths: Vec<&str> = links.iter().map(PagePath::as_str).collect();
+ assert_eq!(
+ paths,
+ vec!["decisions/0001-single-sqlite-file.md", "gotchas/hooks.md"]
+ );
+ }
+
+ #[test]
+ fn extract_links_ignores_fenced_code_blocks() {
+ let path = PagePath::new("notes/a.md").unwrap();
+ let body = "```\n[[notes/ignored]]\n```\n[[notes/kept]]\n";
+ let links = extract_links(body, &path);
+ let paths: Vec<&str> = links.iter().map(PagePath::as_str).collect();
+ assert_eq!(paths, vec!["notes/kept.md"]);
+ }
}
diff --git a/crates/ai-memory-wiki/src/wiki.rs b/crates/ai-memory-wiki/src/wiki.rs
index 70c43394..5dbe4932 100644
--- a/crates/ai-memory-wiki/src/wiki.rs
+++ b/crates/ai-memory-wiki/src/wiki.rs
@@ -10,7 +10,7 @@ use ai_memory_store::{WriterHandle, f32_vec_to_bytes};
use crate::atomic;
use crate::error::WikiResult;
use crate::git::GitAdapter;
-use crate::markdown::{Markdown, derive_title, emit, parse};
+use crate::markdown::{Markdown, derive_title, emit, extract_links, parse};
/// Wiki filesystem handle.
///
@@ -67,10 +67,12 @@ impl Wiki {
self
}
- /// Attach an embedder. When set, every successful `write_page` /
- /// `apply_batch` also computes + stores an embedding for the new
- /// version. Without this, vector search is unavailable and
- /// `ReaderPool::hybrid_search` falls back to pure FTS5.
+ /// Attach an embedder. When set, `write_page` computes + stores an
+ /// embedding for the new version synchronously. `apply_batch` keeps
+ /// the SQL/file fan-out atomic and leaves vector completeness to
+ /// admin or scheduled embedding backfill. Without an embedder,
+ /// vector search is skipped and `ReaderPool::hybrid_search` uses
+ /// FTS5 + graph expansion.
#[must_use]
pub fn with_embedder(mut self, embedder: Arc<dyn Embedder>) -> Self {
self.embedder = Some(embedder);
@@ -189,6 +191,8 @@ impl Wiki {
) -> WikiResult<PageId> {
let md = self.read_page(workspace_id, project_id, &path)?;
let title = derive_title(&md.frontmatter, &md.body, &path);
+ let links = extract_links(&md.body, &path);
+ let pinned = is_slot_path(&path);
let id = self
.writer
.upsert_page(NewPage {
@@ -199,7 +203,8 @@ impl Wiki {
body: md.body,
tier: Tier::Semantic,
frontmatter_json: md.frontmatter,
- pinned: false,
+ pinned,
+ links,
})
.await?;
Ok(id)
@@ -263,7 +268,8 @@ impl Wiki {
body: req.body.clone(),
tier: req.tier,
frontmatter_json: req.frontmatter.clone(),
- pinned: req.pinned,
+ pinned: req.pinned || is_slot_path(&req.path),
+ links: extract_links(&req.body, &req.path),
})
.collect();
@@ -310,6 +316,8 @@ impl Wiki {
.clone()
.map(|t| self.sanitizer.scrub(&t))
.unwrap_or_else(|| derive_title(&frontmatter, &body, &path));
+ let links = extract_links(&body, &path);
+ let pinned = pinned || is_slot_path(&path);
let markdown = Markdown {
frontmatter: frontmatter.clone(),
body: body.clone(),
@@ -332,6 +340,7 @@ impl Wiki {
tier,
frontmatter_json: frontmatter,
pinned,
+ links,
})
.await?;
// Embed if configured. We do this on the caller's task so the
@@ -389,6 +398,10 @@ fn ai_memory_wiki_error(msg: &str) -> crate::WikiError {
crate::WikiError::Io(std::io::Error::other(msg.to_string()))
}
+fn is_slot_path(path: &PagePath) -> bool {
+ path.as_str().starts_with("_slots/")
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -550,6 +563,37 @@ mod tests {
assert!(!hits[0].snippet.contains("hunter2"));
}
+ #[tokio::test]
+ async fn slot_pages_are_pinned_automatically() {
+ let tmp = TempDir::new().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "scratch", None)
+ .await
+ .unwrap();
+ let wiki = Wiki::new(tmp.path(), store.writer.clone()).unwrap();
+
+ wiki.write_page(req(
+ ws,
+ proj,
+ "_slots/current_focus.md",
+ "Keep this tiny and durable.",
+ serde_json::json!({ "title": "Current focus", "kind": "slot" }),
+ ))
+ .await
+ .unwrap();
+
+ let candidates = store.reader.decay_candidates(ws, proj).await.unwrap();
+ assert_eq!(candidates.len(), 1);
+ assert!(candidates[0].pinned, "slot pages should be decay-immune");
+ }
+
#[tokio::test]
async fn apply_batch_persists_all_pages_in_one_transaction() {
let tmp = TempDir::new().unwrap();
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 2ce99121..614cf065 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -85,14 +85,15 @@ markdown stays the source of truth.
4. When `AI_MEMORY_LLM_PROVIDER` is set, `memory_consolidate` rewrites
that summary into a richer durable page or fans out into a
multi-page batch under `concepts/`, `decisions/`, `gotchas/`.
-5. `memory_query` answers via FTS5; when an embedder is configured,
- via RRF fusion of FTS5 + cosine over `page_embeddings`. Every
- query hit bumps `access_count` + `last_accessed_at` on the page -
- the M8 reinforcement term.
-6. The forget sweep runs on demand (or on a future schedule): pages
- with `retention < cold_threshold` are soft-deleted; soft-deletions
- older than `hard_delete_after_days` with no subsequent access get
- purged. Semantic / pinned / freshly-touched pages survive.
+5. `memory_query` answers via FTS5 + link-neighbour RRF; when an
+ embedder is configured, vector cosine over `page_embeddings` joins
+ the same RRF. If compiled wiki pages miss entirely, bounded raw
+ observation FTS returns fallback `raw_hits`. Page hits bump
+ `access_count` + `last_accessed_at` - the M8 reinforcement term.
+6. The forget sweep runs on demand and on the server's `[maintenance]`
+ schedule: pages with `retention < cold_threshold` are soft-deleted;
+ soft-deletions older than `hard_delete_after_days` with no subsequent
+ access get purged. Semantic / pinned / freshly-touched pages survive.
7. Backups: `ai-memory backup --to <tarball>` uses SQLite's online
backup API so the source stays writable; `ai-memory restore`
reverses. Or: `git push` the wiki dir + `rsync` the data dir.
@@ -108,7 +109,8 @@ markdown stays the source of truth.
* `<data_dir>/db/memory.sqlite` - derived index. WAL mode. One
writer actor owns the writer `Connection`; reads go through a
cloneable read-only pool.
-* `<data_dir>/raw/` - reserved for raw session log archives (future).
+* `<data_dir>/raw/` - reserved for raw session log archives; current raw
+ fallback recall searches the durable `observations` table via FTS5.
* `<data_dir>/logs/` - rolling daily `tracing` output.
* `<data_dir>/models/` - reserved for bundled embedding models
(M9.5+, when local `ort` lands).
@@ -121,9 +123,10 @@ markdown stays the source of truth.
| `pages` | Versioned wiki pages with `is_latest` + `supersedes` chain. M8 columns: `last_accessed_at`, `access_count`, `superseded_at`. M9 cols: `embedding_provider`, `embedding_model`, `embedding_dim`. |
| `pages_fts` | FTS5 virtual table over `(title, body)`, auto-synced by triggers. |
| `sessions`, `observations` | Hook capture, full audit log. |
-| `links` | Wiki cross-references with `to_page_id` nullable for unresolved. |
+| `observations_fts` | FTS5 virtual table over raw observation `(title, body)`, used only as bounded fallback. |
+| `links` | Wikilink / markdown cross-references with `to_page_id` nullable for unresolved forward links. |
| `handoffs` | Typed cross-agent handoff records (open / accepted / expired). |
-| `page_embeddings` | One vector per latest page, with `(provider, model, dim)` denormalised for refuse-on-mismatch checks. |
+| `page_embeddings` | Optional vector rows for latest pages, with `(provider, model, dim)` denormalised for refuse-on-mismatch checks and missing-embedding diagnostics. |
| `audit_log` | Every mutation, addressable by `at DESC`. |
**Memory tiers (M8 policy):**
@@ -136,7 +139,8 @@ markdown stays the source of truth.
| Procedural | Indefinite | Frequency-decay if not re-observed |
Pinned pages (`pinned: true` in frontmatter) are exempt from all
-decay paths.
+decay paths. Pages under `_slots/` are pinned automatically and surfaced
+in briefing/explore snapshots as tiny editable memory slots.
## Crate layout
@@ -156,18 +160,21 @@ Each crate has a single responsibility and exposes a typed API. No
circular deps. Inter-crate boundaries enforce the cross-cutting
invariants below.
-## MCP tool surface (10 tools)
+## MCP tool surface (11 tools)
| Tool | Hint | Purpose |
|---|---|---|
-| `memory_query` | read-only | FTS5 or hybrid RRF search. Bumps access counters. |
+| `memory_query` | read-only | FTS5 + graph RRF + optional vector RRF search, with raw fallback. Bumps access counters for page hits. |
| `memory_recent` | read-only | Most-recently-updated `is_latest=1` pages. |
| `memory_status` | read-only | Counts, paths, version. |
+| `memory_briefing` | read-only | Structured counts/activity/rules/slots/recent snapshot. |
+| `memory_explore` | read-only | LLM prose digest over the briefing snapshot, degrading to JSON without a provider. |
| `memory_handoff_begin` | destructive | Open a handoff for the next agent. |
| `memory_handoff_accept` | destructive | Fetch + ack the latest open handoff (auto-cwd-matched). |
| `memory_consolidate` | destructive | LLM-driven page rewrite. `multi_page=true` for atomic fan-out. |
| `memory_forget_sweep` | destructive | M8 retention pass. `dry_run=true` for preview. |
| `memory_lint` | destructive | Rule-based + LLM contradiction findings → `wiki/_lint/`. |
+| `memory_install_self_routing` | read-only | Return the canonical routing snippet for CLAUDE.md / AGENTS.md. |
All MCP tools carry `readOnlyHint` / `destructiveHint` annotations so
the calling agent can pick safely. Parameter aliases (`query|q|search`,
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index 0d267ba8..f1c34926 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -57,7 +57,7 @@ Three options surveyed:
## 4. Database choice - single SQLite file
-**Decision: one SQLite file with FTS5 + `sqlite-vec` extension + JSON columns for graph edges.**
+**Decision: one SQLite file with FTS5, packed-vector embeddings, and SQL tables for graph edges.**
Why not Postgres/pgvector? Cognee's #2717 and basic-memory's #830/#831 show Postgres is a real-deployment-only pain. v1 ships embedded.
@@ -66,13 +66,13 @@ Why not LanceDB/Qdrant/Kuzu/CozoDB/SurrealDB?
- Kuzu / Ladybug: cognee #2098/#2768 (upstream archived, fork-risk realized).
- CozoDB: small bus factor.
- SurrealDB: heavy, multi-mode storage; we'd inherit a lot of surface we don't need.
-- Embedded `sqlite-vec` is well-maintained, single dependency, fits in one file with FTS5 + relational tables.
+- Packed vectors in SQLite keep v1 dependency-light; `sqlite-vec` remains the scale-up path once brute-force cosine stops being enough.
**The graph is just SQL tables.** A `wiki_pages` table, a `wiki_links (from_id, to_id, link_type)` table, optional `wiki_concepts (page_id, concept)`. Graph queries are recursive CTEs in SQLite. Petgraph in-memory for batch traversals. Avoids the entire "embedded graph DB" footgun cognee fell into.
**Crates** (research-backed picks):
-- `rusqlite` + `rusqlite-extension` for sqlite-vec loading. `bundled-sqlcipher` if we want encryption later.
-- `sqlx` for migrations (`sqlx::migrate!`). Async, type-checked.
+- `rusqlite` for embedded SQLite access. `bundled-sqlcipher` if we want encryption later.
+- `refinery` for SQL migrations.
- `tantivy` *not* used initially - sqlite FTS5 is sufficient at the corpus sizes we expect (hundreds to low-thousands of pages per project). Revisit only if FTS5 ranking proves inadequate.
- `petgraph` for in-memory graph algorithms during consolidation.
diff --git a/docs/install.md b/docs/install.md
index 337acf96..a9f3ffec 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -344,11 +344,11 @@ docker run --rm akitaonrails/ai-memory:latest --help # full subcommand tree
| Subcommand | Pattern | What it does |
|---|---|---|
| `serve` | `docker compose up -d` (already done) | Run the HTTP MCP server |
-| `status` | `docker exec` | Counts + paths + provider config |
-| `search "<query>"` | `docker exec` | FTS5 query against the wiki |
+| `status` | `docker exec` | Counts, paths, and derived-index diagnostics |
+| `search "<query>"` | `docker exec` | Wiki search with FTS5 + graph/vector RRF |
| `write-page` | `docker exec` | Manual page write (atomic + indexed) |
| `backup --to` / `restore --from` | `docker exec` | Snapshot or restore the data dir |
-| `forget-sweep` / `lint` / `embed` | `docker exec` | Periodic maintenance |
+| `forget-sweep` / `lint` / `embed` | `docker exec` | Manual maintenance; sweep + lint also run on the server schedule by default |
| `commit -m "…"` | `docker exec` | Stage + commit the wiki tree |
| `reset --confirm` | `docker exec` | Wipe data (refuses while siblings alive) |
| `generate-auth-token` | `docker run --rm` | Print a random hex bearer token |
@@ -360,6 +360,12 @@ docker run --rm akitaonrails/ai-memory:latest --help # full subcommand tree
Data dir inside the container is `/data` (mounted via the compose
volume). Outside docker, override with `AI_MEMORY_DATA_DIR=/path`.
+Scheduled maintenance is configured in `[maintenance]` in `config.toml`.
+By default, rule-based lint and forget sweep run daily outside hook
+latency. Embedding backfill is supported but defaults to off because it
+can call a paid provider; enable it with
+`embedding_backfill_interval_secs` after configuring an embedder.
+
---
## Bootstrap mid-project {#bootstrap-mid-project}
diff --git a/docs/issues-mempalace.md b/docs/issues-mempalace.md
index b2ea100c..2f7bb4a6 100644
--- a/docs/issues-mempalace.md
+++ b/docs/issues-mempalace.md
@@ -1,9 +1,9 @@
# MemPalace - Issue & Architecture Synthesis
-> Source: GitHub `MemPalace/mempalace` + local checkout at `~/Projects/mempalace`.
+> Source: GitHub `MemPalace/mempalace`.
> Captured 2026-05-24. Repo: 52.7k stars, MIT, Python, created 2026-04-05,
-> v3.3.5, 241 open issues, ~1600 PRs in ~7 weeks. Extremely active, unusually
-> transparent maintainers (see `docs/HISTORY.md` — they publicly retract
+> v3.3.5, 241 open issues, ~1600 PRs in ~7 weeks. The maintainers publish
+> unusually candid history notes (see `docs/HISTORY.md` — they publicly retract
> overclaims). Dominant themes: **ChromaDB/HNSW/FTS5 corruption under
> concurrent writes**, **silent persistence failures**, **repair tooling that
> destroys data**, **embedding-model drift**.
diff --git a/docs/lifecycle-ops.md b/docs/lifecycle-ops.md
index f923c162..29f683cb 100644
--- a/docs/lifecycle-ops.md
+++ b/docs/lifecycle-ops.md
@@ -237,11 +237,11 @@ ai-memory purge-project --project experimental --confirm
# Sibling projects (ai-memory, distrobox-gaming, …) untouched.
```
-### "Rename a project after `mv ~/Projects/old ~/Projects/new`"
+### "Rename a project after moving its directory"
```bash
ai-memory rename-project --from old --to new
-# Future sessions in ~/Projects/new will append to the same project
+# Future sessions in /path/to/new will append to the same project
# (the hook router stamps by basename(cwd) = "new"); past
# observations stay under that project too because the project_id
# is stable.
diff --git a/docs/mcp-install.md b/docs/mcp-install.md
index 5162d3e1..e6e8206c 100644
--- a/docs/mcp-install.md
+++ b/docs/mcp-install.md
@@ -235,8 +235,9 @@ You: List the MCP tools you can call. Use one of them to check
ai-memory's status.
Model (any client): I can call: memory_query, memory_recent,
- memory_status, memory_handoff_accept, memory_handoff_begin,
- memory_consolidate, memory_lint, memory_forget_sweep.
+ memory_status, memory_briefing, memory_explore,
+ memory_handoff_accept, memory_handoff_begin, memory_consolidate,
+ memory_lint, memory_forget_sweep, memory_install_self_routing.
memory_status reports: 0 pages, 0 observations, 0 sessions.
```
diff --git a/docs/prior-art-implementation-findings.md b/docs/prior-art-implementation-findings.md
new file mode 100644
index 00000000..419734fb
--- /dev/null
+++ b/docs/prior-art-implementation-findings.md
@@ -0,0 +1,338 @@
+# Prior-Art Implementation Findings
+
+> Scope: compare the checked-in prior-art analyses for agentmemory,
+> basic-memory, cognee, and MemPalace against the current ai-memory
+> implementation. This is an analysis document only; no code changes
+> are implied by this file.
+
+## Executive Summary
+
+ai-memory already captured the most important prior-art lessons:
+automatic capture, a narrow MCP surface, markdown-in-git as the human
+source of truth, SQLite as the derived index/state store, single-writer
+durability, opt-in LLM consolidation, structured JSON outputs, typed
+handoffs, versioned supersession, per-project UUID isolation, and an
+agentmemory-style retention formula.
+
+The remaining high-value improvements were not a rewrite. They were
+mostly lifecycle and retrieval layers on top of the solid base: small
+editable memory slots, scheduled maintenance, first-class graph/link
+retrieval, bounded raw/verbatim fallback recall, diagnostics, and real
+retrieval checks. The most important caution remains keeping the current
+substrate simple. Do not copy agentmemory's sidecar surface, cognee's
+multi-store orchestration, basic-memory's manual write-note workflow, or
+MemPalace's verbatim-everything storage model.
+
+## Implementation Status
+
+The follow-up implementation landed the pragmatic subset of this roadmap:
+
+| Area | Status |
+|---|---|
+| Slots | `_slots/` pages are pinned automatically and surfaced in briefing/explore snapshots. |
+| Scheduled maintenance | Server-side schedule runs lint and forget sweep daily by default; embedding backfill is opt-in. |
+| Vector contract | Status reports missing embeddings and stored triples; batch writes leave vector completeness to backfill. |
+| Links / graph | Markdown and wikilinks are parsed into `links`; unresolved forward links resolve when targets appear; query uses graph-neighbor RRF. |
+| Raw fallback | `observations_fts` provides bounded raw fallback when compiled wiki search misses. |
+| Diagnostics | `status` includes FTS row counts, missing embeddings, embedding triples, and unresolved/stale link counts. |
+| Retrieval evals | Recall harness now covers FTS/vector, graph expansion, and raw fallback. |
+
+## Prior Art Summary
+
+| Project | Main strengths | Main weaknesses | ai-memory stance |
+|---|---|---|---|
+| agentmemory | Automatic hook capture, four memory tiers, versioned supersession, retention scoring, hybrid BM25/vector/graph retrieval, slots, handoff patterns, snapshots, diagnostics. | iii-engine sidecar, JSON-in-KV storage, in-memory indexes, XML parsing, too many tools/endpoints, early default-on LLM costs, hook blocking/config drift bugs. | ai-memory follows the concepts more than any other project, but replaces the substrate with one Rust binary, SQLite, markdown, structured JSON, and a narrow tool surface. |
+| basic-memory | Markdown files as source of truth, derived SQL index, human-editable notes, MCP hints/aliases, `memory://` style navigation, unresolved links, multi-project routing. | Manual `write_note` ceremony, append-only memory, fragile file watcher/sync, markdown grammar burden on the LLM, weak lifecycle/ranking. | ai-memory borrows markdown/source-of-truth and human editability, but rejects manual capture and adds lifecycle, decay, supersession, and hook capture. |
+| cognee | Task-list pipeline shape, structured graph extraction, provenance stamping, triplet embeddings, multi-collection retrieval, feedback-weighted `improve()`. | Heavy Python dependency stack, LiteLLM/Instructor brittleness, multi-store graph/vector/relational sync bugs, SQLite deadlocks, retrieval filter propagation regressions. | ai-memory should borrow only the pipeline and retrieval ideas, not the multi-store architecture or generic LLM gateway. |
+| MemPalace | Strong verbatim/raw recall story, deterministic IDs, resume-safe mining, transparent benchmark culture, temporal KG ideas. | Chroma/HNSW/FTS corruption under concurrent writers, deferred persistence, destructive repair paths, no decay causing bloat, embedding metadata drift. | ai-memory already avoids the durability failures through a single writer, one SQLite file, provider/model/dim metadata, and decay. The useful feature to steal is bounded raw fallback recall. |
+
+## What ai-memory Already Implements Well
+
+### Storage And Durability
+
+ai-memory's store matches the strongest lesson from the issue research:
+keep durable storage and mutation ordering boring. `Store::open` uses
+SQLite in WAL mode with foreign keys and migrations. `WriterHandle`
+serializes all mutations through one writer thread. Page writes go
+through `Wiki::write_page` or `Wiki::apply_batch`, which also update the
+store rows and FTS triggers.
+
+This directly avoids the major failure classes in cognee and MemPalace:
+parallel SQLite deadlocks, uncoordinated writer processes, Chroma/HNSW
+corruption, and write-success-before-durable-index bugs.
+
+### Source Of Truth
+
+The markdown wiki is organized as
+`wiki/<workspace_id>/<project_id>/<page-path>`, so project renames do
+not move files and two projects can use the same page path without
+collision. That preserves basic-memory's best property, human-editable
+plain files, while avoiding the permalink/workspace retrofit problems
+documented in basic-memory's tracker.
+
+### Capture And Handoff
+
+The hook router accepts lifecycle events over HTTP, returns `202` fast,
+and does the writer work outside the agent's critical path. `SessionEnd`
+creates a deterministic session page, ends the session, opens a typed
+handoff, and commits the wiki. `PreCompact` checkpoints state, using the
+LLM consolidator if configured and deterministic synthesis otherwise.
+
+The explicit `memory_handoff_begin` / `memory_handoff_accept` model is a
+cleaner version of agentmemory's informal handoff behavior.
+
+### Consolidation Model
+
+ai-memory implements versioned page supersession (`is_latest`,
+`supersedes`) and structured JSON-schema consolidation. Multi-page
+consolidation can fan out into `sessions/`, `concepts/`, `decisions/`,
+`gotchas/`, and `_rules/`. Rule routing is a useful extension beyond the
+prior art: durable project rules are surfaced as `_rules/` pages and
+lint suggests moving them into `CLAUDE.md` / `AGENTS.md` where agents see
+them every turn.
+
+### Retrieval And Decay
+
+The current retrieval path is FTS5 by default, with graph-neighbor RRF
+and optional RRF fusion against stored page embeddings. Raw observation
+FTS is used as a bounded fallback when compiled wiki pages miss. The
+retention sweep implements the agentmemory-style decay/reinforcement
+formula for episodic pages and exempts semantic, procedural, pinned, and
+slot pages. Query and recent-page reads bump access counters, which feed
+the reinforcement term.
+
+### Operability
+
+The CLI is mostly a thin HTTP client to the server. State-changing admin
+routes cover backup, bootstrap, reorg, lint, forget sweep, embed,
+commit, purge, rename, and write-page. This preserves the single source
+of truth: the running server owns state.
+
+## Gaps And Improvement Opportunities
+
+### P0: Add Bounded Editable Memory Slots
+
+agentmemory's most useful missing idea is the slot system: small,
+human-editable, durable memory blocks for project context, user
+preferences, current focus, and pending items. ai-memory now implements
+this as pinned `_slots/` markdown pages.
+
+Recommended shape:
+
+| Slot | Purpose | Injection policy |
+|---|---|---|
+| `_slots/project_context.md` | Stable project summary and active architecture constraints. | Small budget, eligible for session-start digest. |
+| `_slots/user_preferences.md` | User preferences that are not project rules. | Small budget, never auto-expanded. |
+| `_slots/current_focus.md` | What the next session should know before searching. | Expires or is overwritten frequently. |
+| `_slots/pending_items.md` | Short actionable queue distinct from handoffs. | Prefer bullets, capped length. |
+
+The important constraint is budget. Do not copy agentmemory's broad
+always-injected context problem. Slots should be tiny, explicit, and
+auditable.
+
+### P0: Make Maintenance Scheduled, Not Only Manual
+
+ai-memory already has the machinery for lint, forget sweep, embeddings,
+and consolidation. Most of it is still on-demand. A memory system should
+quietly maintain itself.
+
+Recommended next step: a server-side scheduler with clear intervals,
+logs, and dry-run-equivalent summaries for:
+
+| Job | Why |
+|---|---|
+| forget sweep | Make the retention formula real without relying on the user to ask. |
+| lint | Surface stale pages, duplicate titles, rule suggestions, and contradictions before they accumulate. |
+| embedding backfill | Keep hybrid retrieval complete after pages are created without embeddings or after provider recovery. |
+| optional consolidation queue | Keep session summaries compiled without blocking hooks. |
+
+This should stay outside hook latency. Agentmemory's hook-blocking
+incidents are the warning label.
+
+### P0: Clarify And Fix The Vector Indexing Contract
+
+There was a concrete mismatch in the implementation notes: `Wiki`'s
+comment implied `write_page` / `apply_batch` both computed embeddings
+when an embedder was attached, but the code only embedded in
+`write_page`. The comment now states the contract explicitly:
+`apply_batch` keeps SQL/file fan-out atomic and vector completeness is
+owned by admin or scheduled backfill.
+
+Also, `write_page` logs and continues if the embedder fails. That is a
+reasonable availability tradeoff, but docs and invariants should be
+precise: FTS5/page persistence is transactional; vector completeness is
+best-effort unless a backfill/status mechanism says otherwise.
+
+Recommended options:
+
+| Option | Tradeoff |
+|---|---|
+| Embed synchronously in `apply_batch` too | Stronger completeness, slower consolidation, external provider failures can delay writes. |
+| Add `embedding_status` and scheduled backfill | Better availability and observability, slightly more schema/state. |
+| Document vector indexing as optional best-effort | Minimal change, but less trustworthy without visible status. |
+
+The best pragmatic path is status plus scheduled backfill. It preserves
+the durable FTS path while making vector gaps visible.
+
+### P1: Make Graph/Link Retrieval First-Class
+
+The schema has a `links` table with nullable `to_page_id`. Markdown
+parsing now extracts wikilinks and markdown links into that table, and
+`memory_query` uses graph-neighbor expansion alongside FTS5 and optional
+page embeddings.
+This leaves two prior-art strengths untapped:
+
+| Source | Idea to adopt |
+|---|---|
+| basic-memory | unresolved forward links and graph context building. |
+| agentmemory | graph as a third retrieval stream in RRF. |
+| cognee | triplet-like searchable relationship text, later. |
+
+Recommended sequence:
+
+1. Parse `[[wiki links]]` and ordinary markdown links from page bodies.
+2. Store unresolved links with `to_page_id = NULL` and resolve them when
+ the target appears.
+3. Add a graph-neighbor expansion mode for pages already retrieved by
+ FTS/vector.
+4. Fold graph hits into RRF as the third signal.
+
+Avoid a separate graph database. SQLite tables and recursive CTEs are
+enough for the expected corpus size.
+
+### P1: Add Bounded Raw/Verbatim Fallback Recall
+
+MemPalace's strongest useful idea is not its storage stack. It is the
+evidence that raw text recall can recover details that compiled summaries
+drop. ai-memory reserves `raw/`; the implemented fallback searches the
+durable `observations` table through `observations_fts`.
+
+Recommended shape:
+
+| Layer | Role |
+|---|---|
+| compiled wiki | Default retrieval, high signal, human-readable. |
+| raw observation fallback | Only searched when compiled wiki misses or the user asks for raw/session detail. |
+| retention budget | Prevent MemPalace-style bloat. |
+| privacy boundary | Use the same sanitizer before raw text becomes durable/searchable. |
+
+This gives ai-memory the best of both philosophies: compile first, but
+keep a bounded escape hatch for exact details.
+
+### P1: Add Diagnostics And Safe Heal Paths
+
+The prior-art trackers show that users forgive complexity less than they
+forgive uncertainty. ai-memory has a simple architecture, but it should
+still expose health checks.
+
+Useful diagnostics:
+
+| Check | Why |
+|---|---|
+| wiki file exists for latest page row | Detect file/db drift. |
+| FTS row count matches latest pages | Detect derived-index corruption. |
+| embedding triples are homogeneous | Already checked at startup, also useful on demand. |
+| pages missing embeddings | Supports vector-status/backfill contract. |
+| unresolved/broken links | Once link parsing exists. |
+| orphan sessions/observations/handoffs | Confirms FK/cascade expectations. |
+| watcher degraded streak | Make reconciliation failures visible in status. |
+
+Safe heal should rebuild derived indexes from markdown and backfill
+embeddings. It should not delete source files without a graveyard step.
+
+### P1: Add Real Retrieval Evaluation
+
+The current recall@5 test is useful as a harness, but it is synthetic.
+Do not make LongMemEval-style claims until they are backed by a real
+held-out dataset.
+
+Recommended benchmark matrix:
+
+| Variant | Purpose |
+|---|---|
+| FTS5 only | Baseline, zero-LLM/zero-embedding mode. |
+| vector only | Measures semantic signal independent of keywords. |
+| FTS5 + vector RRF | Current hybrid path. |
+| FTS5 + vector + graph RRF | Future graph path. |
+| compiled wiki + raw fallback | Future MemPalace-inspired path. |
+
+Borrow MemPalace's transparency, not its headline-chasing. Public claims
+should have tests or should not be claims.
+
+### P2: Consider Feedback/Reinforcement Beyond Access Counts
+
+ai-memory already tracks access count and last access time. cognee's
+useful idea is finer-grained feedback: which memory items were used in a
+successful answer or handoff. This can wait until retrieval paths are
+more mature.
+
+Possible future signals:
+
+| Signal | Use |
+|---|---|
+| page included in handoff | Boost because it supported continuity. |
+| page included in explore digest | Boost if repeatedly surfaced. |
+| user marks finding stale/wrong | Lower confidence or route to lint. |
+| query hit ignored repeatedly | Decay ranking without deleting. |
+
+### P2: Temporal Triples Later
+
+MemPalace's temporal triples are useful for facts that change over time,
+but ai-memory should not add them before basic link extraction and graph
+retrieval exist. A future lightweight table could model
+`subject/predicate/object/valid_from/valid_to/source_page_id`, but adding
+that now would be speculative.
+
+## Ideas Not To Copy
+
+| Temptation | Why not |
+|---|---|
+| agentmemory's iii-engine sidecar | Install and ops fragility were the largest pain cluster. |
+| JSON blobs as primary state with in-memory indexes | Scaling, rebuild, and durability issues. |
+| XML or regex-parsed LLM output | Fragile extraction; ai-memory's JSON-schema path is better. |
+| 50+ MCP tools and 100+ REST endpoints | Burns context and confuses agents. |
+| Broad automatic context injection | Token costs and stale context outweigh convenience. |
+| basic-memory's manual `write_note` workflow | Ambient capture is ai-memory's core advantage. |
+| LLM-authored markdown grammar as storage API | Forces models to generate syntax instead of meaning. |
+| cognee's LiteLLM/Instructor gateway | Provider drift and silent kwarg drops are documented failure modes. |
+| cognee's three-store sync | Correctness bugs concentrate at graph/vector/relational seams. |
+| MemPalace's Chroma/HNSW surface | Concurrent-writer corruption and native-binding failures. |
+| Verbatim-everything with no decay | Leads to bloat and later destructive cleanup pressure. |
+| Destructive repair tooling | Any repair/rebuild must be graveyard-first or derived-index-only. |
+
+## Documentation Mismatches To Fix
+
+These are not necessarily product bugs, but docs should match the code.
+
+| Area | Current status |
+|---|---|
+| MCP tool count/list | Fixed: architecture and install docs list the current 11-tool surface. |
+| Lint scope | Some descriptions mention broken cross-reference auditing, but current lint implements stale episodic pages, duplicate titles, rule suggestions, and optional LLM contradictions. Link checking should wait for link parsing. |
+| sqlite-vec status | Some design language frames sqlite-vec as the chosen/current vector index. Current implementation stores packed vectors in SQLite and does brute-force cosine. |
+| raw archive status | Fixed: docs distinguish reserved `raw/` files from implemented observation-FTS fallback. |
+| scheduled maintenance | Fixed: docs describe default scheduled lint/sweep and opt-in embedding backfill. |
+| procedural tier | The type exists, but procedural extraction/frequency decay is not a developed lifecycle path yet. |
+| SessionEnd consolidation | SessionEnd currently writes a deterministic session page and handoff. LLM consolidation is opt-in and available through PreCompact/manual consolidation; docs should avoid implying all SessionEnd pages are LLM-fanned-out automatically. |
+| batch embeddings | Fixed: code comments and status diagnostics now describe batch vector completeness as backfill-owned. |
+
+## Implemented Roadmap Slice
+
+1. **Tighten docs and vector status first.** Missing embeddings are now
+ visible in status diagnostics.
+2. **Add scheduled jobs.** Periodic lint and forget sweep run by default;
+ embedding backfill is available but opt-in.
+3. **Add slots.** `_slots/` pages are pinned and surfaced through briefing.
+4. **Add link extraction and graph RRF.** The existing SQLite `links`
+ table powers graph-neighbor expansion.
+5. **Add bounded raw fallback recall.** Observation FTS handles exact-detail
+ fallback when compiled pages miss.
+6. **Make retrieval claims test-backed.** The recall harness covers the new
+ graph and raw fallback paths.
+
+## Bottom Line
+
+ai-memory followed agentmemory most closely at the idea level and made
+the right substrate choices to avoid agentmemory's worst operational
+costs. The implemented follow-up kept that discipline: slots, scheduled
+decay/lint, graph-aware retrieval, raw fallback, and diagnostics landed
+without adding a sidecar, broad MCP surface, or multi-store sync layer.
diff --git a/docs/research-agentmemory.md b/docs/research-agentmemory.md
index 2535c9a9..90adb071 100644
--- a/docs/research-agentmemory.md
+++ b/docs/research-agentmemory.md
@@ -1,8 +1,8 @@
# agentmemory - Research Report
-> Source project: `~/Projects/agentmemory` (TypeScript, MCP server built on `iii-engine`).
-> This is the user's own existing project. The Rust effort in this repo is a
-> spiritual successor: keep the *ideas*, replace the *substrate*.
+> Source project: `rohitg00/agentmemory` (TypeScript, MCP server built on `iii-engine`).
+> This repo builds on that earlier TypeScript project: keep the *ideas*, replace
+> the *substrate*.
## 1. Purpose & Scope
diff --git a/docs/research-basic-memory.md b/docs/research-basic-memory.md
index a400b2e6..a6fcc05b 100644
--- a/docs/research-basic-memory.md
+++ b/docs/research-basic-memory.md
@@ -1,6 +1,6 @@
# basic-memory - Research Report
-> Source project: `~/Projects/basic-memory` (Python, MCP-native, markdown-on-disk).
+> Source project: `basicmachines-co/basic-memory` (Python, MCP-native, markdown-on-disk).
> Studied as inspiration *and* as the manual-write-note model we explicitly
> diverge from.
diff --git a/docs/research-cognee.md b/docs/research-cognee.md
index 9275a602..12b84462 100644
--- a/docs/research-cognee.md
+++ b/docs/research-cognee.md
@@ -1,6 +1,6 @@
# cognee - Research Report
-> Source project: `~/Projects/cognee` (Python, knowledge-graph + vector + relational, MCP server).
+> Source project: `topoteretes/cognee` (Python, knowledge-graph + vector + relational, MCP server).
## 1. Purpose & Scope