Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(search): add unspaced CJK retrieval (#1433)
AkitaOnRails9fe497047cf0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9098873b..9f30234a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,14 @@ version in [`docs/changelog/`](docs/changelog/):### Added+- Added zero-LLM substring retrieval for unspaced Han, Kana, and Hangul text.+ Pages and observations gain transactionally maintained derived unigram/bigram+ FTS indexes; existing unicode61 search and ranking remain unchanged for+ Latin-only queries, while CJK matches participate in project, scoped,+ global, temporal, raw-observation, and observation-evidence retrieval.+ Upgraded stores backfill missing derived rows before accepting traffic, and+ compaction rebuilds the auxiliary indexes with the existing FTS set.+ ([#1433](https://github.com/akitaonrails/ai-memory/issues/1433))- Added a first-party Hermes Agent lifecycle and an `install-mcp --clienthermes` client. The `hooks:` block `install-hooks --agent hermes` prints nowalso wires `on_session_start`, `pre_llm_call` (the prompt, never thediff --git a/crates/ai-memory-store/migrations/V86__cjk_search_indexes.sql b/crates/ai-memory-store/migrations/V86__cjk_search_indexes.sqlnew file mode 100644index 00000000..285d303b--- /dev/null+++ b/crates/ai-memory-store/migrations/V86__cjk_search_indexes.sql@@ -0,0 +1,32 @@+-- Auxiliary indexes for unspaced Han, Kana, and Hangul text. The regular+-- unicode61 indexes remain the source for every non-CJK query; Rust writes+-- bounded unigram/bigram terms here and repairs missing rows at startup.+CREATE VIRTUAL TABLE pages_cjk_fts USING fts5(+ title,+ body,+ path_search,+ tokenize = 'unicode61 remove_diacritics 2'+);++CREATE VIRTUAL TABLE observations_cjk_fts USING fts5(+ title,+ body,+ tokenize = 'unicode61 remove_diacritics 2'+);++-- Content changes are not part of the runtime model (pages are versioned and+-- observations immutable). Delete stale derived rows defensively if direct+-- maintenance SQL changes that assumption; startup repair recreates them.+CREATE TRIGGER pages_cjk_fts_ad AFTER DELETE ON pages BEGIN+ DELETE FROM pages_cjk_fts WHERE rowid = old.rowid;+END;+CREATE TRIGGER pages_cjk_fts_au AFTER UPDATE OF title, body, path_search ON pages BEGIN+ DELETE FROM pages_cjk_fts WHERE rowid = old.rowid;+END;++CREATE TRIGGER observations_cjk_fts_ad AFTER DELETE ON observations BEGIN+ DELETE FROM observations_cjk_fts WHERE rowid = old.rowid;+END;+CREATE TRIGGER observations_cjk_fts_au AFTER UPDATE OF title, body ON observations BEGIN+ DELETE FROM observations_cjk_fts WHERE rowid = old.rowid;+END;diff --git a/crates/ai-memory-store/src/api_credentials.rs b/crates/ai-memory-store/src/api_credentials.rsindex 8a055ee3..e9d43448 100644--- a/crates/ai-memory-store/src/api_credentials.rs+++ b/crates/ai-memory-store/src/api_credentials.rs@@ -433,7 +433,7 @@ mod tests {|row| row.get(0),).unwrap();- assert_eq!(version, 85, "update the pin when adding a migration");+ assert_eq!(version, 86, "update the pin when adding a migration");let cols: i64 = conn.query_row("SELECT COUNT(*) FROM pragma_table_info('users') \diff --git a/crates/ai-memory-store/src/fts_query.rs b/crates/ai-memory-store/src/fts_query.rsindex 5d494cab..87c75060 100644--- a/crates/ai-memory-store/src/fts_query.rs+++ b/crates/ai-memory-store/src/fts_query.rs@@ -185,6 +185,94 @@ pub fn prepare_fts5_query(raw: &str, stopwords: &FtsStopwords) -> String {.join(" OR ")}+/// Terms stored in the auxiliary CJK index.+///+/// Each maximal Han/Kana/Hangul run contributes its scalar values and its+/// overlapping bigrams. Splitting only those runs keeps punctuation and Latin+/// text out of the derived index, while the bigrams retain enough adjacency+/// to avoid the very broad matches a character-only index would produce.+#[must_use]+pub(crate) fn cjk_index_terms(raw: &str) -> String {+ let mut terms = Vec::new();+ let mut run = Vec::new();+ for ch in raw.chars() {+ if is_cjk(ch) {+ run.push(ch);+ } else {+ append_cjk_terms(&mut terms, &mut run);+ }+ }+ append_cjk_terms(&mut terms, &mut run);+ terms.join(" ")+}++/// A MATCH expression for the CJK portion of a bounded user query.+///+/// `None` is the important common case: callers then execute the existing SQL+/// byte-for-byte. A multi-scalar run becomes an exact phrase of overlapping+/// bigrams; a single scalar uses the unigram fallback.+#[must_use]+pub(crate) fn prepare_cjk_fts5_query(raw: &str) -> Option<String> {+ let bounded = bounded_query_text(raw);+ let mut phrases = Vec::new();+ let mut run = Vec::new();+ for ch in bounded.chars() {+ if is_cjk(ch) {+ run.push(ch);+ } else {+ append_cjk_query(&mut phrases, &mut run);+ }+ }+ append_cjk_query(&mut phrases, &mut run);+ (!phrases.is_empty()).then(|| phrases.join(" OR "))+}++fn append_cjk_terms(terms: &mut Vec<String>, run: &mut Vec<char>) {+ if run.is_empty() {+ return;+ }+ terms.extend(run.iter().map(char::to_string));+ terms.extend(run.windows(2).map(|pair| pair.iter().collect::<String>()));+ run.clear();+}++fn append_cjk_query(phrases: &mut Vec<String>, run: &mut Vec<char>) {+ if run.is_empty() {+ return;+ }+ if run.len() == 1 {+ phrases.push(format!("\"{}\"", run[0]));+ } else {+ let bigrams = run+ .windows(2)+ .map(|pair| pair.iter().collect::<String>())+ .collect::<Vec<_>>()+ .join(" ");+ phrases.push(format!("\"{bigrams}\""));+ }+ run.clear();+}++fn is_cjk(ch: char) -> bool {+ ch.is_alphanumeric()+ && matches!(+ ch as u32,+ 0x1100..=0x11ff+ | 0x2e80..=0x2fff+ | 0x3040..=0x30ff+ | 0x3130..=0x318f+ | 0x31f0..=0x31ff+ | 0x3400..=0x4dbf+ | 0x4e00..=0x9fff+ | 0xa960..=0xa97f+ | 0xac00..=0xd7af+ | 0xd7b0..=0xd7ff+ | 0xf900..=0xfaff+ | 0xff66..=0xff9f+ | 0x20000..=0x323af+ )+}+/// Upper bound (bytes) on the untrusted search text this module tokenises.////// FTS5's query parser is superlinear in the expression it is handed:@@ -364,6 +452,43 @@ mod tests {super::prepare_fts5_query(raw, &FtsStopwords::default())}+ #[test]+ fn cjk_index_terms_are_bounded_to_cjk_runs() {+ assert_eq!(+ cjk_index_terms("pay 支付服务 now"),+ "支 付 服 务 支付 付服 服务"+ );+ assert_eq!(+ cjk_index_terms("日本語・テスト"),+ "日 本 語 日本 本語 テ ス ト テス スト"+ );+ assert_eq!(cjk_index_terms("plain latin"), "");+ }++ #[test]+ fn cjk_queries_require_adjacent_bigrams() {+ assert_eq!(+ prepare_cjk_fts5_query("支付服务 status"),+ Some("\"支付 付服 服务\"".to_string())+ );+ assert_eq!(+ prepare_cjk_fts5_query("支 / 韓"),+ Some("\"支\" OR \"韓\"".to_string())+ );+ assert_eq!(prepare_cjk_fts5_query("latin only"), None);+ }++ #[test]+ fn cjk_query_input_is_bounded_before_expansion() {+ let query = prepare_cjk_fts5_query(&"日".repeat(10_000)).unwrap();+ assert!(+ query.len() < 12_500,+ "expanded query was {} bytes",+ query.len()+ );+ assert_parses(&query);+ }+/// The engine, not this module, is the judge of validity: every/// prepared query must MATCH without error on a real FTS5 table.fn assert_parses(prepared: &str) {diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex dc00a739..96f7279f 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -224,6 +224,14 @@ impl Store {"project coordinate keys backfilled");}+ let cjk_backfill = ops::backfill_cjk_indexes(&mut conn)?;+ if cjk_backfill.pages > 0 || cjk_backfill.observations > 0 {+ tracing::info!(+ pages = cjk_backfill.pages,+ observations = cjk_backfill.observations,+ "CJK search indexes backfilled"+ );+ }let writer = WriterHandle::spawn(conn);let reader = ReaderPool::new(&db_path, READER_POOL_SOFT_CAP)?;diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex e79b1f76..1fed43a6 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -175,6 +175,7 @@ use rusqlite::{Connection, OptionalExtension, Transaction, params};use sha2::{Digest, Sha256};use crate::error::{StoreError, StoreResult};+use crate::fts_query::cjk_index_terms;/// One embedding upsert requested by a backfill or embed command.#[derive(Debug)]@@ -1581,6 +1582,129 @@ fn stamped_frontmatter(mut conformed: serde_json::Value, now_us: i64) -> StoreReOk(serde_json::to_string(&conformed)?)}+/// Rows added by [`backfill_cjk_indexes`] while opening an upgraded store.+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]+pub(crate) struct CjkBackfillSummary {+ pub(crate) pages: u64,+ pub(crate) observations: u64,+}++/// Repair missing rows in the derived CJK indexes before readers start.+///+/// Every source row receives an auxiliary row, including Latin-only content+/// (whose terms are empty). That makes this an idempotent anti-join instead of+/// rescanning every Latin row at each startup. Deletes and direct content+/// updates remove the derived row through V86 triggers, so a crash or an+/// offline maintenance edit converges here on the next open.+pub(crate) fn backfill_cjk_indexes(conn: &mut Connection) -> StoreResult<CjkBackfillSummary> {+ type PageRow = (i64, String, String, String);+ type ObservationRow = (i64, String, String);+ const BATCH: i64 = 256;+ let mut summary = CjkBackfillSummary::default();++ loop {+ let tx = conn.transaction()?;+ let rows: Vec<PageRow> = {+ let mut stmt = tx.prepare(+ "SELECT p.rowid, p.title, p.body, p.path_search FROM pages p \+ WHERE NOT EXISTS (SELECT 1 FROM pages_cjk_fts f WHERE f.rowid = p.rowid) \+ ORDER BY p.rowid LIMIT ?1",+ )?;+ stmt.query_map([BATCH], |row| {+ Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?))+ })?+ .collect::<Result<_, _>>()?+ };+ if rows.is_empty() {+ tx.commit()?;+ break;+ }+ for (rowid, title, body, path_search) in &rows {+ index_page_cjk(&tx, *rowid, title, body, path_search)?;+ }+ summary.pages += rows.len() as u64;+ tx.commit()?;+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();+ }++ loop {+ let tx = conn.transaction()?;+ let rows: Vec<ObservationRow> = {+ let mut stmt = tx.prepare(+ "SELECT o.rowid, o.title, o.body FROM observations o \+ WHERE NOT EXISTS ( \+ SELECT 1 FROM observations_cjk_fts f WHERE f.rowid = o.rowid \+ ) ORDER BY o.rowid LIMIT ?1",+ )?;+ stmt.query_map([BATCH], |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)))?+ .collect::<Result<_, _>>()?+ };+ if rows.is_empty() {+ tx.commit()?;+ break;+ }+ for (rowid, title, body) in &rows {+ index_observation_cjk(&tx, *rowid, title, body)?;+ }+ summary.observations += rows.len() as u64;+ tx.commit()?;+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();+ }+ Ok(summary)+}++fn index_page_cjk(+ conn: &Connection,+ rowid: i64,+ title: &str,+ body: &str,+ path_search: &str,+) -> StoreResult<()> {+ let result = conn.execute(+ "INSERT INTO pages_cjk_fts(rowid, title, body, path_search) VALUES (?1, ?2, ?3, ?4)",+ params![+ rowid,+ cjk_index_terms(title),+ cjk_index_terms(body),+ cjk_index_terms(path_search)+ ],+ );+ finish_cjk_index_write(result)+}++fn index_observation_cjk(+ conn: &Connection,+ rowid: i64,+ title: &str,+ body: &str,+) -> StoreResult<()> {+ let result = conn.execute(+ "INSERT INTO observations_cjk_fts(rowid, title, body) VALUES (?1, ?2, ?3)",+ params![rowid, cjk_index_terms(title), cjk_index_terms(body)],+ );+ finish_cjk_index_write(result)+}++fn finish_cjk_index_write(result: rusqlite::Result<usize>) -> StoreResult<()> {+ match result {+ Ok(_) => Ok(()),+ Err(error) => {+ // Historical migration tests deliberately stop at V18/V19/V33+ // and then exercise current operation helpers against that frozen+ // schema. A real Store migrates through V86 before the writer is+ // spawned, so a missing table remains a fatal integrity error in+ // every production build.+ #[cfg(test)]+ if error.to_string().contains("no such table:")+ && error.to_string().contains("_cjk_fts")+ {+ return Ok(());+ }+ Err(error.into())+ }+ }+}+/// What a [`backfill_entity_index`] pass did.#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]pub struct EntityBackfillSummary {@@ -1917,6 +2041,13 @@ pub(crate) fn upsert_page_in_tx(compacted_at,],)?;+ index_page_cjk(+ tx,+ tx.last_insert_rowid(),+ &page.title,+ &page.body,+ &path_search,+ )?;replace_links_in_tx(tx, &new_id, page)?;attach_entities_in_tx(tx, &new_id, page, now)?;refresh_incoming_links_for_path(tx, page, &new_id)?;@@ -2001,6 +2132,13 @@ pub(crate) fn upsert_page_in_tx(compacted_at,],)?;+ index_page_cjk(+ tx,+ tx.last_insert_rowid(),+ &page.title,+ &page.body,+ &path_search,+ )?;replace_links_in_tx(tx, &new_id, page)?;attach_entities_in_tx(tx, &new_id, page, now)?;refresh_incoming_links_for_path(tx, page, &new_id)?;@@ -3688,6 +3826,7 @@ fn insert_observation_row(conn: &Connection, obs: &NewObservation) -> StoreResulnow,],)?;+ index_observation_cjk(conn, conn.last_insert_rowid(), &obs.title, &obs.body)?;Ok(id)}@@ -5947,7 +6086,13 @@ pub enum PurgeMode {/// being made about it. A list inside a string literal cannot be compared to/// anything, and the failure mode of getting it wrong is silent — text stays/// searchable after an operator asked for it to be reclaimed.-const ALL_FTS_INDEXES: &[&str] = &["observations_fts", "pages_fts", "workstream_events_fts"];+const ALL_FTS_INDEXES: &[&str] = &[+ "observations_cjk_fts",+ "observations_fts",+ "pages_cjk_fts",+ "pages_fts",+ "workstream_events_fts",+];/// Rebuild every FTS5 index, then `VACUUM`. Runs after the caller's/// transaction has committed: `VACUUM` cannot run inside one, and a rebuild isdiff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rsindex 28dca5a5..e1ef81df 100644--- a/crates/ai-memory-store/src/reader.rs+++ b/crates/ai-memory-store/src/reader.rs@@ -33,7 +33,7 @@ use crate::auto_improve::{PendingAutoImproveScope, bytes32, opt_bytes32, summary_from_row, to_sql_err,};use crate::error::{StoreError, StoreResult};-use crate::fts_query::{FtsStopwords, prepare_fts5_query};+use crate::fts_query::{FtsStopwords, prepare_cjk_fts5_query, prepare_fts5_query};use crate::maintenance::MaintenanceJob;use crate::retrieval_tuning::{RetrievalTuning, is_session_recall_query};use crate::users::TOKEN_HASH_LEN;@@ -631,6 +631,117 @@ fn rerank_page_hits_with_meta(hits}+/// Fuse the ordinary unicode61 and opt-in CJK lexical streams before the+/// existing authority adjustment. The ordinary stream is returned untouched+/// when no auxiliary hits exist, which keeps non-CJK ranking byte-identical.+fn fuse_page_candidate_streams(+ primary: Vec<(PageHit, PageAuthority)>,+ auxiliary: Vec<(PageHit, PageAuthority)>,+ limit: usize,+) -> Vec<(PageHit, PageAuthority)> {+ if auxiliary.is_empty() {+ return primary;+ }+ let mut fused: std::collections::HashMap<PageId, (PageHit, PageAuthority, f64)> =+ std::collections::HashMap::new();+ for stream in [primary, auxiliary] {+ for (rank, (hit, authority)) in stream.into_iter().enumerate() {+ let contribution = 1.0 / (60.0 + (rank + 1) as f64);+ fused+ .entry(hit.id)+ .and_modify(|entry| entry.2 += contribution)+ .or_insert((hit, authority, contribution));+ }+ }+ let mut out: Vec<_> = fused+ .into_values()+ .map(|(mut hit, authority, score)| {+ hit.rank = -score;+ (hit, authority)+ })+ .collect();+ out.sort_by(|a, b| {+ a.0.rank+ .partial_cmp(&b.0.rank)+ .unwrap_or(std::cmp::Ordering::Equal)+ .then_with(|| a.0.path.as_str().cmp(b.0.path.as_str()))+ });+ out.truncate(limit);+ out+}++fn fuse_page_meta_candidate_streams(+ primary: Vec<(PageHitWithMeta, PageAuthority)>,+ auxiliary: Vec<(PageHitWithMeta, PageAuthority)>,+ limit: usize,+) -> Vec<(PageHitWithMeta, PageAuthority)> {+ if auxiliary.is_empty() {+ return primary;+ }+ let mut fused: std::collections::HashMap<PageId, (PageHitWithMeta, PageAuthority, f64)> =+ std::collections::HashMap::new();+ for stream in [primary, auxiliary] {+ for (rank, (hit, authority)) in stream.into_iter().enumerate() {+ let contribution = 1.0 / (60.0 + (rank + 1) as f64);+ fused+ .entry(hit.id)+ .and_modify(|entry| entry.2 += contribution)+ .or_insert((hit, authority, contribution));+ }+ }+ let mut out: Vec<_> = fused+ .into_values()+ .map(|(mut hit, authority, score)| {+ hit.rank = -score;+ (hit, authority)+ })+ .collect();+ out.sort_by(|a, b| {+ a.0.rank+ .partial_cmp(&b.0.rank)+ .unwrap_or(std::cmp::Ordering::Equal)+ .then_with(|| a.0.path.as_str().cmp(b.0.path.as_str()))+ });+ out.truncate(limit);+ out+}++fn fuse_observation_streams(+ primary: Vec<ObservationHit>,+ auxiliary: Vec<ObservationHit>,+ limit: usize,+) -> Vec<ObservationHit> {+ if auxiliary.is_empty() {+ return primary;+ }+ let mut fused: std::collections::HashMap<ObservationId, (ObservationHit, f64)> =+ std::collections::HashMap::new();+ for stream in [primary, auxiliary] {+ for (rank, hit) in stream.into_iter().enumerate() {+ let contribution = 1.0 / (60.0 + (rank + 1) as f64);+ fused+ .entry(hit.id)+ .and_modify(|entry| entry.1 += contribution)+ .or_insert((hit, contribution));+ }+ }+ let mut out: Vec<_> = fused+ .into_values()+ .map(|(mut hit, score)| {+ hit.rank = -score;+ hit+ })+ .collect();+ out.sort_by(|a, b| {+ a.rank+ .partial_cmp(&b.rank)+ .unwrap_or(std::cmp::Ordering::Equal)+ .then_with(|| a.created_at.cmp(&b.created_at))+ });+ out.truncate(limit);+ out+}+/// Rows keyed to one session (see [`ReaderPool::session_dependent_rows`]).#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]pub struct SessionDependentRows {@@ -2734,6 +2845,7 @@ impl ReaderPool {viewer: Option<UserId>,) -> StoreResult<Vec<PageHit>> {let fts_query = normalize_fts_query(&query, &self.fts_stopwords);+ let cjk_query = prepare_cjk_fts5_query(&query);if fts_query.is_empty() || limit == 0 {return Ok(Vec::new());}@@ -2760,7 +2872,7 @@ impl ReaderPool {Value::Integer(authority_candidate_limit(limit) as i64),Value::Integer(now_us()),];- bound.extend(visible_params);+ bound.extend(visible_params.iter().cloned());let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {let id_bytes: Vec<u8> = row.get(0)?;let path: String = row.get(1)?;@@ -2803,7 +2915,66 @@ impl ReaderPool {authority,));}- Ok(rerank_page_hits(candidates, limit))+ let mut cjk_candidates = Vec::new();+ if let Some(cjk_query) = cjk_query {+ let descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json");+ let sql = format!(+ "SELECT pages.id, pages.path, pages.title, {descriptor}, \+ pages_cjk_fts.rank, pages.tier, pages.pinned, \+ pages.frontmatter_json, {kind_expr} AS kind \+ FROM pages_cjk_fts \+ JOIN pages ON pages.rowid = pages_cjk_fts.rowid \+ WHERE pages_cjk_fts MATCH ?1 AND pages.is_latest = 1{not_expired}{visible} \+ ORDER BY pages_cjk_fts.rank LIMIT ?2",+ not_expired = not_expired("pages", "?3"),+ );+ let mut stmt = conn.prepare(&sql)?;+ #[allow(clippy::cast_possible_wrap)]+ let mut bound = vec![+ Value::Text(cjk_query),+ Value::Integer(authority_candidate_limit(limit) as i64),+ Value::Integer(now_us()),+ ];+ bound.extend(visible_params);+ let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {+ Ok((+ row.get::<_, Vec<u8>>(0)?,+ row.get::<_, String>(1)?,+ row.get::<_, String>(2)?,+ row.get::<_, String>(3)?,+ row.get::<_, f64>(4)?,+ row.get::<_, String>(5)?,+ row.get::<_, i64>(6)? != 0,+ row.get::<_, String>(7)?,+ row.get::<_, String>(8)?,+ ))+ })?;+ for row in rows {+ let (id, path, title, descriptor, rank, tier, pinned, frontmatter, kind) = row?;+ let authority =+ PageAuthority::from_stored(&path, &kind, &tier, pinned, &frontmatter);+ cjk_candidates.push((+ PageHit {+ id: PageId::from_slice(&id)?,+ path: PagePath::new(path)?,+ snippet: page_descriptor(&descriptor, &title),+ title,+ rank,+ superseded: false,+ pinned: false,+ },+ authority,+ ));+ }+ }+ Ok(rerank_page_hits(+ fuse_page_candidate_streams(+ candidates,+ cjk_candidates,+ authority_candidate_limit(limit),+ ),+ limit,+ ))}).await}@@ -2829,6 +3000,7 @@ impl ReaderPool {filter: PageFilter<'_>,) -> StoreResult<Vec<PageHitWithMeta>> {let fts_query = normalize_fts_query(&query, &self.fts_stopwords);+ let cjk_query = prepare_cjk_fts5_query(&query);if fts_query.is_empty() || limit == 0 {return Ok(Vec::new());}@@ -2859,8 +3031,8 @@ impl ReaderPool {Value::Integer(authority_candidate_limit(limit) as i64),Value::Integer(cutoff),];- bound.extend(visible_params);- bound.extend(kept_params);+ bound.extend(visible_params.iter().cloned());+ bound.extend(kept_params.iter().cloned());let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {let workspace_name: String = row.get(0)?;let project_name: String = row.get(1)?;@@ -2918,7 +3090,71 @@ impl ReaderPool {authority,));}- Ok(rerank_page_hits_with_meta(candidates, limit))+ let mut cjk_candidates = Vec::new();+ if let Some(cjk_query) = cjk_query {+ let descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json");+ let sql = format!(+ "SELECT workspaces.name, projects.name, pages.path, pages.title, \+ {descriptor}, pages_cjk_fts.rank, pages.tier, pages.pinned, \+ pages.frontmatter_json, {kind_expr} AS kind, pages.id \+ FROM pages_cjk_fts \+ JOIN pages ON pages.rowid = pages_cjk_fts.rowid \+ JOIN projects ON projects.id = pages.project_id \+ JOIN workspaces ON workspaces.id = pages.workspace_id \+ WHERE pages_cjk_fts MATCH ?1 AND pages.is_latest = 1{not_expired}{visible}{kept} \+ ORDER BY pages_cjk_fts.rank LIMIT ?2",+ not_expired = not_expired("pages", "?3"),+ );+ let mut stmt = conn.prepare(&sql)?;+ #[allow(clippy::cast_possible_wrap)]+ let mut bound = vec![+ Value::Text(cjk_query),+ Value::Integer(authority_candidate_limit(limit) as i64),+ Value::Integer(cutoff),+ ];+ bound.extend(visible_params);+ bound.extend(kept_params);+ let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {+ Ok((+ row.get::<_, String>(0)?,+ row.get::<_, String>(1)?,+ row.get::<_, String>(2)?,+ row.get::<_, String>(3)?,+ row.get::<_, String>(4)?,+ row.get::<_, f64>(5)?,+ row.get::<_, String>(6)?,+ row.get::<_, i64>(7)? != 0,+ row.get::<_, String>(8)?,+ row.get::<_, String>(9)?,+ row.get::<_, Vec<u8>>(10)?,+ ))+ })?;+ for row in rows {+ let (workspace_name, project_name, path, title, descriptor, rank, tier, pinned, frontmatter, kind, id) = row?;+ let authority =+ PageAuthority::from_stored(&path, &kind, &tier, pinned, &frontmatter);+ cjk_candidates.push((+ PageHitWithMeta {+ id: PageId::from_slice(&id)?,+ workspace_name,+ project_name,+ path: PagePath::new(path)?,+ snippet: page_descriptor(&descriptor, &title),+ title,+ rank,+ },+ authority,+ ));+ }+ }+ Ok(rerank_page_hits_with_meta(+ fuse_page_meta_candidate_streams(+ candidates,+ cjk_candidates,+ authority_candidate_limit(limit),+ ),+ limit,+ ))}).await}@@ -2969,6 +3205,7 @@ impl ReaderPool {filter: PageFilter<'_>,) -> StoreResult<Vec<(PageHit, PageAuthority)>> {let fts_query = normalize_fts_query(&query, &self.fts_stopwords);+ let cjk_query = prepare_cjk_fts5_query(&query);if fts_query.is_empty() || candidate_limit == 0 {return Ok(Vec::new());}@@ -3000,7 +3237,7 @@ impl ReaderPool {Value::Integer(candidate_limit as i64),Value::Integer(cutoff),];- bound.extend(kept_params);+ bound.extend(kept_params.iter().cloned());let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {let id_bytes: Vec<u8> = row.get(0)?;let path: String = row.get(1)?;@@ -3043,7 +3280,68 @@ impl ReaderPool {authority,));}- Ok(candidates)+ let mut cjk_candidates = Vec::new();+ if let Some(cjk_query) = cjk_query {+ let descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json");+ let sql = format!(+ "SELECT pages.id, pages.path, pages.title, {descriptor}, \+ pages_cjk_fts.rank, pages.tier, pages.pinned, \+ pages.frontmatter_json, {kind_expr} AS kind \+ FROM pages_cjk_fts \+ JOIN pages ON pages.rowid = pages_cjk_fts.rowid \+ WHERE pages_cjk_fts MATCH ?1 \+ AND pages.workspace_id = ?2 AND pages.project_id = ?3 \+ {latest}{not_expired}{kept} \+ ORDER BY pages_cjk_fts.rank LIMIT ?4",+ latest = latest_only("pages", include_superseded),+ not_expired = not_expired("pages", "?5"),+ );+ let mut stmt = conn.prepare(&sql)?;+ #[allow(clippy::cast_possible_wrap)]+ let mut bound = vec![+ Value::Text(cjk_query),+ Value::Blob(workspace_id.as_bytes().to_vec()),+ Value::Blob(project_id.as_bytes().to_vec()),+ Value::Integer(candidate_limit as i64),+ Value::Integer(cutoff),+ ];+ bound.extend(kept_params);+ let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {+ Ok((+ row.get::<_, Vec<u8>>(0)?,+ row.get::<_, String>(1)?,+ row.get::<_, String>(2)?,+ row.get::<_, String>(3)?,+ row.get::<_, f64>(4)?,+ row.get::<_, String>(5)?,+ row.get::<_, i64>(6)? != 0,+ row.get::<_, String>(7)?,+ row.get::<_, String>(8)?,+ ))+ })?;+ for row in rows {+ let (id, path, title, descriptor, rank, tier, pinned, frontmatter, kind) = row?;+ let authority =+ PageAuthority::from_stored(&path, &kind, &tier, pinned, &frontmatter);+ cjk_candidates.push((+ PageHit {+ id: PageId::from_slice(&id)?,+ path: PagePath::new(path)?,+ snippet: page_descriptor(&descriptor, &title),+ title,+ rank,+ superseded: false,+ pinned: false,+ },+ authority,+ ));+ }+ }+ Ok(fuse_page_candidate_streams(+ candidates,+ cjk_candidates,+ candidate_limit,+ ))}).await}@@ -3066,6 +3364,7 @@ impl ReaderPool {filter: PageFilter<'_>,) -> StoreResult<Vec<(PageHit, PageAuthority)>> {let fts_query = normalize_fts_query(&query, &self.fts_stopwords);+ let cjk_query = prepare_cjk_fts5_query(&query);if fts_query.is_empty() || candidate_limit == 0 {return Ok(Vec::new());}@@ -3098,7 +3397,7 @@ impl ReaderPool {Value::Integer(as_of_us),Value::Integer(as_of_us),];- sql_params.extend(kept_params);+ sql_params.extend(kept_params.iter().cloned());#[allow(clippy::cast_possible_wrap)]sql_params.push(Value::Integer(candidate_limit as i64));let rows = stmt.query_map(params_from_iter(sql_params), |row| {@@ -3143,7 +3442,70 @@ impl ReaderPool {authority,));}- Ok(candidates)+ let mut cjk_candidates = Vec::new();+ if let Some(cjk_query) = cjk_query {+ let descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json");+ let sql = format!(+ "SELECT pages.id, pages.path, pages.title, {descriptor}, \+ pages_cjk_fts.rank, pages.tier, pages.pinned, \+ pages.frontmatter_json, {kind_expr} AS kind \+ FROM pages_cjk_fts \+ JOIN pages ON pages.rowid = pages_cjk_fts.rowid \+ WHERE pages_cjk_fts MATCH ?1 \+ AND pages.workspace_id = ?2 AND pages.project_id = ?3 \+ AND pages.valid_from <= ?4 \+ AND (pages.valid_to IS NULL OR pages.valid_to > ?5){not_expired}{kept} \+ ORDER BY pages_cjk_fts.rank LIMIT ?{limit_placeholder}",+ not_expired = not_expired("pages", "?6"),+ );+ let mut sql_params = vec![+ Value::Text(cjk_query),+ Value::Blob(workspace_id.as_bytes().to_vec()),+ Value::Blob(project_id.as_bytes().to_vec()),+ Value::Integer(as_of_us),+ Value::Integer(as_of_us),+ Value::Integer(as_of_us),+ ];+ sql_params.extend(kept_params);+ #[allow(clippy::cast_possible_wrap)]+ sql_params.push(Value::Integer(candidate_limit as i64));+ let mut stmt = conn.prepare(&sql)?;+ let rows = stmt.query_map(params_from_iter(sql_params), |row| {+ Ok((+ row.get::<_, Vec<u8>>(0)?,+ row.get::<_, String>(1)?,+ row.get::<_, String>(2)?,+ row.get::<_, String>(3)?,+ row.get::<_, f64>(4)?,+ row.get::<_, String>(5)?,+ row.get::<_, i64>(6)? != 0,+ row.get::<_, String>(7)?,+ row.get::<_, String>(8)?,+ ))+ })?;+ for row in rows {+ let (id, path, title, descriptor, rank, tier, pinned, frontmatter, kind) = row?;+ let authority =+ PageAuthority::from_stored(&path, &kind, &tier, pinned, &frontmatter);+ cjk_candidates.push((+ PageHit {+ id: PageId::from_slice(&id)?,+ path: PagePath::new(path)?,+ snippet: page_descriptor(&descriptor, &title),+ title,+ rank,+ superseded: false,+ pinned: false,+ },+ authority,+ ));+ }+ }+ Ok(fuse_page_candidate_streams(+ candidates,+ cjk_candidates,+ candidate_limit,+ ))}).await}@@ -3367,6 +3729,7 @@ impl ReaderPool {owner_filter: OwnerFilter,) -> StoreResult<Vec<ObservationHit>> {let fts_query = normalize_fts_query(&query, &self.fts_stopwords);+ let cjk_query = prepare_cjk_fts5_query(&query);if fts_query.is_empty() {return Ok(Vec::new());}@@ -3446,7 +3809,55 @@ impl ReaderPool {created_at,});}- Ok(hits)+ let mut cjk_hits = Vec::new();+ if let Some(cjk_query) = cjk_query {+ let mut stmt = conn.prepare(&format!(+ "SELECT observations.id, observations.session_id, observations.kind, \+ observations.title, substr(observations.body, 1, 240), \+ observations_cjk_fts.rank, observations.created_at \+ FROM observations_cjk_fts \+ JOIN observations ON observations.rowid = observations_cjk_fts.rowid \+ WHERE observations_cjk_fts MATCH ?1 \+ AND observations.workspace_id = ?2 \+ AND observations.project_id = ?3{owner_clause} \+ ORDER BY observations_cjk_fts.rank LIMIT ?4",+ ))?;+ let mut sql_params: Vec<&dyn rusqlite::ToSql> = Vec::with_capacity(5);+ sql_params.push(&cjk_query);+ sql_params.push(&ws_bytes);+ sql_params.push(&proj_bytes);+ sql_params.push(&limit);+ if let Some(actor) = &actor {+ sql_params.push(actor);+ }+ let rows = stmt.query_map(sql_params.as_slice(), |row| {+ Ok((+ row.get::<_, Vec<u8>>(0)?,+ row.get::<_, Vec<u8>>(1)?,+ row.get::<_, String>(2)?,+ row.get::<_, String>(3)?,+ row.get::<_, String>(4)?,+ row.get::<_, f64>(5)?,+ row.get::<_, i64>(6)?,+ ))+ })?;+ for row in rows {+ let (id, session, kind, title, snippet, rank, created_us) = row?;+ let created_at = jiff::Timestamp::from_microsecond(created_us)+ .map(|ts| ts.to_string())+ .unwrap_or_default();+ cjk_hits.push(ObservationHit {+ id: ObservationId::from_slice(&id)?,+ session_id: SessionId::from_slice(&session)?,+ kind,+ title,+ snippet,+ rank,+ created_at,+ });+ }+ }+ Ok(fuse_observation_streams(hits, cjk_hits, limit as usize))}).await}@@ -3474,6 +3885,7 @@ impl ReaderPool {filter: PageFilter<'_>,) -> StoreResult<Vec<ObservationVote>> {let fts_query = normalize_fts_query(query, &self.fts_stopwords);+ let cjk_query = prepare_cjk_fts5_query(query);if fts_query.is_empty() || candidate_limit == 0 {return Ok(Vec::new());}@@ -3507,16 +3919,18 @@ impl ReaderPool {// Best-ranked observation first, grouped by session in first-seen// order: the session's stream rank is its best observation's.let mut sessions: Vec<(SessionId, String, Vec<ObservationId>)> = Vec::new();+ let mut index: std::collections::HashMap<SessionId, usize> =+ std::collections::HashMap::new();+ let candidate_path =+ "'sessions/' || substr(lower(hex(observations.session_id)), 1, 8) || '-' || \+ substr(lower(hex(observations.session_id)), 9, 4) || '-' || \+ substr(lower(hex(observations.session_id)), 13, 4) || '-' || \+ substr(lower(hex(observations.session_id)), 17, 4) || '-' || \+ substr(lower(hex(observations.session_id)), 21, 12) || '.md'";{// CROSS JOIN pins the FTS table as the outer loop: left to// itself the planner drives from the observations scope index// and re-enters FTS once per row, which is far slower.- let candidate_path =- "'sessions/' || substr(lower(hex(observations.session_id)), 1, 8) || '-' || \- substr(lower(hex(observations.session_id)), 9, 4) || '-' || \- substr(lower(hex(observations.session_id)), 13, 4) || '-' || \- substr(lower(hex(observations.session_id)), 17, 4) || '-' || \- substr(lower(hex(observations.session_id)), 21, 12) || '.md'";let mut stmt = tx.prepare(&format!("SELECT observations.id, observations.session_id, \snippet(observations_fts, 1, '<mark>', '</mark>', '…', \@@ -3550,7 +3964,7 @@ impl ReaderPool {Value::Integer(last_rowid),Value::Integer(cutoff),];- candidate_params.extend(candidate_kept_params);+ candidate_params.extend(candidate_kept_params.iter().cloned());let rows = stmt.query_map(params_from_iter(candidate_params.iter()), |row| {Ok((row.get::<_, Vec<u8>>(0)?,@@ -3558,8 +3972,6 @@ impl ReaderPool {row.get::<_, String>(2)?,))})?;- let mut index: std::collections::HashMap<SessionId, usize> =- std::collections::HashMap::new();for row in rows {let (id_bytes, session_bytes, snippet) = row?;let id = ObservationId::from_slice(&id_bytes)?;@@ -3578,6 +3990,64 @@ impl ReaderPool {}}}+ if let Some(cjk_query) = cjk_query {+ let mut stmt = tx.prepare(&format!(+ "SELECT observations.id, observations.session_id, \+ substr(observations.body, 1, 240) \+ FROM observations_cjk_fts \+ CROSS JOIN observations ON observations.rowid = observations_cjk_fts.rowid \+ WHERE observations_cjk_fts MATCH ?1 \+ AND observations_cjk_fts.rowid >= ?5 \+ AND observations_cjk_fts.rowid <= ?6 \+ AND observations.workspace_id = ?2 \+ AND observations.project_id = ?3 \+ AND EXISTS ( \+ SELECT 1 FROM pages AS candidate_page \+ WHERE candidate_page.workspace_id = observations.workspace_id \+ AND candidate_page.project_id = observations.project_id \+ AND candidate_page.is_latest = 1 \+ AND candidate_page.path = {candidate_path} \+ {candidate_not_expired}{candidate_kept} \+ ) \+ ORDER BY observations_cjk_fts.rank LIMIT ?4",+ candidate_not_expired = not_expired("candidate_page", "?7"),+ ))?;+ #[allow(clippy::cast_possible_wrap)]+ let mut candidate_params = vec![+ Value::Text(cjk_query),+ Value::Blob(workspace_id.as_bytes().to_vec()),+ Value::Blob(project_id.as_bytes().to_vec()),+ Value::Integer(row_limit as i64),+ Value::Integer(first_rowid),+ Value::Integer(last_rowid),+ Value::Integer(cutoff),+ ];+ candidate_params.extend(candidate_kept_params);+ let rows = stmt.query_map(params_from_iter(candidate_params.iter()), |row| {+ Ok((+ row.get::<_, Vec<u8>>(0)?,+ row.get::<_, Vec<u8>>(1)?,+ row.get::<_, String>(2)?,+ ))+ })?;+ for row in rows {+ let (id_bytes, session_bytes, snippet) = row?;+ let id = ObservationId::from_slice(&id_bytes)?;+ let session_id = SessionId::from_slice(&session_bytes)?;+ match index.get(&session_id) {+ Some(&at) => {+ let ids = &mut sessions[at].2;+ if ids.len() < OBSERVATION_STREAM_IDS_PER_PAGE && !ids.contains(&id) {+ ids.push(id);+ }+ }+ None => {+ index.insert(session_id, sessions.len());+ sessions.push((session_id, snippet, vec![id]));+ }+ }+ }+ }if sessions.is_empty() {return Ok(Vec::new());}@@ -4145,6 +4615,7 @@ impl ReaderPool {session_id: SessionId,page: ObservationPage,) -> StoreResult<ObservationPageResult> {+ let cjk_query = page.query.as_deref().and_then(prepare_cjk_fts5_query);let fts_query = page.query.as_deref()@@ -4170,16 +4641,29 @@ impl ReaderPool {sql_params.push(Value::Blob(project_id.as_bytes().to_vec()));let mut source = String::from("FROM observations");let mut predicate = String::from(- " WHERE observations.session_id = ? \- AND observations.workspace_id = ? \- AND observations.project_id = ?",+ " WHERE observations.session_id = ?1 \+ AND observations.workspace_id = ?2 \+ AND observations.project_id = ?3",);if let Some(q) = fts_query {- source.push_str(- " JOIN observations_fts ON observations_fts.rowid = observations.rowid",- );- predicate.push_str(" AND observations_fts MATCH ?");- sql_params.push(Value::Text(q));+ if let Some(cjk_query) = cjk_query {+ source.push_str(+ " JOIN ( \+ SELECT rowid FROM observations_fts WHERE observations_fts MATCH ?4 \+ UNION \+ SELECT rowid FROM observations_cjk_fts \+ WHERE observations_cjk_fts MATCH ?5 \+ ) AS matched_fts ON matched_fts.rowid = observations.rowid",+ );+ sql_params.push(Value::Text(q));+ sql_params.push(Value::Text(cjk_query));+ } else {+ source.push_str(+ " JOIN observations_fts ON observations_fts.rowid = observations.rowid",+ );+ predicate.push_str(" AND observations_fts MATCH ?");+ sql_params.push(Value::Text(q));+ }}if !kinds.is_empty() {predicate.push_str(" AND observations.kind IN (");diff --git a/crates/ai-memory-store/tests/suite/cjk_search.rs b/crates/ai-memory-store/tests/suite/cjk_search.rsnew file mode 100644index 00000000..90d2db7e--- /dev/null+++ b/crates/ai-memory-store/tests/suite/cjk_search.rs@@ -0,0 +1,238 @@+//! Issue #1433: unicode61 treats an unspaced CJK run as one token. The+//! auxiliary index must recover substring queries without changing Latin-only+//! retrieval, scope filters, supersession, or upgraded-store behavior.++use ai_memory_core::{NewPage, PagePath, Tier};+use ai_memory_store::Store;++use super::retrieval_observation_stream::{observe, scope, search, session, session_page};++fn page(+ ws: ai_memory_core::WorkspaceId,+ project: ai_memory_core::ProjectId,+ path: &str,+ title: &str,+ body: &str,+) -> NewPage {+ NewPage {+ workspace_id: ws,+ project_id: project,+ path: PagePath::new(path).unwrap(),+ title: title.into(),+ body: body.into(),+ tier: Tier::Semantic,+ frontmatter_json: serde_json::json!({}),+ pinned: false,+ links: Vec::new(),+ author_id: None,+ expires_at: None,+ entities: Vec::new(),+ evidence: Vec::new(),+ }+}++#[tokio::test]+async fn page_search_finds_unspaced_cjk_substrings_and_keeps_latin_hits() {+ let tmp = tempfile::tempdir().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let (ws, project) = scope(&store, "default", "app").await;++ store+ .writer+ .upsert_page(page(+ ws,+ project,+ "payments.md",+ "支付说明",+ "启用微信支付服务后检查回调。",+ ))+ .await+ .unwrap();+ store+ .writer+ .upsert_page(page(+ ws,+ project,+ "japanese.md",+ "運用手順",+ "本番環境で自動更新機能を確認する。",+ ))+ .await+ .unwrap();+ store+ .writer+ .upsert_page(page(+ ws,+ project,+ "deploy.md",+ "Deploy guide",+ "deploy the service after tests pass",+ ))+ .await+ .unwrap();++ let chinese = store+ .reader+ .search_pages_for_project(ws, project, "微信支付".into(), 10, None)+ .await+ .unwrap();+ assert_eq!(chinese.len(), 1);+ assert_eq!(chinese[0].path.as_str(), "payments.md");++ let japanese = store+ .reader+ .search_pages_for_project(ws, project, "自動更新".into(), 10, None)+ .await+ .unwrap();+ assert_eq!(japanese.len(), 1);+ assert_eq!(japanese[0].path.as_str(), "japanese.md");++ let global = store+ .reader+ .search_pages_with_meta(+ "微信支付".into(),+ 10,+ None,+ None,+ ai_memory_store::PageFilter::default(),+ )+ .await+ .unwrap();+ assert_eq!(global.len(), 1);+ assert_eq!(global[0].path.as_str(), "payments.md");++ let mixed = store+ .reader+ .search_pages_for_project(ws, project, "deploy 支付".into(), 10, None)+ .await+ .unwrap();+ let paths: Vec<_> = mixed.iter().map(|hit| hit.path.as_str()).collect();+ assert!(+ paths.contains(&"deploy.md"),+ "Latin stream disappeared: {paths:?}"+ );+ assert!(+ paths.contains(&"payments.md"),+ "CJK stream disappeared: {paths:?}"+ );++ let latin = store+ .reader+ .search_pages_for_project(ws, project, "deploy service".into(), 10, None)+ .await+ .unwrap();+ assert_eq!(latin[0].path.as_str(), "deploy.md");+}++#[tokio::test]+async fn observation_search_and_session_votes_find_cjk_substrings() {+ let tmp = tempfile::tempdir().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let (ws, project) = scope(&store, "default", "app").await;+ let session_id = session(&store, ws, project).await;+ observe(+ &store,+ ws,+ project,+ session_id,+ "启用微信支付服务后检查回调",+ )+ .await;+ session_page(+ &store,+ ws,+ project,+ session_id,+ "English-only session summary",+ )+ .await;++ let observations = store+ .reader+ .search_observations_for_project(ws, project, "微信支付".into(), 10)+ .await+ .unwrap();+ assert_eq!(observations.len(), 1);+ assert_eq!(observations[0].session_id, session_id);++ let timeline = store+ .reader+ .session_observations_scoped(+ ws,+ project,+ session_id,+ ai_memory_store::ObservationPage {+ limit: 10,+ offset: 0,+ order: ai_memory_store::ObservationOrder::Asc,+ kinds: None,+ query: Some("微信支付".into()),+ },+ )+ .await+ .unwrap();+ assert_eq!(timeline.total, 1);++ let pages = search(&store, ws, project, "微信支付").await;+ assert!(+ pages+ .iter()+ .any(|(hit, _)| hit.path.as_str() == format!("sessions/{session_id}.md")),+ "the CJK observation stream did not vote for its session page: {pages:?}"+ );+}++#[tokio::test]+async fn superseded_cjk_text_is_hidden_and_startup_repairs_missing_index_rows() {+ let tmp = tempfile::tempdir().unwrap();+ let mut store = Store::open(tmp.path()).unwrap();+ let (ws, project) = scope(&store, "default", "app").await;+ store+ .writer+ .upsert_page(page(ws, project, "state.md", "状态", "旧版支付服务"))+ .await+ .unwrap();+ store+ .writer+ .upsert_page(page(ws, project, "state.md", "State", "new English state"))+ .await+ .unwrap();+ assert!(+ store+ .reader+ .search_pages_for_project(ws, project, "旧版支付".into(), 10, None)+ .await+ .unwrap()+ .is_empty()+ );++ store+ .writer+ .upsert_page(page(ws, project, "repair.md", "恢复", "需要恢复索引"))+ .await+ .unwrap();+ store.close().await;+ drop(store);++ let db = rusqlite::Connection::open(tmp.path().join("db/memory.sqlite")).unwrap();+ db.execute("DELETE FROM pages_cjk_fts", []).unwrap();+ drop(db);++ store = Store::open(tmp.path()).unwrap();+ let repaired = store+ .reader+ .search_pages_for_project(ws, project, "恢复索引".into(), 10, None)+ .await+ .unwrap();+ assert_eq!(repaired.len(), 1);+ assert_eq!(repaired[0].path.as_str(), "repair.md");++ store.writer.compact().await.unwrap();+ let after_rebuild = store+ .reader+ .search_pages_for_project(ws, project, "恢复索引".into(), 10, None)+ .await+ .unwrap();+ assert_eq!(after_rebuild.len(), 1);+ assert_eq!(after_rebuild[0].path.as_str(), "repair.md");+}diff --git a/crates/ai-memory-store/tests/suite/mod.rs b/crates/ai-memory-store/tests/suite/mod.rsindex 129cca28..b922a245 100644--- a/crates/ai-memory-store/tests/suite/mod.rs+++ b/crates/ai-memory-store/tests/suite/mod.rs@@ -9,6 +9,7 @@ mod audit_contamination;mod audit_log;mod auto_improve_staging;mod belief_authority;+mod cjk_search;mod client_activity;mod fts_drift_status;mod fts_stopwords_config;diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 5610ea8f..fd88ea5f 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -166,7 +166,10 @@ from hook paths.trust boundary and delimiters precede automatically injected handoffs,project briefs, and managed-workstream packets; current instructions andcheckout state remain authoritative.-6. `memory_query` answers via FTS5 + entity-match + link-neighbour RRF; when an+6. `memory_query` answers via FTS5 + entity-match + link-neighbour RRF; a+ transactionally maintained auxiliary unigram/bigram FTS stream handles+ unspaced Han, Kana, and Hangul queries without changing the ordinary+ unicode61 path for non-CJK text. When anembedder is configured, vector cosine over `page_embeddings` joins the sameRRF. The entity index is derived from the canonical frontmatter `entities`list, and an empty index contributes no candidates or score. Before final@@ -541,9 +544,11 @@ prunable rather than only after it ages in place.| `workspaces`, `projects` | Top of the 3-tuple identity coordinate. `projects.identity` / `identity_source` (V70, #708) hold the repository identity a project routes by (an explicit marker `identity` or a normalised git remote, resolved client-side), unique per workspace when set; empty until a capture claims the project. Phase-5 clients explicitly send `IdentityStyle::Path` (#1033) for remote-backed projects, naming them from the full repository path without the host (`acme-api`), and send explicit `host_path` for the pre-v3 opt-out. Server-side omission remains `HostPath` because old clients omitted that historical default; no new wire field is needed. Creation or authorized capture promotion uses that canonical name only when no different hostful identity claims it; otherwise resolution falls back to the existing hostful/legacy-safe name. V76 stores indexed canonical and v2-basename keys derived from that full identity: reads are no-create/no-rename, while an authorized explicit write through the canonical name promotes only `projects.name` in the same resolve-or-create writer transaction and preserves the UUID. Project `_meta.md` carries the optional full identity/source plus derived keys; clean reindex validates them and rebuilds the same UUID/name/coordinate lookup, while older manifests without identity remain valid. Ambiguous cross-forge or multi-candidate keys fail closed. A local marker may additionally send up to 16 validated former names as one transient `aliases` JSON-array query value: lookup stays inside its selected workspace, requires the checkout's full hostful git-remote identity to equal the stored row, returns reads without rename, and lets capture perform only the same authorized UUID-preserving canonical promotion. Aliases are not stored and remain distinct in the hook routing cache. The exact operator-home marker may additionally define bounded `[routes.identity."host/path"]` and `[routes.path."absolute-or-tilde-path"]` blocks: an exact hostful remote match wins the longest component-wise lexical path match, any reachable non-home local settings marker wins both, malformed or ambiguous route maps fail closed, and selected fields reuse the existing marker wire without a server/schema change. `AI_MEMORY_HOME` is authoritative for native callers; portable clients use `HOME`/`USERPROFILE`, and Docker compares its forwarded host cwd. See `docs/marker-file.md#repository-identity`. `projects.access_mode` (V68; `open` default / `restricted`) decides whether any authenticated user or only root, the creator (`projects.created_by`, V69) and grant holders (`project_grants`, V68) reach it, decided by `ai_memory_store::authorize_project` (see `docs/users.md#per-project-access`). `projects.profile_contribute` / `profile_consume` (V74) record the marker's `[profile]` opt-outs, forwarded at session start; `profile_candidates`, `profile_harvest_marks` and `profile_entry_ledger` (V75) hold the profile pass's evidence, per-project read position and the entries it wrote. Reserved projects: `scratch`, `_global` (shared preferences; holds the single-user profile under `profile/`), `_profile` (a workspace profile) and `_profile.<user id>` (a private profile, always `restricted`); capture is never attributed to any of them (see `docs/cross-project-profile.md`). || `pages` | Versioned wiki pages with `is_latest` + `supersedes` chain. M8 columns: `last_accessed_at`, `access_count`, and decay-only tombstone marker `superseded_at`. M9 cols: `embedding_provider`, `embedding_model`, `embedding_dim`. V36: `expires_at` (frontmatter TTL). V37: `salience` (NULL = `salience_default`; derived from `page_feedback`). || `pages_fts` | FTS5 virtual table over `(title, body)`, auto-synced by triggers. |+| `pages_cjk_fts` | Derived unigram/bigram FTS5 index over the CJK runs in page title, body, and path. The writer updates it in the page transaction; startup repairs missing rows after an upgrade. Queried only when the bounded query contains Han, Kana, or Hangul. || `sessions`, `observations` | Sanitized, bounded lifecycle-hook projections. `sessions.ended_observation_count` is the stable generation watermark for resumed-session re-end eligibility; wall clocks are not used for that decision. They are an operational audit trail, not a complete native transcript. || `session_consolidation_jobs` | Durable, observation-generation-idempotent queue for automatic LLM consolidation. PreCompact / PostCompaction checkpoint jobs are queued whether or not the session has ended (a resumed session keeps `ended_at`); opt-in SessionEnd jobs need an *ended* session, and observation generations order the two. One bounded server worker leases jobs, retries provider failures with backoff, and recovers expired leases after restart. The worker starts whenever a consolidator is configured; SessionEnd enqueue stays behind `AI_MEMORY_CONSOLIDATE_ON_SESSION_END`. A manual `memory_consolidate` writes its page out-of-band and then reconciles this table (flipping a `failed`/`pending`/`superseded` row for the session to `completed`, never touching a live `running` lease), so the operator does not see a `failed` job for a session that is in fact consolidated. || `observations_fts` | FTS5 virtual table over raw observation `(title, body)`. Feeds the bounded `raw_hits` fallback and, through `[retrieval] observation_stream` (on by default), the observation-evidence RRF stream that votes for session pages. |+| `observations_cjk_fts` | Derived unigram/bigram companion for unspaced CJK observation text. It follows the same transactional write, scope, owner-visibility, raw fallback, timeline, and session-vote boundaries as `observations_fts`. || `workstreams`, `managed_runs`, `workstream_native_sessions` | Optional lease state plus per-harness native source and delivery cursors for `ai-memory run`. || `workstream_events`, `workstream_events_fts` | Append-only normalized visible transcript events and full-text search; immutable sanitized source batches also live under `raw/workstreams/`. || `links` | Wikilink / markdown cross-references. `to_page_id` (a global PageId) is nullable for unresolved forward links. `to_workspace` / `to_project` carry a cross-project scope (NULL = the source page's own project). |