Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(search): add unspaced CJK retrieval (#1433)

AkitaOnRails9fe497047cf0
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9098873b..9f30234a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,14 @@ version in [`docs/changelog/`](docs/changelog/):
### Added
+- Added zero-LLM substring retrieval for unspaced Han, Kana, and Hangul text.
+ Pages and observations gain transactionally maintained derived unigram/bigram
+ FTS indexes; existing unicode61 search and ranking remain unchanged for
+ Latin-only queries, while CJK matches participate in project, scoped,
+ global, temporal, raw-observation, and observation-evidence retrieval.
+ Upgraded stores backfill missing derived rows before accepting traffic, and
+ compaction rebuilds the auxiliary indexes with the existing FTS set.
+ ([#1433](https://github.com/akitaonrails/ai-memory/issues/1433))
- Added a first-party Hermes Agent lifecycle and an `install-mcp --client
hermes` client. The `hooks:` block `install-hooks --agent hermes` prints now
also wires `on_session_start`, `pre_llm_call` (the prompt, never the
diff --git a/crates/ai-memory-store/migrations/V86__cjk_search_indexes.sql b/crates/ai-memory-store/migrations/V86__cjk_search_indexes.sql
new file mode 100644
index 00000000..285d303b
--- /dev/null
+++ b/crates/ai-memory-store/migrations/V86__cjk_search_indexes.sql
@@ -0,0 +1,32 @@
+-- Auxiliary indexes for unspaced Han, Kana, and Hangul text. The regular
+-- unicode61 indexes remain the source for every non-CJK query; Rust writes
+-- bounded unigram/bigram terms here and repairs missing rows at startup.
+CREATE VIRTUAL TABLE pages_cjk_fts USING fts5(
+ title,
+ body,
+ path_search,
+ tokenize = 'unicode61 remove_diacritics 2'
+);
+
+CREATE VIRTUAL TABLE observations_cjk_fts USING fts5(
+ title,
+ body,
+ tokenize = 'unicode61 remove_diacritics 2'
+);
+
+-- Content changes are not part of the runtime model (pages are versioned and
+-- observations immutable). Delete stale derived rows defensively if direct
+-- maintenance SQL changes that assumption; startup repair recreates them.
+CREATE TRIGGER pages_cjk_fts_ad AFTER DELETE ON pages BEGIN
+ DELETE FROM pages_cjk_fts WHERE rowid = old.rowid;
+END;
+CREATE TRIGGER pages_cjk_fts_au AFTER UPDATE OF title, body, path_search ON pages BEGIN
+ DELETE FROM pages_cjk_fts WHERE rowid = old.rowid;
+END;
+
+CREATE TRIGGER observations_cjk_fts_ad AFTER DELETE ON observations BEGIN
+ DELETE FROM observations_cjk_fts WHERE rowid = old.rowid;
+END;
+CREATE TRIGGER observations_cjk_fts_au AFTER UPDATE OF title, body ON observations BEGIN
+ DELETE FROM observations_cjk_fts WHERE rowid = old.rowid;
+END;
diff --git a/crates/ai-memory-store/src/api_credentials.rs b/crates/ai-memory-store/src/api_credentials.rs
index 8a055ee3..e9d43448 100644
--- a/crates/ai-memory-store/src/api_credentials.rs
+++ b/crates/ai-memory-store/src/api_credentials.rs
@@ -433,7 +433,7 @@ mod tests {
|row| row.get(0),
)
.unwrap();
- assert_eq!(version, 85, "update the pin when adding a migration");
+ assert_eq!(version, 86, "update the pin when adding a migration");
let cols: i64 = conn
.query_row(
"SELECT COUNT(*) FROM pragma_table_info('users') \
diff --git a/crates/ai-memory-store/src/fts_query.rs b/crates/ai-memory-store/src/fts_query.rs
index 5d494cab..87c75060 100644
--- a/crates/ai-memory-store/src/fts_query.rs
+++ b/crates/ai-memory-store/src/fts_query.rs
@@ -185,6 +185,94 @@ pub fn prepare_fts5_query(raw: &str, stopwords: &FtsStopwords) -> String {
.join(" OR ")
}
+/// Terms stored in the auxiliary CJK index.
+///
+/// Each maximal Han/Kana/Hangul run contributes its scalar values and its
+/// overlapping bigrams. Splitting only those runs keeps punctuation and Latin
+/// text out of the derived index, while the bigrams retain enough adjacency
+/// to avoid the very broad matches a character-only index would produce.
+#[must_use]
+pub(crate) fn cjk_index_terms(raw: &str) -> String {
+ let mut terms = Vec::new();
+ let mut run = Vec::new();
+ for ch in raw.chars() {
+ if is_cjk(ch) {
+ run.push(ch);
+ } else {
+ append_cjk_terms(&mut terms, &mut run);
+ }
+ }
+ append_cjk_terms(&mut terms, &mut run);
+ terms.join(" ")
+}
+
+/// A MATCH expression for the CJK portion of a bounded user query.
+///
+/// `None` is the important common case: callers then execute the existing SQL
+/// byte-for-byte. A multi-scalar run becomes an exact phrase of overlapping
+/// bigrams; a single scalar uses the unigram fallback.
+#[must_use]
+pub(crate) fn prepare_cjk_fts5_query(raw: &str) -> Option<String> {
+ let bounded = bounded_query_text(raw);
+ let mut phrases = Vec::new();
+ let mut run = Vec::new();
+ for ch in bounded.chars() {
+ if is_cjk(ch) {
+ run.push(ch);
+ } else {
+ append_cjk_query(&mut phrases, &mut run);
+ }
+ }
+ append_cjk_query(&mut phrases, &mut run);
+ (!phrases.is_empty()).then(|| phrases.join(" OR "))
+}
+
+fn append_cjk_terms(terms: &mut Vec<String>, run: &mut Vec<char>) {
+ if run.is_empty() {
+ return;
+ }
+ terms.extend(run.iter().map(char::to_string));
+ terms.extend(run.windows(2).map(|pair| pair.iter().collect::<String>()));
+ run.clear();
+}
+
+fn append_cjk_query(phrases: &mut Vec<String>, run: &mut Vec<char>) {
+ if run.is_empty() {
+ return;
+ }
+ if run.len() == 1 {
+ phrases.push(format!("\"{}\"", run[0]));
+ } else {
+ let bigrams = run
+ .windows(2)
+ .map(|pair| pair.iter().collect::<String>())
+ .collect::<Vec<_>>()
+ .join(" ");
+ phrases.push(format!("\"{bigrams}\""));
+ }
+ run.clear();
+}
+
+fn is_cjk(ch: char) -> bool {
+ ch.is_alphanumeric()
+ && matches!(
+ ch as u32,
+ 0x1100..=0x11ff
+ | 0x2e80..=0x2fff
+ | 0x3040..=0x30ff
+ | 0x3130..=0x318f
+ | 0x31f0..=0x31ff
+ | 0x3400..=0x4dbf
+ | 0x4e00..=0x9fff
+ | 0xa960..=0xa97f
+ | 0xac00..=0xd7af
+ | 0xd7b0..=0xd7ff
+ | 0xf900..=0xfaff
+ | 0xff66..=0xff9f
+ | 0x20000..=0x323af
+ )
+}
+
/// Upper bound (bytes) on the untrusted search text this module tokenises.
///
/// FTS5's query parser is superlinear in the expression it is handed:
@@ -364,6 +452,43 @@ mod tests {
super::prepare_fts5_query(raw, &FtsStopwords::default())
}
+ #[test]
+ fn cjk_index_terms_are_bounded_to_cjk_runs() {
+ assert_eq!(
+ cjk_index_terms("pay 支付服务 now"),
+ "支 付 服 务 支付 付服 服务"
+ );
+ assert_eq!(
+ cjk_index_terms("日本語・テスト"),
+ "日 本 語 日本 本語 テ ス ト テス スト"
+ );
+ assert_eq!(cjk_index_terms("plain latin"), "");
+ }
+
+ #[test]
+ fn cjk_queries_require_adjacent_bigrams() {
+ assert_eq!(
+ prepare_cjk_fts5_query("支付服务 status"),
+ Some("\"支付 付服 服务\"".to_string())
+ );
+ assert_eq!(
+ prepare_cjk_fts5_query("支 / 韓"),
+ Some("\"支\" OR \"韓\"".to_string())
+ );
+ assert_eq!(prepare_cjk_fts5_query("latin only"), None);
+ }
+
+ #[test]
+ fn cjk_query_input_is_bounded_before_expansion() {
+ let query = prepare_cjk_fts5_query(&"日".repeat(10_000)).unwrap();
+ assert!(
+ query.len() < 12_500,
+ "expanded query was {} bytes",
+ query.len()
+ );
+ assert_parses(&query);
+ }
+
/// The engine, not this module, is the judge of validity: every
/// prepared query must MATCH without error on a real FTS5 table.
fn assert_parses(prepared: &str) {
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index dc00a739..96f7279f 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -224,6 +224,14 @@ impl Store {
"project coordinate keys backfilled"
);
}
+ let cjk_backfill = ops::backfill_cjk_indexes(&mut conn)?;
+ if cjk_backfill.pages > 0 || cjk_backfill.observations > 0 {
+ tracing::info!(
+ pages = cjk_backfill.pages,
+ observations = cjk_backfill.observations,
+ "CJK search indexes backfilled"
+ );
+ }
let writer = WriterHandle::spawn(conn);
let reader = ReaderPool::new(&db_path, READER_POOL_SOFT_CAP)?;
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index e79b1f76..1fed43a6 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -175,6 +175,7 @@ use rusqlite::{Connection, OptionalExtension, Transaction, params};
use sha2::{Digest, Sha256};
use crate::error::{StoreError, StoreResult};
+use crate::fts_query::cjk_index_terms;
/// One embedding upsert requested by a backfill or embed command.
#[derive(Debug)]
@@ -1581,6 +1582,129 @@ fn stamped_frontmatter(mut conformed: serde_json::Value, now_us: i64) -> StoreRe
Ok(serde_json::to_string(&conformed)?)
}
+/// Rows added by [`backfill_cjk_indexes`] while opening an upgraded store.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+pub(crate) struct CjkBackfillSummary {
+ pub(crate) pages: u64,
+ pub(crate) observations: u64,
+}
+
+/// Repair missing rows in the derived CJK indexes before readers start.
+///
+/// Every source row receives an auxiliary row, including Latin-only content
+/// (whose terms are empty). That makes this an idempotent anti-join instead of
+/// rescanning every Latin row at each startup. Deletes and direct content
+/// updates remove the derived row through V86 triggers, so a crash or an
+/// offline maintenance edit converges here on the next open.
+pub(crate) fn backfill_cjk_indexes(conn: &mut Connection) -> StoreResult<CjkBackfillSummary> {
+ type PageRow = (i64, String, String, String);
+ type ObservationRow = (i64, String, String);
+ const BATCH: i64 = 256;
+ let mut summary = CjkBackfillSummary::default();
+
+ loop {
+ let tx = conn.transaction()?;
+ let rows: Vec<PageRow> = {
+ let mut stmt = tx.prepare(
+ "SELECT p.rowid, p.title, p.body, p.path_search FROM pages p \
+ WHERE NOT EXISTS (SELECT 1 FROM pages_cjk_fts f WHERE f.rowid = p.rowid) \
+ ORDER BY p.rowid LIMIT ?1",
+ )?;
+ stmt.query_map([BATCH], |row| {
+ Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?))
+ })?
+ .collect::<Result<_, _>>()?
+ };
+ if rows.is_empty() {
+ tx.commit()?;
+ break;
+ }
+ for (rowid, title, body, path_search) in &rows {
+ index_page_cjk(&tx, *rowid, title, body, path_search)?;
+ }
+ summary.pages += rows.len() as u64;
+ tx.commit()?;
+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();
+ }
+
+ loop {
+ let tx = conn.transaction()?;
+ let rows: Vec<ObservationRow> = {
+ let mut stmt = tx.prepare(
+ "SELECT o.rowid, o.title, o.body FROM observations o \
+ WHERE NOT EXISTS ( \
+ SELECT 1 FROM observations_cjk_fts f WHERE f.rowid = o.rowid \
+ ) ORDER BY o.rowid LIMIT ?1",
+ )?;
+ stmt.query_map([BATCH], |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)))?
+ .collect::<Result<_, _>>()?
+ };
+ if rows.is_empty() {
+ tx.commit()?;
+ break;
+ }
+ for (rowid, title, body) in &rows {
+ index_observation_cjk(&tx, *rowid, title, body)?;
+ }
+ summary.observations += rows.len() as u64;
+ tx.commit()?;
+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();
+ }
+ Ok(summary)
+}
+
+fn index_page_cjk(
+ conn: &Connection,
+ rowid: i64,
+ title: &str,
+ body: &str,
+ path_search: &str,
+) -> StoreResult<()> {
+ let result = conn.execute(
+ "INSERT INTO pages_cjk_fts(rowid, title, body, path_search) VALUES (?1, ?2, ?3, ?4)",
+ params![
+ rowid,
+ cjk_index_terms(title),
+ cjk_index_terms(body),
+ cjk_index_terms(path_search)
+ ],
+ );
+ finish_cjk_index_write(result)
+}
+
+fn index_observation_cjk(
+ conn: &Connection,
+ rowid: i64,
+ title: &str,
+ body: &str,
+) -> StoreResult<()> {
+ let result = conn.execute(
+ "INSERT INTO observations_cjk_fts(rowid, title, body) VALUES (?1, ?2, ?3)",
+ params![rowid, cjk_index_terms(title), cjk_index_terms(body)],
+ );
+ finish_cjk_index_write(result)
+}
+
+fn finish_cjk_index_write(result: rusqlite::Result<usize>) -> StoreResult<()> {
+ match result {
+ Ok(_) => Ok(()),
+ Err(error) => {
+ // Historical migration tests deliberately stop at V18/V19/V33
+ // and then exercise current operation helpers against that frozen
+ // schema. A real Store migrates through V86 before the writer is
+ // spawned, so a missing table remains a fatal integrity error in
+ // every production build.
+ #[cfg(test)]
+ if error.to_string().contains("no such table:")
+ && error.to_string().contains("_cjk_fts")
+ {
+ return Ok(());
+ }
+ Err(error.into())
+ }
+ }
+}
+
/// What a [`backfill_entity_index`] pass did.
#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
pub struct EntityBackfillSummary {
@@ -1917,6 +2041,13 @@ pub(crate) fn upsert_page_in_tx(
compacted_at,
],
)?;
+ index_page_cjk(
+ tx,
+ tx.last_insert_rowid(),
+ &page.title,
+ &page.body,
+ &path_search,
+ )?;
replace_links_in_tx(tx, &new_id, page)?;
attach_entities_in_tx(tx, &new_id, page, now)?;
refresh_incoming_links_for_path(tx, page, &new_id)?;
@@ -2001,6 +2132,13 @@ pub(crate) fn upsert_page_in_tx(
compacted_at,
],
)?;
+ index_page_cjk(
+ tx,
+ tx.last_insert_rowid(),
+ &page.title,
+ &page.body,
+ &path_search,
+ )?;
replace_links_in_tx(tx, &new_id, page)?;
attach_entities_in_tx(tx, &new_id, page, now)?;
refresh_incoming_links_for_path(tx, page, &new_id)?;
@@ -3688,6 +3826,7 @@ fn insert_observation_row(conn: &Connection, obs: &NewObservation) -> StoreResul
now,
],
)?;
+ index_observation_cjk(conn, conn.last_insert_rowid(), &obs.title, &obs.body)?;
Ok(id)
}
@@ -5947,7 +6086,13 @@ pub enum PurgeMode {
/// being made about it. A list inside a string literal cannot be compared to
/// anything, and the failure mode of getting it wrong is silent — text stays
/// searchable after an operator asked for it to be reclaimed.
-const ALL_FTS_INDEXES: &[&str] = &["observations_fts", "pages_fts", "workstream_events_fts"];
+const ALL_FTS_INDEXES: &[&str] = &[
+ "observations_cjk_fts",
+ "observations_fts",
+ "pages_cjk_fts",
+ "pages_fts",
+ "workstream_events_fts",
+];
/// Rebuild every FTS5 index, then `VACUUM`. Runs after the caller's
/// transaction has committed: `VACUUM` cannot run inside one, and a rebuild is
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index 28dca5a5..e1ef81df 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -33,7 +33,7 @@ use crate::auto_improve::{
PendingAutoImproveScope, bytes32, opt_bytes32, summary_from_row, to_sql_err,
};
use crate::error::{StoreError, StoreResult};
-use crate::fts_query::{FtsStopwords, prepare_fts5_query};
+use crate::fts_query::{FtsStopwords, prepare_cjk_fts5_query, prepare_fts5_query};
use crate::maintenance::MaintenanceJob;
use crate::retrieval_tuning::{RetrievalTuning, is_session_recall_query};
use crate::users::TOKEN_HASH_LEN;
@@ -631,6 +631,117 @@ fn rerank_page_hits_with_meta(
hits
}
+/// Fuse the ordinary unicode61 and opt-in CJK lexical streams before the
+/// existing authority adjustment. The ordinary stream is returned untouched
+/// when no auxiliary hits exist, which keeps non-CJK ranking byte-identical.
+fn fuse_page_candidate_streams(
+ primary: Vec<(PageHit, PageAuthority)>,
+ auxiliary: Vec<(PageHit, PageAuthority)>,
+ limit: usize,
+) -> Vec<(PageHit, PageAuthority)> {
+ if auxiliary.is_empty() {
+ return primary;
+ }
+ let mut fused: std::collections::HashMap<PageId, (PageHit, PageAuthority, f64)> =
+ std::collections::HashMap::new();
+ for stream in [primary, auxiliary] {
+ for (rank, (hit, authority)) in stream.into_iter().enumerate() {
+ let contribution = 1.0 / (60.0 + (rank + 1) as f64);
+ fused
+ .entry(hit.id)
+ .and_modify(|entry| entry.2 += contribution)
+ .or_insert((hit, authority, contribution));
+ }
+ }
+ let mut out: Vec<_> = fused
+ .into_values()
+ .map(|(mut hit, authority, score)| {
+ hit.rank = -score;
+ (hit, authority)
+ })
+ .collect();
+ out.sort_by(|a, b| {
+ a.0.rank
+ .partial_cmp(&b.0.rank)
+ .unwrap_or(std::cmp::Ordering::Equal)
+ .then_with(|| a.0.path.as_str().cmp(b.0.path.as_str()))
+ });
+ out.truncate(limit);
+ out
+}
+
+fn fuse_page_meta_candidate_streams(
+ primary: Vec<(PageHitWithMeta, PageAuthority)>,
+ auxiliary: Vec<(PageHitWithMeta, PageAuthority)>,
+ limit: usize,
+) -> Vec<(PageHitWithMeta, PageAuthority)> {
+ if auxiliary.is_empty() {
+ return primary;
+ }
+ let mut fused: std::collections::HashMap<PageId, (PageHitWithMeta, PageAuthority, f64)> =
+ std::collections::HashMap::new();
+ for stream in [primary, auxiliary] {
+ for (rank, (hit, authority)) in stream.into_iter().enumerate() {
+ let contribution = 1.0 / (60.0 + (rank + 1) as f64);
+ fused
+ .entry(hit.id)
+ .and_modify(|entry| entry.2 += contribution)
+ .or_insert((hit, authority, contribution));
+ }
+ }
+ let mut out: Vec<_> = fused
+ .into_values()
+ .map(|(mut hit, authority, score)| {
+ hit.rank = -score;
+ (hit, authority)
+ })
+ .collect();
+ out.sort_by(|a, b| {
+ a.0.rank
+ .partial_cmp(&b.0.rank)
+ .unwrap_or(std::cmp::Ordering::Equal)
+ .then_with(|| a.0.path.as_str().cmp(b.0.path.as_str()))
+ });
+ out.truncate(limit);
+ out
+}
+
+fn fuse_observation_streams(
+ primary: Vec<ObservationHit>,
+ auxiliary: Vec<ObservationHit>,
+ limit: usize,
+) -> Vec<ObservationHit> {
+ if auxiliary.is_empty() {
+ return primary;
+ }
+ let mut fused: std::collections::HashMap<ObservationId, (ObservationHit, f64)> =
+ std::collections::HashMap::new();
+ for stream in [primary, auxiliary] {
+ for (rank, hit) in stream.into_iter().enumerate() {
+ let contribution = 1.0 / (60.0 + (rank + 1) as f64);
+ fused
+ .entry(hit.id)
+ .and_modify(|entry| entry.1 += contribution)
+ .or_insert((hit, contribution));
+ }
+ }
+ let mut out: Vec<_> = fused
+ .into_values()
+ .map(|(mut hit, score)| {
+ hit.rank = -score;
+ hit
+ })
+ .collect();
+ out.sort_by(|a, b| {
+ a.rank
+ .partial_cmp(&b.rank)
+ .unwrap_or(std::cmp::Ordering::Equal)
+ .then_with(|| a.created_at.cmp(&b.created_at))
+ });
+ out.truncate(limit);
+ out
+}
+
/// Rows keyed to one session (see [`ReaderPool::session_dependent_rows`]).
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct SessionDependentRows {
@@ -2734,6 +2845,7 @@ impl ReaderPool {
viewer: Option<UserId>,
) -> StoreResult<Vec<PageHit>> {
let fts_query = normalize_fts_query(&query, &self.fts_stopwords);
+ let cjk_query = prepare_cjk_fts5_query(&query);
if fts_query.is_empty() || limit == 0 {
return Ok(Vec::new());
}
@@ -2760,7 +2872,7 @@ impl ReaderPool {
Value::Integer(authority_candidate_limit(limit) as i64),
Value::Integer(now_us()),
];
- bound.extend(visible_params);
+ bound.extend(visible_params.iter().cloned());
let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {
let id_bytes: Vec<u8> = row.get(0)?;
let path: String = row.get(1)?;
@@ -2803,7 +2915,66 @@ impl ReaderPool {
authority,
));
}
- Ok(rerank_page_hits(candidates, limit))
+ let mut cjk_candidates = Vec::new();
+ if let Some(cjk_query) = cjk_query {
+ let descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json");
+ let sql = format!(
+ "SELECT pages.id, pages.path, pages.title, {descriptor}, \
+ pages_cjk_fts.rank, pages.tier, pages.pinned, \
+ pages.frontmatter_json, {kind_expr} AS kind \
+ FROM pages_cjk_fts \
+ JOIN pages ON pages.rowid = pages_cjk_fts.rowid \
+ WHERE pages_cjk_fts MATCH ?1 AND pages.is_latest = 1{not_expired}{visible} \
+ ORDER BY pages_cjk_fts.rank LIMIT ?2",
+ not_expired = not_expired("pages", "?3"),
+ );
+ let mut stmt = conn.prepare(&sql)?;
+ #[allow(clippy::cast_possible_wrap)]
+ let mut bound = vec![
+ Value::Text(cjk_query),
+ Value::Integer(authority_candidate_limit(limit) as i64),
+ Value::Integer(now_us()),
+ ];
+ bound.extend(visible_params);
+ let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {
+ Ok((
+ row.get::<_, Vec<u8>>(0)?,
+ row.get::<_, String>(1)?,
+ row.get::<_, String>(2)?,
+ row.get::<_, String>(3)?,
+ row.get::<_, f64>(4)?,
+ row.get::<_, String>(5)?,
+ row.get::<_, i64>(6)? != 0,
+ row.get::<_, String>(7)?,
+ row.get::<_, String>(8)?,
+ ))
+ })?;
+ for row in rows {
+ let (id, path, title, descriptor, rank, tier, pinned, frontmatter, kind) = row?;
+ let authority =
+ PageAuthority::from_stored(&path, &kind, &tier, pinned, &frontmatter);
+ cjk_candidates.push((
+ PageHit {
+ id: PageId::from_slice(&id)?,
+ path: PagePath::new(path)?,
+ snippet: page_descriptor(&descriptor, &title),
+ title,
+ rank,
+ superseded: false,
+ pinned: false,
+ },
+ authority,
+ ));
+ }
+ }
+ Ok(rerank_page_hits(
+ fuse_page_candidate_streams(
+ candidates,
+ cjk_candidates,
+ authority_candidate_limit(limit),
+ ),
+ limit,
+ ))
})
.await
}
@@ -2829,6 +3000,7 @@ impl ReaderPool {
filter: PageFilter<'_>,
) -> StoreResult<Vec<PageHitWithMeta>> {
let fts_query = normalize_fts_query(&query, &self.fts_stopwords);
+ let cjk_query = prepare_cjk_fts5_query(&query);
if fts_query.is_empty() || limit == 0 {
return Ok(Vec::new());
}
@@ -2859,8 +3031,8 @@ impl ReaderPool {
Value::Integer(authority_candidate_limit(limit) as i64),
Value::Integer(cutoff),
];
- bound.extend(visible_params);
- bound.extend(kept_params);
+ bound.extend(visible_params.iter().cloned());
+ bound.extend(kept_params.iter().cloned());
let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {
let workspace_name: String = row.get(0)?;
let project_name: String = row.get(1)?;
@@ -2918,7 +3090,71 @@ impl ReaderPool {
authority,
));
}
- Ok(rerank_page_hits_with_meta(candidates, limit))
+ let mut cjk_candidates = Vec::new();
+ if let Some(cjk_query) = cjk_query {
+ let descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json");
+ let sql = format!(
+ "SELECT workspaces.name, projects.name, pages.path, pages.title, \
+ {descriptor}, pages_cjk_fts.rank, pages.tier, pages.pinned, \
+ pages.frontmatter_json, {kind_expr} AS kind, pages.id \
+ FROM pages_cjk_fts \
+ JOIN pages ON pages.rowid = pages_cjk_fts.rowid \
+ JOIN projects ON projects.id = pages.project_id \
+ JOIN workspaces ON workspaces.id = pages.workspace_id \
+ WHERE pages_cjk_fts MATCH ?1 AND pages.is_latest = 1{not_expired}{visible}{kept} \
+ ORDER BY pages_cjk_fts.rank LIMIT ?2",
+ not_expired = not_expired("pages", "?3"),
+ );
+ let mut stmt = conn.prepare(&sql)?;
+ #[allow(clippy::cast_possible_wrap)]
+ let mut bound = vec![
+ Value::Text(cjk_query),
+ Value::Integer(authority_candidate_limit(limit) as i64),
+ Value::Integer(cutoff),
+ ];
+ bound.extend(visible_params);
+ bound.extend(kept_params);
+ let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {
+ Ok((
+ row.get::<_, String>(0)?,
+ row.get::<_, String>(1)?,
+ row.get::<_, String>(2)?,
+ row.get::<_, String>(3)?,
+ row.get::<_, String>(4)?,
+ row.get::<_, f64>(5)?,
+ row.get::<_, String>(6)?,
+ row.get::<_, i64>(7)? != 0,
+ row.get::<_, String>(8)?,
+ row.get::<_, String>(9)?,
+ row.get::<_, Vec<u8>>(10)?,
+ ))
+ })?;
+ for row in rows {
+ let (workspace_name, project_name, path, title, descriptor, rank, tier, pinned, frontmatter, kind, id) = row?;
+ let authority =
+ PageAuthority::from_stored(&path, &kind, &tier, pinned, &frontmatter);
+ cjk_candidates.push((
+ PageHitWithMeta {
+ id: PageId::from_slice(&id)?,
+ workspace_name,
+ project_name,
+ path: PagePath::new(path)?,
+ snippet: page_descriptor(&descriptor, &title),
+ title,
+ rank,
+ },
+ authority,
+ ));
+ }
+ }
+ Ok(rerank_page_hits_with_meta(
+ fuse_page_meta_candidate_streams(
+ candidates,
+ cjk_candidates,
+ authority_candidate_limit(limit),
+ ),
+ limit,
+ ))
})
.await
}
@@ -2969,6 +3205,7 @@ impl ReaderPool {
filter: PageFilter<'_>,
) -> StoreResult<Vec<(PageHit, PageAuthority)>> {
let fts_query = normalize_fts_query(&query, &self.fts_stopwords);
+ let cjk_query = prepare_cjk_fts5_query(&query);
if fts_query.is_empty() || candidate_limit == 0 {
return Ok(Vec::new());
}
@@ -3000,7 +3237,7 @@ impl ReaderPool {
Value::Integer(candidate_limit as i64),
Value::Integer(cutoff),
];
- bound.extend(kept_params);
+ bound.extend(kept_params.iter().cloned());
let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {
let id_bytes: Vec<u8> = row.get(0)?;
let path: String = row.get(1)?;
@@ -3043,7 +3280,68 @@ impl ReaderPool {
authority,
));
}
- Ok(candidates)
+ let mut cjk_candidates = Vec::new();
+ if let Some(cjk_query) = cjk_query {
+ let descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json");
+ let sql = format!(
+ "SELECT pages.id, pages.path, pages.title, {descriptor}, \
+ pages_cjk_fts.rank, pages.tier, pages.pinned, \
+ pages.frontmatter_json, {kind_expr} AS kind \
+ FROM pages_cjk_fts \
+ JOIN pages ON pages.rowid = pages_cjk_fts.rowid \
+ WHERE pages_cjk_fts MATCH ?1 \
+ AND pages.workspace_id = ?2 AND pages.project_id = ?3 \
+ {latest}{not_expired}{kept} \
+ ORDER BY pages_cjk_fts.rank LIMIT ?4",
+ latest = latest_only("pages", include_superseded),
+ not_expired = not_expired("pages", "?5"),
+ );
+ let mut stmt = conn.prepare(&sql)?;
+ #[allow(clippy::cast_possible_wrap)]
+ let mut bound = vec![
+ Value::Text(cjk_query),
+ Value::Blob(workspace_id.as_bytes().to_vec()),
+ Value::Blob(project_id.as_bytes().to_vec()),
+ Value::Integer(candidate_limit as i64),
+ Value::Integer(cutoff),
+ ];
+ bound.extend(kept_params);
+ let rows = stmt.query_map(params_from_iter(bound.iter()), |row| {
+ Ok((
+ row.get::<_, Vec<u8>>(0)?,
+ row.get::<_, String>(1)?,
+ row.get::<_, String>(2)?,
+ row.get::<_, String>(3)?,
+ row.get::<_, f64>(4)?,
+ row.get::<_, String>(5)?,
+ row.get::<_, i64>(6)? != 0,
+ row.get::<_, String>(7)?,
+ row.get::<_, String>(8)?,
+ ))
+ })?;
+ for row in rows {
+ let (id, path, title, descriptor, rank, tier, pinned, frontmatter, kind) = row?;
+ let authority =
+ PageAuthority::from_stored(&path, &kind, &tier, pinned, &frontmatter);
+ cjk_candidates.push((
+ PageHit {
+ id: PageId::from_slice(&id)?,
+ path: PagePath::new(path)?,
+ snippet: page_descriptor(&descriptor, &title),
+ title,
+ rank,
+ superseded: false,
+ pinned: false,
+ },
+ authority,
+ ));
+ }
+ }
+ Ok(fuse_page_candidate_streams(
+ candidates,
+ cjk_candidates,
+ candidate_limit,
+ ))
})
.await
}
@@ -3066,6 +3364,7 @@ impl ReaderPool {
filter: PageFilter<'_>,
) -> StoreResult<Vec<(PageHit, PageAuthority)>> {
let fts_query = normalize_fts_query(&query, &self.fts_stopwords);
+ let cjk_query = prepare_cjk_fts5_query(&query);
if fts_query.is_empty() || candidate_limit == 0 {
return Ok(Vec::new());
}
@@ -3098,7 +3397,7 @@ impl ReaderPool {
Value::Integer(as_of_us),
Value::Integer(as_of_us),
];
- sql_params.extend(kept_params);
+ sql_params.extend(kept_params.iter().cloned());
#[allow(clippy::cast_possible_wrap)]
sql_params.push(Value::Integer(candidate_limit as i64));
let rows = stmt.query_map(params_from_iter(sql_params), |row| {
@@ -3143,7 +3442,70 @@ impl ReaderPool {
authority,
));
}
- Ok(candidates)
+ let mut cjk_candidates = Vec::new();
+ if let Some(cjk_query) = cjk_query {
+ let descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json");
+ let sql = format!(
+ "SELECT pages.id, pages.path, pages.title, {descriptor}, \
+ pages_cjk_fts.rank, pages.tier, pages.pinned, \
+ pages.frontmatter_json, {kind_expr} AS kind \
+ FROM pages_cjk_fts \
+ JOIN pages ON pages.rowid = pages_cjk_fts.rowid \
+ WHERE pages_cjk_fts MATCH ?1 \
+ AND pages.workspace_id = ?2 AND pages.project_id = ?3 \
+ AND pages.valid_from <= ?4 \
+ AND (pages.valid_to IS NULL OR pages.valid_to > ?5){not_expired}{kept} \
+ ORDER BY pages_cjk_fts.rank LIMIT ?{limit_placeholder}",
+ not_expired = not_expired("pages", "?6"),
+ );
+ let mut sql_params = vec![
+ Value::Text(cjk_query),
+ Value::Blob(workspace_id.as_bytes().to_vec()),
+ Value::Blob(project_id.as_bytes().to_vec()),
+ Value::Integer(as_of_us),
+ Value::Integer(as_of_us),
+ Value::Integer(as_of_us),
+ ];
+ sql_params.extend(kept_params);
+ #[allow(clippy::cast_possible_wrap)]
+ sql_params.push(Value::Integer(candidate_limit as i64));
+ let mut stmt = conn.prepare(&sql)?;
+ let rows = stmt.query_map(params_from_iter(sql_params), |row| {
+ Ok((
+ row.get::<_, Vec<u8>>(0)?,
+ row.get::<_, String>(1)?,
+ row.get::<_, String>(2)?,
+ row.get::<_, String>(3)?,
+ row.get::<_, f64>(4)?,
+ row.get::<_, String>(5)?,
+ row.get::<_, i64>(6)? != 0,
+ row.get::<_, String>(7)?,
+ row.get::<_, String>(8)?,
+ ))
+ })?;
+ for row in rows {
+ let (id, path, title, descriptor, rank, tier, pinned, frontmatter, kind) = row?;
+ let authority =
+ PageAuthority::from_stored(&path, &kind, &tier, pinned, &frontmatter);
+ cjk_candidates.push((
+ PageHit {
+ id: PageId::from_slice(&id)?,
+ path: PagePath::new(path)?,
+ snippet: page_descriptor(&descriptor, &title),
+ title,
+ rank,
+ superseded: false,
+ pinned: false,
+ },
+ authority,
+ ));
+ }
+ }
+ Ok(fuse_page_candidate_streams(
+ candidates,
+ cjk_candidates,
+ candidate_limit,
+ ))
})
.await
}
@@ -3367,6 +3729,7 @@ impl ReaderPool {
owner_filter: OwnerFilter,
) -> StoreResult<Vec<ObservationHit>> {
let fts_query = normalize_fts_query(&query, &self.fts_stopwords);
+ let cjk_query = prepare_cjk_fts5_query(&query);
if fts_query.is_empty() {
return Ok(Vec::new());
}
@@ -3446,7 +3809,55 @@ impl ReaderPool {
created_at,
});
}
- Ok(hits)
+ let mut cjk_hits = Vec::new();
+ if let Some(cjk_query) = cjk_query {
+ let mut stmt = conn.prepare(&format!(
+ "SELECT observations.id, observations.session_id, observations.kind, \
+ observations.title, substr(observations.body, 1, 240), \
+ observations_cjk_fts.rank, observations.created_at \
+ FROM observations_cjk_fts \
+ JOIN observations ON observations.rowid = observations_cjk_fts.rowid \
+ WHERE observations_cjk_fts MATCH ?1 \
+ AND observations.workspace_id = ?2 \
+ AND observations.project_id = ?3{owner_clause} \
+ ORDER BY observations_cjk_fts.rank LIMIT ?4",
+ ))?;
+ let mut sql_params: Vec<&dyn rusqlite::ToSql> = Vec::with_capacity(5);
+ sql_params.push(&cjk_query);
+ sql_params.push(&ws_bytes);
+ sql_params.push(&proj_bytes);
+ sql_params.push(&limit);
+ if let Some(actor) = &actor {
+ sql_params.push(actor);
+ }
+ let rows = stmt.query_map(sql_params.as_slice(), |row| {
+ Ok((
+ row.get::<_, Vec<u8>>(0)?,
+ row.get::<_, Vec<u8>>(1)?,
+ row.get::<_, String>(2)?,
+ row.get::<_, String>(3)?,
+ row.get::<_, String>(4)?,
+ row.get::<_, f64>(5)?,
+ row.get::<_, i64>(6)?,
+ ))
+ })?;
+ for row in rows {
+ let (id, session, kind, title, snippet, rank, created_us) = row?;
+ let created_at = jiff::Timestamp::from_microsecond(created_us)
+ .map(|ts| ts.to_string())
+ .unwrap_or_default();
+ cjk_hits.push(ObservationHit {
+ id: ObservationId::from_slice(&id)?,
+ session_id: SessionId::from_slice(&session)?,
+ kind,
+ title,
+ snippet,
+ rank,
+ created_at,
+ });
+ }
+ }
+ Ok(fuse_observation_streams(hits, cjk_hits, limit as usize))
})
.await
}
@@ -3474,6 +3885,7 @@ impl ReaderPool {
filter: PageFilter<'_>,
) -> StoreResult<Vec<ObservationVote>> {
let fts_query = normalize_fts_query(query, &self.fts_stopwords);
+ let cjk_query = prepare_cjk_fts5_query(query);
if fts_query.is_empty() || candidate_limit == 0 {
return Ok(Vec::new());
}
@@ -3507,16 +3919,18 @@ impl ReaderPool {
// Best-ranked observation first, grouped by session in first-seen
// order: the session's stream rank is its best observation's.
let mut sessions: Vec<(SessionId, String, Vec<ObservationId>)> = Vec::new();
+ let mut index: std::collections::HashMap<SessionId, usize> =
+ std::collections::HashMap::new();
+ let candidate_path =
+ "'sessions/' || substr(lower(hex(observations.session_id)), 1, 8) || '-' || \
+ substr(lower(hex(observations.session_id)), 9, 4) || '-' || \
+ substr(lower(hex(observations.session_id)), 13, 4) || '-' || \
+ substr(lower(hex(observations.session_id)), 17, 4) || '-' || \
+ substr(lower(hex(observations.session_id)), 21, 12) || '.md'";
{
// CROSS JOIN pins the FTS table as the outer loop: left to
// itself the planner drives from the observations scope index
// and re-enters FTS once per row, which is far slower.
- let candidate_path =
- "'sessions/' || substr(lower(hex(observations.session_id)), 1, 8) || '-' || \
- substr(lower(hex(observations.session_id)), 9, 4) || '-' || \
- substr(lower(hex(observations.session_id)), 13, 4) || '-' || \
- substr(lower(hex(observations.session_id)), 17, 4) || '-' || \
- substr(lower(hex(observations.session_id)), 21, 12) || '.md'";
let mut stmt = tx.prepare(&format!(
"SELECT observations.id, observations.session_id, \
snippet(observations_fts, 1, '<mark>', '</mark>', '…', \
@@ -3550,7 +3964,7 @@ impl ReaderPool {
Value::Integer(last_rowid),
Value::Integer(cutoff),
];
- candidate_params.extend(candidate_kept_params);
+ candidate_params.extend(candidate_kept_params.iter().cloned());
let rows = stmt.query_map(params_from_iter(candidate_params.iter()), |row| {
Ok((
row.get::<_, Vec<u8>>(0)?,
@@ -3558,8 +3972,6 @@ impl ReaderPool {
row.get::<_, String>(2)?,
))
})?;
- let mut index: std::collections::HashMap<SessionId, usize> =
- std::collections::HashMap::new();
for row in rows {
let (id_bytes, session_bytes, snippet) = row?;
let id = ObservationId::from_slice(&id_bytes)?;
@@ -3578,6 +3990,64 @@ impl ReaderPool {
}
}
}
+ if let Some(cjk_query) = cjk_query {
+ let mut stmt = tx.prepare(&format!(
+ "SELECT observations.id, observations.session_id, \
+ substr(observations.body, 1, 240) \
+ FROM observations_cjk_fts \
+ CROSS JOIN observations ON observations.rowid = observations_cjk_fts.rowid \
+ WHERE observations_cjk_fts MATCH ?1 \
+ AND observations_cjk_fts.rowid >= ?5 \
+ AND observations_cjk_fts.rowid <= ?6 \
+ AND observations.workspace_id = ?2 \
+ AND observations.project_id = ?3 \
+ AND EXISTS ( \
+ SELECT 1 FROM pages AS candidate_page \
+ WHERE candidate_page.workspace_id = observations.workspace_id \
+ AND candidate_page.project_id = observations.project_id \
+ AND candidate_page.is_latest = 1 \
+ AND candidate_page.path = {candidate_path} \
+ {candidate_not_expired}{candidate_kept} \
+ ) \
+ ORDER BY observations_cjk_fts.rank LIMIT ?4",
+ candidate_not_expired = not_expired("candidate_page", "?7"),
+ ))?;
+ #[allow(clippy::cast_possible_wrap)]
+ let mut candidate_params = vec![
+ Value::Text(cjk_query),
+ Value::Blob(workspace_id.as_bytes().to_vec()),
+ Value::Blob(project_id.as_bytes().to_vec()),
+ Value::Integer(row_limit as i64),
+ Value::Integer(first_rowid),
+ Value::Integer(last_rowid),
+ Value::Integer(cutoff),
+ ];
+ candidate_params.extend(candidate_kept_params);
+ let rows = stmt.query_map(params_from_iter(candidate_params.iter()), |row| {
+ Ok((
+ row.get::<_, Vec<u8>>(0)?,
+ row.get::<_, Vec<u8>>(1)?,
+ row.get::<_, String>(2)?,
+ ))
+ })?;
+ for row in rows {
+ let (id_bytes, session_bytes, snippet) = row?;
+ let id = ObservationId::from_slice(&id_bytes)?;
+ let session_id = SessionId::from_slice(&session_bytes)?;
+ match index.get(&session_id) {
+ Some(&at) => {
+ let ids = &mut sessions[at].2;
+ if ids.len() < OBSERVATION_STREAM_IDS_PER_PAGE && !ids.contains(&id) {
+ ids.push(id);
+ }
+ }
+ None => {
+ index.insert(session_id, sessions.len());
+ sessions.push((session_id, snippet, vec![id]));
+ }
+ }
+ }
+ }
if sessions.is_empty() {
return Ok(Vec::new());
}
@@ -4145,6 +4615,7 @@ impl ReaderPool {
session_id: SessionId,
page: ObservationPage,
) -> StoreResult<ObservationPageResult> {
+ let cjk_query = page.query.as_deref().and_then(prepare_cjk_fts5_query);
let fts_query = page
.query
.as_deref()
@@ -4170,16 +4641,29 @@ impl ReaderPool {
sql_params.push(Value::Blob(project_id.as_bytes().to_vec()));
let mut source = String::from("FROM observations");
let mut predicate = String::from(
- " WHERE observations.session_id = ? \
- AND observations.workspace_id = ? \
- AND observations.project_id = ?",
+ " WHERE observations.session_id = ?1 \
+ AND observations.workspace_id = ?2 \
+ AND observations.project_id = ?3",
);
if let Some(q) = fts_query {
- source.push_str(
- " JOIN observations_fts ON observations_fts.rowid = observations.rowid",
- );
- predicate.push_str(" AND observations_fts MATCH ?");
- sql_params.push(Value::Text(q));
+ if let Some(cjk_query) = cjk_query {
+ source.push_str(
+ " JOIN ( \
+ SELECT rowid FROM observations_fts WHERE observations_fts MATCH ?4 \
+ UNION \
+ SELECT rowid FROM observations_cjk_fts \
+ WHERE observations_cjk_fts MATCH ?5 \
+ ) AS matched_fts ON matched_fts.rowid = observations.rowid",
+ );
+ sql_params.push(Value::Text(q));
+ sql_params.push(Value::Text(cjk_query));
+ } else {
+ source.push_str(
+ " JOIN observations_fts ON observations_fts.rowid = observations.rowid",
+ );
+ predicate.push_str(" AND observations_fts MATCH ?");
+ sql_params.push(Value::Text(q));
+ }
}
if !kinds.is_empty() {
predicate.push_str(" AND observations.kind IN (");
diff --git a/crates/ai-memory-store/tests/suite/cjk_search.rs b/crates/ai-memory-store/tests/suite/cjk_search.rs
new file mode 100644
index 00000000..90d2db7e
--- /dev/null
+++ b/crates/ai-memory-store/tests/suite/cjk_search.rs
@@ -0,0 +1,238 @@
+//! Issue #1433: unicode61 treats an unspaced CJK run as one token. The
+//! auxiliary index must recover substring queries without changing Latin-only
+//! retrieval, scope filters, supersession, or upgraded-store behavior.
+
+use ai_memory_core::{NewPage, PagePath, Tier};
+use ai_memory_store::Store;
+
+use super::retrieval_observation_stream::{observe, scope, search, session, session_page};
+
+fn page(
+ ws: ai_memory_core::WorkspaceId,
+ project: ai_memory_core::ProjectId,
+ path: &str,
+ title: &str,
+ body: &str,
+) -> NewPage {
+ NewPage {
+ workspace_id: ws,
+ project_id: project,
+ path: PagePath::new(path).unwrap(),
+ title: title.into(),
+ body: body.into(),
+ tier: Tier::Semantic,
+ frontmatter_json: serde_json::json!({}),
+ pinned: false,
+ links: Vec::new(),
+ author_id: None,
+ expires_at: None,
+ entities: Vec::new(),
+ evidence: Vec::new(),
+ }
+}
+
+#[tokio::test]
+async fn page_search_finds_unspaced_cjk_substrings_and_keeps_latin_hits() {
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let (ws, project) = scope(&store, "default", "app").await;
+
+ store
+ .writer
+ .upsert_page(page(
+ ws,
+ project,
+ "payments.md",
+ "支付说明",
+ "启用微信支付服务后检查回调。",
+ ))
+ .await
+ .unwrap();
+ store
+ .writer
+ .upsert_page(page(
+ ws,
+ project,
+ "japanese.md",
+ "運用手順",
+ "本番環境で自動更新機能を確認する。",
+ ))
+ .await
+ .unwrap();
+ store
+ .writer
+ .upsert_page(page(
+ ws,
+ project,
+ "deploy.md",
+ "Deploy guide",
+ "deploy the service after tests pass",
+ ))
+ .await
+ .unwrap();
+
+ let chinese = store
+ .reader
+ .search_pages_for_project(ws, project, "微信支付".into(), 10, None)
+ .await
+ .unwrap();
+ assert_eq!(chinese.len(), 1);
+ assert_eq!(chinese[0].path.as_str(), "payments.md");
+
+ let japanese = store
+ .reader
+ .search_pages_for_project(ws, project, "自動更新".into(), 10, None)
+ .await
+ .unwrap();
+ assert_eq!(japanese.len(), 1);
+ assert_eq!(japanese[0].path.as_str(), "japanese.md");
+
+ let global = store
+ .reader
+ .search_pages_with_meta(
+ "微信支付".into(),
+ 10,
+ None,
+ None,
+ ai_memory_store::PageFilter::default(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(global.len(), 1);
+ assert_eq!(global[0].path.as_str(), "payments.md");
+
+ let mixed = store
+ .reader
+ .search_pages_for_project(ws, project, "deploy 支付".into(), 10, None)
+ .await
+ .unwrap();
+ let paths: Vec<_> = mixed.iter().map(|hit| hit.path.as_str()).collect();
+ assert!(
+ paths.contains(&"deploy.md"),
+ "Latin stream disappeared: {paths:?}"
+ );
+ assert!(
+ paths.contains(&"payments.md"),
+ "CJK stream disappeared: {paths:?}"
+ );
+
+ let latin = store
+ .reader
+ .search_pages_for_project(ws, project, "deploy service".into(), 10, None)
+ .await
+ .unwrap();
+ assert_eq!(latin[0].path.as_str(), "deploy.md");
+}
+
+#[tokio::test]
+async fn observation_search_and_session_votes_find_cjk_substrings() {
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let (ws, project) = scope(&store, "default", "app").await;
+ let session_id = session(&store, ws, project).await;
+ observe(
+ &store,
+ ws,
+ project,
+ session_id,
+ "启用微信支付服务后检查回调",
+ )
+ .await;
+ session_page(
+ &store,
+ ws,
+ project,
+ session_id,
+ "English-only session summary",
+ )
+ .await;
+
+ let observations = store
+ .reader
+ .search_observations_for_project(ws, project, "微信支付".into(), 10)
+ .await
+ .unwrap();
+ assert_eq!(observations.len(), 1);
+ assert_eq!(observations[0].session_id, session_id);
+
+ let timeline = store
+ .reader
+ .session_observations_scoped(
+ ws,
+ project,
+ session_id,
+ ai_memory_store::ObservationPage {
+ limit: 10,
+ offset: 0,
+ order: ai_memory_store::ObservationOrder::Asc,
+ kinds: None,
+ query: Some("微信支付".into()),
+ },
+ )
+ .await
+ .unwrap();
+ assert_eq!(timeline.total, 1);
+
+ let pages = search(&store, ws, project, "微信支付").await;
+ assert!(
+ pages
+ .iter()
+ .any(|(hit, _)| hit.path.as_str() == format!("sessions/{session_id}.md")),
+ "the CJK observation stream did not vote for its session page: {pages:?}"
+ );
+}
+
+#[tokio::test]
+async fn superseded_cjk_text_is_hidden_and_startup_repairs_missing_index_rows() {
+ let tmp = tempfile::tempdir().unwrap();
+ let mut store = Store::open(tmp.path()).unwrap();
+ let (ws, project) = scope(&store, "default", "app").await;
+ store
+ .writer
+ .upsert_page(page(ws, project, "state.md", "状态", "旧版支付服务"))
+ .await
+ .unwrap();
+ store
+ .writer
+ .upsert_page(page(ws, project, "state.md", "State", "new English state"))
+ .await
+ .unwrap();
+ assert!(
+ store
+ .reader
+ .search_pages_for_project(ws, project, "旧版支付".into(), 10, None)
+ .await
+ .unwrap()
+ .is_empty()
+ );
+
+ store
+ .writer
+ .upsert_page(page(ws, project, "repair.md", "恢复", "需要恢复索引"))
+ .await
+ .unwrap();
+ store.close().await;
+ drop(store);
+
+ let db = rusqlite::Connection::open(tmp.path().join("db/memory.sqlite")).unwrap();
+ db.execute("DELETE FROM pages_cjk_fts", []).unwrap();
+ drop(db);
+
+ store = Store::open(tmp.path()).unwrap();
+ let repaired = store
+ .reader
+ .search_pages_for_project(ws, project, "恢复索引".into(), 10, None)
+ .await
+ .unwrap();
+ assert_eq!(repaired.len(), 1);
+ assert_eq!(repaired[0].path.as_str(), "repair.md");
+
+ store.writer.compact().await.unwrap();
+ let after_rebuild = store
+ .reader
+ .search_pages_for_project(ws, project, "恢复索引".into(), 10, None)
+ .await
+ .unwrap();
+ assert_eq!(after_rebuild.len(), 1);
+ assert_eq!(after_rebuild[0].path.as_str(), "repair.md");
+}
diff --git a/crates/ai-memory-store/tests/suite/mod.rs b/crates/ai-memory-store/tests/suite/mod.rs
index 129cca28..b922a245 100644
--- a/crates/ai-memory-store/tests/suite/mod.rs
+++ b/crates/ai-memory-store/tests/suite/mod.rs
@@ -9,6 +9,7 @@ mod audit_contamination;
mod audit_log;
mod auto_improve_staging;
mod belief_authority;
+mod cjk_search;
mod client_activity;
mod fts_drift_status;
mod fts_stopwords_config;
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 5610ea8f..fd88ea5f 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -166,7 +166,10 @@ from hook paths.
trust boundary and delimiters precede automatically injected handoffs,
project briefs, and managed-workstream packets; current instructions and
checkout state remain authoritative.
-6. `memory_query` answers via FTS5 + entity-match + link-neighbour RRF; when an
+6. `memory_query` answers via FTS5 + entity-match + link-neighbour RRF; a
+ transactionally maintained auxiliary unigram/bigram FTS stream handles
+ unspaced Han, Kana, and Hangul queries without changing the ordinary
+ unicode61 path for non-CJK text. When an
embedder is configured, vector cosine over `page_embeddings` joins the same
RRF. The entity index is derived from the canonical frontmatter `entities`
list, and an empty index contributes no candidates or score. Before final
@@ -541,9 +544,11 @@ prunable rather than only after it ages in place.
| `workspaces`, `projects` | Top of the 3-tuple identity coordinate. `projects.identity` / `identity_source` (V70, #708) hold the repository identity a project routes by (an explicit marker `identity` or a normalised git remote, resolved client-side), unique per workspace when set; empty until a capture claims the project. Phase-5 clients explicitly send `IdentityStyle::Path` (#1033) for remote-backed projects, naming them from the full repository path without the host (`acme-api`), and send explicit `host_path` for the pre-v3 opt-out. Server-side omission remains `HostPath` because old clients omitted that historical default; no new wire field is needed. Creation or authorized capture promotion uses that canonical name only when no different hostful identity claims it; otherwise resolution falls back to the existing hostful/legacy-safe name. V76 stores indexed canonical and v2-basename keys derived from that full identity: reads are no-create/no-rename, while an authorized explicit write through the canonical name promotes only `projects.name` in the same resolve-or-create writer transaction and preserves the UUID. Project `_meta.md` carries the optional full identity/source plus derived keys; clean reindex validates them and rebuilds the same UUID/name/coordinate lookup, while older manifests without identity remain valid. Ambiguous cross-forge or multi-candidate keys fail closed. A local marker may additionally send up to 16 validated former names as one transient `aliases` JSON-array query value: lookup stays inside its selected workspace, requires the checkout's full hostful git-remote identity to equal the stored row, returns reads without rename, and lets capture perform only the same authorized UUID-preserving canonical promotion. Aliases are not stored and remain distinct in the hook routing cache. The exact operator-home marker may additionally define bounded `[routes.identity."host/path"]` and `[routes.path."absolute-or-tilde-path"]` blocks: an exact hostful remote match wins the longest component-wise lexical path match, any reachable non-home local settings marker wins both, malformed or ambiguous route maps fail closed, and selected fields reuse the existing marker wire without a server/schema change. `AI_MEMORY_HOME` is authoritative for native callers; portable clients use `HOME`/`USERPROFILE`, and Docker compares its forwarded host cwd. See `docs/marker-file.md#repository-identity`. `projects.access_mode` (V68; `open` default / `restricted`) decides whether any authenticated user or only root, the creator (`projects.created_by`, V69) and grant holders (`project_grants`, V68) reach it, decided by `ai_memory_store::authorize_project` (see `docs/users.md#per-project-access`). `projects.profile_contribute` / `profile_consume` (V74) record the marker's `[profile]` opt-outs, forwarded at session start; `profile_candidates`, `profile_harvest_marks` and `profile_entry_ledger` (V75) hold the profile pass's evidence, per-project read position and the entries it wrote. Reserved projects: `scratch`, `_global` (shared preferences; holds the single-user profile under `profile/`), `_profile` (a workspace profile) and `_profile.<user id>` (a private profile, always `restricted`); capture is never attributed to any of them (see `docs/cross-project-profile.md`). |
| `pages` | Versioned wiki pages with `is_latest` + `supersedes` chain. M8 columns: `last_accessed_at`, `access_count`, and decay-only tombstone marker `superseded_at`. M9 cols: `embedding_provider`, `embedding_model`, `embedding_dim`. V36: `expires_at` (frontmatter TTL). V37: `salience` (NULL = `salience_default`; derived from `page_feedback`). |
| `pages_fts` | FTS5 virtual table over `(title, body)`, auto-synced by triggers. |
+| `pages_cjk_fts` | Derived unigram/bigram FTS5 index over the CJK runs in page title, body, and path. The writer updates it in the page transaction; startup repairs missing rows after an upgrade. Queried only when the bounded query contains Han, Kana, or Hangul. |
| `sessions`, `observations` | Sanitized, bounded lifecycle-hook projections. `sessions.ended_observation_count` is the stable generation watermark for resumed-session re-end eligibility; wall clocks are not used for that decision. They are an operational audit trail, not a complete native transcript. |
| `session_consolidation_jobs` | Durable, observation-generation-idempotent queue for automatic LLM consolidation. PreCompact / PostCompaction checkpoint jobs are queued whether or not the session has ended (a resumed session keeps `ended_at`); opt-in SessionEnd jobs need an *ended* session, and observation generations order the two. One bounded server worker leases jobs, retries provider failures with backoff, and recovers expired leases after restart. The worker starts whenever a consolidator is configured; SessionEnd enqueue stays behind `AI_MEMORY_CONSOLIDATE_ON_SESSION_END`. A manual `memory_consolidate` writes its page out-of-band and then reconciles this table (flipping a `failed`/`pending`/`superseded` row for the session to `completed`, never touching a live `running` lease), so the operator does not see a `failed` job for a session that is in fact consolidated. |
| `observations_fts` | FTS5 virtual table over raw observation `(title, body)`. Feeds the bounded `raw_hits` fallback and, through `[retrieval] observation_stream` (on by default), the observation-evidence RRF stream that votes for session pages. |
+| `observations_cjk_fts` | Derived unigram/bigram companion for unspaced CJK observation text. It follows the same transactional write, scope, owner-visibility, raw fallback, timeline, and session-vote boundaries as `observations_fts`. |
| `workstreams`, `managed_runs`, `workstream_native_sessions` | Optional lease state plus per-harness native source and delivery cursors for `ai-memory run`. |
| `workstream_events`, `workstream_events_fts` | Append-only normalized visible transcript events and full-text search; immutable sanitized source batches also live under `raw/workstreams/`. |
| `links` | Wikilink / markdown cross-references. `to_page_id` (a global PageId) is nullable for unresolved forward links. `to_workspace` / `to_project` carry a cross-project scope (NULL = the source page's own project). |