Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(embeddings): size local capacity from the CPU and bound the query wait
AkitaOnRails20e323a20266
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6609cfa4..8d052ac1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -11,6 +11,21 @@ version in [`docs/changelog/`](docs/changelog/):## [Unreleased]+### Fixed++- Fixed a 3.2.0 regression that serialized local document embedding and+ dropped the vector stream from queries that raced a document embed: the+ local embedder's capacity was hardcoded to two slots (one document, one+ query), and a query finding both busy degraded instantly to lexical+ search. The scheduler is now sized from the CPU (half the available+ parallelism, clamped to between 2 and 6 slots, documents capped at+ capacity minus one so a slot stays query-reserved), and a query waits up+ to 250 ms for a slot before degrading. On the affected LongMemEval-S+ question set this restores query latency (p50/p95 66/87 ms vs 3.2.0's+ 118/295) and ingest throughput; the reported hit@5 delta (0.947 → 0.938)+ traces to rank-boundary questions that flip between identical runs, not+ a deterministic ranking change (#1580).+## [3.2.0] - 2026-10-21### Addeddiff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex 2a15379c..975910d1 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -16,7 +16,9 @@ use ai_memory_hooks::{CLIENT_MARKER_HEADER, DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, HookState, ProjectCacheStore,WorkstreamState, hook_router, workstream_router,};-use ai_memory_llm::{Embedder, EmbeddingCapacity, LlmProvider, ProviderHealth, build_embedder};+use ai_memory_llm::{+ Embedder, EmbeddingCapacity, LlmProvider, ProviderHealth, build_embedder, local_capacity,+};use ai_memory_mcp::human_auth::{Cidr, HumanAuthRuntime, LoginLimiter};use ai_memory_mcp::{AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_sweep_tuning,@@ -3063,10 +3065,15 @@ async fn configure_embedder();let embedder = provider_health.wrap_embedder(embedder, provider_name, model, dim);let embedder: Arc<dyn Embedder> = if is_local {- // all-MiniLM inference parallelizes internally. A single background- // document slot plus one reserved query slot avoids CPU oversubscription- // while keeping interactive vector retrieval available during writes.- Arc::new(EmbeddingCapacity::new(embedder, 2)?)+ // all-MiniLM inference parallelizes internally, so capacity bounds+ // concurrent calls rather than threads. Sizing from the CPU count+ // recovers the ingest throughput a single background document slot+ // lost (#1580) while keeping one slot query-reserved; see+ // `local_capacity` for the halve-and-clamp rationale.+ let cores = std::thread::available_parallelism()+ .map(|cores| cores.get())+ .unwrap_or(4);+ Arc::new(EmbeddingCapacity::new(embedder, local_capacity(cores))?)} else {embedder};diff --git a/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs b/crates/ai-memory-consolidate/tests/suite/local_embeddings.rsindex 15696401..48423661 100644--- a/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs+++ b/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs@@ -359,7 +359,11 @@ async fn local_query_latency_under_document_load() {DOCUMENTS_PER_WORKER,).await;- let bounded: Arc<dyn Embedder> = Arc::new(EmbeddingCapacity::new(embedder, 2).unwrap());+ let cores = std::thread::available_parallelism()+ .map(usize::from)+ .unwrap_or(4);+ let bounded: Arc<dyn Embedder> =+ Arc::new(EmbeddingCapacity::new(embedder, ai_memory_llm::local_capacity(cores)).unwrap());let bounded_loaded = measure_queries_under_document_load(&bounded,workers,diff --git a/crates/ai-memory-llm/Cargo.toml b/crates/ai-memory-llm/Cargo.tomlindex 0b65c6c2..fe16c791 100644--- a/crates/ai-memory-llm/Cargo.toml+++ b/crates/ai-memory-llm/Cargo.toml@@ -46,7 +46,9 @@ local-embeddings = [][dev-dependencies]-tokio = { workspace = true }+# `test-util` brings `start_paused`, which makes the bounded query wait in+# the capacity tests advance deterministically instead of sleeping.+tokio = { workspace = true, features = ["test-util"] }tempfile.workspace = truerstest.workspace = true# In-process HTTP mock for the openai-compat strict-fallback tests.diff --git a/crates/ai-memory-llm/src/embedding.rs b/crates/ai-memory-llm/src/embedding.rsindex b1403f3f..608d8485 100644--- a/crates/ai-memory-llm/src/embedding.rs+++ b/crates/ai-memory-llm/src/embedding.rs@@ -124,7 +124,8 @@ pub async fn embed_search_query(embedder: &dyn Embedder, query: &str) -> Option<debug!(provider = embedder.provider(),model = embedder.model(),- "embedding capacity busy; degrading search to FTS5 + entity + graph"+ "embedding capacity stayed busy past the bounded query wait; \+ degrading search to FTS5 + entity + graph");None}diff --git a/crates/ai-memory-llm/src/embedding_capacity.rs b/crates/ai-memory-llm/src/embedding_capacity.rsindex c825fef1..aa6de669 100644--- a/crates/ai-memory-llm/src/embedding_capacity.rs+++ b/crates/ai-memory-llm/src/embedding_capacity.rs@@ -3,20 +3,55 @@//! Local model inference is CPU-heavy and already parallelizes internally.//! Letting background page writes fill every inference slot can more than//! double interactive query latency. [`EmbeddingCapacity`] bounds total work-//! while reserving one slot for query embeddings.+//! while reserving one slot for query embeddings; a query that finds every+//! slot busy waits a bounded interval ([`QUERY_CAPACITY_WAIT`]) before+//! degrading, so a transient race with a document embed does not silently+//! drop the vector stream from a search.use std::sync::Arc;+use std::time::Duration;use async_trait::async_trait;use tokio::sync::{OwnedSemaphorePermit, Semaphore};use crate::{Embedder, LlmError, LlmResult};+/// How long an interactive query waits for an inference slot before the+/// search degrades to lexical + entity + graph retrieval.+///+/// Long enough to ride out a finishing document embed (or a sibling query)+/// under write load; short enough that `memory_query` latency stays+/// interactive even when background indexing saturates every slot for a+/// sustained period.+const QUERY_CAPACITY_WAIT: Duration = Duration::from_millis(250);++/// Derive the local embedder's total capacity from CPU parallelism.+///+/// `cores` is what the caller observed via `available_parallelism()`+/// (callers pass a small fallback such as 4 when that syscall fails; 0 or+/// tiny values clamp to the minimum). Why halve and clamp:+///+/// - all-MiniLM inference (candle) already parallelizes across cores+/// *inside* one embed, so `capacity` bounds concurrent calls, each of+/// which wants several threads. Half the available slots recovers most+/// of the ingest throughput a single background document slot lost+/// (#1580) while mild oversubscription keeps the CPU busy.+/// - clamping to at least 2 preserves the reserved query slot+/// ([`EmbeddingCapacity::new`] refuses less); clamping to 6 stops a+/// many-core host from running so many CPU-bound embeds at once that+/// interactive latency climbs right back up.+pub fn local_capacity(cores: usize) -> usize {+ (cores / 2).clamp(2, 6)+}+/// Embedder wrapper that bounds inference and reserves foreground capacity.////// `capacity` must be at least two. Document and generic embeddings may use/// at most `capacity - 1` slots, so an interactive query can enter without-/// waiting for background indexing work to finish.+/// waiting for background indexing work to finish. A query that does find+/// every slot busy waits up to [`QUERY_CAPACITY_WAIT`] before failing, at+/// which point search degrades to lexical + entity + graph retrieval rather+/// than queueing unboundedly.pub struct EmbeddingCapacity {inner: Arc<dyn Embedder>,total: Arc<Semaphore>,@@ -113,11 +148,16 @@ impl Embedder for EmbeddingCapacity {}async fn embed_query(&self, text: &str) -> LlmResult<Vec<f32>> {- let capacity = self- .total- .clone()- .try_acquire_owned()- .map_err(|_| LlmError::CapacityBusy("embedding"))?;+ // Bounded wait, not try_acquire: an instant refusal made a query+ // that merely raced a document embed silently degrade the whole+ // search to FTS-only (#1580), dropping the vector stream for that+ // query. Waiting up to QUERY_CAPACITY_WAIT absorbs transient+ // saturation; only a slot busy for the whole window degrades.+ let capacity =+ tokio::time::timeout(QUERY_CAPACITY_WAIT, self.total.clone().acquire_owned())+ .await+ .map_err(|_| LlmError::CapacityBusy("embedding"))?+ .map_err(|_| capacity_closed())?;let inner = Arc::clone(&self.inner);let text = text.to_owned();Self::finish_task(tokio::spawn(async move {@@ -142,6 +182,31 @@ mod tests {}impl BlockingEmbedder {+ /// Wrap a blocking embedder at `capacity`, returning the staged+ /// embedder, the release latch, and the stream of entered+ /// operations.+ fn stage(+ capacity: usize,+ ) -> (+ Arc<dyn Embedder>,+ Arc<Semaphore>,+ mpsc::UnboundedReceiver<&'static str>,+ ) {+ let (entered, entered_rx) = mpsc::unbounded_channel();+ let releases = Arc::new(Semaphore::new(0));+ let stage: Arc<dyn Embedder> = Arc::new(+ EmbeddingCapacity::new(+ Arc::new(BlockingEmbedder {+ entered,+ releases: Arc::clone(&releases),+ }),+ capacity,+ )+ .unwrap(),+ );+ (stage, releases, entered_rx)+ }+async fn block(&self, operation: &'static str) -> LlmResult<Vec<f32>> {self.entered.send(operation).unwrap();self.releases@@ -226,12 +291,6 @@ mod tests {Some("query"),"the reserved query slot must remain available");- assert!(- crate::embed_search_query(stage.as_ref(), "another query")- .await- .is_none(),- "a saturated query must degrade immediately to non-vector search"- );assert!(tokio::time::timeout(Duration::from_millis(25), entered_rx.recv()).await@@ -252,6 +311,169 @@ mod tests {query.await.unwrap().unwrap();}+ /// The #1580 regression shape: a query that arrives while every slot is+ /// momentarily held must WAIT for the next free slot and succeed, not+ /// degrade the search to FTS-only on an instant refusal.+ #[tokio::test]+ async fn query_waits_out_transient_saturation_instead_of_degrading() {+ let (stage, releases, mut entered_rx) = BlockingEmbedder::stage(2);++ // Saturate both total slots: one document embed plus one query,+ // both blocked inside the embedder.+ let document = {+ let stage = Arc::clone(&stage);+ tokio::spawn(async move { stage.embed_document("one").await })+ };+ assert_eq!(entered_rx.recv().await, Some("document"));+ let first_query = {+ let stage = Arc::clone(&stage);+ tokio::spawn(async move { stage.embed_query("first").await })+ };+ assert_eq!(entered_rx.recv().await, Some("query"));++ // A second query cannot start while both slots are held...+ let second_query = {+ let stage = Arc::clone(&stage);+ tokio::spawn(async move { stage.embed_query("second").await })+ };+ assert!(+ tokio::time::timeout(Duration::from_millis(25), entered_rx.recv())+ .await+ .is_err(),+ "the second query must wait for a slot, not start immediately"+ );++ // ...but the moment one holder finishes — well inside the bounded+ // wait — it proceeds and returns a vector. No CapacityBusy, no+ // degraded search.+ releases.add_permits(1);+ assert_eq!(+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())+ .await+ .unwrap(),+ Some("query"),+ "the waiting query must take the slot freed for it"+ );+ releases.add_permits(2);+ assert_eq!(second_query.await.unwrap().unwrap(), vec![1.0]);+ document.await.unwrap().unwrap();+ first_query.await.unwrap().unwrap();+ }++ /// Under *sustained* saturation the query still degrades — but only+ /// after the bounded wait elapses, never before. Paused time makes the+ /// 250 ms window advance deterministically instead of sleeping.+ #[tokio::test(start_paused = true)]+ async fn saturated_query_degrades_only_after_the_bounded_wait() {+ let (stage, releases, mut entered_rx) = BlockingEmbedder::stage(2);++ let document = {+ let stage = Arc::clone(&stage);+ tokio::spawn(async move { stage.embed_document("one").await })+ };+ assert_eq!(entered_rx.recv().await, Some("document"));+ let first_query = {+ let stage = Arc::clone(&stage);+ tokio::spawn(async move { stage.embed_query("first").await })+ };+ assert_eq!(entered_rx.recv().await, Some("query"));++ // Both total slots stay held for the whole window: the wait must+ // elapse before the refusal, and `embed_search_query` surfaces it+ // as FTS-only degradation (None) rather than an error.+ let started = tokio::time::Instant::now();+ assert!(matches!(+ stage.embed_query("saturated").await,+ Err(LlmError::CapacityBusy(_))+ ));+ assert!(+ started.elapsed() >= QUERY_CAPACITY_WAIT,+ "degradation must follow the bounded wait, not race it (took {:?})",+ started.elapsed()+ );+ assert!(+ crate::embed_search_query(stage.as_ref(), "also saturated")+ .await+ .is_none()+ );++ // Control: the saturating holders are still inside the embedder —+ // the refusal came from the wait elapsing, not from them finishing.+ releases.add_permits(2);+ document.await.unwrap().unwrap();+ first_query.await.unwrap().unwrap();+ }++ /// Documents themselves stay bounded: at most `capacity - 1` run+ /// concurrently, and the slot that frees is still query-shaped.+ #[tokio::test]+ async fn documents_run_at_capacity_minus_one_concurrently() {+ let (stage, releases, mut entered_rx) = BlockingEmbedder::stage(4);++ let mut documents = Vec::new();+ for text in ["one", "two", "three", "four"] {+ let stage = Arc::clone(&stage);+ documents.push(tokio::spawn(+ async move { stage.embed_document(text).await },+ ));+ }+ for _ in 0..3 {+ assert_eq!(+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())+ .await+ .unwrap(),+ Some("document"),+ "three of four documents should embed concurrently"+ );+ }+ assert!(+ tokio::time::timeout(Duration::from_millis(25), entered_rx.recv())+ .await+ .is_err(),+ "the fourth document must wait: only capacity-1 document slots exist"+ );++ // The reserved total slot still admits a query alongside three+ // running documents.+ let query = {+ let stage = Arc::clone(&stage);+ tokio::spawn(async move { stage.embed_query("find it").await })+ };+ assert_eq!(+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())+ .await+ .unwrap(),+ Some("query")+ );++ releases.add_permits(3);+ assert_eq!(+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())+ .await+ .unwrap(),+ Some("document"),+ "the fourth document must take the freed document slot"+ );+ releases.add_permits(2);+ for document in documents {+ document.await.unwrap().unwrap();+ }+ query.await.unwrap().unwrap();+ }++ /// The serve.rs sizing table (#1580): half the cores, clamped so the+ /// query reservation survives tiny hosts and many-core hosts alike.+ #[test]+ fn local_capacity_scales_with_cores_and_stays_bounded() {+ assert_eq!(local_capacity(0), 2, "no/failed parallelism clamps up");+ assert_eq!(local_capacity(1), 2);+ assert_eq!(local_capacity(4), 2);+ assert_eq!(local_capacity(8), 4);+ assert_eq!(local_capacity(16), 6);+ assert_eq!(local_capacity(32), 6, "many-core hosts cap at six slots");+ assert_eq!(local_capacity(256), 6);+ }+#[tokio::test]async fn cancelled_caller_does_not_release_capacity_before_inference_finishes() {let (entered, mut entered_rx) = mpsc::unbounded_channel();diff --git a/crates/ai-memory-llm/src/lib.rs b/crates/ai-memory-llm/src/lib.rsindex 5a48365a..1bb2fb92 100644--- a/crates/ai-memory-llm/src/lib.rs+++ b/crates/ai-memory-llm/src/lib.rs@@ -122,7 +122,7 @@ pub use embedding::{Embedder, OpenAiCompatEmbedder, OpenAiEmbedder, QueryVector, SyntheticEmbedder, VoyageEmbedder,cosine, embed_search_query,};-pub use embedding_capacity::EmbeddingCapacity;+pub use embedding_capacity::{EmbeddingCapacity, local_capacity};pub use error::{LlmError, LlmResult};pub use factory::{EmbedderChoice, EmbedderConfig, ProviderChoice, ProviderConfig, build_embedder, build_provider,diff --git a/docs/local-embeddings.md b/docs/local-embeddings.mdindex c8ae59f1..8d2d6d72 100644--- a/docs/local-embeddings.md+++ b/docs/local-embeddings.md@@ -104,13 +104,19 @@ cargo run --release -p ai-memory-eval -- retrieval --embeddings local- Inference is CPU, off the async runtime (`spawn_blocking`);first-load reads ~87 MB into memory once per server process.-- Local inference has two process-wide slots. Background document embeddings- use one slot, leaving the other available to search queries. This avoids- page writes and administrative backfill crowding interactive recall off the- CPU. A query that arrives while both slots are busy falls back- immediately to lexical/entity/graph retrieval instead of building an- unbounded queue. Hosted embedding providers are not subject to this local- scheduler.+- Local inference runs through a small process-wide scheduler sized from+ the CPU: half the available parallelism, clamped to between 2 and 6+ slots (`local_capacity` in `ai-memory-llm`). all-MiniLM inference+ already parallelizes across cores inside one embed, so this bounds+ concurrent calls while mild oversubscription keeps the CPU busy.+ Document embeddings may use all but one slot, so page writes and+ administrative backfill embed several documents at once instead of+ serializing, and one slot stays reserved for search queries. A query+ that arrives while every slot is busy waits up to 250 ms for one before+ falling back to lexical/entity/graph retrieval, so a query that merely+ races a document embed keeps its vector stream; only sustained+ saturation degrades. Hosted embedding providers are not subject to+ this local scheduler.- The `local-embeddings` cargo feature (default on) carries the MLdependency tree; a slim build can disable it and keep every otherprovider.@@ -118,8 +124,9 @@ cargo run --release -p ai-memory-eval -- retrieval --embeddings localLonger page bodies contribute only their head, the same contract as thehosted embedders.-The opt-in contention probe compares the raw local embedder with that capacity-split. It needs the downloaded model and is not part of the routine test tier:+The opt-in contention probe compares the raw local embedder with that+capacity scheduler. It needs the downloaded model and is not part of the+routine test tier:```bashAI_MEMORY_TEST_MODELS_DIR=$PWD/evals/models \