Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(embeddings): size local capacity from the CPU and bound the query wait

AkitaOnRails20e323a20266
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6609cfa4..8d052ac1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,21 @@ version in [`docs/changelog/`](docs/changelog/):
## [Unreleased]
+### Fixed
+
+- Fixed a 3.2.0 regression that serialized local document embedding and
+ dropped the vector stream from queries that raced a document embed: the
+ local embedder's capacity was hardcoded to two slots (one document, one
+ query), and a query finding both busy degraded instantly to lexical
+ search. The scheduler is now sized from the CPU (half the available
+ parallelism, clamped to between 2 and 6 slots, documents capped at
+ capacity minus one so a slot stays query-reserved), and a query waits up
+ to 250 ms for a slot before degrading. On the affected LongMemEval-S
+ question set this restores query latency (p50/p95 66/87 ms vs 3.2.0's
+ 118/295) and ingest throughput; the reported hit@5 delta (0.947 → 0.938)
+ traces to rank-boundary questions that flip between identical runs, not
+ a deterministic ranking change (#1580).
+
## [3.2.0] - 2026-10-21
### Added
diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rs
index 2a15379c..975910d1 100644
--- a/crates/ai-memory-cli/src/commands/serve.rs
+++ b/crates/ai-memory-cli/src/commands/serve.rs
@@ -16,7 +16,9 @@ use ai_memory_hooks::{
CLIENT_MARKER_HEADER, DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, HookState, ProjectCacheStore,
WorkstreamState, hook_router, workstream_router,
};
-use ai_memory_llm::{Embedder, EmbeddingCapacity, LlmProvider, ProviderHealth, build_embedder};
+use ai_memory_llm::{
+ Embedder, EmbeddingCapacity, LlmProvider, ProviderHealth, build_embedder, local_capacity,
+};
use ai_memory_mcp::human_auth::{Cidr, HumanAuthRuntime, LoginLimiter};
use ai_memory_mcp::{
AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_sweep_tuning,
@@ -3063,10 +3065,15 @@ async fn configure_embedder(
);
let embedder = provider_health.wrap_embedder(embedder, provider_name, model, dim);
let embedder: Arc<dyn Embedder> = if is_local {
- // all-MiniLM inference parallelizes internally. A single background
- // document slot plus one reserved query slot avoids CPU oversubscription
- // while keeping interactive vector retrieval available during writes.
- Arc::new(EmbeddingCapacity::new(embedder, 2)?)
+ // all-MiniLM inference parallelizes internally, so capacity bounds
+ // concurrent calls rather than threads. Sizing from the CPU count
+ // recovers the ingest throughput a single background document slot
+ // lost (#1580) while keeping one slot query-reserved; see
+ // `local_capacity` for the halve-and-clamp rationale.
+ let cores = std::thread::available_parallelism()
+ .map(|cores| cores.get())
+ .unwrap_or(4);
+ Arc::new(EmbeddingCapacity::new(embedder, local_capacity(cores))?)
} else {
embedder
};
diff --git a/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs b/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs
index 15696401..48423661 100644
--- a/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs
+++ b/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs
@@ -359,7 +359,11 @@ async fn local_query_latency_under_document_load() {
DOCUMENTS_PER_WORKER,
)
.await;
- let bounded: Arc<dyn Embedder> = Arc::new(EmbeddingCapacity::new(embedder, 2).unwrap());
+ let cores = std::thread::available_parallelism()
+ .map(usize::from)
+ .unwrap_or(4);
+ let bounded: Arc<dyn Embedder> =
+ Arc::new(EmbeddingCapacity::new(embedder, ai_memory_llm::local_capacity(cores)).unwrap());
let bounded_loaded = measure_queries_under_document_load(
&bounded,
workers,
diff --git a/crates/ai-memory-llm/Cargo.toml b/crates/ai-memory-llm/Cargo.toml
index 0b65c6c2..fe16c791 100644
--- a/crates/ai-memory-llm/Cargo.toml
+++ b/crates/ai-memory-llm/Cargo.toml
@@ -46,7 +46,9 @@ local-embeddings = [
]
[dev-dependencies]
-tokio = { workspace = true }
+# `test-util` brings `start_paused`, which makes the bounded query wait in
+# the capacity tests advance deterministically instead of sleeping.
+tokio = { workspace = true, features = ["test-util"] }
tempfile.workspace = true
rstest.workspace = true
# In-process HTTP mock for the openai-compat strict-fallback tests.
diff --git a/crates/ai-memory-llm/src/embedding.rs b/crates/ai-memory-llm/src/embedding.rs
index b1403f3f..608d8485 100644
--- a/crates/ai-memory-llm/src/embedding.rs
+++ b/crates/ai-memory-llm/src/embedding.rs
@@ -124,7 +124,8 @@ pub async fn embed_search_query(embedder: &dyn Embedder, query: &str) -> Option<
debug!(
provider = embedder.provider(),
model = embedder.model(),
- "embedding capacity busy; degrading search to FTS5 + entity + graph"
+ "embedding capacity stayed busy past the bounded query wait; \
+ degrading search to FTS5 + entity + graph"
);
None
}
diff --git a/crates/ai-memory-llm/src/embedding_capacity.rs b/crates/ai-memory-llm/src/embedding_capacity.rs
index c825fef1..aa6de669 100644
--- a/crates/ai-memory-llm/src/embedding_capacity.rs
+++ b/crates/ai-memory-llm/src/embedding_capacity.rs
@@ -3,20 +3,55 @@
//! Local model inference is CPU-heavy and already parallelizes internally.
//! Letting background page writes fill every inference slot can more than
//! double interactive query latency. [`EmbeddingCapacity`] bounds total work
-//! while reserving one slot for query embeddings.
+//! while reserving one slot for query embeddings; a query that finds every
+//! slot busy waits a bounded interval ([`QUERY_CAPACITY_WAIT`]) before
+//! degrading, so a transient race with a document embed does not silently
+//! drop the vector stream from a search.
use std::sync::Arc;
+use std::time::Duration;
use async_trait::async_trait;
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
use crate::{Embedder, LlmError, LlmResult};
+/// How long an interactive query waits for an inference slot before the
+/// search degrades to lexical + entity + graph retrieval.
+///
+/// Long enough to ride out a finishing document embed (or a sibling query)
+/// under write load; short enough that `memory_query` latency stays
+/// interactive even when background indexing saturates every slot for a
+/// sustained period.
+const QUERY_CAPACITY_WAIT: Duration = Duration::from_millis(250);
+
+/// Derive the local embedder's total capacity from CPU parallelism.
+///
+/// `cores` is what the caller observed via `available_parallelism()`
+/// (callers pass a small fallback such as 4 when that syscall fails; 0 or
+/// tiny values clamp to the minimum). Why halve and clamp:
+///
+/// - all-MiniLM inference (candle) already parallelizes across cores
+/// *inside* one embed, so `capacity` bounds concurrent calls, each of
+/// which wants several threads. Half the available slots recovers most
+/// of the ingest throughput a single background document slot lost
+/// (#1580) while mild oversubscription keeps the CPU busy.
+/// - clamping to at least 2 preserves the reserved query slot
+/// ([`EmbeddingCapacity::new`] refuses less); clamping to 6 stops a
+/// many-core host from running so many CPU-bound embeds at once that
+/// interactive latency climbs right back up.
+pub fn local_capacity(cores: usize) -> usize {
+ (cores / 2).clamp(2, 6)
+}
+
/// Embedder wrapper that bounds inference and reserves foreground capacity.
///
/// `capacity` must be at least two. Document and generic embeddings may use
/// at most `capacity - 1` slots, so an interactive query can enter without
-/// waiting for background indexing work to finish.
+/// waiting for background indexing work to finish. A query that does find
+/// every slot busy waits up to [`QUERY_CAPACITY_WAIT`] before failing, at
+/// which point search degrades to lexical + entity + graph retrieval rather
+/// than queueing unboundedly.
pub struct EmbeddingCapacity {
inner: Arc<dyn Embedder>,
total: Arc<Semaphore>,
@@ -113,11 +148,16 @@ impl Embedder for EmbeddingCapacity {
}
async fn embed_query(&self, text: &str) -> LlmResult<Vec<f32>> {
- let capacity = self
- .total
- .clone()
- .try_acquire_owned()
- .map_err(|_| LlmError::CapacityBusy("embedding"))?;
+ // Bounded wait, not try_acquire: an instant refusal made a query
+ // that merely raced a document embed silently degrade the whole
+ // search to FTS-only (#1580), dropping the vector stream for that
+ // query. Waiting up to QUERY_CAPACITY_WAIT absorbs transient
+ // saturation; only a slot busy for the whole window degrades.
+ let capacity =
+ tokio::time::timeout(QUERY_CAPACITY_WAIT, self.total.clone().acquire_owned())
+ .await
+ .map_err(|_| LlmError::CapacityBusy("embedding"))?
+ .map_err(|_| capacity_closed())?;
let inner = Arc::clone(&self.inner);
let text = text.to_owned();
Self::finish_task(tokio::spawn(async move {
@@ -142,6 +182,31 @@ mod tests {
}
impl BlockingEmbedder {
+ /// Wrap a blocking embedder at `capacity`, returning the staged
+ /// embedder, the release latch, and the stream of entered
+ /// operations.
+ fn stage(
+ capacity: usize,
+ ) -> (
+ Arc<dyn Embedder>,
+ Arc<Semaphore>,
+ mpsc::UnboundedReceiver<&'static str>,
+ ) {
+ let (entered, entered_rx) = mpsc::unbounded_channel();
+ let releases = Arc::new(Semaphore::new(0));
+ let stage: Arc<dyn Embedder> = Arc::new(
+ EmbeddingCapacity::new(
+ Arc::new(BlockingEmbedder {
+ entered,
+ releases: Arc::clone(&releases),
+ }),
+ capacity,
+ )
+ .unwrap(),
+ );
+ (stage, releases, entered_rx)
+ }
+
async fn block(&self, operation: &'static str) -> LlmResult<Vec<f32>> {
self.entered.send(operation).unwrap();
self.releases
@@ -226,12 +291,6 @@ mod tests {
Some("query"),
"the reserved query slot must remain available"
);
- assert!(
- crate::embed_search_query(stage.as_ref(), "another query")
- .await
- .is_none(),
- "a saturated query must degrade immediately to non-vector search"
- );
assert!(
tokio::time::timeout(Duration::from_millis(25), entered_rx.recv())
.await
@@ -252,6 +311,169 @@ mod tests {
query.await.unwrap().unwrap();
}
+ /// The #1580 regression shape: a query that arrives while every slot is
+ /// momentarily held must WAIT for the next free slot and succeed, not
+ /// degrade the search to FTS-only on an instant refusal.
+ #[tokio::test]
+ async fn query_waits_out_transient_saturation_instead_of_degrading() {
+ let (stage, releases, mut entered_rx) = BlockingEmbedder::stage(2);
+
+ // Saturate both total slots: one document embed plus one query,
+ // both blocked inside the embedder.
+ let document = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_document("one").await })
+ };
+ assert_eq!(entered_rx.recv().await, Some("document"));
+ let first_query = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_query("first").await })
+ };
+ assert_eq!(entered_rx.recv().await, Some("query"));
+
+ // A second query cannot start while both slots are held...
+ let second_query = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_query("second").await })
+ };
+ assert!(
+ tokio::time::timeout(Duration::from_millis(25), entered_rx.recv())
+ .await
+ .is_err(),
+ "the second query must wait for a slot, not start immediately"
+ );
+
+ // ...but the moment one holder finishes — well inside the bounded
+ // wait — it proceeds and returns a vector. No CapacityBusy, no
+ // degraded search.
+ releases.add_permits(1);
+ assert_eq!(
+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())
+ .await
+ .unwrap(),
+ Some("query"),
+ "the waiting query must take the slot freed for it"
+ );
+ releases.add_permits(2);
+ assert_eq!(second_query.await.unwrap().unwrap(), vec![1.0]);
+ document.await.unwrap().unwrap();
+ first_query.await.unwrap().unwrap();
+ }
+
+ /// Under *sustained* saturation the query still degrades — but only
+ /// after the bounded wait elapses, never before. Paused time makes the
+ /// 250 ms window advance deterministically instead of sleeping.
+ #[tokio::test(start_paused = true)]
+ async fn saturated_query_degrades_only_after_the_bounded_wait() {
+ let (stage, releases, mut entered_rx) = BlockingEmbedder::stage(2);
+
+ let document = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_document("one").await })
+ };
+ assert_eq!(entered_rx.recv().await, Some("document"));
+ let first_query = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_query("first").await })
+ };
+ assert_eq!(entered_rx.recv().await, Some("query"));
+
+ // Both total slots stay held for the whole window: the wait must
+ // elapse before the refusal, and `embed_search_query` surfaces it
+ // as FTS-only degradation (None) rather than an error.
+ let started = tokio::time::Instant::now();
+ assert!(matches!(
+ stage.embed_query("saturated").await,
+ Err(LlmError::CapacityBusy(_))
+ ));
+ assert!(
+ started.elapsed() >= QUERY_CAPACITY_WAIT,
+ "degradation must follow the bounded wait, not race it (took {:?})",
+ started.elapsed()
+ );
+ assert!(
+ crate::embed_search_query(stage.as_ref(), "also saturated")
+ .await
+ .is_none()
+ );
+
+ // Control: the saturating holders are still inside the embedder —
+ // the refusal came from the wait elapsing, not from them finishing.
+ releases.add_permits(2);
+ document.await.unwrap().unwrap();
+ first_query.await.unwrap().unwrap();
+ }
+
+ /// Documents themselves stay bounded: at most `capacity - 1` run
+ /// concurrently, and the slot that frees is still query-shaped.
+ #[tokio::test]
+ async fn documents_run_at_capacity_minus_one_concurrently() {
+ let (stage, releases, mut entered_rx) = BlockingEmbedder::stage(4);
+
+ let mut documents = Vec::new();
+ for text in ["one", "two", "three", "four"] {
+ let stage = Arc::clone(&stage);
+ documents.push(tokio::spawn(
+ async move { stage.embed_document(text).await },
+ ));
+ }
+ for _ in 0..3 {
+ assert_eq!(
+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())
+ .await
+ .unwrap(),
+ Some("document"),
+ "three of four documents should embed concurrently"
+ );
+ }
+ assert!(
+ tokio::time::timeout(Duration::from_millis(25), entered_rx.recv())
+ .await
+ .is_err(),
+ "the fourth document must wait: only capacity-1 document slots exist"
+ );
+
+ // The reserved total slot still admits a query alongside three
+ // running documents.
+ let query = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_query("find it").await })
+ };
+ assert_eq!(
+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())
+ .await
+ .unwrap(),
+ Some("query")
+ );
+
+ releases.add_permits(3);
+ assert_eq!(
+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())
+ .await
+ .unwrap(),
+ Some("document"),
+ "the fourth document must take the freed document slot"
+ );
+ releases.add_permits(2);
+ for document in documents {
+ document.await.unwrap().unwrap();
+ }
+ query.await.unwrap().unwrap();
+ }
+
+ /// The serve.rs sizing table (#1580): half the cores, clamped so the
+ /// query reservation survives tiny hosts and many-core hosts alike.
+ #[test]
+ fn local_capacity_scales_with_cores_and_stays_bounded() {
+ assert_eq!(local_capacity(0), 2, "no/failed parallelism clamps up");
+ assert_eq!(local_capacity(1), 2);
+ assert_eq!(local_capacity(4), 2);
+ assert_eq!(local_capacity(8), 4);
+ assert_eq!(local_capacity(16), 6);
+ assert_eq!(local_capacity(32), 6, "many-core hosts cap at six slots");
+ assert_eq!(local_capacity(256), 6);
+ }
+
#[tokio::test]
async fn cancelled_caller_does_not_release_capacity_before_inference_finishes() {
let (entered, mut entered_rx) = mpsc::unbounded_channel();
diff --git a/crates/ai-memory-llm/src/lib.rs b/crates/ai-memory-llm/src/lib.rs
index 5a48365a..1bb2fb92 100644
--- a/crates/ai-memory-llm/src/lib.rs
+++ b/crates/ai-memory-llm/src/lib.rs
@@ -122,7 +122,7 @@ pub use embedding::{
Embedder, OpenAiCompatEmbedder, OpenAiEmbedder, QueryVector, SyntheticEmbedder, VoyageEmbedder,
cosine, embed_search_query,
};
-pub use embedding_capacity::EmbeddingCapacity;
+pub use embedding_capacity::{EmbeddingCapacity, local_capacity};
pub use error::{LlmError, LlmResult};
pub use factory::{
EmbedderChoice, EmbedderConfig, ProviderChoice, ProviderConfig, build_embedder, build_provider,
diff --git a/docs/local-embeddings.md b/docs/local-embeddings.md
index c8ae59f1..8d2d6d72 100644
--- a/docs/local-embeddings.md
+++ b/docs/local-embeddings.md
@@ -104,13 +104,19 @@ cargo run --release -p ai-memory-eval -- retrieval --embeddings local
- Inference is CPU, off the async runtime (`spawn_blocking`);
first-load reads ~87 MB into memory once per server process.
-- Local inference has two process-wide slots. Background document embeddings
- use one slot, leaving the other available to search queries. This avoids
- page writes and administrative backfill crowding interactive recall off the
- CPU. A query that arrives while both slots are busy falls back
- immediately to lexical/entity/graph retrieval instead of building an
- unbounded queue. Hosted embedding providers are not subject to this local
- scheduler.
+- Local inference runs through a small process-wide scheduler sized from
+ the CPU: half the available parallelism, clamped to between 2 and 6
+ slots (`local_capacity` in `ai-memory-llm`). all-MiniLM inference
+ already parallelizes across cores inside one embed, so this bounds
+ concurrent calls while mild oversubscription keeps the CPU busy.
+ Document embeddings may use all but one slot, so page writes and
+ administrative backfill embed several documents at once instead of
+ serializing, and one slot stays reserved for search queries. A query
+ that arrives while every slot is busy waits up to 250 ms for one before
+ falling back to lexical/entity/graph retrieval, so a query that merely
+ races a document embed keeps its vector stream; only sustained
+ saturation degrades. Hosted embedding providers are not subject to
+ this local scheduler.
- The `local-embeddings` cargo feature (default on) carries the ML
dependency tree; a slim build can disable it and keep every other
provider.
@@ -118,8 +124,9 @@ cargo run --release -p ai-memory-eval -- retrieval --embeddings local
Longer page bodies contribute only their head, the same contract as the
hosted embedders.
-The opt-in contention probe compares the raw local embedder with that capacity
-split. It needs the downloaded model and is not part of the routine test tier:
+The opt-in contention probe compares the raw local embedder with that
+capacity scheduler. It needs the downloaded model and is not part of the
+routine test tier:
```bash
AI_MEMORY_TEST_MODELS_DIR=$PWD/evals/models \