Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(embeddings): reserve local query capacity

AkitaOnRails107ada8db79a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 46bdb924..0c10d013 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -143,6 +143,13 @@ version in [`docs/changelog/`](docs/changelog/):
descriptor's real path. (#1416)
### Fixed
+- Reserved local-embedding capacity for interactive search queries instead of
+ allowing concurrent page writes and backfill work to occupy every inference
+ slot. Excess queries fall back immediately to non-vector retrieval, and a
+ cancelled request retains its slot until its non-cancellable CPU inference
+ actually ends. An opt-in real-model contention probe records idle,
+ unbounded-load, and bounded-load p50/p95 query latency; hosted embedding
+ providers are unchanged.
- Added exact hover timestamps to `/web` page metadata, made the OKF upgrade
dialog keyboard accessible, and added a labelled main-navigation landmark
with current-page state and narrow-screen wrapping.
diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rs
index bf1c3f28..a5ee3477 100644
--- a/crates/ai-memory-cli/src/commands/serve.rs
+++ b/crates/ai-memory-cli/src/commands/serve.rs
@@ -16,7 +16,7 @@ use ai_memory_hooks::{
CLIENT_MARKER_HEADER, DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, HookState, ProjectCacheStore,
WorkstreamState, hook_router, workstream_router,
};
-use ai_memory_llm::{Embedder, LlmProvider, ProviderHealth, build_embedder};
+use ai_memory_llm::{Embedder, EmbeddingCapacity, LlmProvider, ProviderHealth, build_embedder};
use ai_memory_mcp::human_auth::{Cidr, HumanAuthRuntime, LoginLimiter};
use ai_memory_mcp::{
AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_sweep_tuning,
@@ -2916,6 +2916,7 @@ async fn configure_embedder(
return Ok((wiki, None));
};
let provider_name = cfg.provider.name().to_string();
+ let is_local = cfg.provider == ai_memory_llm::EmbedderChoice::Local;
let model = cfg.model.clone();
let dim = cfg.dim;
let defaulted = cfg.defaulted;
@@ -3007,6 +3008,14 @@ async fn configure_embedder(
"embedder enabled"
);
let embedder = provider_health.wrap_embedder(embedder, provider_name, model, dim);
+ let embedder: Arc<dyn Embedder> = if is_local {
+ // all-MiniLM inference parallelizes internally. A single background
+ // document slot plus one reserved query slot avoids CPU oversubscription
+ // while keeping interactive vector retrieval available during writes.
+ Arc::new(EmbeddingCapacity::new(embedder, 2)?)
+ } else {
+ embedder
+ };
Ok((wiki.with_embedder(embedder.clone()), Some(embedder)))
}
diff --git a/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs b/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs
index 00a61187..15696401 100644
--- a/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs
+++ b/crates/ai-memory-consolidate/tests/suite/local_embeddings.rs
@@ -18,9 +18,10 @@
use std::path::Path;
use std::sync::Arc;
+use std::time::{Duration, Instant};
use ai_memory_core::{PagePath, Tier};
-use ai_memory_llm::{Embedder, LocalEmbedder};
+use ai_memory_llm::{Embedder, EmbeddingCapacity, LocalEmbedder};
use ai_memory_store::Store;
use ai_memory_wiki::{Wiki, WritePageRequest};
use tempfile::TempDir;
@@ -268,6 +269,131 @@ async fn embedding_is_deterministic() {
assert_eq!(a, b, "same input must produce identical vectors");
}
+fn percentile(mut samples: Vec<Duration>, percentile: usize) -> Duration {
+ samples.sort_unstable();
+ let index = (samples.len() * percentile).div_ceil(100).saturating_sub(1);
+ samples[index]
+}
+
+async fn measure_queries(embedder: &Arc<dyn Embedder>, count: usize) -> Vec<Duration> {
+ let mut samples = Vec::with_capacity(count);
+ for sequence in 0..count {
+ let started = Instant::now();
+ embedder
+ .embed_query(&format!(
+ "where is the deployment runbook for production? sample {sequence}"
+ ))
+ .await
+ .unwrap();
+ samples.push(started.elapsed());
+ }
+ samples
+}
+
+async fn measure_queries_under_document_load(
+ embedder: &Arc<dyn Embedder>,
+ workers: usize,
+ document: &str,
+ query_samples: usize,
+ documents_per_worker: usize,
+) -> Vec<Duration> {
+ let barrier = Arc::new(tokio::sync::Barrier::new(workers + 1));
+ let mut background = Vec::with_capacity(workers);
+ for worker in 0..workers {
+ let embedder = Arc::clone(embedder);
+ let barrier = Arc::clone(&barrier);
+ let document = document.to_owned();
+ background.push(tokio::spawn(async move {
+ barrier.wait().await;
+ for sequence in 0..documents_per_worker {
+ embedder
+ .embed_document(&format!("{document} worker {worker} item {sequence}"))
+ .await
+ .unwrap();
+ }
+ }));
+ }
+
+ barrier.wait().await;
+ let loaded = measure_queries(embedder, query_samples).await;
+ for task in background {
+ task.await.unwrap();
+ }
+ loaded
+}
+
+/// Manual capacity probe for the shared local embedder. It is deliberately a
+/// benchmark, not a CI assertion: CPU topology and scheduler load determine
+/// the ratio. Run it before adding an admission-control policy so the policy
+/// responds to measured contention rather than an assumed bottleneck.
+#[tokio::test(flavor = "multi_thread")]
+#[ignore = "manual local-model contention benchmark"]
+async fn local_query_latency_under_document_load() {
+ const QUERY_SAMPLES: usize = 9;
+ const DOCUMENTS_PER_WORKER: usize = 12;
+
+ let embedder: Arc<dyn Embedder> =
+ Arc::new(LocalEmbedder::load(Path::new(&models_root())).unwrap());
+ embedder.embed_query("warm the local model").await.unwrap();
+
+ let idle = measure_queries(&embedder, QUERY_SAMPLES).await;
+ let workers = std::env::var("AI_MEMORY_EMBED_BENCH_WORKERS")
+ .ok()
+ .and_then(|value| value.parse().ok())
+ .unwrap_or_else(|| {
+ std::thread::available_parallelism()
+ .map(usize::from)
+ .unwrap_or(2)
+ .clamp(2, 8)
+ });
+ assert!(workers > 0, "benchmark needs at least one worker");
+ let document = "A long-lived project memory page describes architecture decisions, \
+ deployment procedures, incident recovery, database migrations, authentication \
+ boundaries, contributor workflow, release validation, and operational runbooks. "
+ .repeat(32);
+ let loaded = measure_queries_under_document_load(
+ &embedder,
+ workers,
+ &document,
+ QUERY_SAMPLES,
+ DOCUMENTS_PER_WORKER,
+ )
+ .await;
+ let bounded: Arc<dyn Embedder> = Arc::new(EmbeddingCapacity::new(embedder, 2).unwrap());
+ let bounded_loaded = measure_queries_under_document_load(
+ &bounded,
+ workers,
+ &document,
+ QUERY_SAMPLES,
+ DOCUMENTS_PER_WORKER,
+ )
+ .await;
+
+ let idle_p50 = percentile(idle.clone(), 50);
+ let idle_p95 = percentile(idle, 95);
+ let loaded_p50 = percentile(loaded.clone(), 50);
+ let loaded_p95 = percentile(loaded, 95);
+ let bounded_p50 = percentile(bounded_loaded.clone(), 50);
+ let bounded_p95 = percentile(bounded_loaded, 95);
+ eprintln!(
+ "local embedding contention: workers={workers} idle_p50_ms={} \
+ idle_p95_ms={} loaded_p50_ms={} loaded_p95_ms={} \
+ loaded_p50_ratio={:.2} loaded_p95_ratio={:.2} \
+ bounded_p50_ms={} bounded_p95_ms={} bounded_p50_ratio={:.2} \
+ bounded_p95_ratio={:.2}",
+ idle_p50.as_millis(),
+ idle_p95.as_millis(),
+ loaded_p50.as_millis(),
+ loaded_p95.as_millis(),
+ loaded_p50.as_secs_f64() / idle_p50.as_secs_f64(),
+ loaded_p95.as_secs_f64() / idle_p95.as_secs_f64(),
+ bounded_p50.as_millis(),
+ bounded_p95.as_millis(),
+ bounded_p50.as_secs_f64() / idle_p50.as_secs_f64(),
+ bounded_p95.as_secs_f64() / idle_p95.as_secs_f64(),
+ );
+}
+
/// A handful of synonym/paraphrase orderings beyond the single case in
/// the unit test — cheap confidence that the pooling isn't degenerate.
#[tokio::test]
diff --git a/crates/ai-memory-llm/src/embedding.rs b/crates/ai-memory-llm/src/embedding.rs
index ac8c699a..1bc2252e 100644
--- a/crates/ai-memory-llm/src/embedding.rs
+++ b/crates/ai-memory-llm/src/embedding.rs
@@ -120,6 +120,14 @@ pub async fn embed_search_query(embedder: &dyn Embedder, query: &str) -> Option<
model: embedder.model_identity(),
dim: embedder.dim(),
}),
+ Err(LlmError::CapacityBusy(_)) => {
+ debug!(
+ provider = embedder.provider(),
+ model = embedder.model(),
+ "embedding capacity busy; degrading search to FTS5 + entity + graph"
+ );
+ None
+ }
Err(e) => {
tracing::warn!(
provider = embedder.provider(),
diff --git a/crates/ai-memory-llm/src/embedding_capacity.rs b/crates/ai-memory-llm/src/embedding_capacity.rs
new file mode 100644
index 00000000..c825fef1
--- /dev/null
+++ b/crates/ai-memory-llm/src/embedding_capacity.rs
@@ -0,0 +1,336 @@
+//! Capacity isolation for in-process embedding inference.
+//!
+//! Local model inference is CPU-heavy and already parallelizes internally.
+//! Letting background page writes fill every inference slot can more than
+//! double interactive query latency. [`EmbeddingCapacity`] bounds total work
+//! while reserving one slot for query embeddings.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use tokio::sync::{OwnedSemaphorePermit, Semaphore};
+
+use crate::{Embedder, LlmError, LlmResult};
+
+/// Embedder wrapper that bounds inference and reserves foreground capacity.
+///
+/// `capacity` must be at least two. Document and generic embeddings may use
+/// at most `capacity - 1` slots, so an interactive query can enter without
+/// waiting for background indexing work to finish.
+pub struct EmbeddingCapacity {
+ inner: Arc<dyn Embedder>,
+ total: Arc<Semaphore>,
+ documents: Arc<Semaphore>,
+}
+
+impl EmbeddingCapacity {
+ /// Wrap `inner` with a bounded scheduler.
+ ///
+ /// # Errors
+ /// Returns [`LlmError::NotConfigured`] when fewer than two slots are
+ /// requested because that cannot reserve foreground capacity.
+ pub fn new(inner: Arc<dyn Embedder>, capacity: usize) -> LlmResult<Self> {
+ if capacity < 2 {
+ return Err(LlmError::NotConfigured(
+ "embedding capacity must be at least 2".into(),
+ ));
+ }
+ Ok(Self {
+ inner,
+ total: Arc::new(Semaphore::new(capacity)),
+ documents: Arc::new(Semaphore::new(capacity - 1)),
+ })
+ }
+
+ async fn acquire_document_capacity(
+ &self,
+ ) -> LlmResult<(OwnedSemaphorePermit, OwnedSemaphorePermit)> {
+ let document = self
+ .documents
+ .clone()
+ .acquire_owned()
+ .await
+ .map_err(|_| capacity_closed())?;
+ let total = self
+ .total
+ .clone()
+ .acquire_owned()
+ .await
+ .map_err(|_| capacity_closed())?;
+ Ok((document, total))
+ }
+
+ async fn finish_task(
+ task: tokio::task::JoinHandle<LlmResult<Vec<f32>>>,
+ ) -> LlmResult<Vec<f32>> {
+ task.await
+ .map_err(|e| LlmError::UnexpectedShape(format!("embedding task: {e}")))?
+ }
+}
+
+fn capacity_closed() -> LlmError {
+ LlmError::UnexpectedShape("embedding capacity scheduler closed".into())
+}
+
+#[async_trait]
+impl Embedder for EmbeddingCapacity {
+ fn provider(&self) -> &'static str {
+ self.inner.provider()
+ }
+
+ fn model(&self) -> &str {
+ self.inner.model()
+ }
+
+ fn model_identity(&self) -> String {
+ self.inner.model_identity()
+ }
+
+ fn dim(&self) -> u32 {
+ self.inner.dim()
+ }
+
+ async fn embed(&self, text: &str) -> LlmResult<Vec<f32>> {
+ let capacity = self.acquire_document_capacity().await?;
+ let inner = Arc::clone(&self.inner);
+ let text = text.to_owned();
+ Self::finish_task(tokio::spawn(async move {
+ let _capacity = capacity;
+ inner.embed(&text).await
+ }))
+ .await
+ }
+
+ async fn embed_document(&self, text: &str) -> LlmResult<Vec<f32>> {
+ let capacity = self.acquire_document_capacity().await?;
+ let inner = Arc::clone(&self.inner);
+ let text = text.to_owned();
+ Self::finish_task(tokio::spawn(async move {
+ let _capacity = capacity;
+ inner.embed_document(&text).await
+ }))
+ .await
+ }
+
+ async fn embed_query(&self, text: &str) -> LlmResult<Vec<f32>> {
+ let capacity = self
+ .total
+ .clone()
+ .try_acquire_owned()
+ .map_err(|_| LlmError::CapacityBusy("embedding"))?;
+ let inner = Arc::clone(&self.inner);
+ let text = text.to_owned();
+ Self::finish_task(tokio::spawn(async move {
+ let _capacity = capacity;
+ inner.embed_query(&text).await
+ }))
+ .await
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use std::time::Duration;
+
+ use tokio::sync::mpsc;
+
+ use super::*;
+
+ struct BlockingEmbedder {
+ entered: mpsc::UnboundedSender<&'static str>,
+ releases: Arc<Semaphore>,
+ }
+
+ impl BlockingEmbedder {
+ async fn block(&self, operation: &'static str) -> LlmResult<Vec<f32>> {
+ self.entered.send(operation).unwrap();
+ self.releases
+ .acquire()
+ .await
+ .map_err(|_| capacity_closed())?
+ .forget();
+ Ok(vec![1.0])
+ }
+ }
+
+ #[async_trait]
+ impl Embedder for BlockingEmbedder {
+ fn provider(&self) -> &'static str {
+ "blocking"
+ }
+
+ fn model(&self) -> &str {
+ "test"
+ }
+
+ fn model_identity(&self) -> String {
+ "document-identity".into()
+ }
+
+ fn dim(&self) -> u32 {
+ 1
+ }
+
+ async fn embed(&self, _text: &str) -> LlmResult<Vec<f32>> {
+ self.block("generic").await
+ }
+
+ async fn embed_document(&self, _text: &str) -> LlmResult<Vec<f32>> {
+ self.block("document").await
+ }
+
+ async fn embed_query(&self, _text: &str) -> LlmResult<Vec<f32>> {
+ self.block("query").await
+ }
+ }
+
+ #[test]
+ fn requires_a_foreground_reservation() {
+ let (entered, _entered_rx) = mpsc::unbounded_channel();
+ let inner = Arc::new(BlockingEmbedder {
+ entered,
+ releases: Arc::new(Semaphore::new(0)),
+ });
+ assert!(EmbeddingCapacity::new(inner, 1).is_err());
+ }
+
+ #[tokio::test]
+ async fn query_enters_while_second_document_waits() {
+ let (entered, mut entered_rx) = mpsc::unbounded_channel();
+ let releases = Arc::new(Semaphore::new(0));
+ let inner = Arc::new(BlockingEmbedder {
+ entered,
+ releases: Arc::clone(&releases),
+ });
+ let stage: Arc<dyn Embedder> = Arc::new(EmbeddingCapacity::new(inner, 2).unwrap());
+
+ let first_document = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_document("one").await })
+ };
+ assert_eq!(entered_rx.recv().await, Some("document"));
+
+ let second_document = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_document("two").await })
+ };
+ let query = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_query("find it").await })
+ };
+
+ assert_eq!(
+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())
+ .await
+ .unwrap(),
+ Some("query"),
+ "the reserved query slot must remain available"
+ );
+ assert!(
+ crate::embed_search_query(stage.as_ref(), "another query")
+ .await
+ .is_none(),
+ "a saturated query must degrade immediately to non-vector search"
+ );
+ assert!(
+ tokio::time::timeout(Duration::from_millis(25), entered_rx.recv())
+ .await
+ .is_err(),
+ "the second document must wait for the document slot"
+ );
+
+ releases.add_permits(1);
+ assert_eq!(
+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())
+ .await
+ .unwrap(),
+ Some("document")
+ );
+ releases.add_permits(2);
+ first_document.await.unwrap().unwrap();
+ second_document.await.unwrap().unwrap();
+ query.await.unwrap().unwrap();
+ }
+
+ #[tokio::test]
+ async fn cancelled_caller_does_not_release_capacity_before_inference_finishes() {
+ let (entered, mut entered_rx) = mpsc::unbounded_channel();
+ let releases = Arc::new(Semaphore::new(0));
+ let stage: Arc<dyn Embedder> = Arc::new(
+ EmbeddingCapacity::new(
+ Arc::new(BlockingEmbedder {
+ entered,
+ releases: Arc::clone(&releases),
+ }),
+ 2,
+ )
+ .unwrap(),
+ );
+
+ let cancelled = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_document("cancel me").await })
+ };
+ assert_eq!(entered_rx.recv().await, Some("document"));
+ cancelled.abort();
+ assert!(cancelled.await.unwrap_err().is_cancelled());
+
+ let next = {
+ let stage = Arc::clone(&stage);
+ tokio::spawn(async move { stage.embed_document("wait for capacity").await })
+ };
+ assert!(
+ tokio::time::timeout(Duration::from_millis(25), entered_rx.recv())
+ .await
+ .is_err(),
+ "cancelled caller must not free a still-running inference slot"
+ );
+
+ releases.add_permits(1);
+ assert_eq!(
+ tokio::time::timeout(Duration::from_secs(1), entered_rx.recv())
+ .await
+ .unwrap(),
+ Some("document")
+ );
+ releases.add_permits(1);
+ next.await.unwrap().unwrap();
+ }
+
+ #[tokio::test]
+ async fn generic_embedding_keeps_inner_method_semantics() {
+ let (entered, mut entered_rx) = mpsc::unbounded_channel();
+ let releases = Arc::new(Semaphore::new(0));
+ let stage = Arc::new(
+ EmbeddingCapacity::new(
+ Arc::new(BlockingEmbedder {
+ entered,
+ releases: Arc::clone(&releases),
+ }),
+ 2,
+ )
+ .unwrap(),
+ );
+ let task = tokio::spawn(async move { stage.embed("generic input").await });
+ assert_eq!(entered_rx.recv().await, Some("generic"));
+ releases.add_permits(1);
+ task.await.unwrap().unwrap();
+ }
+
+ #[test]
+ fn delegates_embedding_identity() {
+ let (entered, _entered_rx) = mpsc::unbounded_channel();
+ let stage = EmbeddingCapacity::new(
+ Arc::new(BlockingEmbedder {
+ entered,
+ releases: Arc::new(Semaphore::new(0)),
+ }),
+ 2,
+ )
+ .unwrap();
+ assert_eq!(stage.provider(), "blocking");
+ assert_eq!(stage.model(), "test");
+ assert_eq!(stage.model_identity(), "document-identity");
+ assert_eq!(stage.dim(), 1);
+ }
+}
diff --git a/crates/ai-memory-llm/src/error.rs b/crates/ai-memory-llm/src/error.rs
index 66cde7c9..5c13a9dc 100644
--- a/crates/ai-memory-llm/src/error.rs
+++ b/crates/ai-memory-llm/src/error.rs
@@ -65,6 +65,12 @@ pub enum LlmError {
#[error("auth: {0}")]
Auth(String),
+ /// Optional local work could not start without waiting behind already
+ /// active work. Callers should use their non-provider fallback instead of
+ /// retrying or reporting the provider unhealthy.
+ #[error("{0} capacity is busy")]
+ CapacityBusy(&'static str),
+
/// JSON schema for structured output could not be derived.
#[error("schema: {0}")]
Schema(String),
@@ -127,6 +133,7 @@ impl LlmError {
Self::EmptyContent { .. } => "empty-content",
Self::NotConfigured(_) => "not-configured",
Self::Auth(_) => "auth",
+ Self::CapacityBusy(_) => "capacity-busy",
Self::Schema(_) => "schema",
Self::AllCandidatesFailed { .. } => "all-candidates-failed",
}
@@ -178,6 +185,7 @@ mod tests {
#[test]
fn deterministic_errors_are_not_transient() {
assert!(!LlmError::Auth("expired".into()).is_transient());
+ assert!(!LlmError::CapacityBusy("embedding").is_transient());
assert!(!LlmError::Schema("bad".into()).is_transient());
assert!(!LlmError::Serde("nope".into()).is_transient());
assert!(!LlmError::UnexpectedShape("no tool block".into()).is_transient());
@@ -274,6 +282,7 @@ mod tests {
),
(LlmError::NotConfigured("no key".into()), "not-configured"),
(LlmError::Auth("expired".into()), "auth"),
+ (LlmError::CapacityBusy("embedding"), "capacity-busy"),
(LlmError::Schema("bad".into()), "schema"),
(
LlmError::AllCandidatesFailed {
diff --git a/crates/ai-memory-llm/src/lib.rs b/crates/ai-memory-llm/src/lib.rs
index ce9a70c7..6b47ec5c 100644
--- a/crates/ai-memory-llm/src/lib.rs
+++ b/crates/ai-memory-llm/src/lib.rs
@@ -79,6 +79,7 @@ pub mod codex;
pub mod copilot;
pub mod cursor_agent;
pub mod embedding;
+pub mod embedding_capacity;
pub mod error;
pub mod factory;
pub mod fallback;
@@ -121,6 +122,7 @@ pub use embedding::{
Embedder, OpenAiCompatEmbedder, OpenAiEmbedder, QueryVector, SyntheticEmbedder, VoyageEmbedder,
cosine, embed_search_query,
};
+pub use embedding_capacity::EmbeddingCapacity;
pub use error::{LlmError, LlmResult};
pub use factory::{
EmbedderChoice, EmbedderConfig, ProviderChoice, ProviderConfig, build_embedder, build_provider,
diff --git a/docs/comparison.md b/docs/comparison.md
index 7634f2aa..e0148095 100644
--- a/docs/comparison.md
+++ b/docs/comparison.md
@@ -364,9 +364,10 @@ ships something ai-memory does not, it is named.
packaging, its `memu-agent detect` for unknown agents, and its hosted cloud,
and EverOS's ANN vector index, optional Milvus backend and published
end-to-end benchmark. EverOS v1.4.2 also reserves query-embedding capacity
- from background writes; ai-memory's reranker fails open under saturation,
- but embedding backfill and foreground query embedding do not yet have an
- equivalent measured capacity split.
+ from background writes. ai-memory now does the same for in-process local
+ inference, backed by an opt-in real-model contention probe; hosted embedding
+ providers keep their existing concurrency, and the reranker remains bounded
+ and fail-open.
- **From Caura (governed multi-agent fleet memory):** a different buyer. Caura
optimizes org-scale fleet governance (agent/team/org visibility scopes, four
trust tiers, an audit log, PII flagging) on Postgres+pgvector with LLM
@@ -392,19 +393,19 @@ ships something ai-memory does not, it is named.
## Where we're behind, or different by choice
-- **No human workstream operations view.** Managed workstreams, run leases,
- handoffs, and portable event history exist, but `/web` does not turn them
- into a scannable queue of active, blocked, review-ready, and completed work.
- claude-mem v14's local “My Work” view is better for a human supervising
- several requests.
+- **Workstream operations are now visible to operators.** `/web/workstreams`
+ presents trusted lifecycle states for managed workstreams without exposing
+ checkout paths, native session ids, or transcript content. It deliberately
+ does not infer blocked or review-ready states from untrusted prose.
- **No structured page filters on search.** Scope and `as_of` are typed, but
`memory_query` cannot narrow by path prefix, kind, tier, tag, or author
before ranking. Mem0 has directory/category scopes; basic-memory's
unreleased v0.24 adds a broader structured filter set.
-- **Foreground embedding has no measured reservation from backfill.** The LLM
- reranker is bounded and fails open, but query embedding and administrative
- embedding backfill share the configured embedder without a dedicated
- foreground budget. EverOS v1.4.2 shows the latency value of separating them.
+- **Local foreground embedding is isolated from background writes.** A
+ real-model probe demonstrated CPU contention, so local document inference is
+ limited to one slot and a second slot remains available for query embedding.
+ Hosted providers retain their existing concurrency policy; the LLM reranker
+ remains separately bounded and fail-open.
- **Marketplace reach is narrower than harness support.** ai-memory now ships
its own Claude Code and Codex marketplace plugins and `ai-memory setup`
detects and wires installed agents across Linux, macOS, and Windows. It is
diff --git a/docs/competitive-parity.md b/docs/competitive-parity.md
index f6b15993..4d6ed14e 100644
--- a/docs/competitive-parity.md
+++ b/docs/competitive-parity.md
@@ -281,7 +281,7 @@ invariants), **ALREADY** (we have it and should say so).
| Capability seen in the field | Verdict | Why |
|---|---|---|
| Human workstream operations view (claude-mem v14 “My Work”) | ADOPT | The managed-workstream ledger already has runs, leases, events, and handoff evidence, but humans cannot scan active/attention/review-ready work in `/web`. Build a read-only derived view; add only a minimal review marker, not a second task system |
-| Foreground query capacity isolated from embedding backfill (EverOS v1.4.2) | MEASURE-FIRST, then ADOPT if contention is real | The reranker already has a shared fail-open gate. Query embedding and administrative backfill share the embedder without a foreground reservation. Add a contention benchmark; preserve lexical/entity/graph results under saturation |
+| Foreground query capacity isolated from embedding backfill (EverOS v1.4.2) | SHIPPED (3.2) for local inference | A real-model probe measured eight-worker query p50/p95 rising from 36/41 ms idle to 74/76 ms. Limiting local document inference to one slot while reserving one query slot reduced loaded latency to 39/51 ms. Hosted providers keep their existing concurrency; provider failures still degrade to lexical/entity/graph retrieval |
| Structured pre-rank filters (Mem0; basic-memory v0.24 unreleased) | ADOPT | Bounded path-prefix, kind, tier, tag, author, and valid-time filters improve monorepo/tool ergonomics without changing ranking. All paths must reuse `ScopeResolver` and fail closed on incomplete scope |
| Per-operation LLM provider/model profiles (Graphiti v0.31; Hindsight `main`) | MEASURE-FIRST | Potentially lowers cost and latency for consolidation, rerank, and auto-improvement, but expands config and auth surface. Prototype against the evals before adding public settings |
| Retrieval recall through capture (several tools rank whole raw sessions) | SHIPPED (3.0) | The 2026-10-12 decomposition showed our zero-LLM session page indexed only 80-character turn titles; the observation stream recovered it (FTS hit@5 0.662 → 0.951, local 0.800 → 0.945, all 470). Chasing another tool's headline number is declined: those numbers measure different things |
@@ -323,8 +323,8 @@ which should be checked before any work is scoped.
2. **Structured filters before ranking** for path prefix, kind, tier, tag,
author, and valid time (Mem0; basic-memory v0.24, currently unreleased).
3. **Foreground embedding isolation backed by a contention benchmark**
- (EverOS v1.4.2). Reserve query capacity only if the benchmark shows
- administrative backfill can harm interactive latency.
+ (EverOS v1.4.2) is now shipped for the in-process local model. The manual
+ probe remains in-tree so the capacity can be rechecked on other CPU classes.
4. **One reproducible end-to-end QA number** beside hit@5, with reader,
judge, context tokens and latency stated, and more benchmarks than
LongMemEval (Mem0, Zep, Hindsight and EverOS publish QA numbers; Hindsight's
diff --git a/docs/local-embeddings.md b/docs/local-embeddings.md
index 3c9de46a..e4d2fbfc 100644
--- a/docs/local-embeddings.md
+++ b/docs/local-embeddings.md
@@ -104,9 +104,26 @@ cargo run --release -p ai-memory-eval -- retrieval --embeddings local
- Inference is CPU, off the async runtime (`spawn_blocking`);
first-load reads ~87 MB into memory once per server process.
+- Local inference has two process-wide slots. Background document embeddings
+ use one slot, leaving the other available to search queries. This avoids
+ page writes and administrative backfill crowding interactive recall off the
+ CPU. A query that arrives while both slots are serving queries falls back
+ immediately to lexical/entity/graph retrieval instead of building an
+ unbounded queue. Hosted embedding providers are not subject to this local
+ scheduler.
- The `local-embeddings` cargo feature (default on) carries the ML
dependency tree; a slim build can disable it and keep every other
provider.
- Tokenization truncates at 512 tokens, the model's positional limit.
Longer page bodies contribute only their head, the same contract as the
hosted embedders.
+
+The opt-in contention probe compares the raw local embedder with that capacity
+split. It needs the downloaded model and is not part of the routine test tier:
+
+```bash
+AI_MEMORY_TEST_MODELS_DIR=$PWD/evals/models \
+AI_MEMORY_EMBED_BENCH_WORKERS=8 \
+cargo test -p ai-memory-consolidate \
+ local_query_latency_under_document_load -- --ignored --nocapture
+```
diff --git a/docs/research-2026-landscape.md b/docs/research-2026-landscape.md
index 31568718..30370e3f 100644
--- a/docs/research-2026-landscape.md
+++ b/docs/research-2026-landscape.md
@@ -688,7 +688,7 @@ explicitly.
| **Memori** | No commits since 2026-10-03 and no new release. | No material change. |
| **memvid** | No commits since 2026-07-14 and no new release. | Remains quiet; its relative benchmark claim is still not independently useful. |
| **memU** | No feature release. Four Windows path/idempotency fixes landed. | No material change. |
-| **EverOS** | **Released v1.4.2.** Fixes an FD leak, separates query embedding capacity from background writes, batches embeddings, keys indexes by the full scope tuple, self-repairs damaged indexes, and hardens path containment. | The capacity split is worth copying in principle. ai-memory's reranker already fails open behind a shared gate, but query embedding and administrative backfill share the embedder without a foreground reservation. Add a contention benchmark before changing it. |
+| **EverOS** | **Released v1.4.2.** Fixes an FD leak, separates query embedding capacity from background writes, batches embeddings, keys indexes by the full scope tuple, self-repairs damaged indexes, and hardens path containment. | Adopted for the in-process local embedder after measurement: eight background workers raised query p50/p95 from 36/41 ms to 74/76 ms on the reference host; one document slot plus one reserved query slot brought them to 39/51 ms. Hosted providers and the reranker retain their existing policies. |
| **MemOS** | No commits or release after the prior cutoff. Its current headline remains DeepSeek Harness integration, multimodal memory, memory cubes, asynchronous ingestion, and feedback/correction. | No material change. The memory-OS and multimodal tradeoffs remain different from ai-memory's scope. |
| **Honcho** | **Released v3.3.0.** Adds an optional Chroma backend, scoped custom instructions, cursor pagination, prompt-budget handling, and performance fixes; stable object `uid` work followed on `main`. | No core work. Its theory-of-mind buyer remains different. Operation-scoped instructions are a useful reference if per-operation LLM profiles are designed. |
| **Engram** | No new tag after v3.4.0. `main` adds local aliases/routing preview, project provenance, compact search/history clients, capture guards, tombstones, and sync isolation. | No new gap. ai-memory already has stronger repository identity, routing diagnostics, tombstone/supersession semantics, and conflict-aware wiki sync. |
@@ -870,11 +870,12 @@ Each item needs its own design and approval before implementation.
workstreams when the schema has no such relationship. This borrows
claude-mem's useful human visibility without turning ai-memory into a kanban
system or cloud collaboration product.
-2. **P1 - Protect interactive recall from background embedding.** Build a
- contention benchmark first. If administrative backfill can delay query
- embedding, give foreground queries reserved capacity or separate gates.
- Preserve the current fail-open behavior: lexical/entity/graph retrieval
- must still answer when embedding or reranking is saturated.
+2. **P1 - Protect interactive recall from background embedding. Completed.**
+ The opt-in real-model probe found material local CPU contention, so local
+ inference now admits one document embedding and reserves a second slot for
+ queries. The same probe verifies the improvement; hosted providers remain
+ unchanged, and provider failures still degrade to lexical/entity/graph
+ retrieval.
3. **P2 - Structured query filters.** Expose the existing pre-rank kind/tier
filters through `memory_query`, then add bounded path-prefix and exact-tag
filters. Keep `as_of` as the valid-time control and defer author until null