Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
perf(vectordb): micro-batch compatible cuVS searches (#3382)
Yuanqing ZHAO40dd05271cc6
diff --git a/benchmark/cuvs/PRELIMINARY_RESULTS.md b/benchmark/cuvs/PRELIMINARY_RESULTS.mdindex d935fa102..5607307c2 100644--- a/benchmark/cuvs/PRELIMINARY_RESULTS.md+++ b/benchmark/cuvs/PRELIMINARY_RESULTS.md@@ -6,6 +6,14 @@ GPU, one public ANN dataset, and deterministic synthetic vectors for exacthigh-dimensional scaling. Every reported row aggregates five independentprocesses as median +/- median absolute deviation (MAD).+> **Historical scope (pre-microbatch):** Every result below was collected from+> the measured revisions named in this document, before opt-in request+> micro-batching and its warm admission fast path were implemented. Statements+> about the "current integration" describe those measured revisions, not the+> later implementation. The tables are retained as historical engineering+> checkpoints and must not be used to infer current concurrent-request+> throughput.+## Public-dataset setup- Harness revision: `5dbc6a570404c9d1e2c8b584f6447919d0fee62a`@@ -72,10 +80,10 @@ budget keeps the native path for that query and allows a later retry.## Batch size 1-This is the closest index-level approximation of the current OpenViking-single-query integration. It includes Python dispatch, host-to-device query-copy, GPU execution, and result copy back to host. It excludes embedding,-HTTP, record lookup, reranking, and LLM work.+This is the closest index-level approximation of the measured pre-microbatch+OpenViking single-query integration. It includes Python dispatch,+host-to-device query copy, GPU execution, and result copy back to host. It+excludes embedding, HTTP, record lookup, reranking, and LLM work.| Backend | Recall@10 | warm p50 (ms/query) | warm p95 (ms/query) | warm QPS || --- | ---: | ---: | ---: | ---: |@@ -408,8 +416,8 @@ cuVS keeps a strong low-concurrency advantage, but its throughput remains near615--677 QPS as concurrency increases. Native CPU search scales across thethread pool and crosses cuVS between concurrency 4 and 16. cuVS p50 rises from1.5 ms to 92.6 ms at concurrency 64 because requests queue behind the current-index-wide lock; this integration does not yet use concurrent CUDA streams or-micro-batching.+index-wide lock; the measured revision did not yet use concurrent CUDA streams+or micro-batching.### Cached 10% filter@@ -462,7 +470,7 @@ the same rebuild delay amortized across more requests released together.2. CAGRA requires recall-matched tuning; an approximate label alone does notimply better performance.3. CAGRA can improve batched capacity around Recall@10=0.96, but this benefit- requires batching that the current OpenViking integration does not expose.+ required batching that the measured pre-microbatch revision did not expose.4. Collection-level unfiltered lookup retained an approximately 9.0x warm-p50benefit at 100K x 768D in these runs, but native int8 versus the configuredcuVS float32 device representation is not an equal-memory or equal-quality@@ -480,10 +488,10 @@ the same rebuild delay amortized across more requests released together.8. GloVe-100 and normalized Gaussian vectors are engineering datasets, not arepresentative agent-memory corpus.9. At the async service facade, cuVS wins cached tenant search through- concurrency 4, but native crosses it by concurrency 16. The current cuVS- lock holds throughput near 600--680 QPS and turns concurrency into queueing- latency rather than GPU parallelism.-10. The next engineering step is to evaluate a read-safe immutable snapshot- with concurrent searches or micro-batching, plus background rebuild. A- later end-to-end matrix should then add HTTP, embedding, reranking, and a- real agent-memory workload.+ concurrency 4, but native crosses it by concurrency 16. On the measured+ revision, the cuVS lock holds throughput near 600--680 QPS and turns+ concurrency into queueing latency rather than GPU parallelism.+10. For this historical checkpoint, the next engineering step was to evaluate+ a read-safe immutable snapshot with concurrent searches or micro-batching,+ plus background rebuild. A later end-to-end matrix should then add HTTP,+ embedding, reranking, and a real agent-memory workload.diff --git a/benchmark/cuvs/README.md b/benchmark/cuvs/README.mdindex 75117ec98..ac9a6f178 100644--- a/benchmark/cuvs/README.md+++ b/benchmark/cuvs/README.md@@ -181,11 +181,16 @@ vector limit disables the supplied ground truth because neighbor IDs may referto omitted rows; include `native` or `cuvs_brute_force` in that case. A querylimit retains the corresponding prefix of supplied ground truth.-`query_batch_size=1` reflects the current OpenViking integration most closely.-Larger batches measure the vector-index capacity: cuVS executes the batch on-the GPU, while the current native wrapper processes each query sequentially.-Do not present batch results as current server throughput without also running-the collection/server benchmarks.+`query_batch_size=1` most closely represents one public OpenViking API request+and the default configuration, where request micro-batching is disabled.+Opt-in brute-force micro-batching can internally coalesce compatible concurrent+single-row API requests into one matrix-query call; that behavior must be+measured through the collection/service concurrency path rather than inferred+from this index harness. Larger explicit batches measure vector-index capacity:+cuVS executes the batch on the GPU, while the current native wrapper processes+each query sequentially. Do not present explicit batch results as server+throughput without also running the collection/server benchmarks and reporting+whether request micro-batching was enabled.## First comparison matrixdiff --git a/docs/design/openviking-cuvs-benchmark-plan.md b/docs/design/openviking-cuvs-benchmark-plan.mdindex 09d0c0029..aa47e0585 100644--- a/docs/design/openviking-cuvs-benchmark-plan.md+++ b/docs/design/openviking-cuvs-benchmark-plan.md@@ -184,13 +184,22 @@ ground truth 计算 recall。## 当前集成的吞吐限制-当前 Python integration 一次只提交一个 query,并在一次 GPU search 期间持有 index lock;同时-每次调用都会创建 query device array 并把结果同步回 host。因此:--- batch > 1 的 L1 结果代表 cuVS 的能力上限,不等同于当前 OpenViking API 的吞吐;-- L2/L3 必须以 batch=1 为主,并明确记录并发请求是否被 lock 串行化;-- 如果 L1 与 L2 差距明显,下一阶段优先实现 persistent device buffers、CUDA streams 或- dynamic batching,再评估最大吞吐。+默认路径仍按一次公开 API 请求提交一个 query。当前已提供默认关闭、仅适用于 brute-force 的+request micro-batching:它可把兼容的并发单 query 请求合并为最多 8 行的 matrix-query。+对于当前 generation 的 clean immutable snapshot,无 filter 或 device filter cache hit 可通过+warm fast path 直接入队,不获取 caller 侧 device gate;dirty/cold snapshot、filter cache miss+和 device filter preparation 仍先走串行的 gated preparation。入队后只有 micro-batch worker+会在持有 device-search gate 时执行 GPU matrix search,caller 不会持 gate 等待结果。++这消除了旧版“一次 GPU search 全程持有 caller/index lock”的主要并发瓶颈,但每次 dispatch+仍会创建 query/result device array,并在返回前把结果同步到 host。因此:++- batch > 1 的 L1 结果仍代表显式 vector-index batch 的能力上限,不等同于公开 API 吞吐;+- L2/L3 应继续以单 query 请求为主,同时分别测默认关闭和 opt-in micro-batching,并记录+ 实际 batch-size 分布、`batch_wait`、`gpu_gate_queue`、QPS 与 P95;+- 下一阶段优先评估 persistent query/result buffers、allocator reuse 和 host 同步优化,再决定+ 是否支持 CAGRA micro-batching 或多个并行 batch dispatch;OpenViking 的 scheduler 与 cuVS+ 官方的 Dynamic Batching 组件不是同一实现。## 主要图表diff --git a/docs/design/openviking-cuvs-integration-plan.md b/docs/design/openviking-cuvs-integration-plan.mdindex bb48122b1..01f0cb1f4 100644--- a/docs/design/openviking-cuvs-integration-plan.md+++ b/docs/design/openviking-cuvs-integration-plan.md@@ -336,6 +336,9 @@ URI/path 使用更低阈值,是因为宽路径需要 native Trie traversal 和| `algorithm` | `brute_force` | `brute_force` exact 或 `cagra` ANN || `dtype` | `float32` | GPU dataset/query dtype;可显式设为 `float16`,不改变 native CPU dtype || `max_concurrent_gpu_searches` | `1` | 单 index 同时进入 GPU search 的上限;host preflight/filter 仍可并行 |+| `micro_batching_enabled` | `false` | 可选地合并兼容的并发单 query 请求;仅支持 `brute_force`,且要求 `max_concurrent_gpu_searches=1` |+| `micro_batching_max_batch_size` | `8` | 单次 matrix-query 的最大行数,范围 1 到 8 |+| `micro_batching_max_wait_ms` | `1.0` | 等待兼容请求的 collection window,范围 0 到 100 ms;0 表示不主动等待、仅 opportunistic batching || `build_params` | `{}` | 传给 CAGRA `IndexParams` || `search_params` | `{}` | 传给 CAGRA `SearchParams` || `fallback_to_native` | `true` | sparse/hybrid 等非 cuVS dense top-k 能力使用 native |@@ -358,7 +361,11 @@ URI/path 使用更低阈值,是因为宽路径需要 native Trie traversal 和- native mutation 与 cuVS shadow mutation 由跨后端读写锁协调;filter layout 和 bitmappreflight 由 native engine 读写锁及 cuVS generation 校验保证一致性;- GPU index、dataset ownership、label mapping 和 generation 组成 immutable snapshot;-- warmed GPU search 使用不同的 cuVS resources/CUDA stream 并发执行,只短暂串行访问 host cache;+- 默认非 micro-batch 路径可使用复用的 cuVS resources/CUDA stream;启用 micro-batching 后,+ compatible warm query 可在不获取 caller 侧 device gate 的情况下入队,micro-batch worker+ 是唯一在持有 device-search gate 时执行 matrix search 的组件;+- dirty/cold snapshot、device filter cache miss 和 rebuild 仍在同一 gate 内完成 GPU preparation,+ caller 入队后立即释放 gate,不会持 gate 等待 worker;- mutation 不会破坏已被查询持有的 snapshot;同步 rebuild 仍由写锁合并为一次;- 同一 GPU 上不同 collection 的 admission/build 由进程级 coordinator 串行,避免并发超卖显存;- native fallback 不持有 GPU 锁,可以继续并发读取;@@ -390,7 +397,7 @@ URI/path 使用更低阈值,是因为宽路径需要 native Trie traversal 和| --- | --- | --- || 每次 mutation 后整体重建 | 写后首查和 build 为 O(N) | base + delta、阈值重建、后台 build、原子切换 || host shadow 保存完整 dense vectors | Store/native/GPU 之外增加 host memory | 连续 buffer、共享内存、减少 Python object |-| 高并发小 query | 单 query 无法充分占用 GPU | 动态 batching 或显式 query batching |+| 高并发小 query | 已有 opt-in micro-batching,但仍有每次 dispatch 的 device allocation、host sync 和单 worker 上限 | persistent buffers、allocator reuse、CAGRA batching 或多个并行 batch dispatch || Python/CuPy 数据面 | 对象构造、复制和同步有固定开销 | 先 profiling,再决定是否下沉 C++ cuVS C API || 重启后 lazy rebuild | 大 collection 的首次查询延迟高 | 后台预热、版本化派生 cache || native dense 与 GPU shadow 并存 | CPU 内存和写放大 | 覆盖率和回退策略稳定后再评估裁剪 |@@ -429,7 +436,9 @@ URI/path 使用更低阈值,是因为宽路径需要 native Trie traversal 和2. 已保证 in-flight query 在 rebuild 期间安全持有旧 snapshot;后台 build 期间的新请求路由仍待实现;3. 已实现完成后原子交换,并延迟到最后一个持有者退出再回收旧资源;4. 已实现可选的连续 mutation 合并与后台 rebuild;默认关闭以保留现有 auto 行为;-5. 已实现每线程 cuVS resources/CUDA stream;persistent query buffers 和 micro-batching 待评估;+5. 已实现可复用 cuVS resources/CUDA stream,以及默认关闭的 brute-force request+ micro-batching 和 warm admission fast path;persistent query/result buffers、allocator reuse、+ CAGRA batching 和多个并行 batch dispatch 待评估;6. 已补充 build queue、fallback reason、eligible count 和分阶段 latency telemetry;VRAM 指标待统一管理器。### Phase 3:容量与低精度diff --git a/docs/en/guides/07-operation-telemetry.md b/docs/en/guides/07-operation-telemetry.mdindex f71ecfb15..4595ec6ed 100644--- a/docs/en/guides/07-operation-telemetry.md+++ b/docs/en/guides/07-operation-telemetry.md@@ -189,6 +189,11 @@ coalesced into the bounded `other` bucket.| `summary.vector.cuvs.dtypes.<dtype>` | Search count by GPU dataset/query dtype, `float32` or `float16` || `summary.vector.cuvs.max_concurrent_gpu_searches` | Maximum configured per-index in-flight GPU search limit observed || `summary.vector.cuvs.auto_mode_searches` | Number of searches with automatic CPU/GPU routing enabled |+| `summary.vector.cuvs.micro_batching_searches` | Number of searches that used the opt-in OpenViking micro-batch scheduler |+| `summary.vector.cuvs.micro_batched_searches` | Number of those searches dispatched with more than one query row |+| `summary.vector.cuvs.micro_batching_warm_fast_path_searches` | Number of micro-batching-scheduler searches enqueued from a clean current snapshot without caller-side device-gate admission |+| `summary.vector.cuvs.batch_size_max` | Maximum query-row count observed in one shared cuVS call |+| `summary.vector.cuvs.searches_by_batch_size.<size>` | Search count by bounded batch size from 1 through 8 || `summary.vector.cuvs.routes.<reason>` | Search count by route, such as `cuvs`, `native_filter_threshold`, `native_rebuild_pending`, or `native_memory_budget` || `summary.vector.cuvs.filter_kinds.<kind>` | Search count by low-cardinality filter class: `none`, `scalar`, or `path` || `summary.vector.cuvs.filter_cache_hits` | Number of searches that reused a prepared or preflight filter |@@ -200,9 +205,22 @@ coalesced into the bounded `other` bucket.| `summary.vector.cuvs.memory.estimated_peak_bytes_max` | Maximum estimated peak bytes used for auto-build admission || `summary.vector.cuvs.memory.free_bytes_min` | Minimum free device bytes observed inside the per-GPU admission coordinator || `summary.vector.cuvs.memory.usable_bytes_min` | Minimum free bytes observed after applying the configured reserve |-| `summary.vector.cuvs.timings_ms.<stage>.sum` | Sum across searches for `total`, `preflight`, `queue`, `build`, `filter_prepare`, `gpu_search`, or `native_search` |+| `summary.vector.cuvs.timings_ms.<stage>.sum` | Sum across searches for `total`, `preflight`, `queue`, `gpu_gate_queue`, `build`, `filter_prepare`, `batch_wait`, `gpu_search`, or `native_search` || `summary.vector.cuvs.timings_ms.<stage>.max` | Maximum single-search time for the same stage |+For a shared micro-batch, `gpu_search` is the cuVS call's service latency from+each member request's perspective. Its duration is therefore recorded once per+request: `gpu_search.sum` is useful as aggregate request latency, but it is not+GPU busy time and can be approximately batch-size times the physical call+duration. `batch_wait` is the request's time from scheduler enqueue to GPU+dispatch, including worker scheduling and any wait behind current device work.+`gpu_gate_queue` is caller-side time waiting to enter the serialized device+gate before gated rebuild/filter preparation, enqueue, or a non-batched GPU+search. A warm-fast-path request bypasses that caller admission, so its+`gpu_gate_queue` is zero; its worker-side wait remains part of `batch_wait`.+The broader `queue` stage includes both forms of request-visible queueing when+they apply.+### `summary.resource`This group appears on resource ingestion operations such as `resources.add_resource`.diff --git a/docs/en/guides/16-cuvs.md b/docs/en/guides/16-cuvs.mdindex 6e54960fe..baffec25c 100644--- a/docs/en/guides/16-cuvs.md+++ b/docs/en/guides/16-cuvs.md@@ -27,6 +27,7 @@ Start with exact brute-force search:"algorithm": "brute_force","dtype": "float32","max_concurrent_gpu_searches": 1,+ "micro_batching_enabled": false,"fallback_to_native": true,"filter_cache_size": 16}@@ -167,16 +168,16 @@ design because OpenViking uses a per-vector scale that cuVS brute-force doesnot accept directly. CAGRA int8 or PQ compression must likewise be evaluatedas approximate modes with an explicit recall/latency/memory frontier.-The integration uses immutable GPU snapshots. Warmed searches use per-thread-cuVS resources/CUDA streams, while mutation and snapshot commit use a-cross-backend writer lock. Host-side filter and snapshot work can proceed in-parallel, but `max_concurrent_gpu_searches` defaults to 1 because concurrent-single-query brute-force kernels can contend for memory bandwidth and reduce-throughput. Increase it only after measuring the target GPU and workload. By-default, the first query after an upsert or delete rebuilds synchronously;-optional background rebuild changes dirty queries to native fallback until the-new snapshot is ready. On each rebuild it registers the cuVS label order with-the native engine once.+The integration uses immutable GPU snapshots and reusable cuVS resources/CUDA+streams, while mutation and snapshot commit use a cross-backend writer lock.+Host-side filter and snapshot work can proceed in parallel, but+`max_concurrent_gpu_searches` defaults to 1 because concurrent single-query+brute-force kernels can contend for memory bandwidth and reduce throughput.+Increase it only after measuring the target GPU and workload. By default, the+first query after an upsert or delete rebuilds synchronously; optional+background rebuild changes dirty queries to native fallback until the new+snapshot is ready. On each rebuild it registers the cuVS label order with the+native engine once.The first use of a scalar or URI filter then reuses OpenViking's nativescalar/path index and projects its bitmap into cuVS row order; it does not scanall host-side records in Python. `filter_cache_size` retains the resulting@@ -186,6 +187,68 @@ Different unseen filters can use the native engine's shared-read path inparallel, while cached native routing decisions go directly to the nativeindex. A record-generation check prevents a result computed across a mutationfrom entering the route cache.++### Opt-in request micro-batching++Exact brute-force search can coalesce compatible concurrent requests into one+cuVS matrix-query call:++```json+{+ "storage": {+ "vectordb": {+ "backend": "cuvs",+ "cuvs": {+ "algorithm": "brute_force",+ "max_concurrent_gpu_searches": 1,+ "micro_batching_enabled": true,+ "micro_batching_max_batch_size": 8,+ "micro_batching_max_wait_ms": 1.0+ }+ }+ }+}+```++The scheduler batches only requests using the same immutable GPU snapshot,+prepared filter, and effective top-k. Each result row is mapped back to its+original request, so scalar/path filtering and result limits retain their+existing semantics.++A request over a clean, current-generation snapshot can take the warm+admission fast path when it has no filter or its prepared device filter is+already cached. That path pins the immutable snapshot and filter, then enqueues+without the caller acquiring the device-search gate. Dirty, cold, or stale+snapshots, device-filter cache misses or evictions, rebuilds, and device-filter+materialization use the gated preparation path. After preparation, the caller+enqueues and releases the gate before waiting. The micro-batch worker is the+only component that executes matrix search while holding the device-search+gate, so a caller never holds that gate while waiting for the worker.++The collection window is a latency-throughput tradeoff. It bounds the+scheduler's deliberate collection delay to at most the configured value from+the oldest compatible request; it does not bound total enqueue-to-dispatch+latency. Worker scheduling, a preceding GPU call, or gated device preparation+can make dispatch occur later. Under sufficient concurrency up to the+configured maximum number of queries share one GPU call.++The options are validated as follows:++- `micro_batching_max_batch_size` must be from 1 through 8.+- `micro_batching_max_wait_ms` must be from 0 through 100 ms. A value of `0`+ disables deliberate collection delay but can still opportunistically combine+ compatible requests already queued together.+- Micro-batching supports only `algorithm: "brute_force"` and requires+ `max_concurrent_gpu_searches: 1`.++This OpenViking micro-batcher is disabled by default and is distinct from the+cuVS feature named Dynamic Batching. Its first version supports+exact brute-force; CAGRA and multiple concurrent batch dispatches are rejected+until separately validated. Auto mode can use the same options, but requests+routed to the native CPU path never enter the GPU batch queue. Near-tie+neighbor ordering may differ between single-row and matrix-query calls;+validate set overlap and scores when tuning for a workload.+Sparse/hybrid queries fall back to OpenViking's native local index when`fallback_to_native` is enabled. The canonical vectors remain in the localstore and repopulate cuVS after restart.diff --git a/docs/zh/guides/07-operation-telemetry.md b/docs/zh/guides/07-operation-telemetry.mdindex 9633d6a95..5a5e4acb5 100644--- a/docs/zh/guides/07-operation-telemetry.md+++ b/docs/zh/guides/07-operation-telemetry.md@@ -187,6 +187,11 @@ CPU/GPU 路由和分阶段耗时;并发 query 的完成顺序不会改变结| `summary.vector.cuvs.dtypes.<dtype>` | 按 GPU dataset/query dtype 统计的 search 数:`float32` 或 `float16` || `summary.vector.cuvs.max_concurrent_gpu_searches` | 观测到的单 index in-flight GPU search 配置上限最大值 || `summary.vector.cuvs.auto_mode_searches` | 启用自动 CPU/GPU 路由的 search 数量 |+| `summary.vector.cuvs.micro_batching_searches` | 使用 OpenViking 可选 micro-batch scheduler 的 search 数量 |+| `summary.vector.cuvs.micro_batched_searches` | 其中以多于一行 query 共同 dispatch 的 search 数量 |+| `summary.vector.cuvs.micro_batching_warm_fast_path_searches` | 使用 micro-batch scheduler、从 clean current snapshot 入队且未经过 caller 侧 device-gate admission 的 search 数量 |+| `summary.vector.cuvs.batch_size_max` | 单次共享 cuVS call 观测到的最大 query 行数 |+| `summary.vector.cuvs.searches_by_batch_size.<size>` | 按 1 到 8 的有界 batch size 统计 search 数量 || `summary.vector.cuvs.routes.<reason>` | 按路由原因统计的 search 数,例如 `cuvs`、`native_filter_threshold`、`native_rebuild_pending` 或 `native_memory_budget` || `summary.vector.cuvs.filter_kinds.<kind>` | 按低基数 filter 类型统计的 search 数:`none`、`scalar` 或 `path` || `summary.vector.cuvs.filter_cache_hits` | 复用 prepared/preflight filter 的 search 数量 |@@ -198,9 +203,18 @@ CPU/GPU 路由和分阶段耗时;并发 query 的完成顺序不会改变结| `summary.vector.cuvs.memory.estimated_peak_bytes_max` | auto build 准入使用的最大峰值显存估算 || `summary.vector.cuvs.memory.free_bytes_min` | 单 GPU 准入协调器内观测到的最小空闲显存 || `summary.vector.cuvs.memory.usable_bytes_min` | 扣除配置 reserve 后观测到的最小可用显存 |-| `summary.vector.cuvs.timings_ms.<stage>.sum` | `total`、`preflight`、`queue`、`build`、`filter_prepare`、`gpu_search` 或 `native_search` 阶段跨 search 的耗时总和 |+| `summary.vector.cuvs.timings_ms.<stage>.sum` | `total`、`preflight`、`queue`、`gpu_gate_queue`、`build`、`filter_prepare`、`batch_wait`、`gpu_search` 或 `native_search` 阶段跨 search 的耗时总和 || `summary.vector.cuvs.timings_ms.<stage>.max` | 同一阶段单次 search 的最大耗时 |+对于共享 micro-batch,`gpu_search` 表示每个成员请求所感知的同一次 cuVS call+service latency,因此会按请求各记录一次。`gpu_search.sum` 适合解释请求侧累计延迟,+但不等同于 GPU busy time;它可能约为物理 call 时长乘以 batch size。`batch_wait`+表示请求从进入 scheduler 到 GPU dispatch 的等待时间,包括 worker 调度和等待当前 device+work 完成的时间。`gpu_gate_queue` 表示 caller 在 gated rebuild/filter preparation、入队或+非 batch GPU search 前,等待进入串行 device gate 的时间。warm fast path 会跳过这次 caller+admission,所以其 `gpu_gate_queue` 为零;worker 侧等待仍计入 `batch_wait`。更宽泛的 `queue`+阶段会在适用时包含这两类请求可感知的排队时间。+### `summary.resource`这个分组常见于 `resources.add_resource` 这类资源导入操作。diff --git a/docs/zh/guides/16-cuvs.md b/docs/zh/guides/16-cuvs.mdindex a54de4c43..213b546d4 100644--- a/docs/zh/guides/16-cuvs.md+++ b/docs/zh/guides/16-cuvs.md@@ -41,6 +41,7 @@ headers;即使宿主机已有 CUDA driver、但没有完整 toolkit,也建"algorithm": "brute_force","dtype": "float32","max_concurrent_gpu_searches": 1,+ "micro_batching_enabled": false,"fallback_to_native": true,"filter_cache_size": 16}@@ -186,11 +187,62 @@ truth 报告 Recall@K。与 native 兼容的 int8 仍需单独设计,因为 Op逐向量 scale,而 cuVS brute-force 不能直接接收这种 scaled-int8 表示。CAGRAint8 或 PQ compression 也应作为近似模式,单独报告 recall/latency/memory frontier。-集成使用 immutable GPU snapshot 和每线程 cuVS resource/CUDA stream。host 侧+集成使用 immutable GPU snapshot 和可复用的 cuVS resource/CUDA stream。host 侧filter 与 snapshot 工作可以并行,但 `max_concurrent_gpu_searches` 默认是 1:单 query brute-force 通常受显存带宽限制,并发 kernel 可能互相争抢带宽、反而降低吞吐。只有在目标 GPU 与真实 workload 上测得收益后,才建议显式调大该值。+### 可选的请求微批处理++精确 brute-force 路径可以把兼容的并发请求合并为一次 cuVS matrix-query 调用:++```json+{+ "storage": {+ "vectordb": {+ "backend": "cuvs",+ "cuvs": {+ "algorithm": "brute_force",+ "max_concurrent_gpu_searches": 1,+ "micro_batching_enabled": true,+ "micro_batching_max_batch_size": 8,+ "micro_batching_max_wait_ms": 1.0+ }+ }+ }+}+```++scheduler 只会合并使用同一个 immutable GPU snapshot、同一个 prepared filter、+同一个实际 top-k 的请求;GPU 返回的每一行会映射回原请求,因此标量/路径过滤和+结果条数语义不变。++当 immutable snapshot clean、属于当前 generation,且请求没有 filter 或命中已准备好的+device filter cache 时,可走 warm admission fast path。该路径会 pin snapshot/filter,+并在 caller 不获取 device-search gate 的情况下直接入队。dirty、cold 或 stale snapshot,+device filter cache miss/eviction、rebuild 和 device filter materialization 仍走 gated+preparation。准备完成后,caller 先入队并释放 gate,再等待结果;只有 micro-batch worker+会在持有 device-search gate 时执行 matrix search,所以 caller 不会持 gate 等待 worker。++collection window 是延迟与吞吐的权衡。它只限制 scheduler 为收集兼容请求而主动等待的+时间:从最早的 compatible request 起最多主动等待配置值;它不是 enqueue-to-dispatch+latency 上限。worker 调度、前一个 GPU call 或 gated device preparation 都可能使实际+dispatch 更晚。并发充足时,最多由配置上限数量的 query 共用一次 GPU call。++参数约束如下:++- `micro_batching_max_batch_size` 范围为 1 到 8;+- `micro_batching_max_wait_ms` 范围为 0 到 100 ms;设为 `0` 表示不主动等待,但仍可+ opportunistically 合并已经同时在队列中的兼容请求;+- micro-batching 仅支持 `algorithm: "brute_force"`,并要求+ `max_concurrent_gpu_searches: 1`。++该能力默认关闭,是 OpenViking 自己的 micro-batcher,不等同于 cuVS 官方名为+Dynamic Batching 的组件。首版只支持 exact brute-force;CAGRA 和并发 dispatch 多个+batch 会在独立验证后再开放。Auto 模式也可使用这些选项,但被路由到原生 CPU 的请求+不会进入 GPU batch queue。single-row 与 matrix-query 在近似并列分数处可能有顺序+差异,调参时应同时验证结果集合重合度和 score。+## 最小功能验证仓库提供的 smoke test 不依赖 embedding 或 VLM 服务:@@ -260,7 +312,7 @@ collection.close()- local 集成通过 native scalar/path index 生成 prefilter,因此继承原生 DSL、`date_time`、`geo_point` 和 path depth 的过滤语义,而不是在 Python 重复实现。- 每次 GPU rebuild 会向 native engine 注册一次 cuVS label 顺序。新过滤条件直接复用 native scalar/path index 的 bitmap,再投影为 cuVS row bitset,不再用 Python 扫描所有 host-side records。- `filter_cache_size` 会保留最近使用的 GPU bitset 或 native 路由决策,并在数据更新时失效;auto 模式在进入 cuVS search 前预判候选数,不同的首次过滤条件可通过 native engine 的共享读路径并行计算,命中已缓存的 native 路由时则直接进入 native index。generation 校验会阻止跨 mutation 计算出的旧结果写入路由缓存。-- GPU index 使用 immutable snapshot;warmed search 通过不同的 cuVS resources/CUDA stream 并发执行,mutation 和 snapshot commit 使用跨后端写锁。+- GPU index 使用 immutable snapshot 和可复用的 cuVS resources/CUDA stream;默认关闭的 micro-batching 可让 compatible warm request 绕过 caller 侧 gate 入队,并由唯一持有 device-search gate 执行 matrix search 的 worker 合批。mutation 和 snapshot commit 使用跨后端写锁。- 默认情况下,每次 upsert/delete 后仍由下一次查询同步重建;开启 `auto_background_rebuild` 后,dirty 期间查询走 native,连续写被合并为后台重建。- cuVS 索引不作为权威持久化数据;进程重启时会从 OpenViking 本地 store 重建,因此不受 cuVS 跨版本序列化格式变化影响。- `brute_force` 适合功能对齐和 ground truth;CAGRA 的 graph/search 参数需要在后续结合召回率、QPS、延迟和显存进行调优。diff --git a/openviking/storage/vectordb/index/cuvs_index.py b/openviking/storage/vectordb/index/cuvs_index.pyindex 303b1a15c..2e202af54 100644--- a/openviking/storage/vectordb/index/cuvs_index.py+++ b/openviking/storage/vectordb/index/cuvs_index.py@@ -26,6 +26,7 @@ import timeimport tracebackfrom array import arrayfrom collections import OrderedDict+from concurrent.futures import Futurefrom contextlib import nullcontextfrom dataclasses import dataclass, fieldfrom typing import (@@ -99,6 +100,9 @@ class CuVSSearchTelemetry:auto_mode: booldtype: str = "float32"max_concurrent_gpu_searches: int = 1+ micro_batching_enabled: bool = False+ micro_batching_warm_fast_path: bool = False+ batch_size: int = 1route_reason: str = "pending"filter_kind: str = "none"filter_cache_hit: bool = False@@ -119,6 +123,7 @@ class CuVSSearchTelemetry:build_ms: float = 0.0filter_prepare_ms: float = 0.0gpu_search_ms: float = 0.0+ batch_wait_ms: float = 0.0native_search_ms: float = 0.0def as_dict(self) -> Dict[str, Any]:@@ -127,6 +132,9 @@ class CuVSSearchTelemetry:"auto_mode": self.auto_mode,"dtype": self.dtype,"max_concurrent_gpu_searches": self.max_concurrent_gpu_searches,+ "micro_batching_enabled": self.micro_batching_enabled,+ "micro_batching_warm_fast_path": self.micro_batching_warm_fast_path,+ "batch_size": self.batch_size,"route_reason": self.route_reason,"filter_kind": self.filter_kind,"filter_cache_hit": self.filter_cache_hit,@@ -147,6 +155,7 @@ class CuVSSearchTelemetry:"build_ms": round(self.build_ms, 3),"filter_prepare_ms": round(self.filter_prepare_ms, 3),"gpu_search_ms": round(self.gpu_search_ms, 3),+ "batch_wait_ms": round(self.batch_wait_ms, 3),"native_search_ms": round(self.native_search_ms, 3),}@@ -652,11 +661,32 @@ class _CuVSRuntime:limit: int,mask: Optional[Any],) -> Tuple[List[int], List[float]]:+ batch_neighbors, batch_distances = self.search_batch(+ runtime_index,+ [query],+ limit,+ mask,+ )+ return batch_neighbors[0], batch_distances[0]++ def search_batch(+ self,+ runtime_index: "_CuVSRuntimeIndex",+ queries: Sequence[Sequence[float]],+ limit: int,+ mask: Optional[Any],+ ) -> Tuple[List[List[int]], List[List[float]]]:+ """Run compatible query rows in one cuVS call and preserve row order."""++ if not queries:+ return [], []with self.device_scope():- queries = None+ device_queries = Noneprefilter = Nonedistances = Noneneighbors = None+ host_distances = None+ host_neighbors = Noneresources = Noneresource_kwargs = Noneresource_reusable = False@@ -670,7 +700,8 @@ class _CuVSRuntime:# boundary inside this admitted search.resources = self._acquire_resources()resource_kwargs = {"resources": resources}- queries = self.cp.asarray([query], dtype=self.device_dtype)+ query_count = len(queries)+ device_queries = self.cp.asarray(queries, dtype=self.device_dtype)if mask is None:prefilter = Noneelif isinstance(mask, self.cp.ndarray) and mask.dtype == self.cp.uint32:@@ -680,7 +711,7 @@ class _CuVSRuntime:if self.algorithm == "brute_force":distances, neighbors = self.brute_force.search(index,- queries,+ device_queries,limit,prefilter=prefilter,**resource_kwargs,@@ -694,17 +725,23 @@ class _CuVSRuntime:distances, neighbors = self.cagra.search(params,index,- queries,+ device_queries,limit,filter=prefilter,**resource_kwargs,)resources.sync()- host_neighbors = self.cp.asnumpy(neighbors)[0].tolist()- host_distances = self.cp.asnumpy(distances)[0].tolist()+ host_neighbors = self.cp.asnumpy(neighbors)+ host_distances = self.cp.asnumpy(distances)result = (- [int(item) for item in host_neighbors],- [float(item) for item in host_distances],+ [+ [int(item) for item in host_neighbors[row].tolist()]+ for row in range(query_count)+ ],+ [+ [float(item) for item in host_distances[row].tolist()]+ for row in range(query_count)+ ],)# Reuse only after the complete call, including host result# materialization, succeeds. Any exception discards a@@ -717,10 +754,12 @@ class _CuVSRuntime:finally:# Search temporaries otherwise outlive Device.__exit__ as frame# locals and may free allocations on the worker's prior device.- queries = None+ device_queries = Noneprefilter = Nonedistances = Noneneighbors = None+ host_distances = None+ host_neighbors = Nonetry:if resources is not None:self._return_resources(resources, reusable=resource_reusable)@@ -759,6 +798,31 @@ class _CuVSIndexSnapshot:generation: int+@dataclass+class _CuVSMicroBatchRequest:+ """One admitted search waiting for a compatible matrix-query dispatch."""++ snapshot: Optional[_CuVSIndexSnapshot]+ query: Tuple[float, ...]+ result_limit: int+ mask: Optional[Any]+ filter_identity: Union[str, int]+ telemetry: Optional[CuVSSearchTelemetry]+ enqueued_at: float+ future: Future[Tuple[List[int], List[float]]]+ warm_fast_path: bool = False+ released: bool = False++ @property+ def key(self) -> Tuple[int, Union[str, int], int]:+ snapshot = self.snapshot+ return (+ id(snapshot),+ self.filter_identity,+ self.result_limit,+ )++@dataclassclass _CuVSBuildCandidate:runtime_index: Any@@ -869,6 +933,21 @@ class CuVSDenseIndex:self.max_concurrent_gpu_searches = int(config.get("max_concurrent_gpu_searches", 1))if self.max_concurrent_gpu_searches < 1:raise ValueError("cuVS max_concurrent_gpu_searches must be at least 1")+ self.micro_batching_enabled = bool(config.get("micro_batching_enabled", False))+ self.micro_batching_max_batch_size = int(config.get("micro_batching_max_batch_size", 8))+ if not 1 <= self.micro_batching_max_batch_size <= 8:+ raise ValueError("cuVS micro_batching_max_batch_size must be between 1 and 8")+ self.micro_batching_max_wait_ms = float(config.get("micro_batching_max_wait_ms", 1.0))+ if (+ not math.isfinite(self.micro_batching_max_wait_ms)+ or self.micro_batching_max_wait_ms < 0.0+ or self.micro_batching_max_wait_ms > 100.0+ ):+ raise ValueError("cuVS micro_batching_max_wait_ms must be between 0 and 100")+ if self.micro_batching_enabled and self.algorithm != "brute_force":+ raise ValueError("cuVS micro-batching currently supports algorithm='brute_force' only")+ if self.micro_batching_enabled and self.max_concurrent_gpu_searches != 1:+ raise ValueError("cuVS micro-batching currently requires max_concurrent_gpu_searches=1")self.auto_memory_reserve_bytes = (int(config.get("auto_memory_reserve_mb", 1024)) * 1024 * 1024)@@ -906,6 +985,10 @@ class CuVSDenseIndex:set_max_concurrent_searches = getattr(self._runtime, "set_max_concurrent_searches", None)if set_max_concurrent_searches is not None:set_max_concurrent_searches(self.max_concurrent_gpu_searches)+ if self.micro_batching_enabled and not callable(+ getattr(self._runtime, "search_batch", None)+ ):+ raise ValueError("The configured cuVS runtime does not support batched search")self._records: Dict[int, _Record] = {}self._snapshot: Optional[_CuVSIndexSnapshot] = Noneself._dirty = True@@ -919,13 +1002,31 @@ class CuVSDenseIndex:self._filter_layout_lock = threading.Lock()self._records_generation = 0self._filter_layout_generation = -1+ self._closing = Falseself._closed = Falseself._close_complete = False+ self._micro_batch_condition = threading.Condition(threading.Lock())+ self._micro_batch_pending: OrderedDict[+ Tuple[int, Union[str, int], int], List[_CuVSMicroBatchRequest]+ ] = OrderedDict()+ self._micro_batch_warm_lookahead = 0+ self._micro_batch_draining = False+ self._micro_batch_stop = False+ self._micro_batch_worker_failure: Optional[BaseException] = None+ self._micro_batch_worker: Optional[threading.Thread] = None+ if self.micro_batching_enabled:+ self._micro_batch_worker = threading.Thread(+ target=self._micro_batch_worker_main,+ name="openviking-cuvs-microbatch",+ daemon=True,+ )+ self._micro_batch_worker.start()logger.info(- "Initialized cuVS dense index: algorithm=%s metric=%s dimension=%d",+ "Initialized cuVS dense index: algorithm=%s metric=%s dimension=%d micro_batching=%s",self.algorithm,self._metric,self.dimension,+ self.micro_batching_enabled,)def _runtime_device_scope(self):@@ -1678,6 +1779,387 @@ class CuVSDenseIndex:if self.commit_rebuild(candidate, native_filter_layout_registrar):return+ def _enqueue_micro_batch(+ self,+ *,+ snapshot: _CuVSIndexSnapshot,+ query: Tuple[float, ...],+ result_limit: int,+ mask: Optional[Any],+ filter_identity: Union[str, int],+ telemetry: Optional[CuVSSearchTelemetry],+ warm_fast_path: bool = False,+ ) -> Optional[_CuVSMicroBatchRequest]:+ request = _CuVSMicroBatchRequest(+ snapshot=snapshot,+ query=query,+ result_limit=result_limit,+ mask=mask,+ filter_identity=filter_identity,+ telemetry=telemetry,+ enqueued_at=time.perf_counter(),+ future=Future(),+ warm_fast_path=warm_fast_path,+ )+ with self._micro_batch_condition:+ if self._micro_batch_worker_failure is not None:+ raise RuntimeError("The cuVS micro-batch worker is unavailable") from (+ self._micro_batch_worker_failure+ )+ if self._micro_batch_stop:+ raise RuntimeError("The cuVS micro-batch worker is closed")+ if (+ warm_fast_path+ and self._micro_batch_warm_lookahead >= self.micro_batching_max_batch_size+ ):+ return None+ self._micro_batch_pending.setdefault(request.key, []).append(request)+ if warm_fast_path:+ self._micro_batch_warm_lookahead += 1+ self._micro_batch_condition.notify()+ return request++ def _try_enqueue_warm_micro_batch(+ self,+ *,+ query: Tuple[float, ...],+ limit: int,+ filters: Optional[Mapping[str, Any]],+ prepared_filter: Optional[_PreparedHostFilter],+ telemetry: Optional[CuVSSearchTelemetry],+ ) -> Optional[_CuVSMicroBatchRequest]:+ """Queue one warm request without taking the caller-side GPU gate.++ The worker remains the only owner of the device-search permit. This+ path merely pins an immutable, current-generation snapshot and an+ already materialized device filter while the index lock prevents+ mutation/close from racing the borrow. A single batch worth of warm+ requests may look ahead; overflow and every cold/stale case use the+ established gated admission path.+ """++ # LocalIndex normalizes an omitted public-API filter from None to {},+ # so both representations are the canonical no-filter case here.+ no_filter = not filters+ if no_filter:+ cache_key: Optional[str] = None+ else:+ cache_key = (+ prepared_filter.cache_key+ if prepared_filter is not None+ else self._filter_cache_key(filters)+ )+ if cache_key is None:+ return None++ with self._lock:+ if self._closing or self._closed:+ raise RuntimeError("cuVS dense index is closed")+ generation = self._records_generation+ if prepared_filter is not None and prepared_filter.generation != generation:+ # Preserve the established admitted retry path for stale host+ # work instead of creating a second, fast-path-only retry+ # protocol.+ return None+ snapshot = self._snapshot+ if self._dirty or snapshot is None or snapshot.generation != generation:+ return None++ mask: Optional[Any] = None+ filter_identity: Union[str, int] = "no-filter"+ if no_filter:+ result_limit = min(limit, len(snapshot.labels))+ else:+ cached_filter = self._get_cached_filter(cache_key)+ if cached_filter is None:+ # Host metadata can outlive a size-bounded device LRU+ # entry. Preserve the existing one-shot in-gate recovery.+ if (+ telemetry is not None+ and prepared_filter is not None+ and prepared_filter.cached_metadata is not None+ ):+ telemetry.filter_cache_hit = False+ telemetry.filter_cache_eviction_fallback = True+ return None+ if (+ cached_filter.eligible_count == 0+ or cached_filter.route_native+ or cached_filter.prepared is None+ ):+ return None+ mask = cached_filter.prepared+ filter_identity = cache_key+ result_limit = min(limit, cached_filter.eligible_count)+ if telemetry is not None:+ telemetry.filter_cache_hit = True+ telemetry.eligible_count = cached_filter.eligible_count+ telemetry.filter_words_packed = cached_filter.filter_words_packed++ if telemetry is not None:+ telemetry.filter_kind = (+ "path"+ if filters and _filter_uses_field_type(filters, self.field_types, "path")+ else "scalar"+ if filters+ else "none"+ )+ telemetry.records_generation = generation+ telemetry.index_size = len(self._records)++ request = self._enqueue_micro_batch(+ snapshot=snapshot,+ query=query,+ result_limit=result_limit,+ mask=mask,+ filter_identity=filter_identity,+ telemetry=telemetry,+ warm_fast_path=True,+ )+ if request is None:+ return None+ self._active_searches += 1+ if telemetry is not None:+ telemetry.micro_batching_warm_fast_path = True+ return request++ def _release_micro_batch_warm_lookahead(+ self,+ requests: Sequence[_CuVSMicroBatchRequest],+ ) -> None:+ with self._micro_batch_condition:+ released = 0+ for request in requests:+ if request.warm_fast_path:+ request.warm_fast_path = False+ released += 1+ if not released:+ return+ if released > self._micro_batch_warm_lookahead:+ raise RuntimeError("cuVS micro-batch warm look-ahead accounting underflow")+ self._micro_batch_warm_lookahead -= released+ self._micro_batch_condition.notify_all()++ @staticmethod+ def _await_micro_batch_request(+ request: _CuVSMicroBatchRequest,+ ) -> Tuple[List[int], List[float]]:+ try:+ return request.future.result()+ except BaseException:+ request.future.cancel()+ raise++ def _next_micro_batch(self) -> Optional[List[_CuVSMicroBatchRequest]]:+ wait_seconds = self.micro_batching_max_wait_ms / 1000.0+ with self._micro_batch_condition:+ while True:+ if self._micro_batch_stop and not self._micro_batch_pending:+ return None+ if not self._micro_batch_pending:+ self._micro_batch_condition.wait()+ continue++ now = time.perf_counter()+ selected_key: Optional[Tuple[int, Union[str, int], int]] = None+ earliest_deadline: Optional[float] = None+ for key, requests in self._micro_batch_pending.items():+ deadline = requests[0].enqueued_at + wait_seconds+ if (+ len(requests) >= self.micro_batching_max_batch_size+ or now >= deadline+ or self._micro_batch_draining+ or self._micro_batch_stop+ ):+ selected_key = key+ break+ if earliest_deadline is None or deadline < earliest_deadline:+ earliest_deadline = deadline++ if selected_key is None:+ timeout = max((earliest_deadline or now) - now, 0.0)+ self._micro_batch_condition.wait(timeout)+ continue++ requests = self._micro_batch_pending[selected_key]+ batch = requests[: self.micro_batching_max_batch_size]+ del requests[: len(batch)]+ if not requests:+ del self._micro_batch_pending[selected_key]+ return batch++ @staticmethod+ def _copy_micro_batch_exception(exc: BaseException) -> BaseException:+ try:+ return type(exc)(*exc.args)+ except Exception:+ return RuntimeError(f"cuVS micro-batch search failed: {exc}")++ def _complete_micro_batch(+ self,+ requests: Sequence[_CuVSMicroBatchRequest],+ *,+ results: Optional[Sequence[Tuple[List[int], List[float]]]] = None,+ error: Optional[BaseException] = None,+ ) -> None:+ live_requests = [request for request in requests if not request.released]+ if not live_requests:+ return+ # Drop the last references to snapshot-owned CUDA allocations on the+ # captured device before close/rebuild can observe an idle index.+ release_error: Optional[BaseException] = None+ try:+ with self._runtime_device_scope():+ for request in live_requests:+ request.mask = None+ request.snapshot = None+ request.released = True+ except BaseException as exc:+ release_error = exc+ for request in live_requests:+ request.mask = None+ request.snapshot = None+ request.released = True+ with self._lock:+ self._active_searches -= len(live_requests)+ if self._active_searches == 0:+ self._idle_condition.notify_all()++ for index, request in enumerate(live_requests):+ if request.future.cancelled():+ continue+ completion_error = error or release_error+ if completion_error is not None:+ request.future.set_exception(self._copy_micro_batch_exception(completion_error))+ else:+ assert results is not None+ request.future.set_result(results[index])++ def _dispatch_micro_batch(self, requests: List[_CuVSMicroBatchRequest]) -> None:+ runnable = [+ request for request in requests if request.future.set_running_or_notify_cancel()+ ]+ runnable_ids = {id(request) for request in runnable}+ cancelled = [request for request in requests if id(request) not in runnable_ids]+ if cancelled:+ self._release_micro_batch_warm_lookahead(cancelled)+ self._complete_micro_batch(cancelled, results=[([], [])] * len(cancelled))+ if not runnable:+ return++ first = runnable[0]+ snapshot = first.snapshot+ request_snapshot: Optional[_CuVSIndexSnapshot] = None+ if snapshot is None:+ self._release_micro_batch_warm_lookahead(runnable)+ self._complete_micro_batch(+ runnable,+ error=RuntimeError("cuVS micro-batch snapshot was released before dispatch"),+ )+ return++ try:+ self._gpu_search_gate.acquire()+ except BaseException as exc:+ self._release_micro_batch_warm_lookahead(runnable)+ self._complete_micro_batch(runnable, error=exc)+ raise+ try:+ # Once this batch owns the gate it is the current device work, so+ # one subsequent warm batch may begin filling behind it.+ self._release_micro_batch_warm_lookahead(runnable)+ except BaseException as exc:+ self._gpu_search_gate.release()+ self._complete_micro_batch(runnable, error=exc)+ raise+ gpu_started = time.perf_counter()+ results: Optional[List[Tuple[List[int], List[float]]]] = None+ error: Optional[BaseException] = None+ try:+ for request in runnable:+ request_wait_ms = (gpu_started - request.enqueued_at) * 1000.0+ if request.telemetry is not None:+ request.telemetry.micro_batching_enabled = True+ request.telemetry.batch_size = len(runnable)+ request.telemetry.batch_wait_ms += request_wait_ms+ request.telemetry.queue_ms += request_wait_ms+ offsets_by_query, distances_by_query = self._runtime.search_batch(+ snapshot.runtime_index,+ [request.query for request in runnable],+ first.result_limit,+ first.mask,+ )+ if len(offsets_by_query) != len(runnable) or len(distances_by_query) != len(runnable):+ raise RuntimeError("cuVS returned a different number of rows than requested")++ results = []+ for request, offsets, distances in zip(+ runnable,+ offsets_by_query,+ distances_by_query,+ strict=True,+ ):+ request_snapshot = request.snapshot+ if request_snapshot is None:+ raise RuntimeError("cuVS micro-batch snapshot was released during dispatch")+ labels: List[int] = []+ scores: List[float] = []+ for offset, distance in zip(offsets, distances, strict=True):+ if offset < 0 or offset >= len(request_snapshot.labels):+ continue+ labels.append(request_snapshot.labels[offset])+ scores.append(1.0 - distance if self.distance == "l2" else distance)+ results.append((labels, scores))+ except BaseException as exc:+ error = exc+ finally:+ gpu_search_ms = (time.perf_counter() - gpu_started) * 1000.0+ for request in runnable:+ if request.telemetry is not None:+ request.telemetry.gpu_search_ms += gpu_search_ms+ self._gpu_search_gate.release()+ # Requests still own these objects until _complete_micro_batch(),+ # so clearing worker locals here cannot release them prematurely.+ # It does ensure active=0 is never published while this frame keeps+ # an extra CUDA-owned snapshot reference alive.+ try:+ with self._runtime_device_scope():+ request_snapshot = None+ snapshot = None+ except BaseException as cleanup_exc:+ if error is None:+ error = cleanup_exc+ request_snapshot = None+ snapshot = None+ self._complete_micro_batch(runnable, results=results, error=error)+ if error is not None and not isinstance(error, Exception):+ raise error++ def _micro_batch_worker_main(self) -> None:+ try:+ while True:+ requests = self._next_micro_batch()+ if requests is None:+ return+ self._dispatch_micro_batch(requests)+ except BaseException as exc:+ logger.exception("The cuVS micro-batch worker stopped unexpectedly")+ with self._micro_batch_condition:+ self._micro_batch_worker_failure = exc+ pending = [+ request+ for requests in self._micro_batch_pending.values()+ for request in requests+ ]+ for request in pending:+ request.warm_fast_path = False+ self._micro_batch_warm_lookahead = 0+ self._micro_batch_pending.clear()+ self._micro_batch_stop = True+ self._micro_batch_condition.notify_all()+ if pending:+ self._complete_micro_batch(pending, error=exc)+def search(self,query_vector: Sequence[float],@@ -1712,7 +2194,7 @@ class CuVSDenseIndex:) * 1000.0if prepared_filter is None:with self._lock:- if self._closed:+ if self._closing or self._closed:raise RuntimeError("cuVS dense index is closed")# A mutation invalidated the registered native layout while# the resolver ran. Retry before consuming GPU admission.@@ -1736,6 +2218,45 @@ class CuVSDenseIndex:f"({eligible_count} candidates <= {native_threshold})")+ if self.micro_batching_enabled:+ warm_request = self._try_enqueue_warm_micro_batch(+ query=query,+ limit=limit,+ filters=filters,+ prepared_filter=prepared_filter,+ telemetry=telemetry,+ )+ if warm_request is not None:+ return self._await_micro_batch_request(warm_request)++ queue_started = time.perf_counter()+ self._gpu_search_gate.acquire()+ waited_ms = (time.perf_counter() - queue_started) * 1000.0+ if telemetry is not None:+ telemetry.queue_ms += waited_ms+ telemetry.gpu_gate_queue_ms += waited_ms+ try:+ try:+ admitted = self._search_admitted(+ query,+ limit,+ filters,+ native_filter_resolver,+ native_filter_layout_registrar,+ prepared_filter,+ telemetry,+ defer_micro_batch_wait=True,+ )+ except _StalePreparedFilter:+ continue+ finally:+ # GPU rebuild/filter preparation and enqueue are admitted,+ # but waiting for the worker must never retain the gate.+ self._gpu_search_gate.release()+ if isinstance(admitted, _CuVSMicroBatchRequest):+ return self._await_micro_batch_request(admitted)+ return admitted+queue_started = time.perf_counter()self._gpu_search_gate.acquire()waited_ms = (time.perf_counter() - queue_started) * 1000.0@@ -1769,9 +2290,12 @@ class CuVSDenseIndex:native_filter_layout_registrar: Optional[Callable[[Sequence[int]], None]] = None,prepared_filter: Optional[_PreparedHostFilter] = None,telemetry: Optional[CuVSSearchTelemetry] = None,- ) -> Tuple[List[int], List[float]]:+ *,+ defer_micro_batch_wait: bool = False,+ ) -> Union[Tuple[List[int], List[float]], _CuVSMicroBatchRequest]:+ batch_request: Optional[_CuVSMicroBatchRequest] = Nonewith self._lock:- if self._closed:+ if self._closing or self._closed:raise RuntimeError("cuVS dense index is closed")if (prepared_filter is not None@@ -1907,8 +2431,32 @@ class CuVSDenseIndex:result_limit = min(limit, eligible_count)else:result_limit = min(limit, len(snapshot.labels))+ if self.micro_batching_enabled:+ cache_key = self._filter_cache_key(filters) if filters else "no-filter"+ filter_identity: Union[str, int] = (+ cache_key+ if cache_key is not None+ else id(mask)+ if mask is not None+ else id(filters)+ )+ batch_request = self._enqueue_micro_batch(+ snapshot=snapshot,+ query=query,+ result_limit=result_limit,+ mask=mask,+ filter_identity=filter_identity,+ telemetry=telemetry,+ )+ if batch_request is None:+ raise RuntimeError("cuVS gated micro-batch enqueue unexpectedly failed")self._active_searches += 1+ if batch_request is not None:+ if defer_micro_batch_wait:+ return batch_request+ return self._await_micro_batch_request(batch_request)+labels: List[int] = []scores: List[float] = []try:@@ -1950,21 +2498,57 @@ class CuVSDenseIndex:with self._lock:if self._close_complete:return+ while self._closing and not self._close_complete:+ self._idle_condition.wait()+ if self._close_complete:+ return++ # Reject new admission and invalidate any host-filter work that is+ # still outside the index lock. Already accepted micro-batch+ # requests remain active and are drained below.+ self._closing = Trueif not self._closed:self._closed = Trueself._records_generation += 1self._filter_layout_generation = -1self._cancel_preflight_flights_()+ if self.micro_batching_enabled:+ with self._micro_batch_condition:+ self._micro_batch_draining = True+ self._micro_batch_condition.notify_all()while self._active_searches:self._idle_condition.wait()- with self._runtime_device_scope():- self._snapshot = None- self._filter_cache.clear()- self._preflight_filter_cache.clear()- try:- self._runtime.close()- finally:- # A failed recovery can otherwise keep this partially packed- # host shadow alive through the constructor traceback.- self._records.clear()- self._close_complete = True++ if self.micro_batching_enabled:+ with self._micro_batch_condition:+ self._micro_batch_stop = True+ self._micro_batch_condition.notify_all()+ worker = self._micro_batch_worker+ if worker is not None and worker is not threading.current_thread():+ worker.join()++ try:+ with self._lock:+ with self._runtime_device_scope():+ self._snapshot = None+ self._filter_cache.clear()+ self._preflight_filter_cache.clear()+ try:+ self._runtime.close()+ finally:+ # A failed recovery can otherwise keep this partially+ # packed host shadow alive through the constructor+ # traceback.+ self._records.clear()+ except BaseException:+ with self._lock:+ # Keep admission closed, but allow a later close() call to+ # retry runtime cleanup after a transient failure.+ self._closing = False+ self._idle_condition.notify_all()+ raise+ else:+ with self._lock:+ self._close_complete = True+ self._closing = False+ self._idle_condition.notify_all()diff --git a/openviking/telemetry/operation.py b/openviking/telemetry/operation.pyindex 1a5bfdee4..a55242234 100644--- a/openviking/telemetry/operation.py+++ b/openviking/telemetry/operation.py@@ -21,6 +21,7 @@ _CUVS_TIMING_FIELDS = ("build_ms","filter_prepare_ms","gpu_search_ms",+ "batch_wait_ms","native_search_ms",)_CUVS_ROUTE_BUCKETS = {@@ -325,6 +326,26 @@ class TelemetrySummaryBuilder:"memory": cuvs_memory,"timings_ms": cls._cuvs_timing_summary(counters, gauges),}+ micro_batching_searches = cls._i(+ counters.get("vector.cuvs.micro_batching_searches"), 0+ )+ if micro_batching_searches:+ summary["vector"]["cuvs"].update(+ {+ "micro_batching_searches": micro_batching_searches,+ "micro_batched_searches": cls._i(+ counters.get("vector.cuvs.micro_batched_searches"), 0+ ),+ "micro_batching_warm_fast_path_searches": cls._i(+ counters.get("vector.cuvs.micro_batching_warm_fast_path_searches"),+ 0,+ ),+ "batch_size_max": cls._i(gauges.get("vector.cuvs.batch_size.max"), 1),+ "searches_by_batch_size": cls._counter_breakdown(+ "vector.cuvs.searches_by_batch_size", counters+ ),+ }+ )if cls._has_metric_prefix("semantic_nodes", counters, gauges):summary["semantic_nodes"] = {@@ -504,6 +525,21 @@ class OperationTelemetry:self._counters[f"vector.cuvs.dtypes.{dtype}"] += 1if TelemetrySummaryBuilder._bool(metrics.get("auto_mode"), False):self._counters["vector.cuvs.auto_mode_searches"] += 1+ if TelemetrySummaryBuilder._bool(metrics.get("micro_batching_enabled"), False):+ batch_size = min(+ max(TelemetrySummaryBuilder._i(metrics.get("batch_size"), 1), 1),+ 8,+ )+ self._counters["vector.cuvs.micro_batching_searches"] += 1+ self._counters[f"vector.cuvs.searches_by_batch_size.{batch_size}"] += 1+ if batch_size > 1:+ self._counters["vector.cuvs.micro_batched_searches"] += 1+ self._gauges["vector.cuvs.batch_size.max"] = max(+ TelemetrySummaryBuilder._i(self._gauges.get("vector.cuvs.batch_size.max"), 1),+ batch_size,+ )+ if TelemetrySummaryBuilder._bool(metrics.get("micro_batching_warm_fast_path"), False):+ self._counters["vector.cuvs.micro_batching_warm_fast_path_searches"] += 1if TelemetrySummaryBuilder._bool(metrics.get("filter_cache_hit"), False):self._counters["vector.cuvs.filter_cache_hits"] += 1if TelemetrySummaryBuilder._bool(metrics.get("filter_cache_eviction_fallback"), False):diff --git a/openviking_cli/utils/config/vectordb_config.py b/openviking_cli/utils/config/vectordb_config.pyindex b6b4905c4..ef8593e04 100644--- a/openviking_cli/utils/config/vectordb_config.py+++ b/openviking_cli/utils/config/vectordb_config.py@@ -162,6 +162,29 @@ class CuVSConfig(BaseModel):"snapshot work remains concurrent; increase only after hardware-specific tuning."),)+ micro_batching_enabled: bool = Field(+ default=False,+ description=(+ "Coalesce compatible concurrent cuVS dense queries into one matrix-search call. "+ "This OpenViking scheduler is opt-in and distinct from cuVS Dynamic Batching."+ ),+ )+ micro_batching_max_batch_size: int = Field(+ default=8,+ ge=1,+ le=8,+ description="Maximum compatible queries submitted in one cuVS search call.",+ )+ micro_batching_max_wait_ms: float = Field(+ default=1.0,+ ge=0.0,+ le=100.0,+ allow_inf_nan=False,+ description=(+ "Maximum collection window for a compatible cuVS micro-batch, in milliseconds. "+ "Zero performs opportunistic batching without an intentional wait."+ ),+ )auto_background_rebuild: bool = Field(default=False,description=(@@ -180,6 +203,16 @@ class CuVSConfig(BaseModel):model_config = {"extra": "forbid"}+ @model_validator(mode="after")+ def validate_micro_batching(self):+ if not self.micro_batching_enabled:+ return self+ if self.algorithm != "brute_force":+ raise ValueError("cuVS micro-batching currently supports algorithm='brute_force' only")+ if self.max_concurrent_gpu_searches != 1:+ raise ValueError("cuVS micro-batching currently requires max_concurrent_gpu_searches=1")+ return self+_OPENGAUSS_MODES = {"standalone", "distributed"}diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.pyindex 16d487983..8774b9daf 100644--- a/tests/test_telemetry_runtime.py+++ b/tests/test_telemetry_runtime.py@@ -291,6 +291,32 @@ def test_telemetry_summary_includes_cuvs_route_and_stage_timings():}+def test_telemetry_summary_includes_cuvs_micro_batching_fields():+ telemetry = MemoryOperationTelemetry(operation="search.find", enabled=True)+ for batch_size, batch_wait_ms in ((4, 0.8), (4, 0.7), (1, 1.0)):+ telemetry.record_cuvs_search(+ {+ "algorithm": "brute_force",+ "dtype": "float32",+ "route_reason": "cuvs",+ "filter_kind": "none",+ "micro_batching_enabled": True,+ "micro_batching_warm_fast_path": batch_size == 4,+ "batch_size": batch_size,+ "batch_wait_ms": batch_wait_ms,+ }+ )++ cuvs = telemetry.finish().summary["vector"]["cuvs"]++ assert cuvs["micro_batching_searches"] == 3+ assert cuvs["micro_batched_searches"] == 2+ assert cuvs["micro_batching_warm_fast_path_searches"] == 2+ assert cuvs["batch_size_max"] == 4+ assert cuvs["searches_by_batch_size"] == {"1": 1, "4": 2}+ assert cuvs["timings_ms"]["batch_wait"] == {"sum": 2.5, "max": 1.0}++def test_cuvs_telemetry_aggregation_is_completion_order_independent():samples = [{diff --git a/tests/vectordb/test_cuvs_collection.py b/tests/vectordb/test_cuvs_collection.pyindex 7edfb4cba..e32939e3c 100644--- a/tests/vectordb/test_cuvs_collection.py+++ b/tests/vectordb/test_cuvs_collection.py@@ -25,6 +25,7 @@ class FakeCuVSRuntime:def __init__(self, metric):self.metric = metricself.search_count = 0+ self.search_batch_sizes = []def build(self, dataset):return [list(vector) for vector in dataset]@@ -48,10 +49,30 @@ class FakeCuVSRuntime:rows = rows[:limit]return [row[1] for row in rows], [row[2] for row in rows]+ def search_batch(self, index, queries, limit, mask):+ self.search_batch_sizes.append(len(queries))+ results = [self.search(index, query, limit, mask) for query in queries]+ return [result[0] for result in results], [result[1] for result in results]+def close(self):pass+class BlockingFakeCuVSRuntime(FakeCuVSRuntime):+ def __init__(self, metric):+ super().__init__(metric)+ self.block_next_batch = False+ self.search_started = threading.Event()+ self.release_search = threading.Event()++ def search_batch(self, index, queries, limit, mask):+ if self.block_next_batch:+ self.block_next_batch = False+ self.search_started.set()+ assert self.release_search.wait(timeout=5)+ return super().search_batch(index, queries, limit, mask)++class MemoryAwareFakeCuVSRuntime(FakeCuVSRuntime):def __init__(self, metric, free_memory_bytes):super().__init__(metric)@@ -329,6 +350,188 @@ def test_local_collection_allows_concurrent_warmed_cuvs_searches(monkeypatch):collection.close()+def test_local_collection_micro_batches_compatible_offset_and_projection_requests(monkeypatch):+ runtime = FakeCuVSRuntime("inner_product")+ monkeypatch.setattr(+ cuvs_index,+ "_CuVSRuntime",+ lambda _algorithm, _metric, _build_params, _search_params, _dtype: runtime,+ )+ collection = get_or_create_local_collection(+ meta_data={+ "CollectionName": "cuvs_micro_batch_projection",+ "Fields": [+ {"FieldName": "id", "FieldType": "string", "IsPrimaryKey": True},+ {"FieldName": "vector", "FieldType": "vector", "Dim": 4},+ {"FieldName": "account_id", "FieldType": "string"},+ {"FieldName": "rank", "FieldType": "int64"},+ ],+ },+ config={+ "dense_search": {+ "backend": "cuvs",+ "algorithm": "brute_force",+ "micro_batching_enabled": True,+ "micro_batching_max_batch_size": 2,+ "micro_batching_max_wait_ms": 50.0,+ }+ },+ )+ try:+ collection.create_index(+ "default",+ {+ "IndexName": "default",+ "VectorIndex": {"IndexType": "flat", "Distance": "cosine"},+ },+ )+ collection.upsert_data(+ [+ {+ "id": "first",+ "vector": [1.0, 0.0, 0.0, 0.0],+ "account_id": "a",+ "rank": 1,+ },+ {+ "id": "second",+ "vector": [0.8, 0.2, 0.0, 0.0],+ "account_id": "b",+ "rank": 2,+ },+ {+ "id": "third",+ "vector": [0.0, 1.0, 0.0, 0.0],+ "account_id": "c",+ "rank": 3,+ },+ ]+ )+ # Warm the immutable snapshot, then isolate the concurrent measurement.+ assert (+ collection.search_by_vector("default", dense_vector=[1.0, 0.0, 0.0, 0.0], limit=1)+ .data[0]+ .id+ == "first"+ )+ runtime.search_batch_sizes.clear()+ barrier = threading.Barrier(3)++ def search_with_offset():+ barrier.wait(timeout=5)+ return collection.search_by_vector(+ "default",+ dense_vector=[1.0, 0.0, 0.0, 0.0],+ limit=1,+ offset=1,+ output_fields=["rank"],+ )++ def search_two():+ barrier.wait(timeout=5)+ return collection.search_by_vector(+ "default",+ dense_vector=[1.0, 0.0, 0.0, 0.0],+ limit=2,+ output_fields=["account_id"],+ )++ with ThreadPoolExecutor(max_workers=2) as executor:+ offset_future = executor.submit(search_with_offset)+ two_future = executor.submit(search_two)+ barrier.wait(timeout=5)+ offset_result = offset_future.result(timeout=5)+ two_result = two_future.result(timeout=5)++ assert runtime.search_batch_sizes == [2]+ assert [(item.id, item.fields) for item in offset_result.data] == [("second", {"rank": 2})]+ assert [(item.id, item.fields) for item in two_result.data] == [+ ("first", {"account_id": "a"}),+ ("second", {"account_id": "b"}),+ ]+ finally:+ collection.close()+++def test_local_collection_no_filter_requests_fill_warm_micro_batch(monkeypatch):+ runtime = BlockingFakeCuVSRuntime("inner_product")+ monkeypatch.setattr(+ cuvs_index,+ "_CuVSRuntime",+ lambda _algorithm, _metric, _build_params, _search_params, _dtype: runtime,+ )+ collection = get_or_create_local_collection(+ meta_data={+ "CollectionName": "cuvs_micro_batch_no_filter",+ "Fields": [+ {"FieldName": "id", "FieldType": "string", "IsPrimaryKey": True},+ {"FieldName": "vector", "FieldType": "vector", "Dim": 4},+ ],+ },+ config={+ "dense_search": {+ "backend": "cuvs",+ "algorithm": "brute_force",+ "micro_batching_enabled": True,+ "micro_batching_max_batch_size": 2,+ "micro_batching_max_wait_ms": 100.0,+ }+ },+ )+ try:+ local_index = collection.create_index(+ "default",+ {+ "IndexName": "default",+ "VectorIndex": {"IndexType": "flat", "Distance": "cosine"},+ },+ )+ collection.upsert_data(+ [+ {"id": "first", "vector": [1.0, 0.0, 0.0, 0.0]},+ {"id": "second", "vector": [0.0, 1.0, 0.0, 0.0]},+ ]+ )+ assert (+ collection.search_by_vector("default", dense_vector=[1.0, 0.0, 0.0, 0.0], limit=1)+ .data[0]+ .id+ == "first"+ )+ runtime.search_batch_sizes.clear()+ runtime.block_next_batch = True++ def search(vector):+ return collection.search_by_vector("default", dense_vector=vector, limit=1)++ with ThreadPoolExecutor(max_workers=3) as executor:+ active = executor.submit(search, [1.0, 0.0, 0.0, 0.0])+ assert runtime.search_started.wait(timeout=5)+ queued = [+ executor.submit(search, [0.0, 1.0, 0.0, 0.0]),+ executor.submit(search, [1.0, 0.0, 0.0, 0.0]),+ ]+ with local_index.dense_search._micro_batch_condition:+ assert local_index.dense_search._micro_batch_condition.wait_for(+ lambda: local_index.dense_search._micro_batch_warm_lookahead == 2,+ timeout=5,+ )+ assert all(not future.done() for future in queued)++ runtime.release_search.set()+ assert active.result(timeout=5).data[0].id == "first"+ assert [future.result(timeout=5).data[0].id for future in queued] == [+ "second",+ "first",+ ]++ assert runtime.search_batch_sizes == [1, 2]+ assert local_index.dense_search._micro_batch_warm_lookahead == 0+ finally:+ runtime.release_search.set()+ collection.close()++def test_persistent_collection_rehydrates_cuvs_from_local_store(monkeypatch, tmp_path):patch_cuvs_runtime(monkeypatch)path = str(tmp_path / "cuvs-persistent")diff --git a/tests/vectordb/test_cuvs_config.py b/tests/vectordb/test_cuvs_config.pyindex 2dc2dc9e8..7a1d4a5cc 100644--- a/tests/vectordb/test_cuvs_config.py+++ b/tests/vectordb/test_cuvs_config.py@@ -5,6 +5,7 @@ import pytestfrom pydantic import ValidationErrorfrom openviking.storage.vectordb_adapters.local_adapter import (+ CuVSCollectionAdapter,LocalCollectionAdapter,)from openviking_cli.utils.config.vectordb_config import CuVSConfig, VectorDBBackendConfig@@ -14,6 +15,9 @@ def test_cuvs_filter_cache_defaults_and_disable_value():assert CuVSConfig().dtype == "float32"assert CuVSConfig(dtype="float16").dtype == "float16"assert CuVSConfig().max_concurrent_gpu_searches == 1+ assert CuVSConfig().micro_batching_enabled is False+ assert CuVSConfig().micro_batching_max_batch_size == 8+ assert CuVSConfig().micro_batching_max_wait_ms == 1.0assert CuVSConfig().filter_cache_size == 16assert CuVSConfig(filter_cache_size=0).filter_cache_size == 0@@ -25,6 +29,22 @@ def test_cuvs_filter_cache_rejects_negative_size():CuVSConfig(filter_cache_size=-1)with pytest.raises(ValidationError, match="max_concurrent_gpu_searches"):CuVSConfig(max_concurrent_gpu_searches=0)+ with pytest.raises(ValidationError, match="micro_batching_max_batch_size"):+ CuVSConfig(micro_batching_max_batch_size=0)+ with pytest.raises(ValidationError, match="micro_batching_max_batch_size"):+ CuVSConfig(micro_batching_max_batch_size=9)+ with pytest.raises(ValidationError, match="micro_batching_max_wait_ms"):+ CuVSConfig(micro_batching_max_wait_ms=-0.1)+ with pytest.raises(ValidationError, match="micro_batching_max_wait_ms"):+ CuVSConfig(micro_batching_max_wait_ms=float("nan"))+ with pytest.raises(ValidationError, match="micro_batching_max_wait_ms"):+ CuVSConfig(micro_batching_max_wait_ms=100.1)+ with pytest.raises(ValidationError, match="brute_force"):+ CuVSConfig(micro_batching_enabled=True, algorithm="cagra")+ with pytest.raises(ValidationError, match="max_concurrent_gpu_searches=1"):+ CuVSConfig(micro_batching_enabled=True, max_concurrent_gpu_searches=2)+ with pytest.raises(ValidationError, match="dynamic_batching"):+ CuVSConfig(dynamic_batching=True)def test_cuvs_auto_mode_is_opt_in_and_validates_memory_guardrails():@@ -63,6 +83,9 @@ def test_local_adapter_only_passes_auto_cuvs_config_when_enabled():"auto_memory_safety_factor": 1.5,"auto_filter_native_threshold": 1000,"auto_path_filter_native_threshold": 100,+ "micro_batching_enabled": True,+ "micro_batching_max_batch_size": 4,+ "micro_batching_max_wait_ms": 0.5,},))@@ -74,3 +97,24 @@ def test_local_adapter_only_passes_auto_cuvs_config_when_enabled():assert dense_search["auto_memory_safety_factor"] == 1.5assert dense_search["auto_filter_native_threshold"] == 1000assert dense_search["auto_path_filter_native_threshold"] == 100+ assert dense_search["micro_batching_enabled"] is True+ assert dense_search["micro_batching_max_batch_size"] == 4+ assert dense_search["micro_batching_max_wait_ms"] == 0.5+++def test_explicit_cuvs_adapter_forwards_micro_batching_config():+ adapter = CuVSCollectionAdapter.from_config(+ VectorDBBackendConfig(+ backend="cuvs",+ cuvs={+ "micro_batching_enabled": True,+ "micro_batching_max_batch_size": 4,+ "micro_batching_max_wait_ms": 0.25,+ },+ )+ )+ dense_search = adapter._collection_config["dense_search"]+ assert dense_search["backend"] == "cuvs"+ assert dense_search["micro_batching_enabled"] is True+ assert dense_search["micro_batching_max_batch_size"] == 4+ assert dense_search["micro_batching_max_wait_ms"] == 0.25diff --git a/tests/vectordb/test_cuvs_index.py b/tests/vectordb/test_cuvs_index.pyindex 5ded0d3e9..6c6c0a9f0 100644--- a/tests/vectordb/test_cuvs_index.py+++ b/tests/vectordb/test_cuvs_index.py@@ -39,6 +39,7 @@ class FakeCuVSRuntime:self.release_count = 0self.close_count = 0self.close_error = None+ self.search_batch_sizes = []def build(self, dataset):self.dataset = [list(vector) for vector in dataset]@@ -63,6 +64,11 @@ class FakeCuVSRuntime:selected = rows[:limit]return [row[1] for row in selected], [row[2] for row in selected]+ def search_batch(self, index, queries, limit, mask):+ self.search_batch_sizes.append(len(queries))+ results = [self.search(index, query, limit, mask) for query in queries]+ return [result[0] for result in results], [result[1] for result in results]+def prepare_filter(self, mask):self.prepare_filter_count += 1return tuple(mask)@@ -233,8 +239,9 @@ def test_cuvs_runtime_uses_captured_device_from_worker_thread():assert resources.device_id == spy.expected_deviceif spy.fail_search:raise RuntimeError("injected search failure")- return FakeArray([[0.25]], "float32", "distances"), FakeArray(- [[0]], "int64", "neighbors"+ query_count = len(_queries.values)+ return FakeArray([[0.25]] * query_count, "float32", "distances"), FakeArray(+ [[0]] * query_count, "int64", "neighbors")class FakeFilters:@@ -308,6 +315,12 @@ def test_cuvs_runtime_uses_captured_device_from_worker_thread():assert ("array-5", 3) in spy.temporary_releasesassert spy.current() == 9assert spy.releases == [("resources-1", 3)]+ assert runtime.search_batch(+ runtime_index,+ [[1.0, 0.0], [0.0, 1.0]],+ 1,+ [True],+ ) == ([[0], [0]], [[0.25], [0.25]])assert runtime.search(runtime_index, [1.0, 0.0], 1, [True]) == ([0], [0.25])spy.fail_host_copy = Truewith pytest.raises(RuntimeError, match="injected host copy failure"):@@ -1385,6 +1398,910 @@ def test_device_cache_eviction_before_admission_uses_one_in_gate_fallback():assert eviction_telemetry.as_dict()["filter_cache_eviction_fallback"] is True+def make_micro_batch_index(runtime, **config):+ dense_config = {+ "micro_batching_enabled": True,+ "micro_batching_max_batch_size": 4,+ "micro_batching_max_wait_ms": 50.0,+ **config,+ }+ index = CuVSDenseIndex(+ dimension=2,+ distance="ip",+ normalize_vectors=False,+ field_types={"account_id": "string"},+ config=dense_config,+ runtime=runtime,+ )+ index.add_candidates(+ [+ candidate(1, [1.0, 0.0], account_id="a"),+ candidate(2, [0.0, 1.0], account_id="b"),+ candidate(3, [0.8, 0.2], account_id="a"),+ ]+ )+ # Exclude the first lazy build and its unavoidable singleton from assertions.+ assert index.search([1.0, 0.0], 1, None)[0] == [1]+ runtime.search_batch_sizes.clear()+ return index+++class BlockingMicroBatchRuntime(FakeCuVSRuntime):+ def __init__(self):+ super().__init__()+ self.block_next_batch = False+ self.search_started = threading.Event()+ self.release_search = threading.Event()++ def search_batch(self, index, queries, limit, mask):+ if self.block_next_batch:+ self.block_next_batch = False+ self.search_started.set()+ assert self.release_search.wait(timeout=5)+ return super().search_batch(index, queries, limit, mask)+++def wait_for_warm_lookahead(index, expected):+ with index._micro_batch_condition:+ assert index._micro_batch_condition.wait_for(+ lambda: index._micro_batch_warm_lookahead == expected,+ timeout=5,+ )+++def run_simultaneous_searches(index, requests):+ barrier = threading.Barrier(len(requests) + 1)++ def run(request):+ barrier.wait(timeout=5)+ return index.search(*request)++ with ThreadPoolExecutor(max_workers=len(requests)) as executor:+ futures = [executor.submit(run, request) for request in requests]+ barrier.wait(timeout=5)+ return [future.result(timeout=5) for future in futures]+++def test_cuvs_micro_batch_coalesces_compatible_queries_and_demuxes_rows():+ runtime = FakeCuVSRuntime()+ index = make_micro_batch_index(runtime)+ telemetry = [CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False) for _ in range(4)]+ requests = [+ ([1.0, 0.0], 1, None, None, None, telemetry[0]),+ ([0.0, 1.0], 1, None, None, None, telemetry[1]),+ ([0.9, 0.1], 1, None, None, None, telemetry[2]),+ ([0.1, 0.9], 1, None, None, None, telemetry[3]),+ ]++ results = run_simultaneous_searches(index, requests)++ assert [result[0] for result in results] == [[1], [2], [1], [2]]+ assert runtime.search_batch_sizes == [4]+ assert all(item.micro_batching_enabled for item in telemetry)+ assert [item.batch_size for item in telemetry] == [4] * 4+ assert all(item.batch_wait_ms >= 0.0 for item in telemetry)+ index.close()+++def test_cuvs_micro_batch_warm_fast_path_fills_behind_active_batch():+ runtime = BlockingMicroBatchRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_batch_size=2,+ micro_batching_max_wait_ms=100.0,+ )+ runtime.block_next_batch = True+ telemetry = [CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False) for _ in range(3)]++ with ThreadPoolExecutor(max_workers=3) as executor:+ active = executor.submit(index.search, [1.0, 0.0], 1, None, None, None, telemetry[0])+ assert runtime.search_started.wait(timeout=5)+ lookahead = [+ executor.submit(index.search, query, 1, None, None, None, item)+ for query, item in zip(+ ([1.0, 0.0], [0.0, 1.0]),+ telemetry[1:],+ strict=True,+ )+ ]+ wait_for_warm_lookahead(index, 2)++ assert all(not future.done() for future in lookahead)+ assert index._active_searches == 3+ assert all(item.micro_batching_warm_fast_path for item in telemetry)+ assert all(item.gpu_gate_queue_ms == 0.0 for item in telemetry)++ runtime.release_search.set()+ assert active.result(timeout=5)[0] == [1]+ assert [future.result(timeout=5)[0] for future in lookahead] == [[1], [2]]++ assert runtime.search_batch_sizes == [1, 2]+ assert [item.batch_size for item in telemetry] == [1, 2, 2]+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ index.close()+++def test_cuvs_micro_batch_warm_fast_path_caps_one_batch_and_preserves_progress():+ runtime = BlockingMicroBatchRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_batch_size=2,+ micro_batching_max_wait_ms=100.0,+ )+ runtime.block_next_batch = True+ warm_telemetry = [+ CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False) for _ in range(2)+ ]+ overflow_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)++ with ThreadPoolExecutor(max_workers=4) as executor:+ active = executor.submit(index.search, [1.0, 0.0], 1, None)+ assert runtime.search_started.wait(timeout=5)+ warm = [+ executor.submit(index.search, query, 1, None, None, None, item)+ for query, item in zip(+ ([1.0, 0.0], [0.0, 1.0]),+ warm_telemetry,+ strict=True,+ )+ ]+ wait_for_warm_lookahead(index, 2)+ overflow = executor.submit(+ index.search,+ [0.9, 0.1],+ 1,+ None,+ None,+ None,+ overflow_telemetry,+ )++ assert not overflow.done()+ assert index._micro_batch_warm_lookahead == 2+ assert all(item.micro_batching_warm_fast_path for item in warm_telemetry)+ assert overflow_telemetry.micro_batching_warm_fast_path is False++ runtime.release_search.set()+ assert active.result(timeout=5)[0] == [1]+ assert [future.result(timeout=5)[0] for future in warm] == [[1], [2]]+ assert overflow.result(timeout=5)[0] == [1]++ assert runtime.search_batch_sizes == [1, 2, 1]+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ index.close()+++def test_cuvs_micro_batch_empty_filter_uses_warm_fast_path():+ runtime = BlockingMicroBatchRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_batch_size=2,+ micro_batching_max_wait_ms=100.0,+ )+ runtime.block_next_batch = True+ no_filter_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)+ empty_filter_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)++ with ThreadPoolExecutor(max_workers=3) as executor:+ active = executor.submit(index.search, [1.0, 0.0], 1, None)+ assert runtime.search_started.wait(timeout=5)+ no_filter = executor.submit(+ index.search,+ [0.0, 1.0],+ 1,+ None,+ None,+ None,+ no_filter_telemetry,+ )+ wait_for_warm_lookahead(index, 1)+ empty_filter = executor.submit(+ index.search,+ [1.0, 0.0],+ 1,+ {},+ None,+ None,+ empty_filter_telemetry,+ )+ wait_for_warm_lookahead(index, 2)++ assert no_filter_telemetry.micro_batching_warm_fast_path is True+ assert empty_filter_telemetry.micro_batching_warm_fast_path is True+ assert not empty_filter.done()++ runtime.release_search.set()+ assert active.result(timeout=5)[0] == [1]+ assert no_filter.result(timeout=5)[0] == [2]+ assert empty_filter.result(timeout=5)[0] == [1]++ assert runtime.search_batch_sizes == [1, 2]+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ index.close()+++def test_cuvs_micro_batch_groups_same_filter_but_splits_incompatible_keys():+ runtime = FakeCuVSRuntime()+ index = make_micro_batch_index(runtime)+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}+ filter_b = {"op": "must", "field": "account_id", "conds": ["b"]}++ same_filter = run_simultaneous_searches(+ index,+ [+ ([1.0, 0.0], 1, filter_a),+ ([0.0, 1.0], 1, filter_a),+ ],+ )+ assert [result[0] for result in same_filter] == [[1], [3]]+ assert runtime.search_batch_sizes == [2]++ runtime.search_batch_sizes.clear()+ split_filters = run_simultaneous_searches(+ index,+ [+ ([1.0, 0.0], 1, filter_a),+ ([0.0, 1.0], 1, filter_b),+ ],+ )+ assert [result[0] for result in split_filters] == [[1], [2]]+ assert sorted(runtime.search_batch_sizes) == [1, 1]++ runtime.search_batch_sizes.clear()+ split_limits = run_simultaneous_searches(+ index,+ [+ ([1.0, 0.0], 1, None),+ ([1.0, 0.0], 2, None),+ ],+ )+ assert [len(result[0]) for result in split_limits] == [1, 2]+ assert sorted(runtime.search_batch_sizes) == [1, 1]+ index.close()+++def test_cuvs_micro_batch_warm_fast_path_preserves_filter_and_limit_keys():+ runtime = BlockingMicroBatchRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_batch_size=4,+ micro_batching_max_wait_ms=20.0,+ )+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}+ filter_b = {"op": "must", "field": "account_id", "conds": ["b"]}++ def resolve(filters):+ return ([0b101], 2) if filters == filter_a else ([0b010], 1)++ def registrar(_labels):+ return None++ assert index.search([1.0, 0.0], 1, filter_a, resolve, registrar)[0] == [1]+ assert index.search([0.0, 1.0], 1, filter_b, resolve, registrar)[0] == [2]+ runtime.search_batch_sizes.clear()+ runtime.block_next_batch = True+ telemetry = [CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False) for _ in range(4)]++ with ThreadPoolExecutor(max_workers=5) as executor:+ active = executor.submit(index.search, [1.0, 0.0], 1, None)+ assert runtime.search_started.wait(timeout=5)+ requests = [+ executor.submit(+ index.search, [1.0, 0.0], 1, filter_a, resolve, registrar, telemetry[0]+ ),+ executor.submit(+ index.search, [0.0, 1.0], 1, filter_a, resolve, registrar, telemetry[1]+ ),+ executor.submit(+ index.search, [1.0, 0.0], 2, filter_a, resolve, registrar, telemetry[2]+ ),+ executor.submit(+ index.search, [0.0, 1.0], 1, filter_b, resolve, registrar, telemetry[3]+ ),+ ]+ wait_for_warm_lookahead(index, 4)++ assert all(item.micro_batching_warm_fast_path for item in telemetry)+ assert all(item.filter_cache_hit for item in telemetry)+ runtime.release_search.set()++ assert active.result(timeout=5)[0] == [1]+ results = [future.result(timeout=5)[0] for future in requests]++ assert results == [[1], [3], [1, 3], [2]]+ assert runtime.search_batch_sizes == [1, 2, 1, 1]+ assert [item.batch_size for item in telemetry] == [2, 2, 1, 1]+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ index.close()+++def test_cuvs_micro_batch_serializes_device_filter_preparation_with_search():+ class BlockingRuntime(FakeCuVSRuntime):+ def __init__(self):+ super().__init__()+ self.block_next_batch = False+ self.search_started = threading.Event()+ self.release_search = threading.Event()+ self.filter_prepared = threading.Event()++ def search_batch(self, index, queries, limit, mask):+ if self.block_next_batch:+ self.block_next_batch = False+ self.search_started.set()+ assert self.release_search.wait(timeout=5)+ return super().search_batch(index, queries, limit, mask)++ def prepare_filter_words(self, _words):+ self.filter_prepared.set()+ return (True, False, True)++ runtime = BlockingRuntime()+ index = make_micro_batch_index(runtime)+ runtime.block_next_batch = True+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}+ resolver_ran = threading.Event()+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)++ def resolve(_filters):+ resolver_ran.set()+ return [0b101], 2++ with ThreadPoolExecutor(max_workers=2) as executor:+ active_search = executor.submit(index.search, [1.0, 0.0], 1, None)+ assert runtime.search_started.wait(timeout=5)+ filtered_search = executor.submit(+ index.search,+ [1.0, 0.0],+ 1,+ filter_a,+ resolve,+ lambda _labels: None,+ telemetry,+ )+ try:+ # Native filter resolution remains host work outside admission.+ assert resolver_ran.wait(timeout=5)+ # Device materialization waits for the occupied GPU gate.+ assert not runtime.filter_prepared.wait(timeout=0.1)+ finally:+ runtime.release_search.set()+ assert active_search.result(timeout=5)[0] == [1]+ assert filtered_search.result(timeout=5)[0] == [1]++ assert runtime.filter_prepared.is_set()+ assert telemetry.micro_batching_warm_fast_path is False+ index.close()+++def test_cuvs_micro_batch_serializes_rebuild_with_search():+ class BlockingRuntime(FakeCuVSRuntime):+ def __init__(self):+ super().__init__()+ self.block_next_batch = False+ self.observe_build = False+ self.search_started = threading.Event()+ self.release_search = threading.Event()+ self.build_started = threading.Event()++ def search_batch(self, index, queries, limit, mask):+ if self.block_next_batch:+ self.block_next_batch = False+ self.search_started.set()+ assert self.release_search.wait(timeout=5)+ return super().search_batch(index, queries, limit, mask)++ def build(self, dataset):+ if self.observe_build:+ self.build_started.set()+ return super().build(dataset)++ runtime = BlockingRuntime()+ index = make_micro_batch_index(runtime)+ runtime.block_next_batch = True+ runtime.observe_build = True+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)++ with ThreadPoolExecutor(max_workers=2) as executor:+ active_search = executor.submit(index.search, [1.0, 0.0], 1, None)+ assert runtime.search_started.wait(timeout=5)+ index.upsert([delta(4, [2.0, 0.0], account_id="a")])+ rebuilding_search = executor.submit(+ index.search,+ [1.0, 0.0],+ 1,+ None,+ None,+ None,+ telemetry,+ )+ try:+ assert not runtime.build_started.wait(timeout=0.1)+ finally:+ runtime.release_search.set()+ assert active_search.result(timeout=5)[0] == [1]+ assert rebuilding_search.result(timeout=5)[0] == [4]++ assert runtime.build_started.is_set()+ assert telemetry.micro_batching_warm_fast_path is False+ index.close()+++def test_cuvs_micro_batch_device_lru_eviction_falls_back_once():+ class WordPreparingRuntime(FakeCuVSRuntime):+ def __init__(self):+ super().__init__()+ self.prepare_filter_words_count = 0++ def prepare_filter_words(self, words):+ self.prepare_filter_words_count += 1+ return tuple(bool(words[0] & (1 << row)) for row in range(3))++ runtime = WordPreparingRuntime()+ index = make_micro_batch_index(runtime, filter_cache_size=1)+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}+ filter_b = {"op": "must", "field": "account_id", "conds": ["b"]}+ resolver_calls = {"a": 0, "b": 0}++ def resolve(filters):+ account_id = filters["conds"][0]+ resolver_calls[account_id] += 1+ return ([0b101], 2) if account_id == "a" else ([0b010], 1)++ def registrar(_labels):+ return None++ assert index.search([1.0, 0.0], 1, filter_a, resolve, registrar)[0] == [1]+ assert resolver_calls == {"a": 1, "b": 0}++ a_host_prepared = threading.Event()+ release_a = threading.Event()+ original_prepare = index._prepare_host_filter++ def pause_a_after_host_prepare(filters, *args, **kwargs):+ prepared = original_prepare(filters, *args, **kwargs)+ if filters == filter_a:+ a_host_prepared.set()+ assert release_a.wait(timeout=5)+ return prepared++ index._prepare_host_filter = pause_a_after_host_prepare+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)+ with ThreadPoolExecutor(max_workers=1) as executor:+ delayed_a = executor.submit(+ index.search,+ [1.0, 0.0],+ 1,+ filter_a,+ resolve,+ registrar,+ telemetry,+ )+ assert a_host_prepared.wait(timeout=5)+ assert index.search([0.0, 1.0], 1, filter_b, resolve, registrar)[0] == [2]+ release_a.set()+ assert delayed_a.result(timeout=5)[0] == [1]++ assert resolver_calls == {"a": 2, "b": 1}+ assert runtime.prepare_filter_words_count == 3+ assert telemetry.micro_batching_warm_fast_path is False+ assert telemetry.filter_cache_hit is False+ assert telemetry.filter_cache_eviction_fallback is True+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ index.close()+++def test_cuvs_micro_batch_stale_host_filter_uses_admitted_retry():+ class CountingGate:+ def __init__(self, gate):+ self.gate = gate+ self.acquire_count = 0+ self.lock = threading.Lock()++ def acquire(self, *args, **kwargs):+ with self.lock:+ self.acquire_count += 1+ return self.gate.acquire(*args, **kwargs)++ def release(self):+ return self.gate.release()++ runtime = FakeCuVSRuntime()+ index = make_micro_batch_index(runtime)+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}++ def resolve(_filters):+ return ([0b101], 2) if index.size == 3 else ([0b1101], 3)++ def registrar(_labels):+ return None++ assert index.search([1.0, 0.0], 1, filter_a, resolve, registrar)[0] == [1]+ stale = index._prepare_host_filter(filter_a, resolve, registrar)+ assert stale is not None+ index.upsert([delta(4, [2.0, 0.0], account_id="a")])++ original_prepare = index._prepare_host_filter+ prepare_calls = 0++ def return_stale_once(*args, **kwargs):+ nonlocal prepare_calls+ prepare_calls += 1+ if prepare_calls == 1:+ return stale+ return original_prepare(*args, **kwargs)++ index._prepare_host_filter = return_stale_once+ counting_gate = CountingGate(index._gpu_search_gate)+ index._gpu_search_gate = counting_gate+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)++ assert index.search(+ [1.0, 0.0],+ 1,+ filter_a,+ resolve,+ registrar,+ telemetry,+ )[0] == [4]++ # One caller admission rejects the stale host result, the retry performs+ # the cold rebuild/materialization, and the worker owns the final search.+ assert counting_gate.acquire_count == 3+ assert prepare_calls == 2+ assert telemetry.micro_batching_warm_fast_path is False+ assert telemetry.records_generation == index._records_generation+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ index.close()+++def test_cuvs_micro_batch_singleton_deadline_and_exception_recovery():+ class FailingBatchRuntime(FakeCuVSRuntime):+ def __init__(self):+ super().__init__()+ self.fail_next_batch = False++ def search_batch(self, index, queries, limit, mask):+ if self.fail_next_batch:+ self.fail_next_batch = False+ self.search_batch_sizes.append(len(queries))+ raise RuntimeError("injected batch failure")+ return super().search_batch(index, queries, limit, mask)++ runtime = FailingBatchRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_wait_ms=10.0,+ )+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)+ started = time.perf_counter()+ assert index.search([1.0, 0.0], 1, None, telemetry=telemetry)[0] == [1]+ assert time.perf_counter() - started < 1.0+ assert telemetry.batch_size == 1+ assert telemetry.batch_wait_ms >= 0.0++ runtime.search_batch_sizes.clear()+ runtime.fail_next_batch = True+ barrier = threading.Barrier(3)++ def failing_search(query):+ barrier.wait(timeout=5)+ return index.search(query, 1, None)++ with ThreadPoolExecutor(max_workers=2) as executor:+ futures = [+ executor.submit(failing_search, [1.0, 0.0]),+ executor.submit(failing_search, [0.0, 1.0]),+ ]+ barrier.wait(timeout=5)+ for future in futures:+ with pytest.raises(RuntimeError, match="injected batch failure"):+ future.result(timeout=5)++ assert runtime.search_batch_sizes == [2]+ assert index.search([1.0, 0.0], 1, None)[0] == [1]+ index.close()+++def test_cuvs_micro_batch_worker_fatal_error_reaps_current_batch():+ class FatalBatchRuntime(FakeCuVSRuntime):+ def __init__(self):+ super().__init__()+ self.fail_fatally = False++ def search_batch(self, index, queries, limit, mask):+ if self.fail_fatally:+ self.fail_fatally = False+ self.search_batch_sizes.append(len(queries))+ raise KeyboardInterrupt("injected worker failure")+ return super().search_batch(index, queries, limit, mask)++ runtime = FatalBatchRuntime()+ index = make_micro_batch_index(runtime)+ runtime.fail_fatally = True+ barrier = threading.Barrier(3)++ def fatal_search(query):+ barrier.wait(timeout=5)+ return index.search(query, 1, None)++ with ThreadPoolExecutor(max_workers=2) as executor:+ futures = [+ executor.submit(fatal_search, [1.0, 0.0]),+ executor.submit(fatal_search, [0.0, 1.0]),+ ]+ barrier.wait(timeout=5)+ for future in futures:+ with pytest.raises(KeyboardInterrupt, match="injected worker failure"):+ future.result(timeout=5)++ assert runtime.search_batch_sizes == [2]+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ deadline = time.monotonic() + 5+ while index._micro_batch_worker_failure is None and time.monotonic() < deadline:+ time.sleep(0.001)+ assert index._micro_batch_worker_failure is not None+ index.close()+++def test_cuvs_micro_batch_cancelled_warm_request_releases_pins_and_capacity():+ runtime = BlockingMicroBatchRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_batch_size=2,+ micro_batching_max_wait_ms=100.0,+ )+ runtime.block_next_batch = True++ with ThreadPoolExecutor(max_workers=1) as executor:+ active = executor.submit(index.search, [1.0, 0.0], 1, None)+ assert runtime.search_started.wait(timeout=5)+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)+ request = index._try_enqueue_warm_micro_batch(+ query=(0.0, 1.0),+ limit=1,+ filters=None,+ prepared_filter=None,+ telemetry=telemetry,+ )+ assert request is not None+ assert request.future.cancel()+ assert telemetry.micro_batching_warm_fast_path is True+ wait_for_warm_lookahead(index, 1)++ runtime.release_search.set()+ assert active.result(timeout=5)[0] == [1]++ deadline = time.monotonic() + 5+ while not request.released and time.monotonic() < deadline:+ time.sleep(0.001)+ assert request.released is True+ assert request.snapshot is None+ assert request.mask is None+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ index.close()+++def test_cuvs_micro_batch_fatal_worker_releases_warm_lookahead_pins():+ class BlockingFatalRuntime(FakeCuVSRuntime):+ def __init__(self):+ super().__init__()+ self.block_and_fail = False+ self.search_started = threading.Event()+ self.fail_search = threading.Event()++ def search_batch(self, index, queries, limit, mask):+ if self.block_and_fail:+ self.block_and_fail = False+ self.search_started.set()+ assert self.fail_search.wait(timeout=5)+ raise KeyboardInterrupt("injected worker failure")+ return super().search_batch(index, queries, limit, mask)++ runtime = BlockingFatalRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_batch_size=2,+ micro_batching_max_wait_ms=100.0,+ )+ runtime.block_and_fail = True++ with ThreadPoolExecutor(max_workers=1) as executor:+ active = executor.submit(index.search, [1.0, 0.0], 1, None)+ assert runtime.search_started.wait(timeout=5)+ pending = [+ index._try_enqueue_warm_micro_batch(+ query=query,+ limit=1,+ filters=None,+ prepared_filter=None,+ telemetry=None,+ )+ for query in ((1.0, 0.0), (0.0, 1.0))+ ]+ assert all(request is not None for request in pending)+ wait_for_warm_lookahead(index, 2)+ runtime.fail_search.set()++ with pytest.raises(KeyboardInterrupt, match="injected worker failure"):+ active.result(timeout=5)+ for request in pending:+ assert request is not None+ with pytest.raises(KeyboardInterrupt, match="injected worker failure"):+ request.future.result(timeout=5)++ assert all(request.released for request in pending if request is not None)+ assert all(request.snapshot is None for request in pending if request is not None)+ assert all(request.mask is None for request in pending if request is not None)+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ assert index._micro_batch_worker_failure is not None+ index.close()+++def test_cuvs_micro_batch_keeps_snapshot_generations_separate():+ runtime = FakeCuVSRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_wait_ms=100.0,+ )++ old_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)+ new_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)++ with ThreadPoolExecutor(max_workers=2) as executor:+ old_search = executor.submit(+ index.search,+ [1.0, 0.0],+ 1,+ None,+ None,+ None,+ old_telemetry,+ )+ deadline = time.monotonic() + 5+ while time.monotonic() < deadline:+ with index._micro_batch_condition:+ if index._micro_batch_pending:+ break+ time.sleep(0.001)+ else:+ pytest.fail("old-snapshot search did not enter the micro-batch queue")++ index.upsert([delta(4, [2.0, 0.0], account_id="a")])+ new_search = executor.submit(+ index.search,+ [1.0, 0.0],+ 1,+ None,+ None,+ None,+ new_telemetry,+ )+ assert old_search.result(timeout=5)[0] == [1]+ assert new_search.result(timeout=5)[0] == [4]++ assert runtime.search_batch_sizes == [1, 1]+ assert old_telemetry.micro_batching_warm_fast_path is True+ assert new_telemetry.micro_batching_warm_fast_path is False+ assert old_telemetry.records_generation < new_telemetry.records_generation+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ index.close()+++def test_auto_native_route_does_not_report_micro_batching():+ runtime = FakeCuVSRuntime()+ index = CuVSDenseIndex(+ dimension=2,+ distance="ip",+ normalize_vectors=False,+ field_types={"account_id": "string"},+ config={+ "micro_batching_enabled": True,+ "micro_batching_max_wait_ms": 1.0,+ "auto_filter_native_threshold": 10,+ "auto_memory_reserve_mb": 0,+ "auto_memory_safety_factor": 1.0,+ },+ runtime=runtime,+ auto_memory=True,+ )+ index.add_candidates([candidate(1, [1.0, 0.0], account_id="a")])+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=True)++ with pytest.raises(CuVSNativeRouteError):+ index.search(+ [1.0, 0.0],+ 1,+ {"op": "must", "field": "account_id", "conds": ["a"]},+ native_filter_resolver=lambda _filters: ([1], 1),+ native_filter_layout_registrar=lambda _labels: None,+ telemetry=telemetry,+ )++ assert telemetry.micro_batching_enabled is False+ assert telemetry.micro_batching_warm_fast_path is False+ assert telemetry.batch_size == 1+ assert runtime.search_batch_sizes == []+ index.close()+++def test_cuvs_micro_batch_close_drains_queued_search_and_rejects_new_work():+ runtime = FakeCuVSRuntime()+ index = make_micro_batch_index(+ runtime,+ micro_batching_max_wait_ms=100.0,+ )+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)++ with ThreadPoolExecutor(max_workers=2) as executor:+ search_future = executor.submit(+ index.search,+ [1.0, 0.0],+ 1,+ None,+ None,+ None,+ telemetry,+ )+ deadline = time.monotonic() + 5+ while time.monotonic() < deadline:+ with index._micro_batch_condition:+ if index._micro_batch_pending:+ break+ time.sleep(0.001)+ else:+ pytest.fail("search did not enter the micro-batch queue")+ close_future = executor.submit(index.close)+ assert search_future.result(timeout=5)[0] == [1]+ close_future.result(timeout=5)++ assert runtime.closed is True+ assert telemetry.micro_batching_warm_fast_path is True+ assert index._active_searches == 0+ assert index._micro_batch_warm_lookahead == 0+ with pytest.raises(RuntimeError, match="closed"):+ index.search([1.0, 0.0], 1, None)+++ "config, message",+ [+ ({"micro_batching_max_batch_size": 0}, "between 1 and 8"),+ ({"micro_batching_max_batch_size": 9}, "between 1 and 8"),+ ({"micro_batching_max_wait_ms": -1}, "between 0 and 100"),+ ({"micro_batching_max_wait_ms": 100.1}, "between 0 and 100"),+ ({"micro_batching_max_wait_ms": float("inf")}, "between 0 and 100"),+ (+ {"micro_batching_enabled": True, "algorithm": "cagra"},+ "brute_force",+ ),+ (+ {"micro_batching_enabled": True, "max_concurrent_gpu_searches": 2},+ "max_concurrent_gpu_searches=1",+ ),+ ],+)+def test_cuvs_micro_batch_raw_config_validation(config, message):+ with pytest.raises(ValueError, match=message):+ CuVSDenseIndex(+ dimension=2,+ distance="ip",+ normalize_vectors=False,+ field_types={},+ config=config,+ runtime=FakeCuVSRuntime(),+ )++def test_inflight_search_keeps_immutable_snapshot_during_rebuild():class BlockingRuntime(FakeCuVSRuntime):def __init__(self):