Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

perf(vectordb): micro-batch compatible cuVS searches (#3382)

Yuanqing ZHAO40dd05271cc6
diff --git a/benchmark/cuvs/PRELIMINARY_RESULTS.md b/benchmark/cuvs/PRELIMINARY_RESULTS.md
index d935fa102..5607307c2 100644
--- a/benchmark/cuvs/PRELIMINARY_RESULTS.md
+++ b/benchmark/cuvs/PRELIMINARY_RESULTS.md
@@ -6,6 +6,14 @@ GPU, one public ANN dataset, and deterministic synthetic vectors for exact
high-dimensional scaling. Every reported row aggregates five independent
processes as median +/- median absolute deviation (MAD).
+> **Historical scope (pre-microbatch):** Every result below was collected from
+> the measured revisions named in this document, before opt-in request
+> micro-batching and its warm admission fast path were implemented. Statements
+> about the "current integration" describe those measured revisions, not the
+> later implementation. The tables are retained as historical engineering
+> checkpoints and must not be used to infer current concurrent-request
+> throughput.
+
## Public-dataset setup
- Harness revision: `5dbc6a570404c9d1e2c8b584f6447919d0fee62a`
@@ -72,10 +80,10 @@ budget keeps the native path for that query and allows a later retry.
## Batch size 1
-This is the closest index-level approximation of the current OpenViking
-single-query integration. It includes Python dispatch, host-to-device query
-copy, GPU execution, and result copy back to host. It excludes embedding,
-HTTP, record lookup, reranking, and LLM work.
+This is the closest index-level approximation of the measured pre-microbatch
+OpenViking single-query integration. It includes Python dispatch,
+host-to-device query copy, GPU execution, and result copy back to host. It
+excludes embedding, HTTP, record lookup, reranking, and LLM work.
| Backend | Recall@10 | warm p50 (ms/query) | warm p95 (ms/query) | warm QPS |
| --- | ---: | ---: | ---: | ---: |
@@ -408,8 +416,8 @@ cuVS keeps a strong low-concurrency advantage, but its throughput remains near
615--677 QPS as concurrency increases. Native CPU search scales across the
thread pool and crosses cuVS between concurrency 4 and 16. cuVS p50 rises from
1.5 ms to 92.6 ms at concurrency 64 because requests queue behind the current
-index-wide lock; this integration does not yet use concurrent CUDA streams or
-micro-batching.
+index-wide lock; the measured revision did not yet use concurrent CUDA streams
+or micro-batching.
### Cached 10% filter
@@ -462,7 +470,7 @@ the same rebuild delay amortized across more requests released together.
2. CAGRA requires recall-matched tuning; an approximate label alone does not
imply better performance.
3. CAGRA can improve batched capacity around Recall@10=0.96, but this benefit
- requires batching that the current OpenViking integration does not expose.
+ required batching that the measured pre-microbatch revision did not expose.
4. Collection-level unfiltered lookup retained an approximately 9.0x warm-p50
benefit at 100K x 768D in these runs, but native int8 versus the configured
cuVS float32 device representation is not an equal-memory or equal-quality
@@ -480,10 +488,10 @@ the same rebuild delay amortized across more requests released together.
8. GloVe-100 and normalized Gaussian vectors are engineering datasets, not a
representative agent-memory corpus.
9. At the async service facade, cuVS wins cached tenant search through
- concurrency 4, but native crosses it by concurrency 16. The current cuVS
- lock holds throughput near 600--680 QPS and turns concurrency into queueing
- latency rather than GPU parallelism.
-10. The next engineering step is to evaluate a read-safe immutable snapshot
- with concurrent searches or micro-batching, plus background rebuild. A
- later end-to-end matrix should then add HTTP, embedding, reranking, and a
- real agent-memory workload.
+ concurrency 4, but native crosses it by concurrency 16. On the measured
+ revision, the cuVS lock holds throughput near 600--680 QPS and turns
+ concurrency into queueing latency rather than GPU parallelism.
+10. For this historical checkpoint, the next engineering step was to evaluate
+ a read-safe immutable snapshot with concurrent searches or micro-batching,
+ plus background rebuild. A later end-to-end matrix should then add HTTP,
+ embedding, reranking, and a real agent-memory workload.
diff --git a/benchmark/cuvs/README.md b/benchmark/cuvs/README.md
index 75117ec98..ac9a6f178 100644
--- a/benchmark/cuvs/README.md
+++ b/benchmark/cuvs/README.md
@@ -181,11 +181,16 @@ vector limit disables the supplied ground truth because neighbor IDs may refer
to omitted rows; include `native` or `cuvs_brute_force` in that case. A query
limit retains the corresponding prefix of supplied ground truth.
-`query_batch_size=1` reflects the current OpenViking integration most closely.
-Larger batches measure the vector-index capacity: cuVS executes the batch on
-the GPU, while the current native wrapper processes each query sequentially.
-Do not present batch results as current server throughput without also running
-the collection/server benchmarks.
+`query_batch_size=1` most closely represents one public OpenViking API request
+and the default configuration, where request micro-batching is disabled.
+Opt-in brute-force micro-batching can internally coalesce compatible concurrent
+single-row API requests into one matrix-query call; that behavior must be
+measured through the collection/service concurrency path rather than inferred
+from this index harness. Larger explicit batches measure vector-index capacity:
+cuVS executes the batch on the GPU, while the current native wrapper processes
+each query sequentially. Do not present explicit batch results as server
+throughput without also running the collection/server benchmarks and reporting
+whether request micro-batching was enabled.
## First comparison matrix
diff --git a/docs/design/openviking-cuvs-benchmark-plan.md b/docs/design/openviking-cuvs-benchmark-plan.md
index 09d0c0029..aa47e0585 100644
--- a/docs/design/openviking-cuvs-benchmark-plan.md
+++ b/docs/design/openviking-cuvs-benchmark-plan.md
@@ -184,13 +184,22 @@ ground truth 计算 recall。
## 当前集成的吞吐限制
-当前 Python integration 一次只提交一个 query,并在一次 GPU search 期间持有 index lock;同时
-每次调用都会创建 query device array 并把结果同步回 host。因此:
-
-- batch > 1 的 L1 结果代表 cuVS 的能力上限,不等同于当前 OpenViking API 的吞吐;
-- L2/L3 必须以 batch=1 为主,并明确记录并发请求是否被 lock 串行化;
-- 如果 L1 与 L2 差距明显,下一阶段优先实现 persistent device buffers、CUDA streams 或
- dynamic batching,再评估最大吞吐。
+默认路径仍按一次公开 API 请求提交一个 query。当前已提供默认关闭、仅适用于 brute-force 的
+request micro-batching:它可把兼容的并发单 query 请求合并为最多 8 行的 matrix-query。
+对于当前 generation 的 clean immutable snapshot,无 filter 或 device filter cache hit 可通过
+warm fast path 直接入队,不获取 caller 侧 device gate;dirty/cold snapshot、filter cache miss
+和 device filter preparation 仍先走串行的 gated preparation。入队后只有 micro-batch worker
+会在持有 device-search gate 时执行 GPU matrix search,caller 不会持 gate 等待结果。
+
+这消除了旧版“一次 GPU search 全程持有 caller/index lock”的主要并发瓶颈,但每次 dispatch
+仍会创建 query/result device array,并在返回前把结果同步到 host。因此:
+
+- batch > 1 的 L1 结果仍代表显式 vector-index batch 的能力上限,不等同于公开 API 吞吐;
+- L2/L3 应继续以单 query 请求为主,同时分别测默认关闭和 opt-in micro-batching,并记录
+ 实际 batch-size 分布、`batch_wait`、`gpu_gate_queue`、QPS 与 P95;
+- 下一阶段优先评估 persistent query/result buffers、allocator reuse 和 host 同步优化,再决定
+ 是否支持 CAGRA micro-batching 或多个并行 batch dispatch;OpenViking 的 scheduler 与 cuVS
+ 官方的 Dynamic Batching 组件不是同一实现。
## 主要图表
diff --git a/docs/design/openviking-cuvs-integration-plan.md b/docs/design/openviking-cuvs-integration-plan.md
index bb48122b1..01f0cb1f4 100644
--- a/docs/design/openviking-cuvs-integration-plan.md
+++ b/docs/design/openviking-cuvs-integration-plan.md
@@ -336,6 +336,9 @@ URI/path 使用更低阈值,是因为宽路径需要 native Trie traversal 和
| `algorithm` | `brute_force` | `brute_force` exact 或 `cagra` ANN |
| `dtype` | `float32` | GPU dataset/query dtype;可显式设为 `float16`,不改变 native CPU dtype |
| `max_concurrent_gpu_searches` | `1` | 单 index 同时进入 GPU search 的上限;host preflight/filter 仍可并行 |
+| `micro_batching_enabled` | `false` | 可选地合并兼容的并发单 query 请求;仅支持 `brute_force`,且要求 `max_concurrent_gpu_searches=1` |
+| `micro_batching_max_batch_size` | `8` | 单次 matrix-query 的最大行数,范围 1 到 8 |
+| `micro_batching_max_wait_ms` | `1.0` | 等待兼容请求的 collection window,范围 0 到 100 ms;0 表示不主动等待、仅 opportunistic batching |
| `build_params` | `{}` | 传给 CAGRA `IndexParams` |
| `search_params` | `{}` | 传给 CAGRA `SearchParams` |
| `fallback_to_native` | `true` | sparse/hybrid 等非 cuVS dense top-k 能力使用 native |
@@ -358,7 +361,11 @@ URI/path 使用更低阈值,是因为宽路径需要 native Trie traversal 和
- native mutation 与 cuVS shadow mutation 由跨后端读写锁协调;filter layout 和 bitmap
preflight 由 native engine 读写锁及 cuVS generation 校验保证一致性;
- GPU index、dataset ownership、label mapping 和 generation 组成 immutable snapshot;
-- warmed GPU search 使用不同的 cuVS resources/CUDA stream 并发执行,只短暂串行访问 host cache;
+- 默认非 micro-batch 路径可使用复用的 cuVS resources/CUDA stream;启用 micro-batching 后,
+ compatible warm query 可在不获取 caller 侧 device gate 的情况下入队,micro-batch worker
+ 是唯一在持有 device-search gate 时执行 matrix search 的组件;
+- dirty/cold snapshot、device filter cache miss 和 rebuild 仍在同一 gate 内完成 GPU preparation,
+ caller 入队后立即释放 gate,不会持 gate 等待 worker;
- mutation 不会破坏已被查询持有的 snapshot;同步 rebuild 仍由写锁合并为一次;
- 同一 GPU 上不同 collection 的 admission/build 由进程级 coordinator 串行,避免并发超卖显存;
- native fallback 不持有 GPU 锁,可以继续并发读取;
@@ -390,7 +397,7 @@ URI/path 使用更低阈值,是因为宽路径需要 native Trie traversal 和
| --- | --- | --- |
| 每次 mutation 后整体重建 | 写后首查和 build 为 O(N) | base + delta、阈值重建、后台 build、原子切换 |
| host shadow 保存完整 dense vectors | Store/native/GPU 之外增加 host memory | 连续 buffer、共享内存、减少 Python object |
-| 高并发小 query | 单 query 无法充分占用 GPU | 动态 batching 或显式 query batching |
+| 高并发小 query | 已有 opt-in micro-batching,但仍有每次 dispatch 的 device allocation、host sync 和单 worker 上限 | persistent buffers、allocator reuse、CAGRA batching 或多个并行 batch dispatch |
| Python/CuPy 数据面 | 对象构造、复制和同步有固定开销 | 先 profiling,再决定是否下沉 C++ cuVS C API |
| 重启后 lazy rebuild | 大 collection 的首次查询延迟高 | 后台预热、版本化派生 cache |
| native dense 与 GPU shadow 并存 | CPU 内存和写放大 | 覆盖率和回退策略稳定后再评估裁剪 |
@@ -429,7 +436,9 @@ URI/path 使用更低阈值,是因为宽路径需要 native Trie traversal 和
2. 已保证 in-flight query 在 rebuild 期间安全持有旧 snapshot;后台 build 期间的新请求路由仍待实现;
3. 已实现完成后原子交换,并延迟到最后一个持有者退出再回收旧资源;
4. 已实现可选的连续 mutation 合并与后台 rebuild;默认关闭以保留现有 auto 行为;
-5. 已实现每线程 cuVS resources/CUDA stream;persistent query buffers 和 micro-batching 待评估;
+5. 已实现可复用 cuVS resources/CUDA stream,以及默认关闭的 brute-force request
+ micro-batching 和 warm admission fast path;persistent query/result buffers、allocator reuse、
+ CAGRA batching 和多个并行 batch dispatch 待评估;
6. 已补充 build queue、fallback reason、eligible count 和分阶段 latency telemetry;VRAM 指标待统一管理器。
### Phase 3:容量与低精度
diff --git a/docs/en/guides/07-operation-telemetry.md b/docs/en/guides/07-operation-telemetry.md
index f71ecfb15..4595ec6ed 100644
--- a/docs/en/guides/07-operation-telemetry.md
+++ b/docs/en/guides/07-operation-telemetry.md
@@ -189,6 +189,11 @@ coalesced into the bounded `other` bucket.
| `summary.vector.cuvs.dtypes.<dtype>` | Search count by GPU dataset/query dtype, `float32` or `float16` |
| `summary.vector.cuvs.max_concurrent_gpu_searches` | Maximum configured per-index in-flight GPU search limit observed |
| `summary.vector.cuvs.auto_mode_searches` | Number of searches with automatic CPU/GPU routing enabled |
+| `summary.vector.cuvs.micro_batching_searches` | Number of searches that used the opt-in OpenViking micro-batch scheduler |
+| `summary.vector.cuvs.micro_batched_searches` | Number of those searches dispatched with more than one query row |
+| `summary.vector.cuvs.micro_batching_warm_fast_path_searches` | Number of micro-batching-scheduler searches enqueued from a clean current snapshot without caller-side device-gate admission |
+| `summary.vector.cuvs.batch_size_max` | Maximum query-row count observed in one shared cuVS call |
+| `summary.vector.cuvs.searches_by_batch_size.<size>` | Search count by bounded batch size from 1 through 8 |
| `summary.vector.cuvs.routes.<reason>` | Search count by route, such as `cuvs`, `native_filter_threshold`, `native_rebuild_pending`, or `native_memory_budget` |
| `summary.vector.cuvs.filter_kinds.<kind>` | Search count by low-cardinality filter class: `none`, `scalar`, or `path` |
| `summary.vector.cuvs.filter_cache_hits` | Number of searches that reused a prepared or preflight filter |
@@ -200,9 +205,22 @@ coalesced into the bounded `other` bucket.
| `summary.vector.cuvs.memory.estimated_peak_bytes_max` | Maximum estimated peak bytes used for auto-build admission |
| `summary.vector.cuvs.memory.free_bytes_min` | Minimum free device bytes observed inside the per-GPU admission coordinator |
| `summary.vector.cuvs.memory.usable_bytes_min` | Minimum free bytes observed after applying the configured reserve |
-| `summary.vector.cuvs.timings_ms.<stage>.sum` | Sum across searches for `total`, `preflight`, `queue`, `build`, `filter_prepare`, `gpu_search`, or `native_search` |
+| `summary.vector.cuvs.timings_ms.<stage>.sum` | Sum across searches for `total`, `preflight`, `queue`, `gpu_gate_queue`, `build`, `filter_prepare`, `batch_wait`, `gpu_search`, or `native_search` |
| `summary.vector.cuvs.timings_ms.<stage>.max` | Maximum single-search time for the same stage |
+For a shared micro-batch, `gpu_search` is the cuVS call's service latency from
+each member request's perspective. Its duration is therefore recorded once per
+request: `gpu_search.sum` is useful as aggregate request latency, but it is not
+GPU busy time and can be approximately batch-size times the physical call
+duration. `batch_wait` is the request's time from scheduler enqueue to GPU
+dispatch, including worker scheduling and any wait behind current device work.
+`gpu_gate_queue` is caller-side time waiting to enter the serialized device
+gate before gated rebuild/filter preparation, enqueue, or a non-batched GPU
+search. A warm-fast-path request bypasses that caller admission, so its
+`gpu_gate_queue` is zero; its worker-side wait remains part of `batch_wait`.
+The broader `queue` stage includes both forms of request-visible queueing when
+they apply.
+
### `summary.resource`
This group appears on resource ingestion operations such as `resources.add_resource`.
diff --git a/docs/en/guides/16-cuvs.md b/docs/en/guides/16-cuvs.md
index 6e54960fe..baffec25c 100644
--- a/docs/en/guides/16-cuvs.md
+++ b/docs/en/guides/16-cuvs.md
@@ -27,6 +27,7 @@ Start with exact brute-force search:
"algorithm": "brute_force",
"dtype": "float32",
"max_concurrent_gpu_searches": 1,
+ "micro_batching_enabled": false,
"fallback_to_native": true,
"filter_cache_size": 16
}
@@ -167,16 +168,16 @@ design because OpenViking uses a per-vector scale that cuVS brute-force does
not accept directly. CAGRA int8 or PQ compression must likewise be evaluated
as approximate modes with an explicit recall/latency/memory frontier.
-The integration uses immutable GPU snapshots. Warmed searches use per-thread
-cuVS resources/CUDA streams, while mutation and snapshot commit use a
-cross-backend writer lock. Host-side filter and snapshot work can proceed in
-parallel, but `max_concurrent_gpu_searches` defaults to 1 because concurrent
-single-query brute-force kernels can contend for memory bandwidth and reduce
-throughput. Increase it only after measuring the target GPU and workload. By
-default, the first query after an upsert or delete rebuilds synchronously;
-optional background rebuild changes dirty queries to native fallback until the
-new snapshot is ready. On each rebuild it registers the cuVS label order with
-the native engine once.
+The integration uses immutable GPU snapshots and reusable cuVS resources/CUDA
+streams, while mutation and snapshot commit use a cross-backend writer lock.
+Host-side filter and snapshot work can proceed in parallel, but
+`max_concurrent_gpu_searches` defaults to 1 because concurrent single-query
+brute-force kernels can contend for memory bandwidth and reduce throughput.
+Increase it only after measuring the target GPU and workload. By default, the
+first query after an upsert or delete rebuilds synchronously; optional
+background rebuild changes dirty queries to native fallback until the new
+snapshot is ready. On each rebuild it registers the cuVS label order with the
+native engine once.
The first use of a scalar or URI filter then reuses OpenViking's native
scalar/path index and projects its bitmap into cuVS row order; it does not scan
all host-side records in Python. `filter_cache_size` retains the resulting
@@ -186,6 +187,68 @@ Different unseen filters can use the native engine's shared-read path in
parallel, while cached native routing decisions go directly to the native
index. A record-generation check prevents a result computed across a mutation
from entering the route cache.
+
+### Opt-in request micro-batching
+
+Exact brute-force search can coalesce compatible concurrent requests into one
+cuVS matrix-query call:
+
+```json
+{
+ "storage": {
+ "vectordb": {
+ "backend": "cuvs",
+ "cuvs": {
+ "algorithm": "brute_force",
+ "max_concurrent_gpu_searches": 1,
+ "micro_batching_enabled": true,
+ "micro_batching_max_batch_size": 8,
+ "micro_batching_max_wait_ms": 1.0
+ }
+ }
+ }
+}
+```
+
+The scheduler batches only requests using the same immutable GPU snapshot,
+prepared filter, and effective top-k. Each result row is mapped back to its
+original request, so scalar/path filtering and result limits retain their
+existing semantics.
+
+A request over a clean, current-generation snapshot can take the warm
+admission fast path when it has no filter or its prepared device filter is
+already cached. That path pins the immutable snapshot and filter, then enqueues
+without the caller acquiring the device-search gate. Dirty, cold, or stale
+snapshots, device-filter cache misses or evictions, rebuilds, and device-filter
+materialization use the gated preparation path. After preparation, the caller
+enqueues and releases the gate before waiting. The micro-batch worker is the
+only component that executes matrix search while holding the device-search
+gate, so a caller never holds that gate while waiting for the worker.
+
+The collection window is a latency-throughput tradeoff. It bounds the
+scheduler's deliberate collection delay to at most the configured value from
+the oldest compatible request; it does not bound total enqueue-to-dispatch
+latency. Worker scheduling, a preceding GPU call, or gated device preparation
+can make dispatch occur later. Under sufficient concurrency up to the
+configured maximum number of queries share one GPU call.
+
+The options are validated as follows:
+
+- `micro_batching_max_batch_size` must be from 1 through 8.
+- `micro_batching_max_wait_ms` must be from 0 through 100 ms. A value of `0`
+ disables deliberate collection delay but can still opportunistically combine
+ compatible requests already queued together.
+- Micro-batching supports only `algorithm: "brute_force"` and requires
+ `max_concurrent_gpu_searches: 1`.
+
+This OpenViking micro-batcher is disabled by default and is distinct from the
+cuVS feature named Dynamic Batching. Its first version supports
+exact brute-force; CAGRA and multiple concurrent batch dispatches are rejected
+until separately validated. Auto mode can use the same options, but requests
+routed to the native CPU path never enter the GPU batch queue. Near-tie
+neighbor ordering may differ between single-row and matrix-query calls;
+validate set overlap and scores when tuning for a workload.
+
Sparse/hybrid queries fall back to OpenViking's native local index when
`fallback_to_native` is enabled. The canonical vectors remain in the local
store and repopulate cuVS after restart.
diff --git a/docs/zh/guides/07-operation-telemetry.md b/docs/zh/guides/07-operation-telemetry.md
index 9633d6a95..5a5e4acb5 100644
--- a/docs/zh/guides/07-operation-telemetry.md
+++ b/docs/zh/guides/07-operation-telemetry.md
@@ -187,6 +187,11 @@ CPU/GPU 路由和分阶段耗时;并发 query 的完成顺序不会改变结
| `summary.vector.cuvs.dtypes.<dtype>` | 按 GPU dataset/query dtype 统计的 search 数:`float32` 或 `float16` |
| `summary.vector.cuvs.max_concurrent_gpu_searches` | 观测到的单 index in-flight GPU search 配置上限最大值 |
| `summary.vector.cuvs.auto_mode_searches` | 启用自动 CPU/GPU 路由的 search 数量 |
+| `summary.vector.cuvs.micro_batching_searches` | 使用 OpenViking 可选 micro-batch scheduler 的 search 数量 |
+| `summary.vector.cuvs.micro_batched_searches` | 其中以多于一行 query 共同 dispatch 的 search 数量 |
+| `summary.vector.cuvs.micro_batching_warm_fast_path_searches` | 使用 micro-batch scheduler、从 clean current snapshot 入队且未经过 caller 侧 device-gate admission 的 search 数量 |
+| `summary.vector.cuvs.batch_size_max` | 单次共享 cuVS call 观测到的最大 query 行数 |
+| `summary.vector.cuvs.searches_by_batch_size.<size>` | 按 1 到 8 的有界 batch size 统计 search 数量 |
| `summary.vector.cuvs.routes.<reason>` | 按路由原因统计的 search 数,例如 `cuvs`、`native_filter_threshold`、`native_rebuild_pending` 或 `native_memory_budget` |
| `summary.vector.cuvs.filter_kinds.<kind>` | 按低基数 filter 类型统计的 search 数:`none`、`scalar` 或 `path` |
| `summary.vector.cuvs.filter_cache_hits` | 复用 prepared/preflight filter 的 search 数量 |
@@ -198,9 +203,18 @@ CPU/GPU 路由和分阶段耗时;并发 query 的完成顺序不会改变结
| `summary.vector.cuvs.memory.estimated_peak_bytes_max` | auto build 准入使用的最大峰值显存估算 |
| `summary.vector.cuvs.memory.free_bytes_min` | 单 GPU 准入协调器内观测到的最小空闲显存 |
| `summary.vector.cuvs.memory.usable_bytes_min` | 扣除配置 reserve 后观测到的最小可用显存 |
-| `summary.vector.cuvs.timings_ms.<stage>.sum` | `total`、`preflight`、`queue`、`build`、`filter_prepare`、`gpu_search` 或 `native_search` 阶段跨 search 的耗时总和 |
+| `summary.vector.cuvs.timings_ms.<stage>.sum` | `total`、`preflight`、`queue`、`gpu_gate_queue`、`build`、`filter_prepare`、`batch_wait`、`gpu_search` 或 `native_search` 阶段跨 search 的耗时总和 |
| `summary.vector.cuvs.timings_ms.<stage>.max` | 同一阶段单次 search 的最大耗时 |
+对于共享 micro-batch,`gpu_search` 表示每个成员请求所感知的同一次 cuVS call
+service latency,因此会按请求各记录一次。`gpu_search.sum` 适合解释请求侧累计延迟,
+但不等同于 GPU busy time;它可能约为物理 call 时长乘以 batch size。`batch_wait`
+表示请求从进入 scheduler 到 GPU dispatch 的等待时间,包括 worker 调度和等待当前 device
+work 完成的时间。`gpu_gate_queue` 表示 caller 在 gated rebuild/filter preparation、入队或
+非 batch GPU search 前,等待进入串行 device gate 的时间。warm fast path 会跳过这次 caller
+admission,所以其 `gpu_gate_queue` 为零;worker 侧等待仍计入 `batch_wait`。更宽泛的 `queue`
+阶段会在适用时包含这两类请求可感知的排队时间。
+
### `summary.resource`
这个分组常见于 `resources.add_resource` 这类资源导入操作。
diff --git a/docs/zh/guides/16-cuvs.md b/docs/zh/guides/16-cuvs.md
index a54de4c43..213b546d4 100644
--- a/docs/zh/guides/16-cuvs.md
+++ b/docs/zh/guides/16-cuvs.md
@@ -41,6 +41,7 @@ headers;即使宿主机已有 CUDA driver、但没有完整 toolkit,也建
"algorithm": "brute_force",
"dtype": "float32",
"max_concurrent_gpu_searches": 1,
+ "micro_batching_enabled": false,
"fallback_to_native": true,
"filter_cache_size": 16
}
@@ -186,11 +187,62 @@ truth 报告 Recall@K。与 native 兼容的 int8 仍需单独设计,因为 Op
逐向量 scale,而 cuVS brute-force 不能直接接收这种 scaled-int8 表示。CAGRA
int8 或 PQ compression 也应作为近似模式,单独报告 recall/latency/memory frontier。
-集成使用 immutable GPU snapshot 和每线程 cuVS resource/CUDA stream。host 侧
+集成使用 immutable GPU snapshot 和可复用的 cuVS resource/CUDA stream。host 侧
filter 与 snapshot 工作可以并行,但 `max_concurrent_gpu_searches` 默认是 1:
单 query brute-force 通常受显存带宽限制,并发 kernel 可能互相争抢带宽、反而降低
吞吐。只有在目标 GPU 与真实 workload 上测得收益后,才建议显式调大该值。
+### 可选的请求微批处理
+
+精确 brute-force 路径可以把兼容的并发请求合并为一次 cuVS matrix-query 调用:
+
+```json
+{
+ "storage": {
+ "vectordb": {
+ "backend": "cuvs",
+ "cuvs": {
+ "algorithm": "brute_force",
+ "max_concurrent_gpu_searches": 1,
+ "micro_batching_enabled": true,
+ "micro_batching_max_batch_size": 8,
+ "micro_batching_max_wait_ms": 1.0
+ }
+ }
+ }
+}
+```
+
+scheduler 只会合并使用同一个 immutable GPU snapshot、同一个 prepared filter、
+同一个实际 top-k 的请求;GPU 返回的每一行会映射回原请求,因此标量/路径过滤和
+结果条数语义不变。
+
+当 immutable snapshot clean、属于当前 generation,且请求没有 filter 或命中已准备好的
+device filter cache 时,可走 warm admission fast path。该路径会 pin snapshot/filter,
+并在 caller 不获取 device-search gate 的情况下直接入队。dirty、cold 或 stale snapshot,
+device filter cache miss/eviction、rebuild 和 device filter materialization 仍走 gated
+preparation。准备完成后,caller 先入队并释放 gate,再等待结果;只有 micro-batch worker
+会在持有 device-search gate 时执行 matrix search,所以 caller 不会持 gate 等待 worker。
+
+collection window 是延迟与吞吐的权衡。它只限制 scheduler 为收集兼容请求而主动等待的
+时间:从最早的 compatible request 起最多主动等待配置值;它不是 enqueue-to-dispatch
+latency 上限。worker 调度、前一个 GPU call 或 gated device preparation 都可能使实际
+dispatch 更晚。并发充足时,最多由配置上限数量的 query 共用一次 GPU call。
+
+参数约束如下:
+
+- `micro_batching_max_batch_size` 范围为 1 到 8;
+- `micro_batching_max_wait_ms` 范围为 0 到 100 ms;设为 `0` 表示不主动等待,但仍可
+ opportunistically 合并已经同时在队列中的兼容请求;
+- micro-batching 仅支持 `algorithm: "brute_force"`,并要求
+ `max_concurrent_gpu_searches: 1`。
+
+该能力默认关闭,是 OpenViking 自己的 micro-batcher,不等同于 cuVS 官方名为
+Dynamic Batching 的组件。首版只支持 exact brute-force;CAGRA 和并发 dispatch 多个
+batch 会在独立验证后再开放。Auto 模式也可使用这些选项,但被路由到原生 CPU 的请求
+不会进入 GPU batch queue。single-row 与 matrix-query 在近似并列分数处可能有顺序
+差异,调参时应同时验证结果集合重合度和 score。
+
## 最小功能验证
仓库提供的 smoke test 不依赖 embedding 或 VLM 服务:
@@ -260,7 +312,7 @@ collection.close()
- local 集成通过 native scalar/path index 生成 prefilter,因此继承原生 DSL、`date_time`、`geo_point` 和 path depth 的过滤语义,而不是在 Python 重复实现。
- 每次 GPU rebuild 会向 native engine 注册一次 cuVS label 顺序。新过滤条件直接复用 native scalar/path index 的 bitmap,再投影为 cuVS row bitset,不再用 Python 扫描所有 host-side records。
- `filter_cache_size` 会保留最近使用的 GPU bitset 或 native 路由决策,并在数据更新时失效;auto 模式在进入 cuVS search 前预判候选数,不同的首次过滤条件可通过 native engine 的共享读路径并行计算,命中已缓存的 native 路由时则直接进入 native index。generation 校验会阻止跨 mutation 计算出的旧结果写入路由缓存。
-- GPU index 使用 immutable snapshot;warmed search 通过不同的 cuVS resources/CUDA stream 并发执行,mutation 和 snapshot commit 使用跨后端写锁。
+- GPU index 使用 immutable snapshot 和可复用的 cuVS resources/CUDA stream;默认关闭的 micro-batching 可让 compatible warm request 绕过 caller 侧 gate 入队,并由唯一持有 device-search gate 执行 matrix search 的 worker 合批。mutation 和 snapshot commit 使用跨后端写锁。
- 默认情况下,每次 upsert/delete 后仍由下一次查询同步重建;开启 `auto_background_rebuild` 后,dirty 期间查询走 native,连续写被合并为后台重建。
- cuVS 索引不作为权威持久化数据;进程重启时会从 OpenViking 本地 store 重建,因此不受 cuVS 跨版本序列化格式变化影响。
- `brute_force` 适合功能对齐和 ground truth;CAGRA 的 graph/search 参数需要在后续结合召回率、QPS、延迟和显存进行调优。
diff --git a/openviking/storage/vectordb/index/cuvs_index.py b/openviking/storage/vectordb/index/cuvs_index.py
index 303b1a15c..2e202af54 100644
--- a/openviking/storage/vectordb/index/cuvs_index.py
+++ b/openviking/storage/vectordb/index/cuvs_index.py
@@ -26,6 +26,7 @@ import time
import traceback
from array import array
from collections import OrderedDict
+from concurrent.futures import Future
from contextlib import nullcontext
from dataclasses import dataclass, field
from typing import (
@@ -99,6 +100,9 @@ class CuVSSearchTelemetry:
auto_mode: bool
dtype: str = "float32"
max_concurrent_gpu_searches: int = 1
+ micro_batching_enabled: bool = False
+ micro_batching_warm_fast_path: bool = False
+ batch_size: int = 1
route_reason: str = "pending"
filter_kind: str = "none"
filter_cache_hit: bool = False
@@ -119,6 +123,7 @@ class CuVSSearchTelemetry:
build_ms: float = 0.0
filter_prepare_ms: float = 0.0
gpu_search_ms: float = 0.0
+ batch_wait_ms: float = 0.0
native_search_ms: float = 0.0
def as_dict(self) -> Dict[str, Any]:
@@ -127,6 +132,9 @@ class CuVSSearchTelemetry:
"auto_mode": self.auto_mode,
"dtype": self.dtype,
"max_concurrent_gpu_searches": self.max_concurrent_gpu_searches,
+ "micro_batching_enabled": self.micro_batching_enabled,
+ "micro_batching_warm_fast_path": self.micro_batching_warm_fast_path,
+ "batch_size": self.batch_size,
"route_reason": self.route_reason,
"filter_kind": self.filter_kind,
"filter_cache_hit": self.filter_cache_hit,
@@ -147,6 +155,7 @@ class CuVSSearchTelemetry:
"build_ms": round(self.build_ms, 3),
"filter_prepare_ms": round(self.filter_prepare_ms, 3),
"gpu_search_ms": round(self.gpu_search_ms, 3),
+ "batch_wait_ms": round(self.batch_wait_ms, 3),
"native_search_ms": round(self.native_search_ms, 3),
}
@@ -652,11 +661,32 @@ class _CuVSRuntime:
limit: int,
mask: Optional[Any],
) -> Tuple[List[int], List[float]]:
+ batch_neighbors, batch_distances = self.search_batch(
+ runtime_index,
+ [query],
+ limit,
+ mask,
+ )
+ return batch_neighbors[0], batch_distances[0]
+
+ def search_batch(
+ self,
+ runtime_index: "_CuVSRuntimeIndex",
+ queries: Sequence[Sequence[float]],
+ limit: int,
+ mask: Optional[Any],
+ ) -> Tuple[List[List[int]], List[List[float]]]:
+ """Run compatible query rows in one cuVS call and preserve row order."""
+
+ if not queries:
+ return [], []
with self.device_scope():
- queries = None
+ device_queries = None
prefilter = None
distances = None
neighbors = None
+ host_distances = None
+ host_neighbors = None
resources = None
resource_kwargs = None
resource_reusable = False
@@ -670,7 +700,8 @@ class _CuVSRuntime:
# boundary inside this admitted search.
resources = self._acquire_resources()
resource_kwargs = {"resources": resources}
- queries = self.cp.asarray([query], dtype=self.device_dtype)
+ query_count = len(queries)
+ device_queries = self.cp.asarray(queries, dtype=self.device_dtype)
if mask is None:
prefilter = None
elif isinstance(mask, self.cp.ndarray) and mask.dtype == self.cp.uint32:
@@ -680,7 +711,7 @@ class _CuVSRuntime:
if self.algorithm == "brute_force":
distances, neighbors = self.brute_force.search(
index,
- queries,
+ device_queries,
limit,
prefilter=prefilter,
**resource_kwargs,
@@ -694,17 +725,23 @@ class _CuVSRuntime:
distances, neighbors = self.cagra.search(
params,
index,
- queries,
+ device_queries,
limit,
filter=prefilter,
**resource_kwargs,
)
resources.sync()
- host_neighbors = self.cp.asnumpy(neighbors)[0].tolist()
- host_distances = self.cp.asnumpy(distances)[0].tolist()
+ host_neighbors = self.cp.asnumpy(neighbors)
+ host_distances = self.cp.asnumpy(distances)
result = (
- [int(item) for item in host_neighbors],
- [float(item) for item in host_distances],
+ [
+ [int(item) for item in host_neighbors[row].tolist()]
+ for row in range(query_count)
+ ],
+ [
+ [float(item) for item in host_distances[row].tolist()]
+ for row in range(query_count)
+ ],
)
# Reuse only after the complete call, including host result
# materialization, succeeds. Any exception discards a
@@ -717,10 +754,12 @@ class _CuVSRuntime:
finally:
# Search temporaries otherwise outlive Device.__exit__ as frame
# locals and may free allocations on the worker's prior device.
- queries = None
+ device_queries = None
prefilter = None
distances = None
neighbors = None
+ host_distances = None
+ host_neighbors = None
try:
if resources is not None:
self._return_resources(resources, reusable=resource_reusable)
@@ -759,6 +798,31 @@ class _CuVSIndexSnapshot:
generation: int
+@dataclass
+class _CuVSMicroBatchRequest:
+ """One admitted search waiting for a compatible matrix-query dispatch."""
+
+ snapshot: Optional[_CuVSIndexSnapshot]
+ query: Tuple[float, ...]
+ result_limit: int
+ mask: Optional[Any]
+ filter_identity: Union[str, int]
+ telemetry: Optional[CuVSSearchTelemetry]
+ enqueued_at: float
+ future: Future[Tuple[List[int], List[float]]]
+ warm_fast_path: bool = False
+ released: bool = False
+
+ @property
+ def key(self) -> Tuple[int, Union[str, int], int]:
+ snapshot = self.snapshot
+ return (
+ id(snapshot),
+ self.filter_identity,
+ self.result_limit,
+ )
+
+
@dataclass
class _CuVSBuildCandidate:
runtime_index: Any
@@ -869,6 +933,21 @@ class CuVSDenseIndex:
self.max_concurrent_gpu_searches = int(config.get("max_concurrent_gpu_searches", 1))
if self.max_concurrent_gpu_searches < 1:
raise ValueError("cuVS max_concurrent_gpu_searches must be at least 1")
+ self.micro_batching_enabled = bool(config.get("micro_batching_enabled", False))
+ self.micro_batching_max_batch_size = int(config.get("micro_batching_max_batch_size", 8))
+ if not 1 <= self.micro_batching_max_batch_size <= 8:
+ raise ValueError("cuVS micro_batching_max_batch_size must be between 1 and 8")
+ self.micro_batching_max_wait_ms = float(config.get("micro_batching_max_wait_ms", 1.0))
+ if (
+ not math.isfinite(self.micro_batching_max_wait_ms)
+ or self.micro_batching_max_wait_ms < 0.0
+ or self.micro_batching_max_wait_ms > 100.0
+ ):
+ raise ValueError("cuVS micro_batching_max_wait_ms must be between 0 and 100")
+ if self.micro_batching_enabled and self.algorithm != "brute_force":
+ raise ValueError("cuVS micro-batching currently supports algorithm='brute_force' only")
+ if self.micro_batching_enabled and self.max_concurrent_gpu_searches != 1:
+ raise ValueError("cuVS micro-batching currently requires max_concurrent_gpu_searches=1")
self.auto_memory_reserve_bytes = (
int(config.get("auto_memory_reserve_mb", 1024)) * 1024 * 1024
)
@@ -906,6 +985,10 @@ class CuVSDenseIndex:
set_max_concurrent_searches = getattr(self._runtime, "set_max_concurrent_searches", None)
if set_max_concurrent_searches is not None:
set_max_concurrent_searches(self.max_concurrent_gpu_searches)
+ if self.micro_batching_enabled and not callable(
+ getattr(self._runtime, "search_batch", None)
+ ):
+ raise ValueError("The configured cuVS runtime does not support batched search")
self._records: Dict[int, _Record] = {}
self._snapshot: Optional[_CuVSIndexSnapshot] = None
self._dirty = True
@@ -919,13 +1002,31 @@ class CuVSDenseIndex:
self._filter_layout_lock = threading.Lock()
self._records_generation = 0
self._filter_layout_generation = -1
+ self._closing = False
self._closed = False
self._close_complete = False
+ self._micro_batch_condition = threading.Condition(threading.Lock())
+ self._micro_batch_pending: OrderedDict[
+ Tuple[int, Union[str, int], int], List[_CuVSMicroBatchRequest]
+ ] = OrderedDict()
+ self._micro_batch_warm_lookahead = 0
+ self._micro_batch_draining = False
+ self._micro_batch_stop = False
+ self._micro_batch_worker_failure: Optional[BaseException] = None
+ self._micro_batch_worker: Optional[threading.Thread] = None
+ if self.micro_batching_enabled:
+ self._micro_batch_worker = threading.Thread(
+ target=self._micro_batch_worker_main,
+ name="openviking-cuvs-microbatch",
+ daemon=True,
+ )
+ self._micro_batch_worker.start()
logger.info(
- "Initialized cuVS dense index: algorithm=%s metric=%s dimension=%d",
+ "Initialized cuVS dense index: algorithm=%s metric=%s dimension=%d micro_batching=%s",
self.algorithm,
self._metric,
self.dimension,
+ self.micro_batching_enabled,
)
def _runtime_device_scope(self):
@@ -1678,6 +1779,387 @@ class CuVSDenseIndex:
if self.commit_rebuild(candidate, native_filter_layout_registrar):
return
+ def _enqueue_micro_batch(
+ self,
+ *,
+ snapshot: _CuVSIndexSnapshot,
+ query: Tuple[float, ...],
+ result_limit: int,
+ mask: Optional[Any],
+ filter_identity: Union[str, int],
+ telemetry: Optional[CuVSSearchTelemetry],
+ warm_fast_path: bool = False,
+ ) -> Optional[_CuVSMicroBatchRequest]:
+ request = _CuVSMicroBatchRequest(
+ snapshot=snapshot,
+ query=query,
+ result_limit=result_limit,
+ mask=mask,
+ filter_identity=filter_identity,
+ telemetry=telemetry,
+ enqueued_at=time.perf_counter(),
+ future=Future(),
+ warm_fast_path=warm_fast_path,
+ )
+ with self._micro_batch_condition:
+ if self._micro_batch_worker_failure is not None:
+ raise RuntimeError("The cuVS micro-batch worker is unavailable") from (
+ self._micro_batch_worker_failure
+ )
+ if self._micro_batch_stop:
+ raise RuntimeError("The cuVS micro-batch worker is closed")
+ if (
+ warm_fast_path
+ and self._micro_batch_warm_lookahead >= self.micro_batching_max_batch_size
+ ):
+ return None
+ self._micro_batch_pending.setdefault(request.key, []).append(request)
+ if warm_fast_path:
+ self._micro_batch_warm_lookahead += 1
+ self._micro_batch_condition.notify()
+ return request
+
+ def _try_enqueue_warm_micro_batch(
+ self,
+ *,
+ query: Tuple[float, ...],
+ limit: int,
+ filters: Optional[Mapping[str, Any]],
+ prepared_filter: Optional[_PreparedHostFilter],
+ telemetry: Optional[CuVSSearchTelemetry],
+ ) -> Optional[_CuVSMicroBatchRequest]:
+ """Queue one warm request without taking the caller-side GPU gate.
+
+ The worker remains the only owner of the device-search permit. This
+ path merely pins an immutable, current-generation snapshot and an
+ already materialized device filter while the index lock prevents
+ mutation/close from racing the borrow. A single batch worth of warm
+ requests may look ahead; overflow and every cold/stale case use the
+ established gated admission path.
+ """
+
+ # LocalIndex normalizes an omitted public-API filter from None to {},
+ # so both representations are the canonical no-filter case here.
+ no_filter = not filters
+ if no_filter:
+ cache_key: Optional[str] = None
+ else:
+ cache_key = (
+ prepared_filter.cache_key
+ if prepared_filter is not None
+ else self._filter_cache_key(filters)
+ )
+ if cache_key is None:
+ return None
+
+ with self._lock:
+ if self._closing or self._closed:
+ raise RuntimeError("cuVS dense index is closed")
+ generation = self._records_generation
+ if prepared_filter is not None and prepared_filter.generation != generation:
+ # Preserve the established admitted retry path for stale host
+ # work instead of creating a second, fast-path-only retry
+ # protocol.
+ return None
+ snapshot = self._snapshot
+ if self._dirty or snapshot is None or snapshot.generation != generation:
+ return None
+
+ mask: Optional[Any] = None
+ filter_identity: Union[str, int] = "no-filter"
+ if no_filter:
+ result_limit = min(limit, len(snapshot.labels))
+ else:
+ cached_filter = self._get_cached_filter(cache_key)
+ if cached_filter is None:
+ # Host metadata can outlive a size-bounded device LRU
+ # entry. Preserve the existing one-shot in-gate recovery.
+ if (
+ telemetry is not None
+ and prepared_filter is not None
+ and prepared_filter.cached_metadata is not None
+ ):
+ telemetry.filter_cache_hit = False
+ telemetry.filter_cache_eviction_fallback = True
+ return None
+ if (
+ cached_filter.eligible_count == 0
+ or cached_filter.route_native
+ or cached_filter.prepared is None
+ ):
+ return None
+ mask = cached_filter.prepared
+ filter_identity = cache_key
+ result_limit = min(limit, cached_filter.eligible_count)
+ if telemetry is not None:
+ telemetry.filter_cache_hit = True
+ telemetry.eligible_count = cached_filter.eligible_count
+ telemetry.filter_words_packed = cached_filter.filter_words_packed
+
+ if telemetry is not None:
+ telemetry.filter_kind = (
+ "path"
+ if filters and _filter_uses_field_type(filters, self.field_types, "path")
+ else "scalar"
+ if filters
+ else "none"
+ )
+ telemetry.records_generation = generation
+ telemetry.index_size = len(self._records)
+
+ request = self._enqueue_micro_batch(
+ snapshot=snapshot,
+ query=query,
+ result_limit=result_limit,
+ mask=mask,
+ filter_identity=filter_identity,
+ telemetry=telemetry,
+ warm_fast_path=True,
+ )
+ if request is None:
+ return None
+ self._active_searches += 1
+ if telemetry is not None:
+ telemetry.micro_batching_warm_fast_path = True
+ return request
+
+ def _release_micro_batch_warm_lookahead(
+ self,
+ requests: Sequence[_CuVSMicroBatchRequest],
+ ) -> None:
+ with self._micro_batch_condition:
+ released = 0
+ for request in requests:
+ if request.warm_fast_path:
+ request.warm_fast_path = False
+ released += 1
+ if not released:
+ return
+ if released > self._micro_batch_warm_lookahead:
+ raise RuntimeError("cuVS micro-batch warm look-ahead accounting underflow")
+ self._micro_batch_warm_lookahead -= released
+ self._micro_batch_condition.notify_all()
+
+ @staticmethod
+ def _await_micro_batch_request(
+ request: _CuVSMicroBatchRequest,
+ ) -> Tuple[List[int], List[float]]:
+ try:
+ return request.future.result()
+ except BaseException:
+ request.future.cancel()
+ raise
+
+ def _next_micro_batch(self) -> Optional[List[_CuVSMicroBatchRequest]]:
+ wait_seconds = self.micro_batching_max_wait_ms / 1000.0
+ with self._micro_batch_condition:
+ while True:
+ if self._micro_batch_stop and not self._micro_batch_pending:
+ return None
+ if not self._micro_batch_pending:
+ self._micro_batch_condition.wait()
+ continue
+
+ now = time.perf_counter()
+ selected_key: Optional[Tuple[int, Union[str, int], int]] = None
+ earliest_deadline: Optional[float] = None
+ for key, requests in self._micro_batch_pending.items():
+ deadline = requests[0].enqueued_at + wait_seconds
+ if (
+ len(requests) >= self.micro_batching_max_batch_size
+ or now >= deadline
+ or self._micro_batch_draining
+ or self._micro_batch_stop
+ ):
+ selected_key = key
+ break
+ if earliest_deadline is None or deadline < earliest_deadline:
+ earliest_deadline = deadline
+
+ if selected_key is None:
+ timeout = max((earliest_deadline or now) - now, 0.0)
+ self._micro_batch_condition.wait(timeout)
+ continue
+
+ requests = self._micro_batch_pending[selected_key]
+ batch = requests[: self.micro_batching_max_batch_size]
+ del requests[: len(batch)]
+ if not requests:
+ del self._micro_batch_pending[selected_key]
+ return batch
+
+ @staticmethod
+ def _copy_micro_batch_exception(exc: BaseException) -> BaseException:
+ try:
+ return type(exc)(*exc.args)
+ except Exception:
+ return RuntimeError(f"cuVS micro-batch search failed: {exc}")
+
+ def _complete_micro_batch(
+ self,
+ requests: Sequence[_CuVSMicroBatchRequest],
+ *,
+ results: Optional[Sequence[Tuple[List[int], List[float]]]] = None,
+ error: Optional[BaseException] = None,
+ ) -> None:
+ live_requests = [request for request in requests if not request.released]
+ if not live_requests:
+ return
+ # Drop the last references to snapshot-owned CUDA allocations on the
+ # captured device before close/rebuild can observe an idle index.
+ release_error: Optional[BaseException] = None
+ try:
+ with self._runtime_device_scope():
+ for request in live_requests:
+ request.mask = None
+ request.snapshot = None
+ request.released = True
+ except BaseException as exc:
+ release_error = exc
+ for request in live_requests:
+ request.mask = None
+ request.snapshot = None
+ request.released = True
+ with self._lock:
+ self._active_searches -= len(live_requests)
+ if self._active_searches == 0:
+ self._idle_condition.notify_all()
+
+ for index, request in enumerate(live_requests):
+ if request.future.cancelled():
+ continue
+ completion_error = error or release_error
+ if completion_error is not None:
+ request.future.set_exception(self._copy_micro_batch_exception(completion_error))
+ else:
+ assert results is not None
+ request.future.set_result(results[index])
+
+ def _dispatch_micro_batch(self, requests: List[_CuVSMicroBatchRequest]) -> None:
+ runnable = [
+ request for request in requests if request.future.set_running_or_notify_cancel()
+ ]
+ runnable_ids = {id(request) for request in runnable}
+ cancelled = [request for request in requests if id(request) not in runnable_ids]
+ if cancelled:
+ self._release_micro_batch_warm_lookahead(cancelled)
+ self._complete_micro_batch(cancelled, results=[([], [])] * len(cancelled))
+ if not runnable:
+ return
+
+ first = runnable[0]
+ snapshot = first.snapshot
+ request_snapshot: Optional[_CuVSIndexSnapshot] = None
+ if snapshot is None:
+ self._release_micro_batch_warm_lookahead(runnable)
+ self._complete_micro_batch(
+ runnable,
+ error=RuntimeError("cuVS micro-batch snapshot was released before dispatch"),
+ )
+ return
+
+ try:
+ self._gpu_search_gate.acquire()
+ except BaseException as exc:
+ self._release_micro_batch_warm_lookahead(runnable)
+ self._complete_micro_batch(runnable, error=exc)
+ raise
+ try:
+ # Once this batch owns the gate it is the current device work, so
+ # one subsequent warm batch may begin filling behind it.
+ self._release_micro_batch_warm_lookahead(runnable)
+ except BaseException as exc:
+ self._gpu_search_gate.release()
+ self._complete_micro_batch(runnable, error=exc)
+ raise
+ gpu_started = time.perf_counter()
+ results: Optional[List[Tuple[List[int], List[float]]]] = None
+ error: Optional[BaseException] = None
+ try:
+ for request in runnable:
+ request_wait_ms = (gpu_started - request.enqueued_at) * 1000.0
+ if request.telemetry is not None:
+ request.telemetry.micro_batching_enabled = True
+ request.telemetry.batch_size = len(runnable)
+ request.telemetry.batch_wait_ms += request_wait_ms
+ request.telemetry.queue_ms += request_wait_ms
+ offsets_by_query, distances_by_query = self._runtime.search_batch(
+ snapshot.runtime_index,
+ [request.query for request in runnable],
+ first.result_limit,
+ first.mask,
+ )
+ if len(offsets_by_query) != len(runnable) or len(distances_by_query) != len(runnable):
+ raise RuntimeError("cuVS returned a different number of rows than requested")
+
+ results = []
+ for request, offsets, distances in zip(
+ runnable,
+ offsets_by_query,
+ distances_by_query,
+ strict=True,
+ ):
+ request_snapshot = request.snapshot
+ if request_snapshot is None:
+ raise RuntimeError("cuVS micro-batch snapshot was released during dispatch")
+ labels: List[int] = []
+ scores: List[float] = []
+ for offset, distance in zip(offsets, distances, strict=True):
+ if offset < 0 or offset >= len(request_snapshot.labels):
+ continue
+ labels.append(request_snapshot.labels[offset])
+ scores.append(1.0 - distance if self.distance == "l2" else distance)
+ results.append((labels, scores))
+ except BaseException as exc:
+ error = exc
+ finally:
+ gpu_search_ms = (time.perf_counter() - gpu_started) * 1000.0
+ for request in runnable:
+ if request.telemetry is not None:
+ request.telemetry.gpu_search_ms += gpu_search_ms
+ self._gpu_search_gate.release()
+ # Requests still own these objects until _complete_micro_batch(),
+ # so clearing worker locals here cannot release them prematurely.
+ # It does ensure active=0 is never published while this frame keeps
+ # an extra CUDA-owned snapshot reference alive.
+ try:
+ with self._runtime_device_scope():
+ request_snapshot = None
+ snapshot = None
+ except BaseException as cleanup_exc:
+ if error is None:
+ error = cleanup_exc
+ request_snapshot = None
+ snapshot = None
+ self._complete_micro_batch(runnable, results=results, error=error)
+ if error is not None and not isinstance(error, Exception):
+ raise error
+
+ def _micro_batch_worker_main(self) -> None:
+ try:
+ while True:
+ requests = self._next_micro_batch()
+ if requests is None:
+ return
+ self._dispatch_micro_batch(requests)
+ except BaseException as exc:
+ logger.exception("The cuVS micro-batch worker stopped unexpectedly")
+ with self._micro_batch_condition:
+ self._micro_batch_worker_failure = exc
+ pending = [
+ request
+ for requests in self._micro_batch_pending.values()
+ for request in requests
+ ]
+ for request in pending:
+ request.warm_fast_path = False
+ self._micro_batch_warm_lookahead = 0
+ self._micro_batch_pending.clear()
+ self._micro_batch_stop = True
+ self._micro_batch_condition.notify_all()
+ if pending:
+ self._complete_micro_batch(pending, error=exc)
+
def search(
self,
query_vector: Sequence[float],
@@ -1712,7 +2194,7 @@ class CuVSDenseIndex:
) * 1000.0
if prepared_filter is None:
with self._lock:
- if self._closed:
+ if self._closing or self._closed:
raise RuntimeError("cuVS dense index is closed")
# A mutation invalidated the registered native layout while
# the resolver ran. Retry before consuming GPU admission.
@@ -1736,6 +2218,45 @@ class CuVSDenseIndex:
f"({eligible_count} candidates <= {native_threshold})"
)
+ if self.micro_batching_enabled:
+ warm_request = self._try_enqueue_warm_micro_batch(
+ query=query,
+ limit=limit,
+ filters=filters,
+ prepared_filter=prepared_filter,
+ telemetry=telemetry,
+ )
+ if warm_request is not None:
+ return self._await_micro_batch_request(warm_request)
+
+ queue_started = time.perf_counter()
+ self._gpu_search_gate.acquire()
+ waited_ms = (time.perf_counter() - queue_started) * 1000.0
+ if telemetry is not None:
+ telemetry.queue_ms += waited_ms
+ telemetry.gpu_gate_queue_ms += waited_ms
+ try:
+ try:
+ admitted = self._search_admitted(
+ query,
+ limit,
+ filters,
+ native_filter_resolver,
+ native_filter_layout_registrar,
+ prepared_filter,
+ telemetry,
+ defer_micro_batch_wait=True,
+ )
+ except _StalePreparedFilter:
+ continue
+ finally:
+ # GPU rebuild/filter preparation and enqueue are admitted,
+ # but waiting for the worker must never retain the gate.
+ self._gpu_search_gate.release()
+ if isinstance(admitted, _CuVSMicroBatchRequest):
+ return self._await_micro_batch_request(admitted)
+ return admitted
+
queue_started = time.perf_counter()
self._gpu_search_gate.acquire()
waited_ms = (time.perf_counter() - queue_started) * 1000.0
@@ -1769,9 +2290,12 @@ class CuVSDenseIndex:
native_filter_layout_registrar: Optional[Callable[[Sequence[int]], None]] = None,
prepared_filter: Optional[_PreparedHostFilter] = None,
telemetry: Optional[CuVSSearchTelemetry] = None,
- ) -> Tuple[List[int], List[float]]:
+ *,
+ defer_micro_batch_wait: bool = False,
+ ) -> Union[Tuple[List[int], List[float]], _CuVSMicroBatchRequest]:
+ batch_request: Optional[_CuVSMicroBatchRequest] = None
with self._lock:
- if self._closed:
+ if self._closing or self._closed:
raise RuntimeError("cuVS dense index is closed")
if (
prepared_filter is not None
@@ -1907,8 +2431,32 @@ class CuVSDenseIndex:
result_limit = min(limit, eligible_count)
else:
result_limit = min(limit, len(snapshot.labels))
+ if self.micro_batching_enabled:
+ cache_key = self._filter_cache_key(filters) if filters else "no-filter"
+ filter_identity: Union[str, int] = (
+ cache_key
+ if cache_key is not None
+ else id(mask)
+ if mask is not None
+ else id(filters)
+ )
+ batch_request = self._enqueue_micro_batch(
+ snapshot=snapshot,
+ query=query,
+ result_limit=result_limit,
+ mask=mask,
+ filter_identity=filter_identity,
+ telemetry=telemetry,
+ )
+ if batch_request is None:
+ raise RuntimeError("cuVS gated micro-batch enqueue unexpectedly failed")
self._active_searches += 1
+ if batch_request is not None:
+ if defer_micro_batch_wait:
+ return batch_request
+ return self._await_micro_batch_request(batch_request)
+
labels: List[int] = []
scores: List[float] = []
try:
@@ -1950,21 +2498,57 @@ class CuVSDenseIndex:
with self._lock:
if self._close_complete:
return
+ while self._closing and not self._close_complete:
+ self._idle_condition.wait()
+ if self._close_complete:
+ return
+
+ # Reject new admission and invalidate any host-filter work that is
+ # still outside the index lock. Already accepted micro-batch
+ # requests remain active and are drained below.
+ self._closing = True
if not self._closed:
self._closed = True
self._records_generation += 1
self._filter_layout_generation = -1
self._cancel_preflight_flights_()
+ if self.micro_batching_enabled:
+ with self._micro_batch_condition:
+ self._micro_batch_draining = True
+ self._micro_batch_condition.notify_all()
while self._active_searches:
self._idle_condition.wait()
- with self._runtime_device_scope():
- self._snapshot = None
- self._filter_cache.clear()
- self._preflight_filter_cache.clear()
- try:
- self._runtime.close()
- finally:
- # A failed recovery can otherwise keep this partially packed
- # host shadow alive through the constructor traceback.
- self._records.clear()
- self._close_complete = True
+
+ if self.micro_batching_enabled:
+ with self._micro_batch_condition:
+ self._micro_batch_stop = True
+ self._micro_batch_condition.notify_all()
+ worker = self._micro_batch_worker
+ if worker is not None and worker is not threading.current_thread():
+ worker.join()
+
+ try:
+ with self._lock:
+ with self._runtime_device_scope():
+ self._snapshot = None
+ self._filter_cache.clear()
+ self._preflight_filter_cache.clear()
+ try:
+ self._runtime.close()
+ finally:
+ # A failed recovery can otherwise keep this partially
+ # packed host shadow alive through the constructor
+ # traceback.
+ self._records.clear()
+ except BaseException:
+ with self._lock:
+ # Keep admission closed, but allow a later close() call to
+ # retry runtime cleanup after a transient failure.
+ self._closing = False
+ self._idle_condition.notify_all()
+ raise
+ else:
+ with self._lock:
+ self._close_complete = True
+ self._closing = False
+ self._idle_condition.notify_all()
diff --git a/openviking/telemetry/operation.py b/openviking/telemetry/operation.py
index 1a5bfdee4..a55242234 100644
--- a/openviking/telemetry/operation.py
+++ b/openviking/telemetry/operation.py
@@ -21,6 +21,7 @@ _CUVS_TIMING_FIELDS = (
"build_ms",
"filter_prepare_ms",
"gpu_search_ms",
+ "batch_wait_ms",
"native_search_ms",
)
_CUVS_ROUTE_BUCKETS = {
@@ -325,6 +326,26 @@ class TelemetrySummaryBuilder:
"memory": cuvs_memory,
"timings_ms": cls._cuvs_timing_summary(counters, gauges),
}
+ micro_batching_searches = cls._i(
+ counters.get("vector.cuvs.micro_batching_searches"), 0
+ )
+ if micro_batching_searches:
+ summary["vector"]["cuvs"].update(
+ {
+ "micro_batching_searches": micro_batching_searches,
+ "micro_batched_searches": cls._i(
+ counters.get("vector.cuvs.micro_batched_searches"), 0
+ ),
+ "micro_batching_warm_fast_path_searches": cls._i(
+ counters.get("vector.cuvs.micro_batching_warm_fast_path_searches"),
+ 0,
+ ),
+ "batch_size_max": cls._i(gauges.get("vector.cuvs.batch_size.max"), 1),
+ "searches_by_batch_size": cls._counter_breakdown(
+ "vector.cuvs.searches_by_batch_size", counters
+ ),
+ }
+ )
if cls._has_metric_prefix("semantic_nodes", counters, gauges):
summary["semantic_nodes"] = {
@@ -504,6 +525,21 @@ class OperationTelemetry:
self._counters[f"vector.cuvs.dtypes.{dtype}"] += 1
if TelemetrySummaryBuilder._bool(metrics.get("auto_mode"), False):
self._counters["vector.cuvs.auto_mode_searches"] += 1
+ if TelemetrySummaryBuilder._bool(metrics.get("micro_batching_enabled"), False):
+ batch_size = min(
+ max(TelemetrySummaryBuilder._i(metrics.get("batch_size"), 1), 1),
+ 8,
+ )
+ self._counters["vector.cuvs.micro_batching_searches"] += 1
+ self._counters[f"vector.cuvs.searches_by_batch_size.{batch_size}"] += 1
+ if batch_size > 1:
+ self._counters["vector.cuvs.micro_batched_searches"] += 1
+ self._gauges["vector.cuvs.batch_size.max"] = max(
+ TelemetrySummaryBuilder._i(self._gauges.get("vector.cuvs.batch_size.max"), 1),
+ batch_size,
+ )
+ if TelemetrySummaryBuilder._bool(metrics.get("micro_batching_warm_fast_path"), False):
+ self._counters["vector.cuvs.micro_batching_warm_fast_path_searches"] += 1
if TelemetrySummaryBuilder._bool(metrics.get("filter_cache_hit"), False):
self._counters["vector.cuvs.filter_cache_hits"] += 1
if TelemetrySummaryBuilder._bool(metrics.get("filter_cache_eviction_fallback"), False):
diff --git a/openviking_cli/utils/config/vectordb_config.py b/openviking_cli/utils/config/vectordb_config.py
index b6b4905c4..ef8593e04 100644
--- a/openviking_cli/utils/config/vectordb_config.py
+++ b/openviking_cli/utils/config/vectordb_config.py
@@ -162,6 +162,29 @@ class CuVSConfig(BaseModel):
"snapshot work remains concurrent; increase only after hardware-specific tuning."
),
)
+ micro_batching_enabled: bool = Field(
+ default=False,
+ description=(
+ "Coalesce compatible concurrent cuVS dense queries into one matrix-search call. "
+ "This OpenViking scheduler is opt-in and distinct from cuVS Dynamic Batching."
+ ),
+ )
+ micro_batching_max_batch_size: int = Field(
+ default=8,
+ ge=1,
+ le=8,
+ description="Maximum compatible queries submitted in one cuVS search call.",
+ )
+ micro_batching_max_wait_ms: float = Field(
+ default=1.0,
+ ge=0.0,
+ le=100.0,
+ allow_inf_nan=False,
+ description=(
+ "Maximum collection window for a compatible cuVS micro-batch, in milliseconds. "
+ "Zero performs opportunistic batching without an intentional wait."
+ ),
+ )
auto_background_rebuild: bool = Field(
default=False,
description=(
@@ -180,6 +203,16 @@ class CuVSConfig(BaseModel):
model_config = {"extra": "forbid"}
+ @model_validator(mode="after")
+ def validate_micro_batching(self):
+ if not self.micro_batching_enabled:
+ return self
+ if self.algorithm != "brute_force":
+ raise ValueError("cuVS micro-batching currently supports algorithm='brute_force' only")
+ if self.max_concurrent_gpu_searches != 1:
+ raise ValueError("cuVS micro-batching currently requires max_concurrent_gpu_searches=1")
+ return self
+
_OPENGAUSS_MODES = {"standalone", "distributed"}
diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.py
index 16d487983..8774b9daf 100644
--- a/tests/test_telemetry_runtime.py
+++ b/tests/test_telemetry_runtime.py
@@ -291,6 +291,32 @@ def test_telemetry_summary_includes_cuvs_route_and_stage_timings():
}
+def test_telemetry_summary_includes_cuvs_micro_batching_fields():
+ telemetry = MemoryOperationTelemetry(operation="search.find", enabled=True)
+ for batch_size, batch_wait_ms in ((4, 0.8), (4, 0.7), (1, 1.0)):
+ telemetry.record_cuvs_search(
+ {
+ "algorithm": "brute_force",
+ "dtype": "float32",
+ "route_reason": "cuvs",
+ "filter_kind": "none",
+ "micro_batching_enabled": True,
+ "micro_batching_warm_fast_path": batch_size == 4,
+ "batch_size": batch_size,
+ "batch_wait_ms": batch_wait_ms,
+ }
+ )
+
+ cuvs = telemetry.finish().summary["vector"]["cuvs"]
+
+ assert cuvs["micro_batching_searches"] == 3
+ assert cuvs["micro_batched_searches"] == 2
+ assert cuvs["micro_batching_warm_fast_path_searches"] == 2
+ assert cuvs["batch_size_max"] == 4
+ assert cuvs["searches_by_batch_size"] == {"1": 1, "4": 2}
+ assert cuvs["timings_ms"]["batch_wait"] == {"sum": 2.5, "max": 1.0}
+
+
def test_cuvs_telemetry_aggregation_is_completion_order_independent():
samples = [
{
diff --git a/tests/vectordb/test_cuvs_collection.py b/tests/vectordb/test_cuvs_collection.py
index 7edfb4cba..e32939e3c 100644
--- a/tests/vectordb/test_cuvs_collection.py
+++ b/tests/vectordb/test_cuvs_collection.py
@@ -25,6 +25,7 @@ class FakeCuVSRuntime:
def __init__(self, metric):
self.metric = metric
self.search_count = 0
+ self.search_batch_sizes = []
def build(self, dataset):
return [list(vector) for vector in dataset]
@@ -48,10 +49,30 @@ class FakeCuVSRuntime:
rows = rows[:limit]
return [row[1] for row in rows], [row[2] for row in rows]
+ def search_batch(self, index, queries, limit, mask):
+ self.search_batch_sizes.append(len(queries))
+ results = [self.search(index, query, limit, mask) for query in queries]
+ return [result[0] for result in results], [result[1] for result in results]
+
def close(self):
pass
+class BlockingFakeCuVSRuntime(FakeCuVSRuntime):
+ def __init__(self, metric):
+ super().__init__(metric)
+ self.block_next_batch = False
+ self.search_started = threading.Event()
+ self.release_search = threading.Event()
+
+ def search_batch(self, index, queries, limit, mask):
+ if self.block_next_batch:
+ self.block_next_batch = False
+ self.search_started.set()
+ assert self.release_search.wait(timeout=5)
+ return super().search_batch(index, queries, limit, mask)
+
+
class MemoryAwareFakeCuVSRuntime(FakeCuVSRuntime):
def __init__(self, metric, free_memory_bytes):
super().__init__(metric)
@@ -329,6 +350,188 @@ def test_local_collection_allows_concurrent_warmed_cuvs_searches(monkeypatch):
collection.close()
+def test_local_collection_micro_batches_compatible_offset_and_projection_requests(monkeypatch):
+ runtime = FakeCuVSRuntime("inner_product")
+ monkeypatch.setattr(
+ cuvs_index,
+ "_CuVSRuntime",
+ lambda _algorithm, _metric, _build_params, _search_params, _dtype: runtime,
+ )
+ collection = get_or_create_local_collection(
+ meta_data={
+ "CollectionName": "cuvs_micro_batch_projection",
+ "Fields": [
+ {"FieldName": "id", "FieldType": "string", "IsPrimaryKey": True},
+ {"FieldName": "vector", "FieldType": "vector", "Dim": 4},
+ {"FieldName": "account_id", "FieldType": "string"},
+ {"FieldName": "rank", "FieldType": "int64"},
+ ],
+ },
+ config={
+ "dense_search": {
+ "backend": "cuvs",
+ "algorithm": "brute_force",
+ "micro_batching_enabled": True,
+ "micro_batching_max_batch_size": 2,
+ "micro_batching_max_wait_ms": 50.0,
+ }
+ },
+ )
+ try:
+ collection.create_index(
+ "default",
+ {
+ "IndexName": "default",
+ "VectorIndex": {"IndexType": "flat", "Distance": "cosine"},
+ },
+ )
+ collection.upsert_data(
+ [
+ {
+ "id": "first",
+ "vector": [1.0, 0.0, 0.0, 0.0],
+ "account_id": "a",
+ "rank": 1,
+ },
+ {
+ "id": "second",
+ "vector": [0.8, 0.2, 0.0, 0.0],
+ "account_id": "b",
+ "rank": 2,
+ },
+ {
+ "id": "third",
+ "vector": [0.0, 1.0, 0.0, 0.0],
+ "account_id": "c",
+ "rank": 3,
+ },
+ ]
+ )
+ # Warm the immutable snapshot, then isolate the concurrent measurement.
+ assert (
+ collection.search_by_vector("default", dense_vector=[1.0, 0.0, 0.0, 0.0], limit=1)
+ .data[0]
+ .id
+ == "first"
+ )
+ runtime.search_batch_sizes.clear()
+ barrier = threading.Barrier(3)
+
+ def search_with_offset():
+ barrier.wait(timeout=5)
+ return collection.search_by_vector(
+ "default",
+ dense_vector=[1.0, 0.0, 0.0, 0.0],
+ limit=1,
+ offset=1,
+ output_fields=["rank"],
+ )
+
+ def search_two():
+ barrier.wait(timeout=5)
+ return collection.search_by_vector(
+ "default",
+ dense_vector=[1.0, 0.0, 0.0, 0.0],
+ limit=2,
+ output_fields=["account_id"],
+ )
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ offset_future = executor.submit(search_with_offset)
+ two_future = executor.submit(search_two)
+ barrier.wait(timeout=5)
+ offset_result = offset_future.result(timeout=5)
+ two_result = two_future.result(timeout=5)
+
+ assert runtime.search_batch_sizes == [2]
+ assert [(item.id, item.fields) for item in offset_result.data] == [("second", {"rank": 2})]
+ assert [(item.id, item.fields) for item in two_result.data] == [
+ ("first", {"account_id": "a"}),
+ ("second", {"account_id": "b"}),
+ ]
+ finally:
+ collection.close()
+
+
+def test_local_collection_no_filter_requests_fill_warm_micro_batch(monkeypatch):
+ runtime = BlockingFakeCuVSRuntime("inner_product")
+ monkeypatch.setattr(
+ cuvs_index,
+ "_CuVSRuntime",
+ lambda _algorithm, _metric, _build_params, _search_params, _dtype: runtime,
+ )
+ collection = get_or_create_local_collection(
+ meta_data={
+ "CollectionName": "cuvs_micro_batch_no_filter",
+ "Fields": [
+ {"FieldName": "id", "FieldType": "string", "IsPrimaryKey": True},
+ {"FieldName": "vector", "FieldType": "vector", "Dim": 4},
+ ],
+ },
+ config={
+ "dense_search": {
+ "backend": "cuvs",
+ "algorithm": "brute_force",
+ "micro_batching_enabled": True,
+ "micro_batching_max_batch_size": 2,
+ "micro_batching_max_wait_ms": 100.0,
+ }
+ },
+ )
+ try:
+ local_index = collection.create_index(
+ "default",
+ {
+ "IndexName": "default",
+ "VectorIndex": {"IndexType": "flat", "Distance": "cosine"},
+ },
+ )
+ collection.upsert_data(
+ [
+ {"id": "first", "vector": [1.0, 0.0, 0.0, 0.0]},
+ {"id": "second", "vector": [0.0, 1.0, 0.0, 0.0]},
+ ]
+ )
+ assert (
+ collection.search_by_vector("default", dense_vector=[1.0, 0.0, 0.0, 0.0], limit=1)
+ .data[0]
+ .id
+ == "first"
+ )
+ runtime.search_batch_sizes.clear()
+ runtime.block_next_batch = True
+
+ def search(vector):
+ return collection.search_by_vector("default", dense_vector=vector, limit=1)
+
+ with ThreadPoolExecutor(max_workers=3) as executor:
+ active = executor.submit(search, [1.0, 0.0, 0.0, 0.0])
+ assert runtime.search_started.wait(timeout=5)
+ queued = [
+ executor.submit(search, [0.0, 1.0, 0.0, 0.0]),
+ executor.submit(search, [1.0, 0.0, 0.0, 0.0]),
+ ]
+ with local_index.dense_search._micro_batch_condition:
+ assert local_index.dense_search._micro_batch_condition.wait_for(
+ lambda: local_index.dense_search._micro_batch_warm_lookahead == 2,
+ timeout=5,
+ )
+ assert all(not future.done() for future in queued)
+
+ runtime.release_search.set()
+ assert active.result(timeout=5).data[0].id == "first"
+ assert [future.result(timeout=5).data[0].id for future in queued] == [
+ "second",
+ "first",
+ ]
+
+ assert runtime.search_batch_sizes == [1, 2]
+ assert local_index.dense_search._micro_batch_warm_lookahead == 0
+ finally:
+ runtime.release_search.set()
+ collection.close()
+
+
def test_persistent_collection_rehydrates_cuvs_from_local_store(monkeypatch, tmp_path):
patch_cuvs_runtime(monkeypatch)
path = str(tmp_path / "cuvs-persistent")
diff --git a/tests/vectordb/test_cuvs_config.py b/tests/vectordb/test_cuvs_config.py
index 2dc2dc9e8..7a1d4a5cc 100644
--- a/tests/vectordb/test_cuvs_config.py
+++ b/tests/vectordb/test_cuvs_config.py
@@ -5,6 +5,7 @@ import pytest
from pydantic import ValidationError
from openviking.storage.vectordb_adapters.local_adapter import (
+ CuVSCollectionAdapter,
LocalCollectionAdapter,
)
from openviking_cli.utils.config.vectordb_config import CuVSConfig, VectorDBBackendConfig
@@ -14,6 +15,9 @@ def test_cuvs_filter_cache_defaults_and_disable_value():
assert CuVSConfig().dtype == "float32"
assert CuVSConfig(dtype="float16").dtype == "float16"
assert CuVSConfig().max_concurrent_gpu_searches == 1
+ assert CuVSConfig().micro_batching_enabled is False
+ assert CuVSConfig().micro_batching_max_batch_size == 8
+ assert CuVSConfig().micro_batching_max_wait_ms == 1.0
assert CuVSConfig().filter_cache_size == 16
assert CuVSConfig(filter_cache_size=0).filter_cache_size == 0
@@ -25,6 +29,22 @@ def test_cuvs_filter_cache_rejects_negative_size():
CuVSConfig(filter_cache_size=-1)
with pytest.raises(ValidationError, match="max_concurrent_gpu_searches"):
CuVSConfig(max_concurrent_gpu_searches=0)
+ with pytest.raises(ValidationError, match="micro_batching_max_batch_size"):
+ CuVSConfig(micro_batching_max_batch_size=0)
+ with pytest.raises(ValidationError, match="micro_batching_max_batch_size"):
+ CuVSConfig(micro_batching_max_batch_size=9)
+ with pytest.raises(ValidationError, match="micro_batching_max_wait_ms"):
+ CuVSConfig(micro_batching_max_wait_ms=-0.1)
+ with pytest.raises(ValidationError, match="micro_batching_max_wait_ms"):
+ CuVSConfig(micro_batching_max_wait_ms=float("nan"))
+ with pytest.raises(ValidationError, match="micro_batching_max_wait_ms"):
+ CuVSConfig(micro_batching_max_wait_ms=100.1)
+ with pytest.raises(ValidationError, match="brute_force"):
+ CuVSConfig(micro_batching_enabled=True, algorithm="cagra")
+ with pytest.raises(ValidationError, match="max_concurrent_gpu_searches=1"):
+ CuVSConfig(micro_batching_enabled=True, max_concurrent_gpu_searches=2)
+ with pytest.raises(ValidationError, match="dynamic_batching"):
+ CuVSConfig(dynamic_batching=True)
def test_cuvs_auto_mode_is_opt_in_and_validates_memory_guardrails():
@@ -63,6 +83,9 @@ def test_local_adapter_only_passes_auto_cuvs_config_when_enabled():
"auto_memory_safety_factor": 1.5,
"auto_filter_native_threshold": 1000,
"auto_path_filter_native_threshold": 100,
+ "micro_batching_enabled": True,
+ "micro_batching_max_batch_size": 4,
+ "micro_batching_max_wait_ms": 0.5,
},
)
)
@@ -74,3 +97,24 @@ def test_local_adapter_only_passes_auto_cuvs_config_when_enabled():
assert dense_search["auto_memory_safety_factor"] == 1.5
assert dense_search["auto_filter_native_threshold"] == 1000
assert dense_search["auto_path_filter_native_threshold"] == 100
+ assert dense_search["micro_batching_enabled"] is True
+ assert dense_search["micro_batching_max_batch_size"] == 4
+ assert dense_search["micro_batching_max_wait_ms"] == 0.5
+
+
+def test_explicit_cuvs_adapter_forwards_micro_batching_config():
+ adapter = CuVSCollectionAdapter.from_config(
+ VectorDBBackendConfig(
+ backend="cuvs",
+ cuvs={
+ "micro_batching_enabled": True,
+ "micro_batching_max_batch_size": 4,
+ "micro_batching_max_wait_ms": 0.25,
+ },
+ )
+ )
+ dense_search = adapter._collection_config["dense_search"]
+ assert dense_search["backend"] == "cuvs"
+ assert dense_search["micro_batching_enabled"] is True
+ assert dense_search["micro_batching_max_batch_size"] == 4
+ assert dense_search["micro_batching_max_wait_ms"] == 0.25
diff --git a/tests/vectordb/test_cuvs_index.py b/tests/vectordb/test_cuvs_index.py
index 5ded0d3e9..6c6c0a9f0 100644
--- a/tests/vectordb/test_cuvs_index.py
+++ b/tests/vectordb/test_cuvs_index.py
@@ -39,6 +39,7 @@ class FakeCuVSRuntime:
self.release_count = 0
self.close_count = 0
self.close_error = None
+ self.search_batch_sizes = []
def build(self, dataset):
self.dataset = [list(vector) for vector in dataset]
@@ -63,6 +64,11 @@ class FakeCuVSRuntime:
selected = rows[:limit]
return [row[1] for row in selected], [row[2] for row in selected]
+ def search_batch(self, index, queries, limit, mask):
+ self.search_batch_sizes.append(len(queries))
+ results = [self.search(index, query, limit, mask) for query in queries]
+ return [result[0] for result in results], [result[1] for result in results]
+
def prepare_filter(self, mask):
self.prepare_filter_count += 1
return tuple(mask)
@@ -233,8 +239,9 @@ def test_cuvs_runtime_uses_captured_device_from_worker_thread():
assert resources.device_id == spy.expected_device
if spy.fail_search:
raise RuntimeError("injected search failure")
- return FakeArray([[0.25]], "float32", "distances"), FakeArray(
- [[0]], "int64", "neighbors"
+ query_count = len(_queries.values)
+ return FakeArray([[0.25]] * query_count, "float32", "distances"), FakeArray(
+ [[0]] * query_count, "int64", "neighbors"
)
class FakeFilters:
@@ -308,6 +315,12 @@ def test_cuvs_runtime_uses_captured_device_from_worker_thread():
assert ("array-5", 3) in spy.temporary_releases
assert spy.current() == 9
assert spy.releases == [("resources-1", 3)]
+ assert runtime.search_batch(
+ runtime_index,
+ [[1.0, 0.0], [0.0, 1.0]],
+ 1,
+ [True],
+ ) == ([[0], [0]], [[0.25], [0.25]])
assert runtime.search(runtime_index, [1.0, 0.0], 1, [True]) == ([0], [0.25])
spy.fail_host_copy = True
with pytest.raises(RuntimeError, match="injected host copy failure"):
@@ -1385,6 +1398,910 @@ def test_device_cache_eviction_before_admission_uses_one_in_gate_fallback():
assert eviction_telemetry.as_dict()["filter_cache_eviction_fallback"] is True
+def make_micro_batch_index(runtime, **config):
+ dense_config = {
+ "micro_batching_enabled": True,
+ "micro_batching_max_batch_size": 4,
+ "micro_batching_max_wait_ms": 50.0,
+ **config,
+ }
+ index = CuVSDenseIndex(
+ dimension=2,
+ distance="ip",
+ normalize_vectors=False,
+ field_types={"account_id": "string"},
+ config=dense_config,
+ runtime=runtime,
+ )
+ index.add_candidates(
+ [
+ candidate(1, [1.0, 0.0], account_id="a"),
+ candidate(2, [0.0, 1.0], account_id="b"),
+ candidate(3, [0.8, 0.2], account_id="a"),
+ ]
+ )
+ # Exclude the first lazy build and its unavoidable singleton from assertions.
+ assert index.search([1.0, 0.0], 1, None)[0] == [1]
+ runtime.search_batch_sizes.clear()
+ return index
+
+
+class BlockingMicroBatchRuntime(FakeCuVSRuntime):
+ def __init__(self):
+ super().__init__()
+ self.block_next_batch = False
+ self.search_started = threading.Event()
+ self.release_search = threading.Event()
+
+ def search_batch(self, index, queries, limit, mask):
+ if self.block_next_batch:
+ self.block_next_batch = False
+ self.search_started.set()
+ assert self.release_search.wait(timeout=5)
+ return super().search_batch(index, queries, limit, mask)
+
+
+def wait_for_warm_lookahead(index, expected):
+ with index._micro_batch_condition:
+ assert index._micro_batch_condition.wait_for(
+ lambda: index._micro_batch_warm_lookahead == expected,
+ timeout=5,
+ )
+
+
+def run_simultaneous_searches(index, requests):
+ barrier = threading.Barrier(len(requests) + 1)
+
+ def run(request):
+ barrier.wait(timeout=5)
+ return index.search(*request)
+
+ with ThreadPoolExecutor(max_workers=len(requests)) as executor:
+ futures = [executor.submit(run, request) for request in requests]
+ barrier.wait(timeout=5)
+ return [future.result(timeout=5) for future in futures]
+
+
+def test_cuvs_micro_batch_coalesces_compatible_queries_and_demuxes_rows():
+ runtime = FakeCuVSRuntime()
+ index = make_micro_batch_index(runtime)
+ telemetry = [CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False) for _ in range(4)]
+ requests = [
+ ([1.0, 0.0], 1, None, None, None, telemetry[0]),
+ ([0.0, 1.0], 1, None, None, None, telemetry[1]),
+ ([0.9, 0.1], 1, None, None, None, telemetry[2]),
+ ([0.1, 0.9], 1, None, None, None, telemetry[3]),
+ ]
+
+ results = run_simultaneous_searches(index, requests)
+
+ assert [result[0] for result in results] == [[1], [2], [1], [2]]
+ assert runtime.search_batch_sizes == [4]
+ assert all(item.micro_batching_enabled for item in telemetry)
+ assert [item.batch_size for item in telemetry] == [4] * 4
+ assert all(item.batch_wait_ms >= 0.0 for item in telemetry)
+ index.close()
+
+
+def test_cuvs_micro_batch_warm_fast_path_fills_behind_active_batch():
+ runtime = BlockingMicroBatchRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_batch_size=2,
+ micro_batching_max_wait_ms=100.0,
+ )
+ runtime.block_next_batch = True
+ telemetry = [CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False) for _ in range(3)]
+
+ with ThreadPoolExecutor(max_workers=3) as executor:
+ active = executor.submit(index.search, [1.0, 0.0], 1, None, None, None, telemetry[0])
+ assert runtime.search_started.wait(timeout=5)
+ lookahead = [
+ executor.submit(index.search, query, 1, None, None, None, item)
+ for query, item in zip(
+ ([1.0, 0.0], [0.0, 1.0]),
+ telemetry[1:],
+ strict=True,
+ )
+ ]
+ wait_for_warm_lookahead(index, 2)
+
+ assert all(not future.done() for future in lookahead)
+ assert index._active_searches == 3
+ assert all(item.micro_batching_warm_fast_path for item in telemetry)
+ assert all(item.gpu_gate_queue_ms == 0.0 for item in telemetry)
+
+ runtime.release_search.set()
+ assert active.result(timeout=5)[0] == [1]
+ assert [future.result(timeout=5)[0] for future in lookahead] == [[1], [2]]
+
+ assert runtime.search_batch_sizes == [1, 2]
+ assert [item.batch_size for item in telemetry] == [1, 2, 2]
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ index.close()
+
+
+def test_cuvs_micro_batch_warm_fast_path_caps_one_batch_and_preserves_progress():
+ runtime = BlockingMicroBatchRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_batch_size=2,
+ micro_batching_max_wait_ms=100.0,
+ )
+ runtime.block_next_batch = True
+ warm_telemetry = [
+ CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False) for _ in range(2)
+ ]
+ overflow_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+
+ with ThreadPoolExecutor(max_workers=4) as executor:
+ active = executor.submit(index.search, [1.0, 0.0], 1, None)
+ assert runtime.search_started.wait(timeout=5)
+ warm = [
+ executor.submit(index.search, query, 1, None, None, None, item)
+ for query, item in zip(
+ ([1.0, 0.0], [0.0, 1.0]),
+ warm_telemetry,
+ strict=True,
+ )
+ ]
+ wait_for_warm_lookahead(index, 2)
+ overflow = executor.submit(
+ index.search,
+ [0.9, 0.1],
+ 1,
+ None,
+ None,
+ None,
+ overflow_telemetry,
+ )
+
+ assert not overflow.done()
+ assert index._micro_batch_warm_lookahead == 2
+ assert all(item.micro_batching_warm_fast_path for item in warm_telemetry)
+ assert overflow_telemetry.micro_batching_warm_fast_path is False
+
+ runtime.release_search.set()
+ assert active.result(timeout=5)[0] == [1]
+ assert [future.result(timeout=5)[0] for future in warm] == [[1], [2]]
+ assert overflow.result(timeout=5)[0] == [1]
+
+ assert runtime.search_batch_sizes == [1, 2, 1]
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ index.close()
+
+
+def test_cuvs_micro_batch_empty_filter_uses_warm_fast_path():
+ runtime = BlockingMicroBatchRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_batch_size=2,
+ micro_batching_max_wait_ms=100.0,
+ )
+ runtime.block_next_batch = True
+ no_filter_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+ empty_filter_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+
+ with ThreadPoolExecutor(max_workers=3) as executor:
+ active = executor.submit(index.search, [1.0, 0.0], 1, None)
+ assert runtime.search_started.wait(timeout=5)
+ no_filter = executor.submit(
+ index.search,
+ [0.0, 1.0],
+ 1,
+ None,
+ None,
+ None,
+ no_filter_telemetry,
+ )
+ wait_for_warm_lookahead(index, 1)
+ empty_filter = executor.submit(
+ index.search,
+ [1.0, 0.0],
+ 1,
+ {},
+ None,
+ None,
+ empty_filter_telemetry,
+ )
+ wait_for_warm_lookahead(index, 2)
+
+ assert no_filter_telemetry.micro_batching_warm_fast_path is True
+ assert empty_filter_telemetry.micro_batching_warm_fast_path is True
+ assert not empty_filter.done()
+
+ runtime.release_search.set()
+ assert active.result(timeout=5)[0] == [1]
+ assert no_filter.result(timeout=5)[0] == [2]
+ assert empty_filter.result(timeout=5)[0] == [1]
+
+ assert runtime.search_batch_sizes == [1, 2]
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ index.close()
+
+
+def test_cuvs_micro_batch_groups_same_filter_but_splits_incompatible_keys():
+ runtime = FakeCuVSRuntime()
+ index = make_micro_batch_index(runtime)
+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}
+ filter_b = {"op": "must", "field": "account_id", "conds": ["b"]}
+
+ same_filter = run_simultaneous_searches(
+ index,
+ [
+ ([1.0, 0.0], 1, filter_a),
+ ([0.0, 1.0], 1, filter_a),
+ ],
+ )
+ assert [result[0] for result in same_filter] == [[1], [3]]
+ assert runtime.search_batch_sizes == [2]
+
+ runtime.search_batch_sizes.clear()
+ split_filters = run_simultaneous_searches(
+ index,
+ [
+ ([1.0, 0.0], 1, filter_a),
+ ([0.0, 1.0], 1, filter_b),
+ ],
+ )
+ assert [result[0] for result in split_filters] == [[1], [2]]
+ assert sorted(runtime.search_batch_sizes) == [1, 1]
+
+ runtime.search_batch_sizes.clear()
+ split_limits = run_simultaneous_searches(
+ index,
+ [
+ ([1.0, 0.0], 1, None),
+ ([1.0, 0.0], 2, None),
+ ],
+ )
+ assert [len(result[0]) for result in split_limits] == [1, 2]
+ assert sorted(runtime.search_batch_sizes) == [1, 1]
+ index.close()
+
+
+def test_cuvs_micro_batch_warm_fast_path_preserves_filter_and_limit_keys():
+ runtime = BlockingMicroBatchRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_batch_size=4,
+ micro_batching_max_wait_ms=20.0,
+ )
+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}
+ filter_b = {"op": "must", "field": "account_id", "conds": ["b"]}
+
+ def resolve(filters):
+ return ([0b101], 2) if filters == filter_a else ([0b010], 1)
+
+ def registrar(_labels):
+ return None
+
+ assert index.search([1.0, 0.0], 1, filter_a, resolve, registrar)[0] == [1]
+ assert index.search([0.0, 1.0], 1, filter_b, resolve, registrar)[0] == [2]
+ runtime.search_batch_sizes.clear()
+ runtime.block_next_batch = True
+ telemetry = [CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False) for _ in range(4)]
+
+ with ThreadPoolExecutor(max_workers=5) as executor:
+ active = executor.submit(index.search, [1.0, 0.0], 1, None)
+ assert runtime.search_started.wait(timeout=5)
+ requests = [
+ executor.submit(
+ index.search, [1.0, 0.0], 1, filter_a, resolve, registrar, telemetry[0]
+ ),
+ executor.submit(
+ index.search, [0.0, 1.0], 1, filter_a, resolve, registrar, telemetry[1]
+ ),
+ executor.submit(
+ index.search, [1.0, 0.0], 2, filter_a, resolve, registrar, telemetry[2]
+ ),
+ executor.submit(
+ index.search, [0.0, 1.0], 1, filter_b, resolve, registrar, telemetry[3]
+ ),
+ ]
+ wait_for_warm_lookahead(index, 4)
+
+ assert all(item.micro_batching_warm_fast_path for item in telemetry)
+ assert all(item.filter_cache_hit for item in telemetry)
+ runtime.release_search.set()
+
+ assert active.result(timeout=5)[0] == [1]
+ results = [future.result(timeout=5)[0] for future in requests]
+
+ assert results == [[1], [3], [1, 3], [2]]
+ assert runtime.search_batch_sizes == [1, 2, 1, 1]
+ assert [item.batch_size for item in telemetry] == [2, 2, 1, 1]
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ index.close()
+
+
+def test_cuvs_micro_batch_serializes_device_filter_preparation_with_search():
+ class BlockingRuntime(FakeCuVSRuntime):
+ def __init__(self):
+ super().__init__()
+ self.block_next_batch = False
+ self.search_started = threading.Event()
+ self.release_search = threading.Event()
+ self.filter_prepared = threading.Event()
+
+ def search_batch(self, index, queries, limit, mask):
+ if self.block_next_batch:
+ self.block_next_batch = False
+ self.search_started.set()
+ assert self.release_search.wait(timeout=5)
+ return super().search_batch(index, queries, limit, mask)
+
+ def prepare_filter_words(self, _words):
+ self.filter_prepared.set()
+ return (True, False, True)
+
+ runtime = BlockingRuntime()
+ index = make_micro_batch_index(runtime)
+ runtime.block_next_batch = True
+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}
+ resolver_ran = threading.Event()
+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+
+ def resolve(_filters):
+ resolver_ran.set()
+ return [0b101], 2
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ active_search = executor.submit(index.search, [1.0, 0.0], 1, None)
+ assert runtime.search_started.wait(timeout=5)
+ filtered_search = executor.submit(
+ index.search,
+ [1.0, 0.0],
+ 1,
+ filter_a,
+ resolve,
+ lambda _labels: None,
+ telemetry,
+ )
+ try:
+ # Native filter resolution remains host work outside admission.
+ assert resolver_ran.wait(timeout=5)
+ # Device materialization waits for the occupied GPU gate.
+ assert not runtime.filter_prepared.wait(timeout=0.1)
+ finally:
+ runtime.release_search.set()
+ assert active_search.result(timeout=5)[0] == [1]
+ assert filtered_search.result(timeout=5)[0] == [1]
+
+ assert runtime.filter_prepared.is_set()
+ assert telemetry.micro_batching_warm_fast_path is False
+ index.close()
+
+
+def test_cuvs_micro_batch_serializes_rebuild_with_search():
+ class BlockingRuntime(FakeCuVSRuntime):
+ def __init__(self):
+ super().__init__()
+ self.block_next_batch = False
+ self.observe_build = False
+ self.search_started = threading.Event()
+ self.release_search = threading.Event()
+ self.build_started = threading.Event()
+
+ def search_batch(self, index, queries, limit, mask):
+ if self.block_next_batch:
+ self.block_next_batch = False
+ self.search_started.set()
+ assert self.release_search.wait(timeout=5)
+ return super().search_batch(index, queries, limit, mask)
+
+ def build(self, dataset):
+ if self.observe_build:
+ self.build_started.set()
+ return super().build(dataset)
+
+ runtime = BlockingRuntime()
+ index = make_micro_batch_index(runtime)
+ runtime.block_next_batch = True
+ runtime.observe_build = True
+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ active_search = executor.submit(index.search, [1.0, 0.0], 1, None)
+ assert runtime.search_started.wait(timeout=5)
+ index.upsert([delta(4, [2.0, 0.0], account_id="a")])
+ rebuilding_search = executor.submit(
+ index.search,
+ [1.0, 0.0],
+ 1,
+ None,
+ None,
+ None,
+ telemetry,
+ )
+ try:
+ assert not runtime.build_started.wait(timeout=0.1)
+ finally:
+ runtime.release_search.set()
+ assert active_search.result(timeout=5)[0] == [1]
+ assert rebuilding_search.result(timeout=5)[0] == [4]
+
+ assert runtime.build_started.is_set()
+ assert telemetry.micro_batching_warm_fast_path is False
+ index.close()
+
+
+def test_cuvs_micro_batch_device_lru_eviction_falls_back_once():
+ class WordPreparingRuntime(FakeCuVSRuntime):
+ def __init__(self):
+ super().__init__()
+ self.prepare_filter_words_count = 0
+
+ def prepare_filter_words(self, words):
+ self.prepare_filter_words_count += 1
+ return tuple(bool(words[0] & (1 << row)) for row in range(3))
+
+ runtime = WordPreparingRuntime()
+ index = make_micro_batch_index(runtime, filter_cache_size=1)
+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}
+ filter_b = {"op": "must", "field": "account_id", "conds": ["b"]}
+ resolver_calls = {"a": 0, "b": 0}
+
+ def resolve(filters):
+ account_id = filters["conds"][0]
+ resolver_calls[account_id] += 1
+ return ([0b101], 2) if account_id == "a" else ([0b010], 1)
+
+ def registrar(_labels):
+ return None
+
+ assert index.search([1.0, 0.0], 1, filter_a, resolve, registrar)[0] == [1]
+ assert resolver_calls == {"a": 1, "b": 0}
+
+ a_host_prepared = threading.Event()
+ release_a = threading.Event()
+ original_prepare = index._prepare_host_filter
+
+ def pause_a_after_host_prepare(filters, *args, **kwargs):
+ prepared = original_prepare(filters, *args, **kwargs)
+ if filters == filter_a:
+ a_host_prepared.set()
+ assert release_a.wait(timeout=5)
+ return prepared
+
+ index._prepare_host_filter = pause_a_after_host_prepare
+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ delayed_a = executor.submit(
+ index.search,
+ [1.0, 0.0],
+ 1,
+ filter_a,
+ resolve,
+ registrar,
+ telemetry,
+ )
+ assert a_host_prepared.wait(timeout=5)
+ assert index.search([0.0, 1.0], 1, filter_b, resolve, registrar)[0] == [2]
+ release_a.set()
+ assert delayed_a.result(timeout=5)[0] == [1]
+
+ assert resolver_calls == {"a": 2, "b": 1}
+ assert runtime.prepare_filter_words_count == 3
+ assert telemetry.micro_batching_warm_fast_path is False
+ assert telemetry.filter_cache_hit is False
+ assert telemetry.filter_cache_eviction_fallback is True
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ index.close()
+
+
+def test_cuvs_micro_batch_stale_host_filter_uses_admitted_retry():
+ class CountingGate:
+ def __init__(self, gate):
+ self.gate = gate
+ self.acquire_count = 0
+ self.lock = threading.Lock()
+
+ def acquire(self, *args, **kwargs):
+ with self.lock:
+ self.acquire_count += 1
+ return self.gate.acquire(*args, **kwargs)
+
+ def release(self):
+ return self.gate.release()
+
+ runtime = FakeCuVSRuntime()
+ index = make_micro_batch_index(runtime)
+ filter_a = {"op": "must", "field": "account_id", "conds": ["a"]}
+
+ def resolve(_filters):
+ return ([0b101], 2) if index.size == 3 else ([0b1101], 3)
+
+ def registrar(_labels):
+ return None
+
+ assert index.search([1.0, 0.0], 1, filter_a, resolve, registrar)[0] == [1]
+ stale = index._prepare_host_filter(filter_a, resolve, registrar)
+ assert stale is not None
+ index.upsert([delta(4, [2.0, 0.0], account_id="a")])
+
+ original_prepare = index._prepare_host_filter
+ prepare_calls = 0
+
+ def return_stale_once(*args, **kwargs):
+ nonlocal prepare_calls
+ prepare_calls += 1
+ if prepare_calls == 1:
+ return stale
+ return original_prepare(*args, **kwargs)
+
+ index._prepare_host_filter = return_stale_once
+ counting_gate = CountingGate(index._gpu_search_gate)
+ index._gpu_search_gate = counting_gate
+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+
+ assert index.search(
+ [1.0, 0.0],
+ 1,
+ filter_a,
+ resolve,
+ registrar,
+ telemetry,
+ )[0] == [4]
+
+ # One caller admission rejects the stale host result, the retry performs
+ # the cold rebuild/materialization, and the worker owns the final search.
+ assert counting_gate.acquire_count == 3
+ assert prepare_calls == 2
+ assert telemetry.micro_batching_warm_fast_path is False
+ assert telemetry.records_generation == index._records_generation
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ index.close()
+
+
+def test_cuvs_micro_batch_singleton_deadline_and_exception_recovery():
+ class FailingBatchRuntime(FakeCuVSRuntime):
+ def __init__(self):
+ super().__init__()
+ self.fail_next_batch = False
+
+ def search_batch(self, index, queries, limit, mask):
+ if self.fail_next_batch:
+ self.fail_next_batch = False
+ self.search_batch_sizes.append(len(queries))
+ raise RuntimeError("injected batch failure")
+ return super().search_batch(index, queries, limit, mask)
+
+ runtime = FailingBatchRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_wait_ms=10.0,
+ )
+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+ started = time.perf_counter()
+ assert index.search([1.0, 0.0], 1, None, telemetry=telemetry)[0] == [1]
+ assert time.perf_counter() - started < 1.0
+ assert telemetry.batch_size == 1
+ assert telemetry.batch_wait_ms >= 0.0
+
+ runtime.search_batch_sizes.clear()
+ runtime.fail_next_batch = True
+ barrier = threading.Barrier(3)
+
+ def failing_search(query):
+ barrier.wait(timeout=5)
+ return index.search(query, 1, None)
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ futures = [
+ executor.submit(failing_search, [1.0, 0.0]),
+ executor.submit(failing_search, [0.0, 1.0]),
+ ]
+ barrier.wait(timeout=5)
+ for future in futures:
+ with pytest.raises(RuntimeError, match="injected batch failure"):
+ future.result(timeout=5)
+
+ assert runtime.search_batch_sizes == [2]
+ assert index.search([1.0, 0.0], 1, None)[0] == [1]
+ index.close()
+
+
+def test_cuvs_micro_batch_worker_fatal_error_reaps_current_batch():
+ class FatalBatchRuntime(FakeCuVSRuntime):
+ def __init__(self):
+ super().__init__()
+ self.fail_fatally = False
+
+ def search_batch(self, index, queries, limit, mask):
+ if self.fail_fatally:
+ self.fail_fatally = False
+ self.search_batch_sizes.append(len(queries))
+ raise KeyboardInterrupt("injected worker failure")
+ return super().search_batch(index, queries, limit, mask)
+
+ runtime = FatalBatchRuntime()
+ index = make_micro_batch_index(runtime)
+ runtime.fail_fatally = True
+ barrier = threading.Barrier(3)
+
+ def fatal_search(query):
+ barrier.wait(timeout=5)
+ return index.search(query, 1, None)
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ futures = [
+ executor.submit(fatal_search, [1.0, 0.0]),
+ executor.submit(fatal_search, [0.0, 1.0]),
+ ]
+ barrier.wait(timeout=5)
+ for future in futures:
+ with pytest.raises(KeyboardInterrupt, match="injected worker failure"):
+ future.result(timeout=5)
+
+ assert runtime.search_batch_sizes == [2]
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ deadline = time.monotonic() + 5
+ while index._micro_batch_worker_failure is None and time.monotonic() < deadline:
+ time.sleep(0.001)
+ assert index._micro_batch_worker_failure is not None
+ index.close()
+
+
+def test_cuvs_micro_batch_cancelled_warm_request_releases_pins_and_capacity():
+ runtime = BlockingMicroBatchRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_batch_size=2,
+ micro_batching_max_wait_ms=100.0,
+ )
+ runtime.block_next_batch = True
+
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ active = executor.submit(index.search, [1.0, 0.0], 1, None)
+ assert runtime.search_started.wait(timeout=5)
+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+ request = index._try_enqueue_warm_micro_batch(
+ query=(0.0, 1.0),
+ limit=1,
+ filters=None,
+ prepared_filter=None,
+ telemetry=telemetry,
+ )
+ assert request is not None
+ assert request.future.cancel()
+ assert telemetry.micro_batching_warm_fast_path is True
+ wait_for_warm_lookahead(index, 1)
+
+ runtime.release_search.set()
+ assert active.result(timeout=5)[0] == [1]
+
+ deadline = time.monotonic() + 5
+ while not request.released and time.monotonic() < deadline:
+ time.sleep(0.001)
+ assert request.released is True
+ assert request.snapshot is None
+ assert request.mask is None
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ index.close()
+
+
+def test_cuvs_micro_batch_fatal_worker_releases_warm_lookahead_pins():
+ class BlockingFatalRuntime(FakeCuVSRuntime):
+ def __init__(self):
+ super().__init__()
+ self.block_and_fail = False
+ self.search_started = threading.Event()
+ self.fail_search = threading.Event()
+
+ def search_batch(self, index, queries, limit, mask):
+ if self.block_and_fail:
+ self.block_and_fail = False
+ self.search_started.set()
+ assert self.fail_search.wait(timeout=5)
+ raise KeyboardInterrupt("injected worker failure")
+ return super().search_batch(index, queries, limit, mask)
+
+ runtime = BlockingFatalRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_batch_size=2,
+ micro_batching_max_wait_ms=100.0,
+ )
+ runtime.block_and_fail = True
+
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ active = executor.submit(index.search, [1.0, 0.0], 1, None)
+ assert runtime.search_started.wait(timeout=5)
+ pending = [
+ index._try_enqueue_warm_micro_batch(
+ query=query,
+ limit=1,
+ filters=None,
+ prepared_filter=None,
+ telemetry=None,
+ )
+ for query in ((1.0, 0.0), (0.0, 1.0))
+ ]
+ assert all(request is not None for request in pending)
+ wait_for_warm_lookahead(index, 2)
+ runtime.fail_search.set()
+
+ with pytest.raises(KeyboardInterrupt, match="injected worker failure"):
+ active.result(timeout=5)
+ for request in pending:
+ assert request is not None
+ with pytest.raises(KeyboardInterrupt, match="injected worker failure"):
+ request.future.result(timeout=5)
+
+ assert all(request.released for request in pending if request is not None)
+ assert all(request.snapshot is None for request in pending if request is not None)
+ assert all(request.mask is None for request in pending if request is not None)
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ assert index._micro_batch_worker_failure is not None
+ index.close()
+
+
+def test_cuvs_micro_batch_keeps_snapshot_generations_separate():
+ runtime = FakeCuVSRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_wait_ms=100.0,
+ )
+
+ old_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+ new_telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ old_search = executor.submit(
+ index.search,
+ [1.0, 0.0],
+ 1,
+ None,
+ None,
+ None,
+ old_telemetry,
+ )
+ deadline = time.monotonic() + 5
+ while time.monotonic() < deadline:
+ with index._micro_batch_condition:
+ if index._micro_batch_pending:
+ break
+ time.sleep(0.001)
+ else:
+ pytest.fail("old-snapshot search did not enter the micro-batch queue")
+
+ index.upsert([delta(4, [2.0, 0.0], account_id="a")])
+ new_search = executor.submit(
+ index.search,
+ [1.0, 0.0],
+ 1,
+ None,
+ None,
+ None,
+ new_telemetry,
+ )
+ assert old_search.result(timeout=5)[0] == [1]
+ assert new_search.result(timeout=5)[0] == [4]
+
+ assert runtime.search_batch_sizes == [1, 1]
+ assert old_telemetry.micro_batching_warm_fast_path is True
+ assert new_telemetry.micro_batching_warm_fast_path is False
+ assert old_telemetry.records_generation < new_telemetry.records_generation
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ index.close()
+
+
+def test_auto_native_route_does_not_report_micro_batching():
+ runtime = FakeCuVSRuntime()
+ index = CuVSDenseIndex(
+ dimension=2,
+ distance="ip",
+ normalize_vectors=False,
+ field_types={"account_id": "string"},
+ config={
+ "micro_batching_enabled": True,
+ "micro_batching_max_wait_ms": 1.0,
+ "auto_filter_native_threshold": 10,
+ "auto_memory_reserve_mb": 0,
+ "auto_memory_safety_factor": 1.0,
+ },
+ runtime=runtime,
+ auto_memory=True,
+ )
+ index.add_candidates([candidate(1, [1.0, 0.0], account_id="a")])
+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=True)
+
+ with pytest.raises(CuVSNativeRouteError):
+ index.search(
+ [1.0, 0.0],
+ 1,
+ {"op": "must", "field": "account_id", "conds": ["a"]},
+ native_filter_resolver=lambda _filters: ([1], 1),
+ native_filter_layout_registrar=lambda _labels: None,
+ telemetry=telemetry,
+ )
+
+ assert telemetry.micro_batching_enabled is False
+ assert telemetry.micro_batching_warm_fast_path is False
+ assert telemetry.batch_size == 1
+ assert runtime.search_batch_sizes == []
+ index.close()
+
+
+def test_cuvs_micro_batch_close_drains_queued_search_and_rejects_new_work():
+ runtime = FakeCuVSRuntime()
+ index = make_micro_batch_index(
+ runtime,
+ micro_batching_max_wait_ms=100.0,
+ )
+ telemetry = CuVSSearchTelemetry(algorithm="brute_force", auto_mode=False)
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ search_future = executor.submit(
+ index.search,
+ [1.0, 0.0],
+ 1,
+ None,
+ None,
+ None,
+ telemetry,
+ )
+ deadline = time.monotonic() + 5
+ while time.monotonic() < deadline:
+ with index._micro_batch_condition:
+ if index._micro_batch_pending:
+ break
+ time.sleep(0.001)
+ else:
+ pytest.fail("search did not enter the micro-batch queue")
+ close_future = executor.submit(index.close)
+ assert search_future.result(timeout=5)[0] == [1]
+ close_future.result(timeout=5)
+
+ assert runtime.closed is True
+ assert telemetry.micro_batching_warm_fast_path is True
+ assert index._active_searches == 0
+ assert index._micro_batch_warm_lookahead == 0
+ with pytest.raises(RuntimeError, match="closed"):
+ index.search([1.0, 0.0], 1, None)
+
+
+ "config, message",
+ [
+ ({"micro_batching_max_batch_size": 0}, "between 1 and 8"),
+ ({"micro_batching_max_batch_size": 9}, "between 1 and 8"),
+ ({"micro_batching_max_wait_ms": -1}, "between 0 and 100"),
+ ({"micro_batching_max_wait_ms": 100.1}, "between 0 and 100"),
+ ({"micro_batching_max_wait_ms": float("inf")}, "between 0 and 100"),
+ (
+ {"micro_batching_enabled": True, "algorithm": "cagra"},
+ "brute_force",
+ ),
+ (
+ {"micro_batching_enabled": True, "max_concurrent_gpu_searches": 2},
+ "max_concurrent_gpu_searches=1",
+ ),
+ ],
+)
+def test_cuvs_micro_batch_raw_config_validation(config, message):
+ with pytest.raises(ValueError, match=message):
+ CuVSDenseIndex(
+ dimension=2,
+ distance="ip",
+ normalize_vectors=False,
+ field_types={},
+ config=config,
+ runtime=FakeCuVSRuntime(),
+ )
+
+
def test_inflight_search_keeps_immutable_snapshot_during_rebuild():
class BlockingRuntime(FakeCuVSRuntime):
def __init__(self):