Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

perf(rag): hash segments once and scan orphans periodically for large FAISS collections (#7867)

LearningCircuit1ff6ee8bc2b3
diff --git a/.github/workflows/rag-vector-scale.yml b/.github/workflows/rag-vector-scale.yml
new file mode 100644
index 0000000000..8dc71ff95e
--- /dev/null
+++ b/.github/workflows/rag-vector-scale.yml
@@ -0,0 +1,80 @@
+name: RAG Vector Scale
+
+# Manual latency benchmark for large segmented FAISS collections (#7810).
+# Vectors are synthetic and go straight into the store: no PDFs, no model.
+on:
+ workflow_dispatch:
+ inputs:
+ vectors:
+ description: Vectors in the benchmark collection
+ default: '1000000'
+ required: true
+ dimension:
+ description: Embedding dimension
+ default: '1024'
+ required: true
+
+permissions: {}
+
+jobs:
+ rag-vector-scale:
+ name: Vector scale (${{ matrix.index-type }})
+ runs-on: ubuntu-latest
+ timeout-minutes: 150
+ permissions:
+ contents: read
+ strategy:
+ fail-fast: false
+ matrix:
+ index-type: [flat, hnsw]
+ env:
+ OMP_NUM_THREADS: '2'
+ OPENBLAS_NUM_THREADS: '2'
+ HF_HUB_OFFLINE: '1'
+ TRANSFORMERS_OFFLINE: '1'
+ RAG_INDEX_TYPE: ${{ matrix.index-type }}
+ RAG_VECTORS: ${{ inputs.vectors }}
+ RAG_DIMENSION: ${{ inputs.dimension }}
+
+ steps:
+ - name: Harden runner
+ uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
+ with:
+ egress-policy: audit
+ - name: Checkout
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ with:
+ persist-credentials: false
+ - name: Set up PDM
+ uses: pdm-project/setup-pdm@544d7237314ee09c256785bd360f6b30add38b37 # v4.6
+ with:
+ python-version: '3.12'
+ version: 2.26.2
+ cache: true
+ cache-restore-exact-match: true
+ - name: Install SQLCipher
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y libsqlcipher-dev
+ - name: Install dependencies
+ run: pdm sync -d --clean
+ - name: Benchmark writes and searches
+ shell: bash
+ run: |
+ [[ "$RAG_VECTORS" =~ ^[0-9]+$ && "$RAG_DIMENSION" =~ ^[0-9]+$ ]] || {
+ echo "vectors and dimension must be integers" >&2
+ exit 1
+ }
+ mkdir -p test-results
+ pdm run python scripts/ci/rag_vector_scale_benchmark.py \
+ --vectors "$RAG_VECTORS" --dimension "$RAG_DIMENSION" \
+ --index-type "$RAG_INDEX_TYPE" \
+ --report "test-results/rag-vector-scale-$RAG_INDEX_TYPE.json" \
+ 2>&1 | tee "test-results/rag-vector-scale-$RAG_INDEX_TYPE.log"
+ - name: Upload benchmark report
+ if: always()
+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+ with:
+ name: rag-vector-scale-${{ matrix.index-type }}
+ path: test-results/rag-vector-scale-*
+ retention-days: 30
diff --git a/changelog.d/7810.bugfix.md b/changelog.d/7810.bugfix.md
new file mode 100644
index 0000000000..47f68c61b5
--- /dev/null
+++ b/changelog.d/7810.bugfix.md
@@ -0,0 +1 @@
+Speed up large RAG collections. Each vector segment is hashed once per process and re-hashed whenever its device, inode, size, modification time or change time differs from the values seen at that verification, so searches no longer re-hash the whole index on every query. Those fields move only with the filesystem's timestamp resolution, so a segment whose stored times are whole seconds (coarse-timestamp filesystems such as NFSv2/3, CIFS/SMB, VFAT, HFS+, some FUSE and overlay mounts) is hashed on every read instead of cached, closing the same-tick rewrite gap at pre-caching cost on those mounts. Writes run the full orphan-vector scan at the latest every 65th write — 64 writes raise the scan counter to its threshold and the next write scans — instead of on every write, and in between run it only when the index holds vector IDs above the highest chunk row, the only IDs a new chunk row could reuse. Segments shrunk by deletions merge with their neighbours. Add a manual large-collection latency benchmark.
diff --git a/docs/ci/workflow-status.md b/docs/ci/workflow-status.md
index 060dc7dc58..3bbdc9b161 100644
--- a/docs/ci/workflow-status.md
+++ b/docs/ci/workflow-status.md
@@ -196,6 +196,11 @@ the table above.
scanner binary and complete scan results before artifact and SARIF uploads.
Its status is reported in the calling workflows.
+- `rag-vector-scale.yml` — manual large-collection latency benchmark for
+ segmented FAISS (#7810). Flat and HNSW shards build a synthetic collection
+ (1,000,000 vectors of 1,024 dimensions by default) without PDFs or an
+ embedding model, and report per-document write time, search time and peak
+ RSS. It is not called by `release.yml` and never runs on pull requests.
- `rag-index-scale.yml` — required synthetic-PDF scale gate called by
`release.yml`, with manual dispatch available. Separate flat and HNSW
shards check bounded index writes, memory, search, rollback, deletion,
diff --git a/docs/library-and-rag.md b/docs/library-and-rag.md
index b0ced18b2a..767057a6f1 100644
--- a/docs/library-and-rag.md
+++ b/docs/library-and-rag.md
@@ -503,12 +503,20 @@ Uses OpenAI's embedding API.
FAISS indexes use segments of at most 4,096 vectors. Appends rewrite the
partially filled segment and write new segments; they do not rewrite the whole
collection. Searches visit segments sequentially and merge their best matches.
-Searches also re-verify (hash) every segment file against the authenticated
-manifest on each query. Full chunk records are loaded only for the requested
-results after their IDs are selected. This bounds the vector working set, at the
-cost of reading multiple segments for a large collection. Embedding models, PDF
-extraction, individual document preparation, and reconciliation metadata still
-contribute to memory use.
+Each segment file is verified (hashed) against the authenticated manifest the
+first time the server process reads it, and again whenever the file changes:
+a later read compares the file's device, inode, size, modification time and
+change time with those seen at verification and re-hashes on any difference.
+Since those fields move only with the filesystem's timestamp resolution, a
+segment whose stored times are whole seconds (coarse-timestamp filesystems
+such as NFSv2/3, CIFS/SMB, VFAT, HFS+, some FUSE and overlay mounts) is hashed on every
+read instead of cached. Full chunk records are loaded only
+for the requested results after their IDs are selected. This bounds the vector
+working set, at the cost of reading every segment for each search of a large
+collection. A deletion that leaves a segment less than half full merges it with
+a neighbouring segment when both fit. Embedding models, PDF extraction,
+individual document preparation, and reconciliation metadata still contribute
+to memory use.
Existing native `.faiss` files remain readable and convert on their next write
without re-embedding. Conversion reads the old payload through read-only memory
@@ -525,7 +533,13 @@ indexing replaces only that configuration's chunks, and document deletion purges
every retained configuration. A successful full collection rebuild replaces the
previous published configurations. Writes and reconciliation remove
unowned vectors; surviving chunks from partially indexed documents are retained
-and reported separately. A rollback checkpoint preserves the previous vector
+and reported separately. Reconciliation always checks every vector. A write
+checks every vector at the latest every 65th write — 64 writes raise the scan
+counter to its threshold and the next write scans — and sooner on the first
+write after an upgrade (whose index has no scan record) or when the index
+holds vector IDs above the highest chunk row, the only IDs a new chunk could
+reuse; a write that converts a legacy index checks every vector as part of the
+conversion, and a fresh empty index has nothing to check on its first write. A rollback checkpoint preserves the previous vector
generation until the database commits. Crash recovery restores a checkpoint
only when it matches the checksum in the encrypted database.
@@ -573,6 +587,27 @@ embedding quality or the memory requirements of a real embedding model. The
large shards run on releases or manual workflow requests; focused lifecycle
regressions and the small PDF generator test run in the regular pytest suite.
+### Large-collection latency benchmark
+
+The synthetic PDF test bounds memory but not time. A separate benchmark builds
+a large collection of synthetic vectors directly in the store, with real
+SQLCipher storage and no PDFs or embedding model, then times per-document
+writes and searches:
+
+```bash
+OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 pdm run python scripts/ci/rag_vector_scale_benchmark.py \
+ --vectors 300000 --dimension 1024 --index-type flat \
+ --report /tmp/rag-vector-scale.json
+```
+
+The report includes the first write after the bulk build (which runs the
+periodic full orphan scan), median and maximum steady write time, the cold
+first search, median and maximum warm search time, and peak RSS. Optional
+`--max-write-seconds`, `--max-search-seconds` and `--max-rss-mib` arguments
+turn these into limits. The default of 657 chunks per document matches the
+1,024-dimension library measured in #7628. The manual **RAG Vector Scale**
+workflow runs it with 1,000,000 vectors for flat and HNSW indexes.
+
---
## See Also
diff --git a/scripts/ci/rag_vector_scale_benchmark.py b/scripts/ci/rag_vector_scale_benchmark.py
new file mode 100644
index 0000000000..33e78d70ea
--- /dev/null
+++ b/scripts/ci/rag_vector_scale_benchmark.py
@@ -0,0 +1,11 @@
+"""Run the offline segmented FAISS latency benchmark from a source checkout."""
+
+from pathlib import Path
+import sys
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+from tests.performance.rag_vector_scale_benchmark import main # noqa: E402
+
+
+if __name__ == "__main__":
+ main()
diff --git a/src/local_deep_research/vector_stores/facade.py b/src/local_deep_research/vector_stores/facade.py
index 3941ab1942..230afab384 100644
--- a/src/local_deep_research/vector_stores/facade.py
+++ b/src/local_deep_research/vector_stores/facade.py
@@ -37,7 +37,7 @@ from typing import (
import numpy as np
from langchain_core.embeddings import Embeddings
from loguru import logger
-from sqlalchemy import event, exists, or_, select
+from sqlalchemy import event, exists, func, or_, select
from sqlalchemy.orm import Session
from ..database.models.library import (
@@ -264,11 +264,12 @@ class VectorIndex:
owner,
)
- def _apply(self, session: Session, **kwargs):
+ def _apply(self, session: Session, *, orphan_scanned=False, **kwargs):
"""Settle vector checkpoints with the caller's outer DB transaction."""
from .implementations.segmented_faiss import SegmentedFaissStore
if isinstance(self._store, SegmentedFaissStore):
+ kwargs["orphan_scanned"] = orphan_scanned
if session.in_nested_transaction():
raise RuntimeError(
"Vector writes require an outer database transaction"
@@ -297,13 +298,24 @@ class VectorIndex:
),
)
- def _prepare_cleanup(self, session: Session) -> tuple[list[int], list[int]]:
+ def _prepare_cleanup(
+ self, session: Session, *, force: bool = False
+ ) -> tuple[list[int], list[int], bool]:
"""Discover real orphans before allocating IDs that a rollback reused.
Partial documents remain owned: completeness is a status concern, not
a garbage-collection criterion. Legacy ownership is recovered only
from membership in this physical index. Queries use bounded IN lists.
+
+ A store that tracks its scans (segmented FAISS) is fully scanned only
+ periodically, or when it holds IDs above the highest chunk row; the
+ last value says whether this call scanned.
"""
+ due = getattr(self._store, "orphan_scan_due", None)
+ if not force and due is not None:
+ highest = session.scalar(select(func.max(DocumentChunk.id)))
+ if not due(highest):
+ return [], [], False
orphan_vectors: list[int] = []
orphan_rows: list[int] = []
for ids in self._store.iter_id_batches():
@@ -334,15 +346,19 @@ class VectorIndex:
)
orphan_vectors.extend(set(ids) - valid)
orphan_rows.extend(owned - valid)
- return orphan_vectors, orphan_rows
+ return orphan_vectors, orphan_rows, True
def compact_orphans(self, session: Optional[Session] = None) -> int:
"""Remove only unowned vectors; retain every valid partial document."""
with self._db(session) as (sess, owns):
- vectors, rows = self._prepare_cleanup(sess)
+ vectors, rows, scanned = self._prepare_cleanup(sess, force=True)
if vectors:
self._apply(
- sess, add_ids=[], add_vectors=None, remove_ids=vectors
+ sess,
+ orphan_scanned=scanned,
+ add_ids=[],
+ add_vectors=None,
+ remove_ids=vectors,
)
self._finalize_db(sess, owns, rows, "compact orphan vectors")
return len(vectors)
@@ -376,7 +392,7 @@ class VectorIndex:
sid = str(source_id) if source_id is not None else None
with self._db(session) as (sess, owns):
- orphan_vectors, orphan_rows = self._prepare_cleanup(sess)
+ orphan_vectors, orphan_rows, scanned = self._prepare_cleanup(sess)
prior_ids: List[int] = []
if replace and sid is not None:
prior_ids = list(
@@ -400,6 +416,7 @@ class VectorIndex:
try:
stats = self._apply(
sess,
+ orphan_scanned=scanned,
add_ids=new_ids,
add_vectors=vectors,
remove_ids=prior_ids + orphan_vectors,
@@ -468,7 +485,7 @@ class VectorIndex:
sid = str(source_id) if source_id is not None else None
with self._db(session) as (sess, owns):
- orphan_vectors, orphan_rows = self._prepare_cleanup(sess)
+ orphan_vectors, orphan_rows, scanned = self._prepare_cleanup(sess)
prior_ids: List[int] = []
if replace and sid is not None:
# Scope the prior-chunk lookup to THIS store's embedding-model
@@ -505,6 +522,7 @@ class VectorIndex:
try:
stats = self._apply(
sess,
+ orphan_scanned=scanned,
add_ids=new_ids,
add_vectors=vectors,
remove_ids=prior_ids + orphan_vectors,
@@ -738,7 +756,7 @@ class VectorIndex:
if not source_ids:
return DeleteStats(removed=0, rows_deleted=0)
with self._db(session) as (sess, owns):
- orphan_vectors, orphan_rows = self._prepare_cleanup(sess)
+ orphan_vectors, orphan_rows, scanned = self._prepare_cleanup(sess)
ids: list[int] = []
for start in range(0, len(source_ids), 500):
ids.extend(
@@ -758,6 +776,7 @@ class VectorIndex:
try:
stats = self._apply(
sess,
+ orphan_scanned=scanned,
add_ids=[],
add_vectors=None,
remove_ids=removed,
diff --git a/src/local_deep_research/vector_stores/implementations/segmented_faiss.py b/src/local_deep_research/vector_stores/implementations/segmented_faiss.py
index 401f5b782d..fb64b2176c 100644
--- a/src/local_deep_research/vector_stores/implementations/segmented_faiss.py
+++ b/src/local_deep_research/vector_stores/implementations/segmented_faiss.py
@@ -10,6 +10,20 @@ Like the existing FAISS store, coordination is per-process. All readers reopen
the manifest under the injected index lock, so a successful writer can retire
superseded files without invalidating a reader's snapshot.
+Each segment file is hashed against the manifest the first time this process
+reads it. Later reads skip the hash while the file keeps the device, inode,
+size, modification time and change time seen at that verification. Writing
+to a file changes its change time, which ``utime`` cannot set back, so a
+rewrite is hashed again before use whenever it moves one of those fields.
+A replacement usually changes the inode, but the inode can be reused, so
+both a same-size rewrite whose modification time is restored within one
+filesystem timestamp tick and a same-size replacement that reuses the inode
+and restores the modification time within that tick move none of the five
+fields; ticks are microseconds on typical local filesystems but whole
+seconds on coarse-timestamp mounts (NFSv2/3, CIFS/SMB, VFAT, HFS+, some FUSE
+and overlay mounts), so a segment whose stored times are whole seconds
+forfeits the cache and is hashed on every read.
+
Crash safety additionally assumes that every writer of a published
generation, and every non-nested ``verify_file()`` / ``recover_checkpoint``
caller, holds the collection transaction lock
@@ -27,6 +41,7 @@ the transaction lock, stage cleanup under the rebuild lock, while
from __future__ import annotations
+from collections import OrderedDict
from contextlib import contextmanager, nullcontext
import hashlib
import json
@@ -35,6 +50,7 @@ from pathlib import Path
import re
import shutil
import tempfile
+import threading
import uuid
import faiss
@@ -61,6 +77,12 @@ SEGMENT_VECTORS = 4096
_NAME = re.compile(r"[0-9a-f]{32}\.faiss\Z")
_HASH = re.compile(r"[0-9a-f]{64}\Z")
_MAX_MANIFEST_BYTES = 16 * 1024 * 1024
+# A full orphan scan reads every vector ID; between scans a write only checks
+# for IDs above the highest chunk row, the only orphans a new row could reuse.
+ORPHAN_SCAN_INTERVAL = 64
+_SCAN_COUNTER = "writes_since_orphan_scan"
+# Verified segment files remembered per process (about 200 bytes each).
+_VERIFIED_FILES = 65536
def is_segmented(path: Path) -> bool:
@@ -114,6 +136,93 @@ def _validate_entries(entries) -> None:
raise ValueError("Invalid vector segment bounds")
+def _file_signature(path: Path):
+ """Identity of a file's current contents, or None if it cannot be read."""
+ try:
+ info = os.lstat(path)
+ except OSError:
+ return None
+ return (
+ info.st_dev,
+ info.st_ino,
+ info.st_size,
+ info.st_mtime_ns,
+ info.st_ctime_ns,
+ )
+
+
+_SECOND_NS = 1_000_000_000
+
+
+def _coarse_timestamps(mtime_ns: int, ctime_ns: int) -> bool:
+ """Whether stored times look whole-second, hiding same-tick rewrites.
+
+ Filesystems that store only whole-second timestamps (NFSv2/3, CIFS/SMB,
+ VFAT, HFS+, some FUSE and overlay mounts) report every time on a whole
+ second, so a same-size rewrite whose modification time is restored
+ within the same second leaves every recorded field unchanged and the
+ verification record cannot tell it from an unchanged file. On such
+ mounts the record is worthless: a whole-second time in EITHER field
+ (a rewrite cannot forge the change time finer than the mount stores
+ it) forfeits the cache, at the cost of one extra hash per read.
+ """
+ return mtime_ns % _SECOND_NS == 0 or ctime_ns % _SECOND_NS == 0
+
+
+class _VerifiedFiles:
+ """Bounded record of files whose digest matched, keyed by digest."""
+
+ def __init__(self, limit):
+ self.limit = limit
+ self._entries = OrderedDict()
+ self._lock = threading.Lock()
+
+ def matches(self, path, digest):
+ signature = _file_signature(path)
+ with self._lock:
+ known = self._entries.get((str(path), digest))
+ if known is None or signature is None or known != signature:
+ return False
+ self._entries.move_to_end((str(path), digest))
+ return True
+
+ def remember(self, path, digest, signature):
+ with self._lock:
+ self._entries[(str(path), digest)] = signature
+ self._entries.move_to_end((str(path), digest))
+ while len(self._entries) > self.limit:
+ self._entries.popitem(last=False)
+
+ def clear(self):
+ with self._lock:
+ self._entries.clear()
+
+
+_VERIFIED = _VerifiedFiles(_VERIFIED_FILES)
+
+
+def _verified_digest(path: Path, expected: str) -> bool:
+ """Whether ``path`` holds ``expected``, hashing only unseen contents.
+
+ Contents whose stored times look whole-second are never cached — see
+ ``_coarse_timestamps`` — so they are hashed on every read.
+ """
+ path = path.absolute()
+ before = _file_signature(path)
+ cacheable = before is not None and not _coarse_timestamps(
+ before[3], before[4]
+ )
+ if cacheable and _VERIFIED.matches(path, expected):
+ return True
+ if _digest(path) != expected:
+ return False
+ # Remember only cacheable contents that did not change while being
+ # hashed.
+ if cacheable and before == _file_signature(path):
+ _VERIFIED.remember(path, expected, before)
+ return True
+
+
def _sync_directory(path: Path) -> None:
try:
descriptor = os.open(path, os.O_RDONLY)
@@ -214,8 +323,12 @@ class SegmentedFaissStore(BaseVectorStore):
)
def _state(self, *, verify=True):
+ manifest = self._manifest(verify=verify)
+ return manifest if manifest is None else manifest["segments"]
+
+ def _manifest(self, *, verify=True):
if not self._path.exists():
- return []
+ return {"segments": []}
if verify and self._verify is not None:
verified, reason = self._verify(self._path)
if not verified:
@@ -249,7 +362,7 @@ class SegmentedFaissStore(BaseVectorStore):
if not isinstance(entries, list):
raise ValueError("Invalid vector segment list")
_validate_entries(entries)
- return entries
+ return state
def begin_transaction(self, *, first=True):
"""Retain the prior root and its segments until the owning DB settles."""
@@ -428,7 +541,7 @@ class SegmentedFaissStore(BaseVectorStore):
"Missing or symlinked vector segment"
)
expected = entry["ids_sha256" if ids else "sha256"]
- if _digest(path) != expected:
+ if not _verified_digest(path, expected):
raise VectorStoreIntegrityError("Vector segment checksum mismatch")
return path
@@ -491,7 +604,7 @@ class SegmentedFaissStore(BaseVectorStore):
"max_id": max(ids),
}
- def _publish(self, entries):
+ def _publish(self, entries, scan_count=None):
# Publish only a generation every later read accepts: this is the
# exact validation _state() applies, run BEFORE the manifest is
# written and authenticated.
@@ -507,6 +620,8 @@ class SegmentedFaissStore(BaseVectorStore):
"normalize": self.normalize,
"segments": entries,
}
+ if scan_count is not None:
+ manifest[_SCAN_COUNTER] = scan_count
data = json.dumps(manifest, separators=(",", ":")).encode()
if len(data) > _MAX_MANIFEST_BYTES:
raise ValueError("Vector index manifest exceeds its metadata limit")
@@ -604,9 +719,69 @@ class SegmentedFaissStore(BaseVectorStore):
"A segmented index must be persisted at its bound path"
)
with self._guard():
- entries = self._state()
- if entries is not None:
- self._publish(entries)
+ manifest = self._manifest()
+ if manifest is not None:
+ self._publish(manifest["segments"], manifest.get(_SCAN_COUNTER))
+
+ def orphan_scan_due(self, max_chunk_id):
+ """Whether the next write must check every vector ID for orphans.
+
+ Between periodic full scans only vectors above the highest chunk row
+ matter before a write: their IDs are the only ones a new row can
+ receive. A legacy file is fully checked by the write that converts
+ it, and a manifest with no scan record yet (an upgraded index) by
+ the next write; an index with no segments has nothing to check.
+ Thereafter a scan is due at the latest every 65th write: 64 writes
+ raise the counter to its threshold and the next write scans.
+ """
+ with self._guard():
+ manifest = self._manifest()
+ if manifest is None:
+ return True
+ entries = manifest["segments"]
+ if not entries:
+ return False
+ count = manifest.get(_SCAN_COUNTER)
+ if type(count) is not int or not 0 <= count < ORPHAN_SCAN_INTERVAL:
+ return True
+ ceiling = -1 if max_chunk_id is None else int(max_chunk_id)
+ return any(entry["max_id"] > ceiling for entry in entries)
+
+ def _merge_small(self, entries, changed):
+ """Fold a rewritten segment into an adjacent one when both fit.
+
+ A pair merges only when this write produced at least one of its two
+ segments, so each write does bounded extra work and old fragmentation
+ merges as it is touched: a rewritten small segment may merge into an
+ untouched neighbour, rewriting it, while untouched full segments are
+ never rewritten.
+ """
+ merged: list[dict] = []
+ for entry in entries:
+ if merged:
+ previous = merged[-1]
+ if (
+ (previous["file"] in changed or entry["file"] in changed)
+ and min(previous["count"], entry["count"])
+ < SEGMENT_VECTORS // 2
+ and previous["count"] + entry["count"] <= SEGMENT_VECTORS
+ ):
+ target = self._read(previous)
+ source = self._read(entry)
+ ids = self._ids(entry)
+ vectors = np.asarray(
+ [source._index.reconstruct(value) for value in ids],
+ dtype="float32",
+ )
+ target._index.add_with_ids(
+ vectors, np.asarray(ids, dtype="int64")
+ )
+ merged[-1] = self._write(target)
+ changed.add(merged[-1]["file"])
+ del target, source, vectors
+ continue
+ merged.append(entry)
+ return merged
@contextmanager
def _mutation(self):
@@ -630,7 +805,15 @@ class SegmentedFaissStore(BaseVectorStore):
"Deferred retirement of superseded vector segments"
)
- def apply(self, *, add_ids, add_vectors, remove_ids=(), dedup=True):
+ def apply(
+ self,
+ *,
+ add_ids,
+ add_vectors,
+ remove_ids=(),
+ dedup=True,
+ orphan_scanned=False,
+ ):
if self._lock is None or self._record is None or self._verify is None:
raise RuntimeError(
"Vector writes require lock and integrity callbacks"
@@ -656,6 +839,17 @@ class SegmentedFaissStore(BaseVectorStore):
removals = set(map(int, remove_ids)) | set(add_ids)
removed = 0
with self._mutation() as previous:
+ if orphan_scanned:
+ scan_count = 0
+ else:
+ count = None
+ if previous is not None and self._path.exists():
+ count = self._manifest(verify=False).get(_SCAN_COUNTER)
+ scan_count = (
+ count + 1
+ if type(count) is int and 0 <= count < ORPHAN_SCAN_INTERVAL
+ else ORPHAN_SCAN_INTERVAL
+ )
if previous is None:
# Legacy conversion. The id map is normalized with the same
# discipline as the add path before anything is written, so a
@@ -746,6 +940,7 @@ class SegmentedFaissStore(BaseVectorStore):
removed = len(removed_ids)
else:
entries = []
+ changed = set()
for entry in previous:
candidates = {
value
@@ -768,7 +963,9 @@ class SegmentedFaissStore(BaseVectorStore):
store._rebuild_dropping(actual)
if store.count():
entries.append(self._write(store))
+ changed.add(entries[-1]["file"])
del store
+ entries = self._merge_small(entries, changed)
position = 0
if add_ids and entries and entries[-1]["count"] < SEGMENT_VECTORS:
tail = self._read(entries.pop())
@@ -789,5 +986,5 @@ class SegmentedFaissStore(BaseVectorStore):
entries.append(self._write(store))
del store
position = end
- self._publish(entries)
+ self._publish(entries, scan_count)
return {"added": len(add_ids), "removed": removed}
diff --git a/tests/performance/_shared/resources.py b/tests/performance/_shared/resources.py
new file mode 100644
index 0000000000..772bb9a877
--- /dev/null
+++ b/tests/performance/_shared/resources.py
@@ -0,0 +1,25 @@
+"""Peak memory of the workload executable, in MiB."""
+
+from pathlib import Path
+import sys
+
+
+def peak_rss_mib():
+ if sys.platform == "linux":
+ # ru_maxrss can retain the spawning process's pre-exec high-water
+ # mark. VmHWM describes the current executable image instead.
+ try:
+ for line in Path("/proc/self/status").read_text().splitlines():
+ if line.startswith("VmHWM:"):
+ return int(line.split()[1]) / 1024
+ except (OSError, ValueError, IndexError):
+ pass
+ try:
+ import resource
+ except ImportError:
+ raise RuntimeError(
+ "RSS guard requires procfs or resource.getrusage"
+ ) from None
+ peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+ # The fallback is conservative on Linux when procfs is unavailable.
+ return peak / (1024 * 1024 if sys.platform == "darwin" else 1024)
diff --git a/tests/performance/rag_index_workload.py b/tests/performance/rag_index_workload.py
index 0ae7aa0d8a..c9d8a1b3f7 100644
--- a/tests/performance/rag_index_workload.py
+++ b/tests/performance/rag_index_workload.py
@@ -15,7 +15,6 @@ import hashlib
import json
import os
from pathlib import Path
-import resource
import sys
import tempfile
import time
@@ -48,6 +47,7 @@ from local_deep_research.vector_stores.implementations import (
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from tests.fixtures.synthetic_pdf import synthetic_pdf # noqa: E402
+from tests.performance._shared.resources import peak_rss_mib # noqa: E402
def require(condition, detail):
@@ -72,7 +72,13 @@ class EmbeddingManager:
self.embeddings = SyntheticEmbeddings(dimension)
-def run(args, root):
+@contextmanager
+def workload_session(root):
+ """Yield a session on a real encrypted database that every caller shares.
+
+ Yields ``(session, key, shared_session, db_path)``. Production modules
+ that open their own user session receive this one instead.
+ """
os.environ["LDR_DATA_DIR"] = str(root)
logger.disable("local_deep_research")
db_path = root / "documents.encrypted.db"
@@ -94,6 +100,30 @@ def run(args, root):
def shared_session(*_args, **_kwargs):
yield session
+ with ExitStack() as stack:
+ stack.callback(engine.dispose)
+ stack.callback(session.close)
+ for module in (
+ "research_library.services.library_rag_service",
+ "vector_stores.facade",
+ "security.file_integrity.integrity_manager",
+ ):
+ stack.enter_context(
+ patch(
+ f"local_deep_research.{module}.get_user_db_session",
+ shared_session,
+ )
+ )
+ stack.enter_context(
+ patch(
+ "local_deep_research.database.thread_local_session.get_current_thread_session",
+ return_value=session,
+ )
+ )
+ yield session, key, shared_session, db_path
+
+
+def run(args, root):
metrics = {
"faiss_bytes_written": 0,
"metadata_bytes_written": 0,
@@ -123,26 +153,10 @@ def run(args, root):
return original_atomic(path, data)
started = time.monotonic()
- with ExitStack() as stack:
- stack.callback(engine.dispose)
- stack.callback(session.close)
- for module in (
- "research_library.services.library_rag_service",
- "vector_stores.facade",
- "security.file_integrity.integrity_manager",
- ):
- stack.enter_context(
- patch(
- f"local_deep_research.{module}.get_user_db_session",
- shared_session,
- )
- )
- stack.enter_context(
- patch(
- "local_deep_research.database.thread_local_session.get_current_thread_session",
- return_value=session,
- )
- )
+ with (
+ workload_session(root) as (session, key, shared_session, db_path),
+ ExitStack() as stack,
+ ):
stack.enter_context(
patch.object(faiss_store, "write_index", write_index)
)
@@ -372,10 +386,7 @@ def run(args, root):
)
metrics["elapsed_seconds"] = round(time.monotonic() - started, 3)
- peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
- metrics["peak_rss_mib"] = round(
- peak / (1024 * 1024 if sys.platform == "darwin" else 1024), 2
- )
+ metrics["peak_rss_mib"] = round(peak_rss_mib(), 2)
require(metrics["peak_rss_mib"] <= args.max_rss_mib, metrics)
# No generated PDF or plaintext chunk cache may have reached disk.
for path in root.rglob("*"):
diff --git a/tests/performance/rag_scale_smoke/README.md b/tests/performance/rag_scale_smoke/README.md
new file mode 100644
index 0000000000..3deb93ae29
--- /dev/null
+++ b/tests/performance/rag_scale_smoke/README.md
@@ -0,0 +1,24 @@
+This automatic CI shard runs the original synthetic-PDF ingestion workload and
+the original vector-scale workload for both flat and HNSW. Each CLI runs in an
+owned subprocess, serially, with native thread counts set to one. Reports and
+JUnit evidence are retained by the Performance Assertions workflow.
+
+CI defaults: eight 16-page PDFs, 8,192 vectors, 32 dimensions, 64 chunks/write,
+66 writes, four searches and a 768 MiB per-child RSS ceiling. The data crosses
+the real 4,096-vector segment boundary and the 64-write orphan-scan interval.
+The PDF workload's bounded-write, rollback, reconciliation and rebuild checks
+remain active. Vector checks assert added/final counts, segment size, unique
+finite search results and a known inserted vector ranking first.
+
+```bash
+pdm run pytest tests/performance/rag_scale_smoke -q -n 0 \
+ --ragscale-profile ci
+```
+
+The `local` profile uses 16 PDFs, 12,000 vectors, 128 dimensions, 70 writes and
+eight searches. CLI `--ragscale-{profile,documents,pages,vectors,dimension,
+chunks-per-document,writes,searches,max-rss-mib}` overrides the matching
+`LDR_PERF_RAGSCALE_*` environment value. `LDR_PERF_PROFILE` is the profile
+fallback. `LDR_PERF_RAGSCALE_REPORT_DIR` selects persistent report storage.
+Sizes must retain the asserted segment/scan coverage. Larger manual/release
+workflows keep their existing workloads; this shard supplies bounded PR CI.
diff --git a/tests/performance/rag_scale_smoke/__init__.py b/tests/performance/rag_scale_smoke/__init__.py
new file mode 100644
index 0000000000..998f3c4aae
--- /dev/null
+++ b/tests/performance/rag_scale_smoke/__init__.py
@@ -0,0 +1 @@
+"""Bounded automatic coverage for the original PDF and vector workloads."""
diff --git a/tests/performance/rag_scale_smoke/conftest.py b/tests/performance/rag_scale_smoke/conftest.py
new file mode 100644
index 0000000000..6a2b873914
--- /dev/null
+++ b/tests/performance/rag_scale_smoke/conftest.py
@@ -0,0 +1,76 @@
+import math
+import os
+
+import pytest
+
+PROFILES = {
+ "ci": dict(
+ documents=8,
+ pages=16,
+ vectors=8192,
+ dimension=32,
+ chunks_per_document=64,
+ writes=66,
+ searches=4,
+ max_rss_mib=768,
+ ),
+ "local": dict(
+ documents=16,
+ pages=16,
+ vectors=12000,
+ dimension=128,
+ chunks_per_document=657,
+ writes=70,
+ searches=8,
+ max_rss_mib=1536,
+ ),
+}
+
+
+def pytest_addoption(parser):
+ group = parser.getgroup("original RAG workload profiles")
+ for field in (
+ "profile",
+ "documents",
+ "pages",
+ "vectors",
+ "dimension",
+ "chunks-per-document",
+ "writes",
+ "searches",
+ "max-rss-mib",
+ ):
+ group.addoption(f"--ragscale-{field}", default=None)
+
+
[email protected](scope="session")
+def rag_scale_settings(request):
+ def value(field, default):
+ cli = request.config.getoption(f"--ragscale-{field.replace('_', '-')}")
+ return (
+ cli
+ if cli is not None
+ else os.environ.get(f"LDR_PERF_RAGSCALE_{field.upper()}") or default
+ )
+
+ profile = value("profile", os.environ.get("LDR_PERF_PROFILE") or "ci")
+ if profile not in PROFILES:
+ raise pytest.UsageError("ragscale profile must be ci or local")
+ result = dict(profile=profile)
+ for field, default in PROFILES[profile].items():
+ try:
+ number = int(value(field, default))
+ except (ValueError, TypeError):
+ raise pytest.UsageError(f"Invalid ragscale {field}") from None
+ if not math.isfinite(number) or number <= 0:
+ raise pytest.UsageError(f"Invalid ragscale {field}")
+ result[field] = number
+ if (
+ result["documents"] < 4
+ or result["dimension"] < 8
+ or result["vectors"] < 1000
+ ):
+ raise pytest.UsageError(
+ "ragscale needs >=4 documents, >=8 dimensions and >=1000 vectors"
+ )
+ return result
diff --git a/tests/performance/rag_scale_smoke/test_offline.py b/tests/performance/rag_scale_smoke/test_offline.py
new file mode 100644
index 0000000000..4f1ea01703
--- /dev/null
+++ b/tests/performance/rag_scale_smoke/test_offline.py
@@ -0,0 +1,116 @@
+"""Assert the actual original CLI workloads in owned, serial child processes."""
+
+import json
+import os
+from pathlib import Path
+import subprocess
+import sys
+
+import pytest
+
+from local_deep_research.vector_stores.implementations.segmented_faiss import (
+ SEGMENT_VECTORS,
+ ORPHAN_SCAN_INTERVAL,
+)
+
+# The dedicated performance job selects this integration explicitly. Keeping
+# it out of the parallel default lane isolates native RSS and global caches.
+pytestmark = [pytest.mark.integration, pytest.mark.timeout(360)]
+
+
[email protected]("index_type", ["flat", "hnsw"])
[email protected]("workload", ["pdf", "vectors"])
+def test_original_workload_has_asserted_bounded_ci_coverage(
+ tmp_path, rag_scale_settings, index_type, workload
+):
+ root = Path(__file__).resolve().parents[3]
+ settings = rag_scale_settings
+ reports = Path(os.environ.get("LDR_PERF_RAGSCALE_REPORT_DIR") or tmp_path)
+ reports.mkdir(parents=True, exist_ok=True)
+ report = reports / f"{workload}-{index_type}.json"
+ script = (
+ "rag_index_workload.py"
+ if workload == "pdf"
+ else "rag_vector_scale_benchmark.py"
+ )
+ command = [
+ sys.executable,
+ str(root / "scripts/ci" / script),
+ "--dimension",
+ str(settings["dimension"]),
+ "--index-type",
+ index_type,
+ "--max-rss-mib",
+ str(settings["max_rss_mib"]),
+ "--report",
+ str(report),
+ ]
+ if workload == "pdf":
+ command.extend(
+ [
+ "--documents",
+ str(settings["documents"]),
+ "--pages",
+ str(settings["pages"]),
+ ]
+ )
+ else:
+ command.extend(
+ [
+ "--vectors",
+ str(settings["vectors"]),
+ "--writes",
+ str(settings["writes"]),
+ "--chunks-per-document",
+ str(settings["chunks_per_document"]),
+ "--searches",
+ str(settings["searches"]),
+ "--verify-known-query",
+ ]
+ )
+ environment = dict(
+ os.environ,
+ PYTHONPATH=f"{root / 'src'}{os.pathsep}{root}",
+ OMP_NUM_THREADS="1",
+ OPENBLAS_NUM_THREADS="1",
+ MKL_NUM_THREADS="1",
+ HF_HUB_OFFLINE="1",
+ TRANSFORMERS_OFFLINE="1",
+ TOKENIZERS_PARALLELISM="false",
+ )
+ completed = subprocess.run(
+ command,
+ cwd=root,
+ env=environment,
+ capture_output=True,
+ text=True,
+ timeout=120 if settings["profile"] == "ci" else 300,
+ )
+ assert completed.returncode == 0, (
+ completed.stdout[-4000:] + completed.stderr[-4000:]
+ )
+ result = json.loads(report.read_text(encoding="utf-8"))
+ assert result["dimension"] == settings["dimension"]
+ assert result["index_type"] == index_type
+ assert result["peak_rss_mib"] <= settings["max_rss_mib"]
+ assert result["segments"] >= 2 # CI must really cross a segment boundary.
+ assert result["largest_segment_vectors"] <= SEGMENT_VECTORS
+ assert result["elapsed_seconds"] > 0
+ if workload == "pdf":
+ assert result["document_count"] == settings["documents"]
+ assert result["pages_per_document"] == settings["pages"]
+ assert result["initial_vectors"] > SEGMENT_VECTORS
+ assert 0 < result["remaining_vectors"] == result["rebuilt_vectors"]
+ assert result["faiss_bytes_written"] > 0
+ assert result["metadata_bytes_written"] > 0
+ else:
+ assert result["initial_vectors"] == settings["vectors"]
+ assert result["writes_completed"] == settings["writes"]
+ assert result["writes_completed"] > ORPHAN_SCAN_INTERVAL
+ assert (
+ result["final_vectors"]
+ == settings["vectors"]
+ + settings["writes"] * settings["chunks_per_document"]
+ )
+ assert result["searches_completed"] == settings["searches"]
+ assert result["known_query_top1_matches"] == 1
diff --git a/tests/performance/rag_vector_scale_benchmark.py b/tests/performance/rag_vector_scale_benchmark.py
new file mode 100644
index 0000000000..2ad497c6e9
--- /dev/null
+++ b/tests/performance/rag_vector_scale_benchmark.py
@@ -0,0 +1,296 @@
+"""Offline latency probe for segmented FAISS at large vector counts.
+
+Builds a collection of synthetic vectors through the production vector store
+and a real encrypted SQLCipher database, without PDFs or an embedding model.
+It then times per-document writes through ``VectorIndex.index_prepared`` and
+searches through ``LibraryRAGService.search``, the paths whose cost grows
+with collection size. The report contains numeric diagnostics only.
+"""
+
+from __future__ import annotations
+
+import argparse
+from contextlib import ExitStack
+import hashlib
+import json
+import math
+from pathlib import Path
+import statistics
+import sys
+import tempfile
+import time
+
+import numpy as np
+
+from local_deep_research.database.models.library import (
+ Collection,
+ Document,
+ DocumentCollection,
+ DocumentStatus,
+ SourceType,
+)
+from local_deep_research.research_library.services.library_rag_service import (
+ LibraryRAGService,
+)
+from local_deep_research.vector_stores.facade import ChunkInput
+from local_deep_research.vector_stores.implementations import segmented_faiss
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+from tests.performance._shared.resources import peak_rss_mib # noqa: E402
+from tests.performance.rag_index_workload import ( # noqa: E402
+ EmbeddingManager,
+ require,
+ workload_session,
+)
+
+# Documents added per bulk-build transaction.
+_BUILD_BATCH = 32
+
+
+def _add_documents(session, source_id, collection_id, numbers):
+ for number in numbers:
+ document_id = f"bench-{number:07d}"
+ session.add(
+ Document(
+ id=document_id,
+ source_type_id=source_id,
+ title="Synthetic benchmark document",
+ document_hash=hashlib.sha256(document_id.encode()).hexdigest(),
+ text_content=document_id,
+ file_type="pdf",
+ file_size=len(document_id),
+ status=DocumentStatus.COMPLETED,
+ )
+ )
+ session.add(
+ DocumentCollection(
+ document_id=document_id, collection_id=collection_id
+ )
+ )
+ return [f"bench-{number:07d}" for number in numbers]
+
+
+def _chunks(document_id, count):
+ return [
+ ChunkInput(
+ text=f"{document_id} chunk {position}",
+ metadata={"chunk_index": position},
+ )
+ for position in range(count)
+ ]
+
+
+def _seconds(values):
+ return {
+ "p50": round(statistics.median(values), 4),
+ "max": round(max(values), 4),
+ }
+
+
+def run(args, root):
+ verify_known = getattr(args, "verify_known_query", False)
+ require(
+ not verify_known or args.dimension >= 8,
+ "Known-query checks require at least 8 dimensions",
+ )
+ rng = np.random.default_rng(7)
+ metrics = {
+ "target_vectors": args.vectors,
+ "dimension": args.dimension,
+ "index_type": args.index_type,
+ "chunks_per_document": args.chunks_per_document,
+ "orphan_scan_interval": segmented_faiss.ORPHAN_SCAN_INTERVAL,
+ }
+ started = time.monotonic()
+ with (
+ workload_session(root) as (session, key, _shared, _db_path),
+ ExitStack() as stack,
+ ):
+ source = SourceType(
+ id="benchmark-source", name="document", display_name="Benchmark"
+ )
+ collection = Collection(id="benchmark", name="Vector scale benchmark")
+ session.add_all([source, collection])
+ session.commit()
+ source_id, collection_id = source.id, collection.id
+ rag = LibraryRAGService(
+ username="synthetic",
+ db_password=key,
+ embedding_model="synthetic-sha256",
+ embedding_provider="sentence_transformers",
+ embedding_manager=EmbeddingManager(args.dimension),
+ chunk_size=1000,
+ chunk_overlap=200,
+ index_type=args.index_type,
+ distance_metric="cosine",
+ normalize_vectors=True,
+ )
+ stack.callback(rag.close)
+ name = f"collection_{collection_id}"
+ index = rag._get_vector_index(collection_id, name)
+
+ # Bulk build through the same row and vector path as a write, but
+ # without the per-write orphan check: the build is setup, not the
+ # measured workload.
+ documents = math.ceil(args.vectors / args.chunks_per_document)
+ built = 0
+ for start in range(0, documents, _BUILD_BATCH):
+ numbers = range(start, min(documents, start + _BUILD_BATCH))
+ rows = []
+ for document_id in _add_documents(
+ session, source_id, collection_id, numbers
+ ):
+ count = min(args.chunks_per_document, args.vectors - built)
+ rows.extend(
+ index._build_row(chunk, "document", document_id)
+ for chunk in _chunks(document_id, count)
+ )
+ built += count
+ session.add_all(rows)
+ session.flush()
+ index._apply(
+ session,
+ add_ids=[row.id for row in rows],
+ add_vectors=rng.standard_normal(
+ (len(rows), args.dimension), dtype=np.float32
+ ),
+ )
+ session.commit()
+ del rows
+ if (start // _BUILD_BATCH) % 20 == 0:
+ print(json.dumps({"built_vectors": built}), flush=True)
+ metrics["build_seconds"] = round(time.monotonic() - started, 1)
+ metrics["initial_vectors"] = index.count()
+ metrics["segments"] = len(index._store._state())
+ require(metrics["initial_vectors"] == args.vectors, metrics)
+
+ # Per-document writes. The first one after the bulk build also runs
+ # the periodic full orphan scan, so it is reported separately.
+ write_seconds = []
+ numbers = range(documents, documents + args.writes)
+ _add_documents(session, source_id, collection_id, numbers)
+ session.commit()
+ known_document = f"bench-{documents:07d}"
+ known_query = "benchmark known-vector query"
+ metrics["writes_completed"] = 0
+ for number in numbers:
+ document_id = f"bench-{number:07d}"
+ vectors = rng.standard_normal(
+ (args.chunks_per_document, args.dimension), dtype=np.float32
+ )
+ if verify_known and number == documents:
+ vectors[0] = np.asarray(
+ index.embeddings.embed_query(known_query), dtype=np.float32
+ )
+ chunks = _chunks(document_id, args.chunks_per_document)
+ begun = time.monotonic()
+ stats = index.index_prepared(
+ source_type="document",
+ source_id=document_id,
+ chunks=chunks,
+ vectors=vectors,
+ )
+ write_seconds.append(time.monotonic() - begun)
+ require(stats.added == args.chunks_per_document, stats)
+ metrics["writes_completed"] += 1
+ metrics["final_vectors"] = index.count()
+ require(
+ metrics["final_vectors"]
+ == args.vectors + args.writes * args.chunks_per_document,
+ metrics,
+ )
+ entries = index._store._state()
+ metrics["largest_segment_vectors"] = max(
+ entry["count"] for entry in entries
+ )
+ require(
+ metrics["largest_segment_vectors"]
+ <= segmented_faiss.SEGMENT_VECTORS,
+ metrics,
+ )
+ metrics["first_write_seconds"] = round(write_seconds[0], 4)
+ metrics["write_seconds"] = _seconds(write_seconds[1:] or write_seconds)
+
+ # Searches through the service, as the research tools call them.
+ # The first search starts with no segment verified in this process.
+ segmented_faiss._VERIFIED.clear()
+ search_seconds = []
+ metrics["searches_completed"] = metrics["known_query_top1_matches"] = 0
+ for number in range(args.searches):
+ query = (
+ known_query
+ if verify_known and number == 0
+ else f"benchmark query {number}"
+ )
+ begun = time.monotonic()
+ hits = rag.search(query, collection_id, 10)
+ search_seconds.append(time.monotonic() - begun)
+ require(len(hits) == 10, len(hits))
+ require(
+ len({hit.chunk_id for hit in hits}) == 10,
+ "Duplicate search results",
+ )
+ require(
+ all(math.isfinite(hit.distance) for hit in hits),
+ "Non-finite search score",
+ )
+ if verify_known and number == 0:
+ require(
+ hits[0].source_id == known_document
+ and hits[0].text == f"{known_document} chunk 0",
+ "Known vector did not rank first",
+ )
+ metrics["known_query_top1_matches"] += 1
+ metrics["searches_completed"] += 1
+ metrics["cold_search_seconds"] = round(search_seconds[0], 4)
+ metrics["search_seconds"] = _seconds(
+ search_seconds[1:] or search_seconds
+ )
+
+ metrics["elapsed_seconds"] = round(time.monotonic() - started, 1)
+ metrics["peak_rss_mib"] = round(peak_rss_mib(), 2)
+ for limit, value, label in (
+ (args.max_write_seconds, metrics["write_seconds"]["p50"], "write"),
+ (args.max_search_seconds, metrics["search_seconds"]["p50"], "search"),
+ (args.max_rss_mib, metrics["peak_rss_mib"], "peak RSS"),
+ ):
+ require(limit is None or value <= limit, (label, value, limit))
+ return metrics
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--vectors", type=int, default=20000)
+ parser.add_argument("--dimension", type=int, default=128)
+ parser.add_argument(
+ "--index-type", choices=("flat", "hnsw"), default="flat"
+ )
+ # 657 chunks per document matches the 1024-dimension library in #7628.
+ parser.add_argument("--chunks-per-document", type=int, default=657)
+ parser.add_argument("--writes", type=int, default=6)
+ parser.add_argument("--searches", type=int, default=8)
+ parser.add_argument("--max-write-seconds", type=float)
+ parser.add_argument("--max-search-seconds", type=float)
+ parser.add_argument("--max-rss-mib", type=float)
+ parser.add_argument("--verify-known-query", action="store_true")
+ parser.add_argument("--report", type=Path, required=True)
+ args = parser.parse_args()
+ if (
+ not 1000 <= args.vectors <= 10_000_000
+ or not 1 <= args.dimension <= 4096
+ or not 1 <= args.chunks_per_document <= 10_000
+ or args.writes < 1
+ or args.searches < 1
+ ):
+ parser.error("Invalid benchmark size")
+ with tempfile.TemporaryDirectory(prefix="ldr-vector-scale-") as temporary:
+ metrics = run(args, Path(temporary))
+ args.report.parent.mkdir(parents=True, exist_ok=True)
+ args.report.write_text(
+ json.dumps(metrics, indent=2) + "\n", encoding="utf-8"
+ )
+ print(json.dumps(metrics), flush=True)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/performance/test_resources.py b/tests/performance/test_resources.py
new file mode 100644
index 0000000000..83bbdb62c7
--- /dev/null
+++ b/tests/performance/test_resources.py
@@ -0,0 +1,65 @@
+"""Keep child memory gates independent from the spawning test runner."""
+# allow: no-sut-import — tests the shared workload memory reader in owned children.
+
+from pathlib import Path
+import subprocess
+import sys
+from types import SimpleNamespace
+
+import pytest
+
+from tests.performance._shared import resources
+
+
+ sys.platform != "linux" or not Path("/proc/self/status").is_file(),
+ reason="Linux procfs memory accounting",
+)
+def test_child_peak_excludes_parent_and_detects_own_allocation():
+ child = """
+from tests.performance._shared.resources import peak_rss_mib
+before = peak_rss_mib()
+assert before < 24, before
+payload = bytearray(32 * 1024 * 1024)
+payload[::4096] = b'x' * len(payload[::4096])
+after = peak_rss_mib()
+assert after >= before + 24, (before, after)
+"""
+ parent = """
+import subprocess
+import sys
+payload = bytearray(32 * 1024 * 1024)
+payload[::4096] = b'x' * len(payload[::4096])
+subprocess.run([sys.executable, '-c', sys.argv[1]], check=True)
+"""
+ result = subprocess.run(
+ [sys.executable, "-c", parent, child],
+ capture_output=True,
+ text=True,
+ timeout=30,
+ )
+ assert result.returncode == 0, result.stdout + result.stderr
+
+
+ "platform,peak", [("linux", 2048), ("darwin", 2 * 1024 * 1024)]
+)
+def test_fallback_keeps_platform_specific_rss_units(
+ monkeypatch, platform, peak
+):
+ def unavailable():
+ raise OSError("procfs unavailable")
+
+ monkeypatch.setattr(resources, "sys", SimpleNamespace(platform=platform))
+ monkeypatch.setattr(
+ resources, "Path", lambda _: SimpleNamespace(read_text=unavailable)
+ )
+ monkeypatch.setitem(
+ sys.modules,
+ "resource",
+ SimpleNamespace(
+ RUSAGE_SELF=0,
+ getrusage=lambda _: SimpleNamespace(ru_maxrss=peak),
+ ),
+ )
+ assert resources.peak_rss_mib() == 2
diff --git a/tests/research_library/services/test_rag_vector_lifecycle.py b/tests/research_library/services/test_rag_vector_lifecycle.py
index 4de44ade53..55bd2ff987 100644
--- a/tests/research_library/services/test_rag_vector_lifecycle.py
+++ b/tests/research_library/services/test_rag_vector_lifecycle.py
@@ -261,6 +261,35 @@ def test_cleanup_repairs_orphans_from_older_writers_before_id_reuse(
assert len(rows) == 1
+def test_writes_skip_the_full_orphan_scan_between_intervals(
+ tmp_path, shared_world, monkeypatch
+):
+ from local_deep_research.vector_stores.implementations.segmented_faiss import (
+ SegmentedFaissStore,
+ )
+
+ index = make_index(tmp_path)
+ scans = []
+ original = SegmentedFaissStore.iter_id_batches
+
+ def counting(store):
+ scans.append(True)
+ return original(store)
+
+ monkeypatch.setattr(SegmentedFaissStore, "iter_id_batches", counting)
+ for number in range(4):
+ index.index(
+ source_type="document",
+ source_id=f"document-{number}",
+ chunks=[ChunkInput(text=f"chunk {number}")],
+ )
+ # An empty index needs no scan and the second write records the first
+ # one; later writes only compare IDs against the highest chunk row.
+ assert len(scans) == 1
+ rows = {row.id for row in shared_world.query(DocumentChunk)}
+ assert set(original(index._store).__next__()) == rows
+
+
@pytest.mark.parametrize("delete_parent", [False, True])
def test_reconciliation_collects_deleted_or_unlinked_documents(
shared_world, seeded_collection, delete_parent
diff --git a/tests/vector_stores/test_segmented_faiss.py b/tests/vector_stores/test_segmented_faiss.py
index f0110204f1..cc006d99c0 100644
--- a/tests/vector_stores/test_segmented_faiss.py
+++ b/tests/vector_stores/test_segmented_faiss.py
@@ -412,3 +412,206 @@ def test_index_storage_size_skips_files_retired_mid_scan(tmp_path, monkeypatch):
assert segment_files.index_storage_size(root) == 10
root.unlink()
assert segment_files.index_storage_size(root) is None
+
+
+def fresh_verification(monkeypatch):
+ """Isolate the process-wide verified-file record from other tests."""
+ monkeypatch.setattr(module, "_VERIFIED", module._VerifiedFiles(1024))
+
+
+def _count_digests(monkeypatch):
+ digested = []
+ original = module._digest
+
+ def counting(path):
+ digested.append(Path(path).suffix)
+ return original(path)
+
+ monkeypatch.setattr(module, "_digest", counting)
+ return digested
+
+
+def _whole_second_times():
+ """Emulate a filesystem storing whole-second timestamps (NFSv2/3,
+ CIFS, VFAT, HFS+): every recorded time lands on a whole second."""
+
+ original = module._file_signature
+
+ def coarse(path):
+ signature = original(path)
+ if signature is None:
+ return None
+ return signature[:3] + tuple(
+ value - value % 1_000_000_000 for value in signature[3:]
+ )
+
+ return coarse
+
+
+def _one_coarse_tick():
+ """Emulate one whole-second tick that spans both a verification and a
+ rewrite inside it: the mount reports the same recorded fields for both."""
+
+ original = module._file_signature
+ remembered = {}
+
+ def coarse(path):
+ if "signature" not in remembered:
+ signature = original(path)
+ if signature is not None:
+ remembered["signature"] = signature[:3] + tuple(
+ value - value % 1_000_000_000 for value in signature[3:]
+ )
+ return remembered.get("signature")
+
+ return coarse
+
+
+def test_repeated_search_hashes_each_segment_once(
+ binding, monkeypatch, fresh_verification
+):
+ monkeypatch.setattr(module, "SEGMENT_VECTORS", 4)
+ store = module.SegmentedFaissStore.create(**binding)
+ store.apply(add_ids=list(range(1, 13)), add_vectors=vectors(12))
+ reader = module.SegmentedFaissStore.load(**binding)
+ module._VERIFIED.clear()
+ digested = _count_digests(monkeypatch)
+
+ first = reader.search(vectors(1, 3)[0], 5)
+ assert sorted(digested) == [".faiss"] * 3 + [".ids"] * 3
+ digested.clear()
+ assert reader.search(vectors(1, 3)[0], 5) == first
+ assert digested == []
+
+
[email protected]("suffix", [".faiss", ".ids"])
[email protected]("restore_mtime", [False, True])
+def test_rewriting_a_verified_segment_is_detected(
+ binding, fresh_verification, suffix, restore_mtime
+):
+ store = module.SegmentedFaissStore.create(**binding)
+ store.apply(add_ids=[1, 2], add_vectors=vectors(2))
+ store.search(vectors(1)[0], 1)
+ entry = state(binding["path"])[0]
+ path = (
+ module.segment_directory(binding["path"]) / entry["file"]
+ ).with_suffix(suffix)
+ before = os.stat(path)
+ data = bytearray(path.read_bytes())
+ data[-1] ^= 0xFF # Same size, same inode.
+ time.sleep(0.01)
+ path.write_bytes(bytes(data))
+ if restore_mtime:
+ # utime can restore the modification time but not the change time.
+ os.utime(path, ns=(before.st_atime_ns, before.st_mtime_ns))
+ with pytest.raises(ValueError, match="checksum"):
+ store.search(vectors(1)[0], 1)
+
+
+def test_coarse_stored_times_re_hash_segments_on_every_read(
+ binding, monkeypatch, fresh_verification
+):
+ # Whole-second stored times cannot show a same-tick rewrite in the
+ # recorded fields, so segments on such mounts forfeit the verification
+ # cache and are hashed on every read (safe direction: pre-caching cost).
+ monkeypatch.setattr(module, "SEGMENT_VECTORS", 4)
+ store = module.SegmentedFaissStore.create(**binding)
+ store.apply(add_ids=list(range(1, 13)), add_vectors=vectors(12))
+ reader = module.SegmentedFaissStore.load(**binding)
+ module._VERIFIED.clear()
+ digested = _count_digests(monkeypatch)
+
+ first = reader.search(vectors(1, 3)[0], 5)
+ assert sorted(digested) == [".faiss"] * 3 + [".ids"] * 3
+ digested.clear()
+ assert reader.search(vectors(1, 3)[0], 5) == first
+ assert digested == [] # Fine-grained times: the cache serves the read.
+
+ monkeypatch.setattr(module, "_file_signature", _whole_second_times())
+ for _ in range(2):
+ digested.clear()
+ assert reader.search(vectors(1, 3)[0], 5) == first
+ assert sorted(digested) == [".faiss"] * 3 + [".ids"] * 3
+
+
+def test_same_tick_rewrite_with_coarse_times_is_re_hashed(
+ binding, fresh_verification, monkeypatch
+):
+ # The worst case: on a whole-second-timestamp filesystem, a same-size
+ # rewrite and a utime restore inside one tick leave every recorded
+ # field unchanged. The cache must not serve that read unhashed.
+ store = module.SegmentedFaissStore.create(**binding)
+ store.apply(add_ids=[1, 2], add_vectors=vectors(2))
+ entry = state(binding["path"])[0]
+ monkeypatch.setattr(module, "_file_signature", _one_coarse_tick())
+ store.search(vectors(1)[0], 1) # Verifies within the coarse tick.
+ needle = np.asarray(store.reconstruct(1), dtype="<f4").tobytes()
+ path = (
+ module.segment_directory(binding["path"]) / entry["file"]
+ ).with_suffix(".faiss")
+ data = bytearray(path.read_bytes())
+ # Corrupt a stored vector value in place: same size, same inode, ids
+ # and bounds untouched, so the hash is the only thing that can notice.
+ # (Corrupting the id map instead would be caught by the membership
+ # cross-check even without the hash.)
+ offset = bytes(data).find(needle)
+ assert offset > 0
+ data[offset] ^= 0xFF
+ path.write_bytes(bytes(data))
+ with pytest.raises(ValueError, match="checksum"):
+ store.search(vectors(1)[0], 1)
+
+
+def test_verified_files_are_bounded():
+ record = module._VerifiedFiles(limit=2)
+ for name in ("a", "b", "c"):
+ record.remember(Path(name), "digest", ("signature",))
+ assert [key[0] for key in record._entries] == ["b", "c"]
+
+
+def test_orphan_scan_runs_periodically_or_for_reusable_ids(
+ binding, monkeypatch
+):
+ monkeypatch.setattr(module, "ORPHAN_SCAN_INTERVAL", 3)
+ store = module.SegmentedFaissStore.create(**binding)
+ assert not store.orphan_scan_due(None) # Nothing indexed yet.
+ store.apply(add_ids=[1], add_vectors=vectors(1))
+ assert store.orphan_scan_due(1) # No scan has been recorded.
+ store.apply(add_ids=[2], add_vectors=vectors(1, 2), orphan_scanned=True)
+ assert not store.orphan_scan_due(2)
+ # A vector above the highest chunk row could collide with a new row.
+ assert store.orphan_scan_due(1)
+ assert store.orphan_scan_due(None)
+ for value in (3, 4):
+ assert not store.orphan_scan_due(value - 1)
+ store.apply(add_ids=[value], add_vectors=vectors(1, value))
+ assert not store.orphan_scan_due(4)
+ store.apply(add_ids=[5], add_vectors=vectors(1, 5))
+ assert store.orphan_scan_due(5) # The interval has elapsed.
+ # Republishing an unchanged generation keeps its scan record.
+ store.apply(add_ids=[6], add_vectors=vectors(1, 6), orphan_scanned=True)
+ store.persist(binding["path"])
+ assert not store.orphan_scan_due(6)
+
+
+def test_small_rewritten_segments_merge_with_their_neighbour(
+ binding, monkeypatch
+):
+ monkeypatch.setattr(module, "SEGMENT_VECTORS", 4)
+ store = module.SegmentedFaissStore.create(**binding)
+ original = vectors(12)
+ store.apply(add_ids=list(range(1, 13)), add_vectors=original)
+ assert [entry["count"] for entry in state(binding["path"])] == [4, 4, 4]
+
+ store.apply(add_ids=[], add_vectors=None, remove_ids=[1, 2, 3, 5, 6])
+ assert [entry["count"] for entry in state(binding["path"])] == [3, 4]
+ assert sorted(store.live_ids()) == [4, 7, 8, 9, 10, 11, 12]
+ for value in (4, 7, 9):
+ np.testing.assert_allclose(
+ store.reconstruct(value), original[value - 1], rtol=1e-6
+ )
+ # Untouched full segments are never rewritten to merge.
+ untouched = state(binding["path"])[-1]["file"]
+ store.apply(add_ids=[], add_vectors=None, remove_ids=[4])
+ assert state(binding["path"])[-1]["file"] == untouched