~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
perf(rag): hash segments once and scan orphans periodically for large FAISS collections (#7867)
LearningCircuit1ff6ee8bc2b3
diff --git a/.github/workflows/rag-vector-scale.yml b/.github/workflows/rag-vector-scale.ymlnew file mode 100644index 0000000000..8dc71ff95e--- /dev/null+++ b/.github/workflows/rag-vector-scale.yml@@ -0,0 +1,80 @@+name: RAG Vector Scale++# Manual latency benchmark for large segmented FAISS collections (#7810).+# Vectors are synthetic and go straight into the store: no PDFs, no model.+on:+ workflow_dispatch:+ inputs:+ vectors:+ description: Vectors in the benchmark collection+ default: '1000000'+ required: true+ dimension:+ description: Embedding dimension+ default: '1024'+ required: true++permissions: {}++jobs:+ rag-vector-scale:+ name: Vector scale (${{ matrix.index-type }})+ runs-on: ubuntu-latest+ timeout-minutes: 150+ permissions:+ contents: read+ strategy:+ fail-fast: false+ matrix:+ index-type: [flat, hnsw]+ env:+ OMP_NUM_THREADS: '2'+ OPENBLAS_NUM_THREADS: '2'+ HF_HUB_OFFLINE: '1'+ TRANSFORMERS_OFFLINE: '1'+ RAG_INDEX_TYPE: ${{ matrix.index-type }}+ RAG_VECTORS: ${{ inputs.vectors }}+ RAG_DIMENSION: ${{ inputs.dimension }}++ steps:+ - name: Harden runner+ uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1+ with:+ egress-policy: audit+ - name: Checkout+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ with:+ persist-credentials: false+ - name: Set up PDM+ uses: pdm-project/setup-pdm@544d7237314ee09c256785bd360f6b30add38b37 # v4.6+ with:+ python-version: '3.12'+ version: 2.26.2+ cache: true+ cache-restore-exact-match: true+ - name: Install SQLCipher+ run: |+ sudo apt-get update+ sudo apt-get install -y libsqlcipher-dev+ - name: Install dependencies+ run: pdm sync -d --clean+ - name: Benchmark writes and searches+ shell: bash+ run: |+ [[ "$RAG_VECTORS" =~ ^[0-9]+$ && "$RAG_DIMENSION" =~ ^[0-9]+$ ]] || {+ echo "vectors and dimension must be integers" >&2+ exit 1+ }+ mkdir -p test-results+ pdm run python scripts/ci/rag_vector_scale_benchmark.py \+ --vectors "$RAG_VECTORS" --dimension "$RAG_DIMENSION" \+ --index-type "$RAG_INDEX_TYPE" \+ --report "test-results/rag-vector-scale-$RAG_INDEX_TYPE.json" \+ 2>&1 | tee "test-results/rag-vector-scale-$RAG_INDEX_TYPE.log"+ - name: Upload benchmark report+ if: always()+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2+ with:+ name: rag-vector-scale-${{ matrix.index-type }}+ path: test-results/rag-vector-scale-*+ retention-days: 30diff --git a/changelog.d/7810.bugfix.md b/changelog.d/7810.bugfix.mdnew file mode 100644index 0000000000..47f68c61b5--- /dev/null+++ b/changelog.d/7810.bugfix.md@@ -0,0 +1 @@+Speed up large RAG collections. Each vector segment is hashed once per process and re-hashed whenever its device, inode, size, modification time or change time differs from the values seen at that verification, so searches no longer re-hash the whole index on every query. Those fields move only with the filesystem's timestamp resolution, so a segment whose stored times are whole seconds (coarse-timestamp filesystems such as NFSv2/3, CIFS/SMB, VFAT, HFS+, some FUSE and overlay mounts) is hashed on every read instead of cached, closing the same-tick rewrite gap at pre-caching cost on those mounts. Writes run the full orphan-vector scan at the latest every 65th write — 64 writes raise the scan counter to its threshold and the next write scans — instead of on every write, and in between run it only when the index holds vector IDs above the highest chunk row, the only IDs a new chunk row could reuse. Segments shrunk by deletions merge with their neighbours. Add a manual large-collection latency benchmark.diff --git a/docs/ci/workflow-status.md b/docs/ci/workflow-status.mdindex 060dc7dc58..3bbdc9b161 100644--- a/docs/ci/workflow-status.md+++ b/docs/ci/workflow-status.md@@ -196,6 +196,11 @@ the table above.scanner binary and complete scan results before artifact and SARIF uploads.Its status is reported in the calling workflows.+- `rag-vector-scale.yml` — manual large-collection latency benchmark for+ segmented FAISS (#7810). Flat and HNSW shards build a synthetic collection+ (1,000,000 vectors of 1,024 dimensions by default) without PDFs or an+ embedding model, and report per-document write time, search time and peak+ RSS. It is not called by `release.yml` and never runs on pull requests.- `rag-index-scale.yml` — required synthetic-PDF scale gate called by`release.yml`, with manual dispatch available. Separate flat and HNSWshards check bounded index writes, memory, search, rollback, deletion,diff --git a/docs/library-and-rag.md b/docs/library-and-rag.mdindex b0ced18b2a..767057a6f1 100644--- a/docs/library-and-rag.md+++ b/docs/library-and-rag.md@@ -503,12 +503,20 @@ Uses OpenAI's embedding API.FAISS indexes use segments of at most 4,096 vectors. Appends rewrite thepartially filled segment and write new segments; they do not rewrite the wholecollection. Searches visit segments sequentially and merge their best matches.-Searches also re-verify (hash) every segment file against the authenticated-manifest on each query. Full chunk records are loaded only for the requested-results after their IDs are selected. This bounds the vector working set, at the-cost of reading multiple segments for a large collection. Embedding models, PDF-extraction, individual document preparation, and reconciliation metadata still-contribute to memory use.+Each segment file is verified (hashed) against the authenticated manifest the+first time the server process reads it, and again whenever the file changes:+a later read compares the file's device, inode, size, modification time and+change time with those seen at verification and re-hashes on any difference.+Since those fields move only with the filesystem's timestamp resolution, a+segment whose stored times are whole seconds (coarse-timestamp filesystems+such as NFSv2/3, CIFS/SMB, VFAT, HFS+, some FUSE and overlay mounts) is hashed on every+read instead of cached. Full chunk records are loaded only+for the requested results after their IDs are selected. This bounds the vector+working set, at the cost of reading every segment for each search of a large+collection. A deletion that leaves a segment less than half full merges it with+a neighbouring segment when both fit. Embedding models, PDF extraction,+individual document preparation, and reconciliation metadata still contribute+to memory use.Existing native `.faiss` files remain readable and convert on their next writewithout re-embedding. Conversion reads the old payload through read-only memory@@ -525,7 +533,13 @@ indexing replaces only that configuration's chunks, and document deletion purgesevery retained configuration. A successful full collection rebuild replaces theprevious published configurations. Writes and reconciliation removeunowned vectors; surviving chunks from partially indexed documents are retained-and reported separately. A rollback checkpoint preserves the previous vector+and reported separately. Reconciliation always checks every vector. A write+checks every vector at the latest every 65th write — 64 writes raise the scan+counter to its threshold and the next write scans — and sooner on the first+write after an upgrade (whose index has no scan record) or when the index+holds vector IDs above the highest chunk row, the only IDs a new chunk could+reuse; a write that converts a legacy index checks every vector as part of the+conversion, and a fresh empty index has nothing to check on its first write. A rollback checkpoint preserves the previous vectorgeneration until the database commits. Crash recovery restores a checkpointonly when it matches the checksum in the encrypted database.@@ -573,6 +587,27 @@ embedding quality or the memory requirements of a real embedding model. Thelarge shards run on releases or manual workflow requests; focused lifecycleregressions and the small PDF generator test run in the regular pytest suite.+### Large-collection latency benchmark++The synthetic PDF test bounds memory but not time. A separate benchmark builds+a large collection of synthetic vectors directly in the store, with real+SQLCipher storage and no PDFs or embedding model, then times per-document+writes and searches:++```bash+OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 pdm run python scripts/ci/rag_vector_scale_benchmark.py \+ --vectors 300000 --dimension 1024 --index-type flat \+ --report /tmp/rag-vector-scale.json+```++The report includes the first write after the bulk build (which runs the+periodic full orphan scan), median and maximum steady write time, the cold+first search, median and maximum warm search time, and peak RSS. Optional+`--max-write-seconds`, `--max-search-seconds` and `--max-rss-mib` arguments+turn these into limits. The default of 657 chunks per document matches the+1,024-dimension library measured in #7628. The manual **RAG Vector Scale**+workflow runs it with 1,000,000 vectors for flat and HNSW indexes.+---## See Alsodiff --git a/scripts/ci/rag_vector_scale_benchmark.py b/scripts/ci/rag_vector_scale_benchmark.pynew file mode 100644index 0000000000..33e78d70ea--- /dev/null+++ b/scripts/ci/rag_vector_scale_benchmark.py@@ -0,0 +1,11 @@+"""Run the offline segmented FAISS latency benchmark from a source checkout."""++from pathlib import Path+import sys++sys.path.insert(0, str(Path(__file__).resolve().parents[2]))+from tests.performance.rag_vector_scale_benchmark import main # noqa: E402+++if __name__ == "__main__":+ main()diff --git a/src/local_deep_research/vector_stores/facade.py b/src/local_deep_research/vector_stores/facade.pyindex 3941ab1942..230afab384 100644--- a/src/local_deep_research/vector_stores/facade.py+++ b/src/local_deep_research/vector_stores/facade.py@@ -37,7 +37,7 @@ from typing import (import numpy as npfrom langchain_core.embeddings import Embeddingsfrom loguru import logger-from sqlalchemy import event, exists, or_, select+from sqlalchemy import event, exists, func, or_, selectfrom sqlalchemy.orm import Sessionfrom ..database.models.library import (@@ -264,11 +264,12 @@ class VectorIndex:owner,)- def _apply(self, session: Session, **kwargs):+ def _apply(self, session: Session, *, orphan_scanned=False, **kwargs):"""Settle vector checkpoints with the caller's outer DB transaction."""from .implementations.segmented_faiss import SegmentedFaissStoreif isinstance(self._store, SegmentedFaissStore):+ kwargs["orphan_scanned"] = orphan_scannedif session.in_nested_transaction():raise RuntimeError("Vector writes require an outer database transaction"@@ -297,13 +298,24 @@ class VectorIndex:),)- def _prepare_cleanup(self, session: Session) -> tuple[list[int], list[int]]:+ def _prepare_cleanup(+ self, session: Session, *, force: bool = False+ ) -> tuple[list[int], list[int], bool]:"""Discover real orphans before allocating IDs that a rollback reused.Partial documents remain owned: completeness is a status concern, nota garbage-collection criterion. Legacy ownership is recovered onlyfrom membership in this physical index. Queries use bounded IN lists.++ A store that tracks its scans (segmented FAISS) is fully scanned only+ periodically, or when it holds IDs above the highest chunk row; the+ last value says whether this call scanned."""+ due = getattr(self._store, "orphan_scan_due", None)+ if not force and due is not None:+ highest = session.scalar(select(func.max(DocumentChunk.id)))+ if not due(highest):+ return [], [], Falseorphan_vectors: list[int] = []orphan_rows: list[int] = []for ids in self._store.iter_id_batches():@@ -334,15 +346,19 @@ class VectorIndex:)orphan_vectors.extend(set(ids) - valid)orphan_rows.extend(owned - valid)- return orphan_vectors, orphan_rows+ return orphan_vectors, orphan_rows, Truedef compact_orphans(self, session: Optional[Session] = None) -> int:"""Remove only unowned vectors; retain every valid partial document."""with self._db(session) as (sess, owns):- vectors, rows = self._prepare_cleanup(sess)+ vectors, rows, scanned = self._prepare_cleanup(sess, force=True)if vectors:self._apply(- sess, add_ids=[], add_vectors=None, remove_ids=vectors+ sess,+ orphan_scanned=scanned,+ add_ids=[],+ add_vectors=None,+ remove_ids=vectors,)self._finalize_db(sess, owns, rows, "compact orphan vectors")return len(vectors)@@ -376,7 +392,7 @@ class VectorIndex:sid = str(source_id) if source_id is not None else Nonewith self._db(session) as (sess, owns):- orphan_vectors, orphan_rows = self._prepare_cleanup(sess)+ orphan_vectors, orphan_rows, scanned = self._prepare_cleanup(sess)prior_ids: List[int] = []if replace and sid is not None:prior_ids = list(@@ -400,6 +416,7 @@ class VectorIndex:try:stats = self._apply(sess,+ orphan_scanned=scanned,add_ids=new_ids,add_vectors=vectors,remove_ids=prior_ids + orphan_vectors,@@ -468,7 +485,7 @@ class VectorIndex:sid = str(source_id) if source_id is not None else Nonewith self._db(session) as (sess, owns):- orphan_vectors, orphan_rows = self._prepare_cleanup(sess)+ orphan_vectors, orphan_rows, scanned = self._prepare_cleanup(sess)prior_ids: List[int] = []if replace and sid is not None:# Scope the prior-chunk lookup to THIS store's embedding-model@@ -505,6 +522,7 @@ class VectorIndex:try:stats = self._apply(sess,+ orphan_scanned=scanned,add_ids=new_ids,add_vectors=vectors,remove_ids=prior_ids + orphan_vectors,@@ -738,7 +756,7 @@ class VectorIndex:if not source_ids:return DeleteStats(removed=0, rows_deleted=0)with self._db(session) as (sess, owns):- orphan_vectors, orphan_rows = self._prepare_cleanup(sess)+ orphan_vectors, orphan_rows, scanned = self._prepare_cleanup(sess)ids: list[int] = []for start in range(0, len(source_ids), 500):ids.extend(@@ -758,6 +776,7 @@ class VectorIndex:try:stats = self._apply(sess,+ orphan_scanned=scanned,add_ids=[],add_vectors=None,remove_ids=removed,diff --git a/src/local_deep_research/vector_stores/implementations/segmented_faiss.py b/src/local_deep_research/vector_stores/implementations/segmented_faiss.pyindex 401f5b782d..fb64b2176c 100644--- a/src/local_deep_research/vector_stores/implementations/segmented_faiss.py+++ b/src/local_deep_research/vector_stores/implementations/segmented_faiss.py@@ -10,6 +10,20 @@ Like the existing FAISS store, coordination is per-process. All readers reopenthe manifest under the injected index lock, so a successful writer can retiresuperseded files without invalidating a reader's snapshot.+Each segment file is hashed against the manifest the first time this process+reads it. Later reads skip the hash while the file keeps the device, inode,+size, modification time and change time seen at that verification. Writing+to a file changes its change time, which ``utime`` cannot set back, so a+rewrite is hashed again before use whenever it moves one of those fields.+A replacement usually changes the inode, but the inode can be reused, so+both a same-size rewrite whose modification time is restored within one+filesystem timestamp tick and a same-size replacement that reuses the inode+and restores the modification time within that tick move none of the five+fields; ticks are microseconds on typical local filesystems but whole+seconds on coarse-timestamp mounts (NFSv2/3, CIFS/SMB, VFAT, HFS+, some FUSE+and overlay mounts), so a segment whose stored times are whole seconds+forfeits the cache and is hashed on every read.+Crash safety additionally assumes that every writer of a publishedgeneration, and every non-nested ``verify_file()`` / ``recover_checkpoint``caller, holds the collection transaction lock@@ -27,6 +41,7 @@ the transaction lock, stage cleanup under the rebuild lock, whilefrom __future__ import annotations+from collections import OrderedDictfrom contextlib import contextmanager, nullcontextimport hashlibimport json@@ -35,6 +50,7 @@ from pathlib import Pathimport reimport shutilimport tempfile+import threadingimport uuidimport faiss@@ -61,6 +77,12 @@ SEGMENT_VECTORS = 4096_NAME = re.compile(r"[0-9a-f]{32}\.faiss\Z")_HASH = re.compile(r"[0-9a-f]{64}\Z")_MAX_MANIFEST_BYTES = 16 * 1024 * 1024+# A full orphan scan reads every vector ID; between scans a write only checks+# for IDs above the highest chunk row, the only orphans a new row could reuse.+ORPHAN_SCAN_INTERVAL = 64+_SCAN_COUNTER = "writes_since_orphan_scan"+# Verified segment files remembered per process (about 200 bytes each).+_VERIFIED_FILES = 65536def is_segmented(path: Path) -> bool:@@ -114,6 +136,93 @@ def _validate_entries(entries) -> None:raise ValueError("Invalid vector segment bounds")+def _file_signature(path: Path):+ """Identity of a file's current contents, or None if it cannot be read."""+ try:+ info = os.lstat(path)+ except OSError:+ return None+ return (+ info.st_dev,+ info.st_ino,+ info.st_size,+ info.st_mtime_ns,+ info.st_ctime_ns,+ )+++_SECOND_NS = 1_000_000_000+++def _coarse_timestamps(mtime_ns: int, ctime_ns: int) -> bool:+ """Whether stored times look whole-second, hiding same-tick rewrites.++ Filesystems that store only whole-second timestamps (NFSv2/3, CIFS/SMB,+ VFAT, HFS+, some FUSE and overlay mounts) report every time on a whole+ second, so a same-size rewrite whose modification time is restored+ within the same second leaves every recorded field unchanged and the+ verification record cannot tell it from an unchanged file. On such+ mounts the record is worthless: a whole-second time in EITHER field+ (a rewrite cannot forge the change time finer than the mount stores+ it) forfeits the cache, at the cost of one extra hash per read.+ """+ return mtime_ns % _SECOND_NS == 0 or ctime_ns % _SECOND_NS == 0+++class _VerifiedFiles:+ """Bounded record of files whose digest matched, keyed by digest."""++ def __init__(self, limit):+ self.limit = limit+ self._entries = OrderedDict()+ self._lock = threading.Lock()++ def matches(self, path, digest):+ signature = _file_signature(path)+ with self._lock:+ known = self._entries.get((str(path), digest))+ if known is None or signature is None or known != signature:+ return False+ self._entries.move_to_end((str(path), digest))+ return True++ def remember(self, path, digest, signature):+ with self._lock:+ self._entries[(str(path), digest)] = signature+ self._entries.move_to_end((str(path), digest))+ while len(self._entries) > self.limit:+ self._entries.popitem(last=False)++ def clear(self):+ with self._lock:+ self._entries.clear()+++_VERIFIED = _VerifiedFiles(_VERIFIED_FILES)+++def _verified_digest(path: Path, expected: str) -> bool:+ """Whether ``path`` holds ``expected``, hashing only unseen contents.++ Contents whose stored times look whole-second are never cached — see+ ``_coarse_timestamps`` — so they are hashed on every read.+ """+ path = path.absolute()+ before = _file_signature(path)+ cacheable = before is not None and not _coarse_timestamps(+ before[3], before[4]+ )+ if cacheable and _VERIFIED.matches(path, expected):+ return True+ if _digest(path) != expected:+ return False+ # Remember only cacheable contents that did not change while being+ # hashed.+ if cacheable and before == _file_signature(path):+ _VERIFIED.remember(path, expected, before)+ return True++def _sync_directory(path: Path) -> None:try:descriptor = os.open(path, os.O_RDONLY)@@ -214,8 +323,12 @@ class SegmentedFaissStore(BaseVectorStore):)def _state(self, *, verify=True):+ manifest = self._manifest(verify=verify)+ return manifest if manifest is None else manifest["segments"]++ def _manifest(self, *, verify=True):if not self._path.exists():- return []+ return {"segments": []}if verify and self._verify is not None:verified, reason = self._verify(self._path)if not verified:@@ -249,7 +362,7 @@ class SegmentedFaissStore(BaseVectorStore):if not isinstance(entries, list):raise ValueError("Invalid vector segment list")_validate_entries(entries)- return entries+ return statedef begin_transaction(self, *, first=True):"""Retain the prior root and its segments until the owning DB settles."""@@ -428,7 +541,7 @@ class SegmentedFaissStore(BaseVectorStore):"Missing or symlinked vector segment")expected = entry["ids_sha256" if ids else "sha256"]- if _digest(path) != expected:+ if not _verified_digest(path, expected):raise VectorStoreIntegrityError("Vector segment checksum mismatch")return path@@ -491,7 +604,7 @@ class SegmentedFaissStore(BaseVectorStore):"max_id": max(ids),}- def _publish(self, entries):+ def _publish(self, entries, scan_count=None):# Publish only a generation every later read accepts: this is the# exact validation _state() applies, run BEFORE the manifest is# written and authenticated.@@ -507,6 +620,8 @@ class SegmentedFaissStore(BaseVectorStore):"normalize": self.normalize,"segments": entries,}+ if scan_count is not None:+ manifest[_SCAN_COUNTER] = scan_countdata = json.dumps(manifest, separators=(",", ":")).encode()if len(data) > _MAX_MANIFEST_BYTES:raise ValueError("Vector index manifest exceeds its metadata limit")@@ -604,9 +719,69 @@ class SegmentedFaissStore(BaseVectorStore):"A segmented index must be persisted at its bound path")with self._guard():- entries = self._state()- if entries is not None:- self._publish(entries)+ manifest = self._manifest()+ if manifest is not None:+ self._publish(manifest["segments"], manifest.get(_SCAN_COUNTER))++ def orphan_scan_due(self, max_chunk_id):+ """Whether the next write must check every vector ID for orphans.++ Between periodic full scans only vectors above the highest chunk row+ matter before a write: their IDs are the only ones a new row can+ receive. A legacy file is fully checked by the write that converts+ it, and a manifest with no scan record yet (an upgraded index) by+ the next write; an index with no segments has nothing to check.+ Thereafter a scan is due at the latest every 65th write: 64 writes+ raise the counter to its threshold and the next write scans.+ """+ with self._guard():+ manifest = self._manifest()+ if manifest is None:+ return True+ entries = manifest["segments"]+ if not entries:+ return False+ count = manifest.get(_SCAN_COUNTER)+ if type(count) is not int or not 0 <= count < ORPHAN_SCAN_INTERVAL:+ return True+ ceiling = -1 if max_chunk_id is None else int(max_chunk_id)+ return any(entry["max_id"] > ceiling for entry in entries)++ def _merge_small(self, entries, changed):+ """Fold a rewritten segment into an adjacent one when both fit.++ A pair merges only when this write produced at least one of its two+ segments, so each write does bounded extra work and old fragmentation+ merges as it is touched: a rewritten small segment may merge into an+ untouched neighbour, rewriting it, while untouched full segments are+ never rewritten.+ """+ merged: list[dict] = []+ for entry in entries:+ if merged:+ previous = merged[-1]+ if (+ (previous["file"] in changed or entry["file"] in changed)+ and min(previous["count"], entry["count"])+ < SEGMENT_VECTORS // 2+ and previous["count"] + entry["count"] <= SEGMENT_VECTORS+ ):+ target = self._read(previous)+ source = self._read(entry)+ ids = self._ids(entry)+ vectors = np.asarray(+ [source._index.reconstruct(value) for value in ids],+ dtype="float32",+ )+ target._index.add_with_ids(+ vectors, np.asarray(ids, dtype="int64")+ )+ merged[-1] = self._write(target)+ changed.add(merged[-1]["file"])+ del target, source, vectors+ continue+ merged.append(entry)+ return merged@contextmanagerdef _mutation(self):@@ -630,7 +805,15 @@ class SegmentedFaissStore(BaseVectorStore):"Deferred retirement of superseded vector segments")- def apply(self, *, add_ids, add_vectors, remove_ids=(), dedup=True):+ def apply(+ self,+ *,+ add_ids,+ add_vectors,+ remove_ids=(),+ dedup=True,+ orphan_scanned=False,+ ):if self._lock is None or self._record is None or self._verify is None:raise RuntimeError("Vector writes require lock and integrity callbacks"@@ -656,6 +839,17 @@ class SegmentedFaissStore(BaseVectorStore):removals = set(map(int, remove_ids)) | set(add_ids)removed = 0with self._mutation() as previous:+ if orphan_scanned:+ scan_count = 0+ else:+ count = None+ if previous is not None and self._path.exists():+ count = self._manifest(verify=False).get(_SCAN_COUNTER)+ scan_count = (+ count + 1+ if type(count) is int and 0 <= count < ORPHAN_SCAN_INTERVAL+ else ORPHAN_SCAN_INTERVAL+ )if previous is None:# Legacy conversion. The id map is normalized with the same# discipline as the add path before anything is written, so a@@ -746,6 +940,7 @@ class SegmentedFaissStore(BaseVectorStore):removed = len(removed_ids)else:entries = []+ changed = set()for entry in previous:candidates = {value@@ -768,7 +963,9 @@ class SegmentedFaissStore(BaseVectorStore):store._rebuild_dropping(actual)if store.count():entries.append(self._write(store))+ changed.add(entries[-1]["file"])del store+ entries = self._merge_small(entries, changed)position = 0if add_ids and entries and entries[-1]["count"] < SEGMENT_VECTORS:tail = self._read(entries.pop())@@ -789,5 +986,5 @@ class SegmentedFaissStore(BaseVectorStore):entries.append(self._write(store))del storeposition = end- self._publish(entries)+ self._publish(entries, scan_count)return {"added": len(add_ids), "removed": removed}diff --git a/tests/performance/_shared/resources.py b/tests/performance/_shared/resources.pynew file mode 100644index 0000000000..772bb9a877--- /dev/null+++ b/tests/performance/_shared/resources.py@@ -0,0 +1,25 @@+"""Peak memory of the workload executable, in MiB."""++from pathlib import Path+import sys+++def peak_rss_mib():+ if sys.platform == "linux":+ # ru_maxrss can retain the spawning process's pre-exec high-water+ # mark. VmHWM describes the current executable image instead.+ try:+ for line in Path("/proc/self/status").read_text().splitlines():+ if line.startswith("VmHWM:"):+ return int(line.split()[1]) / 1024+ except (OSError, ValueError, IndexError):+ pass+ try:+ import resource+ except ImportError:+ raise RuntimeError(+ "RSS guard requires procfs or resource.getrusage"+ ) from None+ peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss+ # The fallback is conservative on Linux when procfs is unavailable.+ return peak / (1024 * 1024 if sys.platform == "darwin" else 1024)diff --git a/tests/performance/rag_index_workload.py b/tests/performance/rag_index_workload.pyindex 0ae7aa0d8a..c9d8a1b3f7 100644--- a/tests/performance/rag_index_workload.py+++ b/tests/performance/rag_index_workload.py@@ -15,7 +15,6 @@ import hashlibimport jsonimport osfrom pathlib import Path-import resourceimport sysimport tempfileimport time@@ -48,6 +47,7 @@ from local_deep_research.vector_stores.implementations import (sys.path.insert(0, str(Path(__file__).resolve().parents[2]))from tests.fixtures.synthetic_pdf import synthetic_pdf # noqa: E402+from tests.performance._shared.resources import peak_rss_mib # noqa: E402def require(condition, detail):@@ -72,7 +72,13 @@ class EmbeddingManager:self.embeddings = SyntheticEmbeddings(dimension)-def run(args, root):+@contextmanager+def workload_session(root):+ """Yield a session on a real encrypted database that every caller shares.++ Yields ``(session, key, shared_session, db_path)``. Production modules+ that open their own user session receive this one instead.+ """os.environ["LDR_DATA_DIR"] = str(root)logger.disable("local_deep_research")db_path = root / "documents.encrypted.db"@@ -94,6 +100,30 @@ def run(args, root):def shared_session(*_args, **_kwargs):yield session+ with ExitStack() as stack:+ stack.callback(engine.dispose)+ stack.callback(session.close)+ for module in (+ "research_library.services.library_rag_service",+ "vector_stores.facade",+ "security.file_integrity.integrity_manager",+ ):+ stack.enter_context(+ patch(+ f"local_deep_research.{module}.get_user_db_session",+ shared_session,+ )+ )+ stack.enter_context(+ patch(+ "local_deep_research.database.thread_local_session.get_current_thread_session",+ return_value=session,+ )+ )+ yield session, key, shared_session, db_path+++def run(args, root):metrics = {"faiss_bytes_written": 0,"metadata_bytes_written": 0,@@ -123,26 +153,10 @@ def run(args, root):return original_atomic(path, data)started = time.monotonic()- with ExitStack() as stack:- stack.callback(engine.dispose)- stack.callback(session.close)- for module in (- "research_library.services.library_rag_service",- "vector_stores.facade",- "security.file_integrity.integrity_manager",- ):- stack.enter_context(- patch(- f"local_deep_research.{module}.get_user_db_session",- shared_session,- )- )- stack.enter_context(- patch(- "local_deep_research.database.thread_local_session.get_current_thread_session",- return_value=session,- )- )+ with (+ workload_session(root) as (session, key, shared_session, db_path),+ ExitStack() as stack,+ ):stack.enter_context(patch.object(faiss_store, "write_index", write_index))@@ -372,10 +386,7 @@ def run(args, root):)metrics["elapsed_seconds"] = round(time.monotonic() - started, 3)- peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss- metrics["peak_rss_mib"] = round(- peak / (1024 * 1024 if sys.platform == "darwin" else 1024), 2- )+ metrics["peak_rss_mib"] = round(peak_rss_mib(), 2)require(metrics["peak_rss_mib"] <= args.max_rss_mib, metrics)# No generated PDF or plaintext chunk cache may have reached disk.for path in root.rglob("*"):diff --git a/tests/performance/rag_scale_smoke/README.md b/tests/performance/rag_scale_smoke/README.mdnew file mode 100644index 0000000000..3deb93ae29--- /dev/null+++ b/tests/performance/rag_scale_smoke/README.md@@ -0,0 +1,24 @@+This automatic CI shard runs the original synthetic-PDF ingestion workload and+the original vector-scale workload for both flat and HNSW. Each CLI runs in an+owned subprocess, serially, with native thread counts set to one. Reports and+JUnit evidence are retained by the Performance Assertions workflow.++CI defaults: eight 16-page PDFs, 8,192 vectors, 32 dimensions, 64 chunks/write,+66 writes, four searches and a 768 MiB per-child RSS ceiling. The data crosses+the real 4,096-vector segment boundary and the 64-write orphan-scan interval.+The PDF workload's bounded-write, rollback, reconciliation and rebuild checks+remain active. Vector checks assert added/final counts, segment size, unique+finite search results and a known inserted vector ranking first.++```bash+pdm run pytest tests/performance/rag_scale_smoke -q -n 0 \+ --ragscale-profile ci+```++The `local` profile uses 16 PDFs, 12,000 vectors, 128 dimensions, 70 writes and+eight searches. CLI `--ragscale-{profile,documents,pages,vectors,dimension,+chunks-per-document,writes,searches,max-rss-mib}` overrides the matching+`LDR_PERF_RAGSCALE_*` environment value. `LDR_PERF_PROFILE` is the profile+fallback. `LDR_PERF_RAGSCALE_REPORT_DIR` selects persistent report storage.+Sizes must retain the asserted segment/scan coverage. Larger manual/release+workflows keep their existing workloads; this shard supplies bounded PR CI.diff --git a/tests/performance/rag_scale_smoke/__init__.py b/tests/performance/rag_scale_smoke/__init__.pynew file mode 100644index 0000000000..998f3c4aae--- /dev/null+++ b/tests/performance/rag_scale_smoke/__init__.py@@ -0,0 +1 @@+"""Bounded automatic coverage for the original PDF and vector workloads."""diff --git a/tests/performance/rag_scale_smoke/conftest.py b/tests/performance/rag_scale_smoke/conftest.pynew file mode 100644index 0000000000..6a2b873914--- /dev/null+++ b/tests/performance/rag_scale_smoke/conftest.py@@ -0,0 +1,76 @@+import math+import os++import pytest++PROFILES = {+ "ci": dict(+ documents=8,+ pages=16,+ vectors=8192,+ dimension=32,+ chunks_per_document=64,+ writes=66,+ searches=4,+ max_rss_mib=768,+ ),+ "local": dict(+ documents=16,+ pages=16,+ vectors=12000,+ dimension=128,+ chunks_per_document=657,+ writes=70,+ searches=8,+ max_rss_mib=1536,+ ),+}+++def pytest_addoption(parser):+ group = parser.getgroup("original RAG workload profiles")+ for field in (+ "profile",+ "documents",+ "pages",+ "vectors",+ "dimension",+ "chunks-per-document",+ "writes",+ "searches",+ "max-rss-mib",+ ):+ group.addoption(f"--ragscale-{field}", default=None)++[email protected](scope="session")+def rag_scale_settings(request):+ def value(field, default):+ cli = request.config.getoption(f"--ragscale-{field.replace('_', '-')}")+ return (+ cli+ if cli is not None+ else os.environ.get(f"LDR_PERF_RAGSCALE_{field.upper()}") or default+ )++ profile = value("profile", os.environ.get("LDR_PERF_PROFILE") or "ci")+ if profile not in PROFILES:+ raise pytest.UsageError("ragscale profile must be ci or local")+ result = dict(profile=profile)+ for field, default in PROFILES[profile].items():+ try:+ number = int(value(field, default))+ except (ValueError, TypeError):+ raise pytest.UsageError(f"Invalid ragscale {field}") from None+ if not math.isfinite(number) or number <= 0:+ raise pytest.UsageError(f"Invalid ragscale {field}")+ result[field] = number+ if (+ result["documents"] < 4+ or result["dimension"] < 8+ or result["vectors"] < 1000+ ):+ raise pytest.UsageError(+ "ragscale needs >=4 documents, >=8 dimensions and >=1000 vectors"+ )+ return resultdiff --git a/tests/performance/rag_scale_smoke/test_offline.py b/tests/performance/rag_scale_smoke/test_offline.pynew file mode 100644index 0000000000..4f1ea01703--- /dev/null+++ b/tests/performance/rag_scale_smoke/test_offline.py@@ -0,0 +1,116 @@+"""Assert the actual original CLI workloads in owned, serial child processes."""++import json+import os+from pathlib import Path+import subprocess+import sys++import pytest++from local_deep_research.vector_stores.implementations.segmented_faiss import (+ SEGMENT_VECTORS,+ ORPHAN_SCAN_INTERVAL,+)++# The dedicated performance job selects this integration explicitly. Keeping+# it out of the parallel default lane isolates native RSS and global caches.+pytestmark = [pytest.mark.integration, pytest.mark.timeout(360)]++[email protected]("index_type", ["flat", "hnsw"])[email protected]("workload", ["pdf", "vectors"])+def test_original_workload_has_asserted_bounded_ci_coverage(+ tmp_path, rag_scale_settings, index_type, workload+):+ root = Path(__file__).resolve().parents[3]+ settings = rag_scale_settings+ reports = Path(os.environ.get("LDR_PERF_RAGSCALE_REPORT_DIR") or tmp_path)+ reports.mkdir(parents=True, exist_ok=True)+ report = reports / f"{workload}-{index_type}.json"+ script = (+ "rag_index_workload.py"+ if workload == "pdf"+ else "rag_vector_scale_benchmark.py"+ )+ command = [+ sys.executable,+ str(root / "scripts/ci" / script),+ "--dimension",+ str(settings["dimension"]),+ "--index-type",+ index_type,+ "--max-rss-mib",+ str(settings["max_rss_mib"]),+ "--report",+ str(report),+ ]+ if workload == "pdf":+ command.extend(+ [+ "--documents",+ str(settings["documents"]),+ "--pages",+ str(settings["pages"]),+ ]+ )+ else:+ command.extend(+ [+ "--vectors",+ str(settings["vectors"]),+ "--writes",+ str(settings["writes"]),+ "--chunks-per-document",+ str(settings["chunks_per_document"]),+ "--searches",+ str(settings["searches"]),+ "--verify-known-query",+ ]+ )+ environment = dict(+ os.environ,+ PYTHONPATH=f"{root / 'src'}{os.pathsep}{root}",+ OMP_NUM_THREADS="1",+ OPENBLAS_NUM_THREADS="1",+ MKL_NUM_THREADS="1",+ HF_HUB_OFFLINE="1",+ TRANSFORMERS_OFFLINE="1",+ TOKENIZERS_PARALLELISM="false",+ )+ completed = subprocess.run(+ command,+ cwd=root,+ env=environment,+ capture_output=True,+ text=True,+ timeout=120 if settings["profile"] == "ci" else 300,+ )+ assert completed.returncode == 0, (+ completed.stdout[-4000:] + completed.stderr[-4000:]+ )+ result = json.loads(report.read_text(encoding="utf-8"))+ assert result["dimension"] == settings["dimension"]+ assert result["index_type"] == index_type+ assert result["peak_rss_mib"] <= settings["max_rss_mib"]+ assert result["segments"] >= 2 # CI must really cross a segment boundary.+ assert result["largest_segment_vectors"] <= SEGMENT_VECTORS+ assert result["elapsed_seconds"] > 0+ if workload == "pdf":+ assert result["document_count"] == settings["documents"]+ assert result["pages_per_document"] == settings["pages"]+ assert result["initial_vectors"] > SEGMENT_VECTORS+ assert 0 < result["remaining_vectors"] == result["rebuilt_vectors"]+ assert result["faiss_bytes_written"] > 0+ assert result["metadata_bytes_written"] > 0+ else:+ assert result["initial_vectors"] == settings["vectors"]+ assert result["writes_completed"] == settings["writes"]+ assert result["writes_completed"] > ORPHAN_SCAN_INTERVAL+ assert (+ result["final_vectors"]+ == settings["vectors"]+ + settings["writes"] * settings["chunks_per_document"]+ )+ assert result["searches_completed"] == settings["searches"]+ assert result["known_query_top1_matches"] == 1diff --git a/tests/performance/rag_vector_scale_benchmark.py b/tests/performance/rag_vector_scale_benchmark.pynew file mode 100644index 0000000000..2ad497c6e9--- /dev/null+++ b/tests/performance/rag_vector_scale_benchmark.py@@ -0,0 +1,296 @@+"""Offline latency probe for segmented FAISS at large vector counts.++Builds a collection of synthetic vectors through the production vector store+and a real encrypted SQLCipher database, without PDFs or an embedding model.+It then times per-document writes through ``VectorIndex.index_prepared`` and+searches through ``LibraryRAGService.search``, the paths whose cost grows+with collection size. The report contains numeric diagnostics only.+"""++from __future__ import annotations++import argparse+from contextlib import ExitStack+import hashlib+import json+import math+from pathlib import Path+import statistics+import sys+import tempfile+import time++import numpy as np++from local_deep_research.database.models.library import (+ Collection,+ Document,+ DocumentCollection,+ DocumentStatus,+ SourceType,+)+from local_deep_research.research_library.services.library_rag_service import (+ LibraryRAGService,+)+from local_deep_research.vector_stores.facade import ChunkInput+from local_deep_research.vector_stores.implementations import segmented_faiss++sys.path.insert(0, str(Path(__file__).resolve().parents[2]))+from tests.performance._shared.resources import peak_rss_mib # noqa: E402+from tests.performance.rag_index_workload import ( # noqa: E402+ EmbeddingManager,+ require,+ workload_session,+)++# Documents added per bulk-build transaction.+_BUILD_BATCH = 32+++def _add_documents(session, source_id, collection_id, numbers):+ for number in numbers:+ document_id = f"bench-{number:07d}"+ session.add(+ Document(+ id=document_id,+ source_type_id=source_id,+ title="Synthetic benchmark document",+ document_hash=hashlib.sha256(document_id.encode()).hexdigest(),+ text_content=document_id,+ file_type="pdf",+ file_size=len(document_id),+ status=DocumentStatus.COMPLETED,+ )+ )+ session.add(+ DocumentCollection(+ document_id=document_id, collection_id=collection_id+ )+ )+ return [f"bench-{number:07d}" for number in numbers]+++def _chunks(document_id, count):+ return [+ ChunkInput(+ text=f"{document_id} chunk {position}",+ metadata={"chunk_index": position},+ )+ for position in range(count)+ ]+++def _seconds(values):+ return {+ "p50": round(statistics.median(values), 4),+ "max": round(max(values), 4),+ }+++def run(args, root):+ verify_known = getattr(args, "verify_known_query", False)+ require(+ not verify_known or args.dimension >= 8,+ "Known-query checks require at least 8 dimensions",+ )+ rng = np.random.default_rng(7)+ metrics = {+ "target_vectors": args.vectors,+ "dimension": args.dimension,+ "index_type": args.index_type,+ "chunks_per_document": args.chunks_per_document,+ "orphan_scan_interval": segmented_faiss.ORPHAN_SCAN_INTERVAL,+ }+ started = time.monotonic()+ with (+ workload_session(root) as (session, key, _shared, _db_path),+ ExitStack() as stack,+ ):+ source = SourceType(+ id="benchmark-source", name="document", display_name="Benchmark"+ )+ collection = Collection(id="benchmark", name="Vector scale benchmark")+ session.add_all([source, collection])+ session.commit()+ source_id, collection_id = source.id, collection.id+ rag = LibraryRAGService(+ username="synthetic",+ db_password=key,+ embedding_model="synthetic-sha256",+ embedding_provider="sentence_transformers",+ embedding_manager=EmbeddingManager(args.dimension),+ chunk_size=1000,+ chunk_overlap=200,+ index_type=args.index_type,+ distance_metric="cosine",+ normalize_vectors=True,+ )+ stack.callback(rag.close)+ name = f"collection_{collection_id}"+ index = rag._get_vector_index(collection_id, name)++ # Bulk build through the same row and vector path as a write, but+ # without the per-write orphan check: the build is setup, not the+ # measured workload.+ documents = math.ceil(args.vectors / args.chunks_per_document)+ built = 0+ for start in range(0, documents, _BUILD_BATCH):+ numbers = range(start, min(documents, start + _BUILD_BATCH))+ rows = []+ for document_id in _add_documents(+ session, source_id, collection_id, numbers+ ):+ count = min(args.chunks_per_document, args.vectors - built)+ rows.extend(+ index._build_row(chunk, "document", document_id)+ for chunk in _chunks(document_id, count)+ )+ built += count+ session.add_all(rows)+ session.flush()+ index._apply(+ session,+ add_ids=[row.id for row in rows],+ add_vectors=rng.standard_normal(+ (len(rows), args.dimension), dtype=np.float32+ ),+ )+ session.commit()+ del rows+ if (start // _BUILD_BATCH) % 20 == 0:+ print(json.dumps({"built_vectors": built}), flush=True)+ metrics["build_seconds"] = round(time.monotonic() - started, 1)+ metrics["initial_vectors"] = index.count()+ metrics["segments"] = len(index._store._state())+ require(metrics["initial_vectors"] == args.vectors, metrics)++ # Per-document writes. The first one after the bulk build also runs+ # the periodic full orphan scan, so it is reported separately.+ write_seconds = []+ numbers = range(documents, documents + args.writes)+ _add_documents(session, source_id, collection_id, numbers)+ session.commit()+ known_document = f"bench-{documents:07d}"+ known_query = "benchmark known-vector query"+ metrics["writes_completed"] = 0+ for number in numbers:+ document_id = f"bench-{number:07d}"+ vectors = rng.standard_normal(+ (args.chunks_per_document, args.dimension), dtype=np.float32+ )+ if verify_known and number == documents:+ vectors[0] = np.asarray(+ index.embeddings.embed_query(known_query), dtype=np.float32+ )+ chunks = _chunks(document_id, args.chunks_per_document)+ begun = time.monotonic()+ stats = index.index_prepared(+ source_type="document",+ source_id=document_id,+ chunks=chunks,+ vectors=vectors,+ )+ write_seconds.append(time.monotonic() - begun)+ require(stats.added == args.chunks_per_document, stats)+ metrics["writes_completed"] += 1+ metrics["final_vectors"] = index.count()+ require(+ metrics["final_vectors"]+ == args.vectors + args.writes * args.chunks_per_document,+ metrics,+ )+ entries = index._store._state()+ metrics["largest_segment_vectors"] = max(+ entry["count"] for entry in entries+ )+ require(+ metrics["largest_segment_vectors"]+ <= segmented_faiss.SEGMENT_VECTORS,+ metrics,+ )+ metrics["first_write_seconds"] = round(write_seconds[0], 4)+ metrics["write_seconds"] = _seconds(write_seconds[1:] or write_seconds)++ # Searches through the service, as the research tools call them.+ # The first search starts with no segment verified in this process.+ segmented_faiss._VERIFIED.clear()+ search_seconds = []+ metrics["searches_completed"] = metrics["known_query_top1_matches"] = 0+ for number in range(args.searches):+ query = (+ known_query+ if verify_known and number == 0+ else f"benchmark query {number}"+ )+ begun = time.monotonic()+ hits = rag.search(query, collection_id, 10)+ search_seconds.append(time.monotonic() - begun)+ require(len(hits) == 10, len(hits))+ require(+ len({hit.chunk_id for hit in hits}) == 10,+ "Duplicate search results",+ )+ require(+ all(math.isfinite(hit.distance) for hit in hits),+ "Non-finite search score",+ )+ if verify_known and number == 0:+ require(+ hits[0].source_id == known_document+ and hits[0].text == f"{known_document} chunk 0",+ "Known vector did not rank first",+ )+ metrics["known_query_top1_matches"] += 1+ metrics["searches_completed"] += 1+ metrics["cold_search_seconds"] = round(search_seconds[0], 4)+ metrics["search_seconds"] = _seconds(+ search_seconds[1:] or search_seconds+ )++ metrics["elapsed_seconds"] = round(time.monotonic() - started, 1)+ metrics["peak_rss_mib"] = round(peak_rss_mib(), 2)+ for limit, value, label in (+ (args.max_write_seconds, metrics["write_seconds"]["p50"], "write"),+ (args.max_search_seconds, metrics["search_seconds"]["p50"], "search"),+ (args.max_rss_mib, metrics["peak_rss_mib"], "peak RSS"),+ ):+ require(limit is None or value <= limit, (label, value, limit))+ return metrics+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--vectors", type=int, default=20000)+ parser.add_argument("--dimension", type=int, default=128)+ parser.add_argument(+ "--index-type", choices=("flat", "hnsw"), default="flat"+ )+ # 657 chunks per document matches the 1024-dimension library in #7628.+ parser.add_argument("--chunks-per-document", type=int, default=657)+ parser.add_argument("--writes", type=int, default=6)+ parser.add_argument("--searches", type=int, default=8)+ parser.add_argument("--max-write-seconds", type=float)+ parser.add_argument("--max-search-seconds", type=float)+ parser.add_argument("--max-rss-mib", type=float)+ parser.add_argument("--verify-known-query", action="store_true")+ parser.add_argument("--report", type=Path, required=True)+ args = parser.parse_args()+ if (+ not 1000 <= args.vectors <= 10_000_000+ or not 1 <= args.dimension <= 4096+ or not 1 <= args.chunks_per_document <= 10_000+ or args.writes < 1+ or args.searches < 1+ ):+ parser.error("Invalid benchmark size")+ with tempfile.TemporaryDirectory(prefix="ldr-vector-scale-") as temporary:+ metrics = run(args, Path(temporary))+ args.report.parent.mkdir(parents=True, exist_ok=True)+ args.report.write_text(+ json.dumps(metrics, indent=2) + "\n", encoding="utf-8"+ )+ print(json.dumps(metrics), flush=True)+++if __name__ == "__main__":+ main()diff --git a/tests/performance/test_resources.py b/tests/performance/test_resources.pynew file mode 100644index 0000000000..83bbdb62c7--- /dev/null+++ b/tests/performance/test_resources.py@@ -0,0 +1,65 @@+"""Keep child memory gates independent from the spawning test runner."""+# allow: no-sut-import — tests the shared workload memory reader in owned children.++from pathlib import Path+import subprocess+import sys+from types import SimpleNamespace++import pytest++from tests.performance._shared import resources+++ sys.platform != "linux" or not Path("/proc/self/status").is_file(),+ reason="Linux procfs memory accounting",+)+def test_child_peak_excludes_parent_and_detects_own_allocation():+ child = """+from tests.performance._shared.resources import peak_rss_mib+before = peak_rss_mib()+assert before < 24, before+payload = bytearray(32 * 1024 * 1024)+payload[::4096] = b'x' * len(payload[::4096])+after = peak_rss_mib()+assert after >= before + 24, (before, after)+"""+ parent = """+import subprocess+import sys+payload = bytearray(32 * 1024 * 1024)+payload[::4096] = b'x' * len(payload[::4096])+subprocess.run([sys.executable, '-c', sys.argv[1]], check=True)+"""+ result = subprocess.run(+ [sys.executable, "-c", parent, child],+ capture_output=True,+ text=True,+ timeout=30,+ )+ assert result.returncode == 0, result.stdout + result.stderr+++ "platform,peak", [("linux", 2048), ("darwin", 2 * 1024 * 1024)]+)+def test_fallback_keeps_platform_specific_rss_units(+ monkeypatch, platform, peak+):+ def unavailable():+ raise OSError("procfs unavailable")++ monkeypatch.setattr(resources, "sys", SimpleNamespace(platform=platform))+ monkeypatch.setattr(+ resources, "Path", lambda _: SimpleNamespace(read_text=unavailable)+ )+ monkeypatch.setitem(+ sys.modules,+ "resource",+ SimpleNamespace(+ RUSAGE_SELF=0,+ getrusage=lambda _: SimpleNamespace(ru_maxrss=peak),+ ),+ )+ assert resources.peak_rss_mib() == 2diff --git a/tests/research_library/services/test_rag_vector_lifecycle.py b/tests/research_library/services/test_rag_vector_lifecycle.pyindex 4de44ade53..55bd2ff987 100644--- a/tests/research_library/services/test_rag_vector_lifecycle.py+++ b/tests/research_library/services/test_rag_vector_lifecycle.py@@ -261,6 +261,35 @@ def test_cleanup_repairs_orphans_from_older_writers_before_id_reuse(assert len(rows) == 1+def test_writes_skip_the_full_orphan_scan_between_intervals(+ tmp_path, shared_world, monkeypatch+):+ from local_deep_research.vector_stores.implementations.segmented_faiss import (+ SegmentedFaissStore,+ )++ index = make_index(tmp_path)+ scans = []+ original = SegmentedFaissStore.iter_id_batches++ def counting(store):+ scans.append(True)+ return original(store)++ monkeypatch.setattr(SegmentedFaissStore, "iter_id_batches", counting)+ for number in range(4):+ index.index(+ source_type="document",+ source_id=f"document-{number}",+ chunks=[ChunkInput(text=f"chunk {number}")],+ )+ # An empty index needs no scan and the second write records the first+ # one; later writes only compare IDs against the highest chunk row.+ assert len(scans) == 1+ rows = {row.id for row in shared_world.query(DocumentChunk)}+ assert set(original(index._store).__next__()) == rows++@pytest.mark.parametrize("delete_parent", [False, True])def test_reconciliation_collects_deleted_or_unlinked_documents(shared_world, seeded_collection, delete_parentdiff --git a/tests/vector_stores/test_segmented_faiss.py b/tests/vector_stores/test_segmented_faiss.pyindex f0110204f1..cc006d99c0 100644--- a/tests/vector_stores/test_segmented_faiss.py+++ b/tests/vector_stores/test_segmented_faiss.py@@ -412,3 +412,206 @@ def test_index_storage_size_skips_files_retired_mid_scan(tmp_path, monkeypatch):assert segment_files.index_storage_size(root) == 10root.unlink()assert segment_files.index_storage_size(root) is None+++def fresh_verification(monkeypatch):+ """Isolate the process-wide verified-file record from other tests."""+ monkeypatch.setattr(module, "_VERIFIED", module._VerifiedFiles(1024))+++def _count_digests(monkeypatch):+ digested = []+ original = module._digest++ def counting(path):+ digested.append(Path(path).suffix)+ return original(path)++ monkeypatch.setattr(module, "_digest", counting)+ return digested+++def _whole_second_times():+ """Emulate a filesystem storing whole-second timestamps (NFSv2/3,+ CIFS, VFAT, HFS+): every recorded time lands on a whole second."""++ original = module._file_signature++ def coarse(path):+ signature = original(path)+ if signature is None:+ return None+ return signature[:3] + tuple(+ value - value % 1_000_000_000 for value in signature[3:]+ )++ return coarse+++def _one_coarse_tick():+ """Emulate one whole-second tick that spans both a verification and a+ rewrite inside it: the mount reports the same recorded fields for both."""++ original = module._file_signature+ remembered = {}++ def coarse(path):+ if "signature" not in remembered:+ signature = original(path)+ if signature is not None:+ remembered["signature"] = signature[:3] + tuple(+ value - value % 1_000_000_000 for value in signature[3:]+ )+ return remembered.get("signature")++ return coarse+++def test_repeated_search_hashes_each_segment_once(+ binding, monkeypatch, fresh_verification+):+ monkeypatch.setattr(module, "SEGMENT_VECTORS", 4)+ store = module.SegmentedFaissStore.create(**binding)+ store.apply(add_ids=list(range(1, 13)), add_vectors=vectors(12))+ reader = module.SegmentedFaissStore.load(**binding)+ module._VERIFIED.clear()+ digested = _count_digests(monkeypatch)++ first = reader.search(vectors(1, 3)[0], 5)+ assert sorted(digested) == [".faiss"] * 3 + [".ids"] * 3+ digested.clear()+ assert reader.search(vectors(1, 3)[0], 5) == first+ assert digested == []++[email protected]("suffix", [".faiss", ".ids"])[email protected]("restore_mtime", [False, True])+def test_rewriting_a_verified_segment_is_detected(+ binding, fresh_verification, suffix, restore_mtime+):+ store = module.SegmentedFaissStore.create(**binding)+ store.apply(add_ids=[1, 2], add_vectors=vectors(2))+ store.search(vectors(1)[0], 1)+ entry = state(binding["path"])[0]+ path = (+ module.segment_directory(binding["path"]) / entry["file"]+ ).with_suffix(suffix)+ before = os.stat(path)+ data = bytearray(path.read_bytes())+ data[-1] ^= 0xFF # Same size, same inode.+ time.sleep(0.01)+ path.write_bytes(bytes(data))+ if restore_mtime:+ # utime can restore the modification time but not the change time.+ os.utime(path, ns=(before.st_atime_ns, before.st_mtime_ns))+ with pytest.raises(ValueError, match="checksum"):+ store.search(vectors(1)[0], 1)+++def test_coarse_stored_times_re_hash_segments_on_every_read(+ binding, monkeypatch, fresh_verification+):+ # Whole-second stored times cannot show a same-tick rewrite in the+ # recorded fields, so segments on such mounts forfeit the verification+ # cache and are hashed on every read (safe direction: pre-caching cost).+ monkeypatch.setattr(module, "SEGMENT_VECTORS", 4)+ store = module.SegmentedFaissStore.create(**binding)+ store.apply(add_ids=list(range(1, 13)), add_vectors=vectors(12))+ reader = module.SegmentedFaissStore.load(**binding)+ module._VERIFIED.clear()+ digested = _count_digests(monkeypatch)++ first = reader.search(vectors(1, 3)[0], 5)+ assert sorted(digested) == [".faiss"] * 3 + [".ids"] * 3+ digested.clear()+ assert reader.search(vectors(1, 3)[0], 5) == first+ assert digested == [] # Fine-grained times: the cache serves the read.++ monkeypatch.setattr(module, "_file_signature", _whole_second_times())+ for _ in range(2):+ digested.clear()+ assert reader.search(vectors(1, 3)[0], 5) == first+ assert sorted(digested) == [".faiss"] * 3 + [".ids"] * 3+++def test_same_tick_rewrite_with_coarse_times_is_re_hashed(+ binding, fresh_verification, monkeypatch+):+ # The worst case: on a whole-second-timestamp filesystem, a same-size+ # rewrite and a utime restore inside one tick leave every recorded+ # field unchanged. The cache must not serve that read unhashed.+ store = module.SegmentedFaissStore.create(**binding)+ store.apply(add_ids=[1, 2], add_vectors=vectors(2))+ entry = state(binding["path"])[0]+ monkeypatch.setattr(module, "_file_signature", _one_coarse_tick())+ store.search(vectors(1)[0], 1) # Verifies within the coarse tick.+ needle = np.asarray(store.reconstruct(1), dtype="<f4").tobytes()+ path = (+ module.segment_directory(binding["path"]) / entry["file"]+ ).with_suffix(".faiss")+ data = bytearray(path.read_bytes())+ # Corrupt a stored vector value in place: same size, same inode, ids+ # and bounds untouched, so the hash is the only thing that can notice.+ # (Corrupting the id map instead would be caught by the membership+ # cross-check even without the hash.)+ offset = bytes(data).find(needle)+ assert offset > 0+ data[offset] ^= 0xFF+ path.write_bytes(bytes(data))+ with pytest.raises(ValueError, match="checksum"):+ store.search(vectors(1)[0], 1)+++def test_verified_files_are_bounded():+ record = module._VerifiedFiles(limit=2)+ for name in ("a", "b", "c"):+ record.remember(Path(name), "digest", ("signature",))+ assert [key[0] for key in record._entries] == ["b", "c"]+++def test_orphan_scan_runs_periodically_or_for_reusable_ids(+ binding, monkeypatch+):+ monkeypatch.setattr(module, "ORPHAN_SCAN_INTERVAL", 3)+ store = module.SegmentedFaissStore.create(**binding)+ assert not store.orphan_scan_due(None) # Nothing indexed yet.+ store.apply(add_ids=[1], add_vectors=vectors(1))+ assert store.orphan_scan_due(1) # No scan has been recorded.+ store.apply(add_ids=[2], add_vectors=vectors(1, 2), orphan_scanned=True)+ assert not store.orphan_scan_due(2)+ # A vector above the highest chunk row could collide with a new row.+ assert store.orphan_scan_due(1)+ assert store.orphan_scan_due(None)+ for value in (3, 4):+ assert not store.orphan_scan_due(value - 1)+ store.apply(add_ids=[value], add_vectors=vectors(1, value))+ assert not store.orphan_scan_due(4)+ store.apply(add_ids=[5], add_vectors=vectors(1, 5))+ assert store.orphan_scan_due(5) # The interval has elapsed.+ # Republishing an unchanged generation keeps its scan record.+ store.apply(add_ids=[6], add_vectors=vectors(1, 6), orphan_scanned=True)+ store.persist(binding["path"])+ assert not store.orphan_scan_due(6)+++def test_small_rewritten_segments_merge_with_their_neighbour(+ binding, monkeypatch+):+ monkeypatch.setattr(module, "SEGMENT_VECTORS", 4)+ store = module.SegmentedFaissStore.create(**binding)+ original = vectors(12)+ store.apply(add_ids=list(range(1, 13)), add_vectors=original)+ assert [entry["count"] for entry in state(binding["path"])] == [4, 4, 4]++ store.apply(add_ids=[], add_vectors=None, remove_ids=[1, 2, 3, 5, 6])+ assert [entry["count"] for entry in state(binding["path"])] == [3, 4]+ assert sorted(store.live_ids()) == [4, 7, 8, 9, 10, 11, 12]+ for value in (4, 7, 9):+ np.testing.assert_allclose(+ store.reconstruct(value), original[value - 1], rtol=1e-6+ )+ # Untouched full segments are never rewritten to merge.+ untouched = state(binding["path"])[-1]["file"]+ store.apply(add_ids=[], add_vectors=None, remove_ids=[4])+ assert state(binding["path"])[-1]["file"] == untouched