Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

ci+test: retry transient network installs, fix patch.object race (#4302)

LearningCircuite1bc52904e2b
diff --git a/.github/workflows/playwright-webkit-tests.yml b/.github/workflows/playwright-webkit-tests.yml
index 440add070f..2bc9fdc2e8 100644
--- a/.github/workflows/playwright-webkit-tests.yml
+++ b/.github/workflows/playwright-webkit-tests.yml
@@ -50,12 +50,28 @@ jobs:
npm ci
npm run build
- - name: Install Playwright dependencies
+ - name: Install Playwright dependencies (npm)
working-directory: tests/ui_tests/playwright
- run: |
- npm ci
+ run: npm ci
+
+ # ``playwright install --with-deps`` downloads ~150 MB of browser
+ # binaries from the playwright CDN per attempt. Single CDN
+ # hiccups have failed the whole release pipeline in the past
+ # (e.g. transient 5xx mid-download); a single retry costs little
+ # and removes the dominant transient-network failure mode for
+ # this workflow. Two attempts only — three attempts of a 150 MB
+ # download starts hiding real outages.
+ - name: Install Playwright browsers
+ uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
+ with:
+ timeout_minutes: 10
+ max_attempts: 2
+ retry_on: error
+ shell: bash
# Install chromium for auth setup step + webkit for Safari tests
- npx playwright install chromium webkit --with-deps
+ command: |
+ cd tests/ui_tests/playwright
+ npx playwright install chromium webkit --with-deps
- name: Setup test directories
run: |
@@ -190,12 +206,28 @@ jobs:
npm ci
npm run build
- - name: Install Playwright dependencies
+ - name: Install Playwright dependencies (npm)
working-directory: tests/ui_tests/playwright
- run: |
- npm ci
+ run: npm ci
+
+ # ``playwright install --with-deps`` downloads ~150 MB of browser
+ # binaries from the playwright CDN per attempt. Single CDN
+ # hiccups have failed the whole release pipeline in the past
+ # (e.g. transient 5xx mid-download); a single retry costs little
+ # and removes the dominant transient-network failure mode for
+ # this workflow. Two attempts only — three attempts of a 150 MB
+ # download starts hiding real outages.
+ - name: Install Playwright browsers
+ uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
+ with:
+ timeout_minutes: 10
+ max_attempts: 2
+ retry_on: error
+ shell: bash
# Install chromium for auth setup step + webkit for Safari tests
- npx playwright install chromium webkit --with-deps
+ command: |
+ cd tests/ui_tests/playwright
+ npx playwright install chromium webkit --with-deps
- name: Setup test directories
run: |
diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml
index c98766046d..984784d34f 100644
--- a/.github/workflows/pre-commit.yml
+++ b/.github/workflows/pre-commit.yml
@@ -56,7 +56,37 @@ jobs:
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
+ # Replicate pre-commit/[email protected] manually so we can wrap the
+ # actual `pre-commit run` invocation in nick-fields/retry. The
+ # upstream action is a thin composite (set PY for cache key →
+ # cache `~/.cache/pre-commit` → `pip install pre-commit` →
+ # `pre-commit run`), and the failure that motivates the retry
+ # always happens in that last step: pre-commit lazily downloads
+ # hook environments from external sources (PyPI for ruff, GitHub
+ # release binaries for shellcheck, etc.), and a single HTTP 5xx
+ # from any one of them fails the whole job. See run #2524 where
+ # `Building wheel for shellcheck_py` hit HTTP 502 fetching the
+ # shellcheck binary during the wheel build. A second attempt
+ # benefits from the partially-populated cache and almost always
+ # succeeds. Two attempts is enough: a hook environment that
+ # fails to install twice in a row is not a transient outage.
+ - name: Compute pre-commit cache key
+ run: echo "PY=$(python -VV | sha256sum | cut -d' ' -f1)" >> "$GITHUB_ENV"
+
+ - name: Cache pre-commit hook environments
+ uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+ with:
+ path: ~/.cache/pre-commit
+ key: pre-commit|${{ env.PY }}|${{ hashFiles('.pre-commit-config.yaml') }}
+
+ - name: Install pre-commit
+ run: python -m pip install pre-commit
+
- name: Run pre-commit
- uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1
+ uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
with:
- extra_args: --all-files
+ timeout_minutes: 15
+ max_attempts: 2
+ retry_on: error
+ shell: bash
+ command: pre-commit run --show-diff-on-failure --color=always --all-files
diff --git a/tests/research_library/services/test_library_rag_service_coverage.py b/tests/research_library/services/test_library_rag_service_coverage.py
index f5bfcd81c3..e90f37138f 100644
--- a/tests/research_library/services/test_library_rag_service_coverage.py
+++ b/tests/research_library/services/test_library_rag_service_coverage.py
@@ -1467,26 +1467,46 @@ class TestMergeAndPersistLocked:
svc.faiss_index.docstore = MagicMock()
svc.faiss_index.docstore._dict = {}
start_event.wait() # Race start
- with patch.object(
- mod.FAISS,
- "load_local",
- side_effect=lambda *a, **kw: make_index_proxy(),
- ):
- svc._merge_and_persist_locked(
- index_path,
- [LangchainDocument(page_content=f"chunk-{cid}")],
- [cid],
- force_reindex=False,
- )
+ svc._merge_and_persist_locked(
+ index_path,
+ [LangchainDocument(page_content=f"chunk-{cid}")],
+ [cid],
+ force_reindex=False,
+ )
go = threading.Event()
t1 = threading.Thread(target=worker, args=("A", go))
t2 = threading.Thread(target=worker, args=("B", go))
- t1.start()
- t2.start()
- go.set()
- t1.join(timeout=5.0)
- t2.join(timeout=5.0)
+
+ # Patch FAISS.load_local ONCE, around both threads. Patching
+ # inside each worker (the original shape) is a hidden race:
+ # ``patch.object`` rewrites a module attribute on the FAISS
+ # class and is not thread-safe — when the first worker's
+ # ``with patch.object(...)`` block exits, it restores whatever
+ # it captured as "the original" at entry, which can be the
+ # second worker's lambda OR the real ``load_local`` (depending
+ # on which thread entered patch first). If the real
+ # ``load_local`` becomes active mid-test, the second worker's
+ # reload step raises on the empty touched ``shared.faiss``,
+ # the production code's ``except`` branch falls back to the
+ # caller's stale in-memory MagicMock, and the worker's save is
+ # a no-op (the setup MagicMock has no ``save_local`` side
+ # effect). Result: that worker's chunk never reaches
+ # ``disk_state``, the assertion fails, and the test wrongly
+ # blames a regression in the production lock. Applying the
+ # patch around both threads removes the race entirely — the
+ # patched value is the only value visible while either worker
+ # runs.
+ with patch.object(
+ mod.FAISS,
+ "load_local",
+ side_effect=lambda *a, **kw: make_index_proxy(),
+ ):
+ t1.start()
+ t2.start()
+ go.set()
+ t1.join(timeout=5.0)
+ t2.join(timeout=5.0)
# Both writers' chunks present — read-modify-write under the
# lock did its job. Without the fix, only one of {A, B} would