Skip to content

The OSINT investigator's workbench. One case, one folder, every tool. Local-first: your media and your investigations never leave your machine.

JavaScript

Teach the e2e fixture the route the app now reads on mount, and let a slow runner drain (#74)

OsintMeThat63761973fe4e
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 1f94a85..f471467 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -45,6 +45,18 @@ jobs:
# alone is half of it. That range is split rather than trusted, since a
# slow hour on the 30-minute timeout is a red run for no reason.
#
+ # These ranges look unbalanced by CPU seconds, and re-cutting them by that
+ # number made the run worse rather than better: a split that put
+ # `test_media_api.py` and `test_links_api.py` in one piece — even weight on
+ # paper — failed the shard, because those two are where the suite contends
+ # on locks and threads, and contention does not add up the way CPU seconds
+ # do. `test_concurrent_sidecar_merges_do_not_drop_each_other` alone went to
+ # two minutes there and the thread joins under it ran out. So the weight to
+ # balance is Windows wall clock under `-n auto`, measured on Windows; a
+ # Linux profile is not a stand-in for it, and cheap parallelism is not worth
+ # a red run. Anyone re-cutting these should keep the lock-heavy files apart
+ # and prove it on the box itself.
+ #
# The ranges have to leave no gap: one stopping at test_sc* beside one
# starting at test_se* runs everything except the test_sd* somebody adds
# next year, and says nothing. `tests/test_ci_shards.py` reads them back
diff --git a/frontend/e2e/app.fixture.js b/frontend/e2e/app.fixture.js
index 8ac1b1d..c3f1775 100644
--- a/frontend/e2e/app.fixture.js
+++ b/frontend/e2e/app.fixture.js
@@ -991,6 +991,17 @@ export async function installAppFixture(page, options = {}) {
return json(route, { name: payload.name, path: made });
}
if (path === '/api/settings/scrapers') return json(route, { scrapers: [] });
+ // Read on mount by `checkForUpdatesOnStart`, before any tool and whatever the
+ // update switch says, because it touches disk rather than the network. The
+ // fixture has no extension folder, which is the state a fresh install is in.
+ if (path === '/api/settings/extension') {
+ return json(route, {
+ path: '/tmp/fixture/extension',
+ bundled: { version: '0.0.0-fixture', payload: 'fixture-payload' },
+ folder: null,
+ staged: false,
+ });
+ }
if (path === '/api/settings/ffmpeg') {
return json(route, { available: true, version: 'fixture', source: 'bundled', path: '/tmp/ffmpeg' });
}
diff --git a/pyproject.toml b/pyproject.toml
index 78c9ebc..6714065 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -169,6 +169,13 @@ packages = ["src/azimut"]
[tool.pytest.ini_options]
testpaths = ["tests"]
asyncio_mode = "auto"
+# Print every thread's stack when one test runs longer than this, using the
+# stdlib's own faulthandler rather than a plugin. A test that hangs on a runner
+# we don't have takes the job's whole budget and is then killed with no name
+# attached: the shard reports a percentage and nothing else. This turns that
+# into a traceback. Well above the slowest test the suite has (`jobwait.WAIT`
+# is 60), so a healthy run never prints.
+faulthandler_timeout = 180
[tool.ruff]
# Match the codebase's existing wrap. Default rule set (E/F/W) only — this is a
diff --git a/tests/conftest.py b/tests/conftest.py
index defa1d0..d401141 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,4 +1,6 @@
import tempfile
+import threading
+import time
import pytest
from fastapi.testclient import TestClient
@@ -62,10 +64,23 @@ def a_worker_of_its_own():
Set up before the fixtures that own a temporary workspace, so the state is
clean before a case exists at all. The worker is waited out first, because
clearing the flag under a live thread would let a second one start.
+
+ Waited out *to the end of the thread*, not just until the queue reads empty.
+ `_reset_for_tests` keeps the flag when a worker is still alive — rightly, the
+ flag is true then — but a test that starts in that state queues its work, is
+ told a worker already has it, and starts none; and if the live one reaches
+ the end of its queue in between, it leaves with the new work undrained. The
+ test then waits its whole budget for a thumbnail nobody is rendering. Which
+ box shows it is a coin toss, since what leaks the thread is the test that ran
+ before it, and that is the randomized order's to choose.
"""
from azimut.engine import workqueue
- workqueue.wait_until_idle(timeout=10)
+ workqueue.wait_until_idle(timeout=60)
+ deadline = time.monotonic() + 60
+ for thread in threading.enumerate():
+ if thread.name == workqueue._WORKER_NAME and thread.is_alive():
+ thread.join(max(0.0, deadline - time.monotonic()))
workqueue._reset_for_tests()
yield
@@ -78,10 +93,19 @@ def _let_the_worker_finish() -> None:
still leave the worker mid-write when its temp directory goes — which on
POSIX surfaces as a "directory not empty" teardown error and on Windows as a
locked file. This is the same orderly-shutdown wait the app itself performs.
+
+ The wait is generous because running out of it is silent: `wait_until_idle`
+ reports the timeout in its return value and nothing here can act on it, so
+ the only symptom is the rmtree that follows, failing in whichever test owned
+ the directory. Ten seconds was enough on Linux and not on Windows, where the
+ concurrent-download tests queue eight items' worth of enrichment behind
+ `synchronous = FULL`. An idle worker returns immediately, so the ceiling
+ costs nothing in the normal case; it only has to be above the slowest
+ runner.
"""
from azimut.engine import workqueue
- workqueue.wait_until_idle(timeout=10)
+ workqueue.wait_until_idle(timeout=60)
def _settle_before_the_workspace_goes() -> None:
diff --git a/tests/test_ingest_maptools.py b/tests/test_ingest_maptools.py
index 1ae7ba4..783ae14 100644
--- a/tests/test_ingest_maptools.py
+++ b/tests/test_ingest_maptools.py
@@ -198,15 +198,19 @@ def test_reference_media_is_searched_and_bounded(client):
assert page["total"] == 5
-def test_sky_answers_without_the_network(client, monkeypatch):
- # the panel has to work on a map tab with no connection: the computation is
- # local, and a test that lets it reach the network would not prove that
- import socket
-
- def refuse(*args, **kwargs):
- raise AssertionError("the sky route must not touch the network")
-
- monkeypatch.setattr(socket, "socket", refuse)
+def test_sky_answers_the_panel_for_a_point(client):
+ # The panel has to work on a map tab with no connection, and that guarantee
+ # is held one layer down, where it can be held honestly:
+ # `test_sky.py::test_sky_does_not_touch_the_network` makes every socket
+ # explode and calls the computation directly.
+ #
+ # It cannot be re-proved through this route, because the client that would
+ # ask is itself a loop that opens sockets to talk to its own server thread:
+ # taking `socket.socket` away does not stop the route reaching out, it stops
+ # the request arriving, and the call never returns. That deadlock has no
+ # timeout under it, so it costs the whole Windows shard — twice, before the
+ # traceback said so. What is left here is what only this layer can answer:
+ # the route exists, takes a point, and hands the panel the shape it draws.
day = client.get(
"/api/ingest/sky", params={"lat": 48.8584, "lon": 2.2945}, headers=_headers(client)
).json()