Dodaios is an open-source agent OS for your desktop: it runs your coding agents, browser and apps by voice or chat, on your own machine. Formerly Personal Jarvis.
Python agent-orchestrationai-agentai-coding-assistantclaude-codecoding-agentcomputer-usedesktop-applocal-firstmcppersonal-jarvisvoice-assistantvoice-coding
fix(net): stop emptying the machine's socket pool on wake (BUG-215, AP-33)
rubenluetke10-beep7da809f7944b
diff --git a/AGENTS.md b/AGENTS.mdindex aea4705e4f..1e660480dd 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -35,6 +35,13 @@ These cost real bugs. Nothing catches them but you.- **A stall watchdog resets its counter per unit of work** (AP-19); a WebSocketreceive loop treats ANY read error as terminal and breaks (AP-20); a subscriberexception never leaves `EventBus._safe_dispatch` (AP-18).+- **Every reconnect is jittered and pays the shared connect budget.** A retry+ grid with no jitter, or a wake handler that opens a socket directly, is not+ an app bug: N panes x M windows x 2 instances firing on one+ `visibilitychange` empties the OS ephemeral-port pool and NOTHING on the+ machine can connect for two minutes. Frontend goes through+ `lib/connectBudget.ts`; a poll reuses a client from `jarvis/core/http_pool.py`.+ (AP-33)- **No Windows Service** — SYSTEM has no microphone. (AP-17)- **Never gate a CI check on `isinstance` against an unpinned library.** Greenlocally, red in CI on the next release. Discriminate by capability. (AP-28)@@ -112,7 +119,9 @@ it (AP-14); new `[phase6.*]` / `[memory.wiki.*]` keys needwake upgrade only on the out-of-process inference probe, never on CUDA presence(AP-25); verify a wake word on audio energy and candidate shape, never ontranscript content (AP-27); a WebGL scene releases its context and survives-losing it (AP-32). Detail and history for any of them: `docs/BUGS.md`.+losing it (AP-32); a reconnect without jitter and without a shared connect+budget is an outage of the whole machine, not an app bug (AP-33). Detail and+history for any of them: `docs/BUGS.md`.## 4. How work shipsdiff --git a/CLAUDE.md b/CLAUDE.mdindex aea4705e4f..1e660480dd 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -35,6 +35,13 @@ These cost real bugs. Nothing catches them but you.- **A stall watchdog resets its counter per unit of work** (AP-19); a WebSocketreceive loop treats ANY read error as terminal and breaks (AP-20); a subscriberexception never leaves `EventBus._safe_dispatch` (AP-18).+- **Every reconnect is jittered and pays the shared connect budget.** A retry+ grid with no jitter, or a wake handler that opens a socket directly, is not+ an app bug: N panes x M windows x 2 instances firing on one+ `visibilitychange` empties the OS ephemeral-port pool and NOTHING on the+ machine can connect for two minutes. Frontend goes through+ `lib/connectBudget.ts`; a poll reuses a client from `jarvis/core/http_pool.py`.+ (AP-33)- **No Windows Service** — SYSTEM has no microphone. (AP-17)- **Never gate a CI check on `isinstance` against an unpinned library.** Greenlocally, red in CI on the next release. Discriminate by capability. (AP-28)@@ -112,7 +119,9 @@ it (AP-14); new `[phase6.*]` / `[memory.wiki.*]` keys needwake upgrade only on the out-of-process inference probe, never on CUDA presence(AP-25); verify a wake word on audio energy and candidate shape, never ontranscript content (AP-27); a WebGL scene releases its context and survives-losing it (AP-32). Detail and history for any of them: `docs/BUGS.md`.+losing it (AP-32); a reconnect without jitter and without a shared connect+budget is an outage of the whole machine, not an app bug (AP-33). Detail and+history for any of them: `docs/BUGS.md`.## 4. How work shipsdiff --git a/docs/BUGS.md b/docs/BUGS.mdindex 17b4755254..47179e04d6 100644--- a/docs/BUGS.md+++ b/docs/BUGS.md@@ -15081,3 +15081,143 @@ reach), AP-21 (gate on capability, never a provider name), AP-24 (a wedgednative engine is replaced, never waited on), AP-26 (nothing initialises on theboot critical path — this removes work from it), AP-30 (every new skip branchlogs its reason).++## BUG-215: the whole PC became unusable after time away, then healed itself — Jarvis emptied the machine's ephemeral-port pool (HIGH, FIXED 2026-09-03)++**Symptom.** The maintainer came back to the machine after a while away and+could barely operate it: windows unresponsive, the start menu dead, the browser+stuck. After a few minutes it recovered on its own, with nothing to show for+it. Reported 2026-09-03 ~10:20.++This was not the first time. Windows logs event 4231 ("failed to allocate an+ephemeral port from the global TCP port pool — all ports are in use") and it had+fired on **23 of the previous 31 days**, most recently that morning at 09:30;+the UDP twin (4266) fired about as often. The oldest entry the log still held+was 2026-08-04, so the bug is at least a month old and probably older than the+retention window.++**What it was not** — measured on the live box while the forensics ran, because+each of these is the usual suspect and each was innocent:++| Suspect | Reading |+|---|---|+| GPU memory | 6 145 of 16 303 MiB, no model loaded |+| RAM / paging | pagefile peak 5 970 MB against 5 196 MB current — no paging storm |+| CPU | 3.1 of 16 cores under load |+| Disk | ~0 MB/s on every candidate process |+| Sleep / resume | no Kernel-Power transition; the machine was awake throughout |+| App hangs | no Application-log hang or crash entry in the window |++Every other machine-wide resource was fine. The one resource Windows itself+reported running out of was ephemeral ports.++**Why that reads as "the whole computer is broken".** Every outbound connection+borrows a port from a pool the OS owns, and a *closed* connection keeps its port+through TIME_WAIT — 120 s on Windows, against a default pool of 16 384. Drain+the pool and nothing on the machine can open a socket: not Jarvis, not Explorer,+not the start menu, not the browser. And then TIME_WAIT expires, the pool+refills, and the evidence is gone. Windows throttles event 4231 to roughly one+an hour, so the entry usually lands in the wrong minute and never names a+process — which is why a month of this produced no diagnosis.++**Root cause — a wake storm on top of a standing idle burn.**++*The storm.* While nobody is at the machine, windows are hidden, timers are+clamped, and open WebSockets die without a `close` event — the peer is gone but+`readyState` still says OPEN. `lib/ws.ts:69-71` had already documented exactly+this. The instant the user comes back, `visibilitychange` and `online` fire in+the same millisecond and **every socket client reconnects at once, with no delay+and no jitter**:++| Client | Behaviour before |+|---|---|+| `lib/ws.ts` (app socket) | reconnected immediately on wake; plain doubling, no jitter |+| `components/agentic/paneSocket.ts` | **one socket per terminal pane**, each reconnecting immediately on wake; no jitter |+| `store/agentChat.ts` | flat 1 500 ms retry, **no cap, no backoff, no attempt limit** |+| `lib/realtimeTransportBroker.ts` | exponential and capped, but no jitter |+| `jarvis/plugins/realtime/openai_realtime.py:2293` | flat 2 s step, up to 60 attempts in a 120 s window |++Multiply by the panes in a workspace, by the open windows, and by the two app+instances that run on this box (prod + dev). A search for jitter over the whole+tree returned only audio jitter buffers: there was no spreading anywhere. The+codebase had even *measured* this shape once before — `paneSocket.ts:33-44`+records five panes knocking at 10.174, 11.174, 12.162, 13.170 on 2026-07-27,+"a flat one-second grid where the backoff asked for 0.5 s, 1 s, 2 s, 4 s" — and+fixed the symptom for panes without ever adding jitter or a shared ceiling.++*The standing burn the storm landed on top of.* Measured 2026-09-03 on an idle+app with nobody using local models: **32 requests a minute to Ollama**, 421 of+the last 549 of them `POST /api/show`. `ollama_inventory._make_client` opened a+NEW client — and so a new connection — per call ("One client per call", said its+own docstring), a sweep is `/api/tags` plus one `/api/show` per installed+download, and `DEFAULT_SNAPSHOT_MAX_AGE_S` was **5 s against a 15 s UI poll**, so+the cache could never hit and every poll swept. Alongside it,+`desktop_app._wait_for_backend` probed `127.0.0.1/api/health` every 250 ms with a+fresh `httpx.get()` — while the reused client it should have used sat ten lines+below it — and the local-realtime monitor opened a fresh+`socket.create_connection` plus a fresh `http.client` connection every 5 s, for+as long as a voice server lived.++**Fix.**++1. `jarvis/ui/web/frontend/src/lib/connectBudget.ts` — one shared speed limit+ for every socket the tab opens. Two mechanisms, both needed: full jitter+ (`jitteredDelay`) so clients that started together do not stay together, and+ a token bucket (`requestConnect`, 6 connects/s with a burst of 8) so that+ even when they collide the total is survivable. It delays connections; it+ never drops them. `spreadDelay` keeps a *repeating* knock's average interval+ while destroying its phase.+2. All four socket clients now go through it, and waking **queues** a reconnect+ instead of opening one from the event handler. `store/agentChat.ts` also+ gained the backoff and the cap it never had.+3. Connections are reused rather than rebuilt: `jarvis/core/http_pool.py`+ (promoted from `jarvis/plugins/tool/_http_pool.py`, now with explicit+ `httpx.Limits`) backs the Ollama inventory, the desktop health probe and the+ local-realtime readiness probe. `DEFAULT_SNAPSHOT_MAX_AGE_S` is 60 s — longer+ than any poll that reads it, which is the only way a cache helps — and safe+ because the three moments the inventory really changes all call+ `invalidate_snapshot`.+4. `supervisor._port_open` closes with `SO_LINGER = 0`, so the kernel sends RST+ instead of FIN and the probe leaves **no** TIME_WAIT at all. Nothing about an+ "is anyone listening" check wants a graceful shutdown.+5. `jarvis/core/socket_budget.py` — the watchdog that makes the next occurrence+ diagnosable instead of invisible. Off the event loop (like+ `core/loop_watchdog.py`, and for the same reason), one census every 30 s,+ armed by both entry points. It reads the ceiling per platform, warns at 60 %+ with a census naming the busiest destinations and processes, and stands+ optional traffic down at 80 % while voice and brain keep working.++**Three operating systems, three ceilings, one failure.** The fix is to cap the+burst, never to widen a limit:++| System | Ephemeral ports | TIME_WAIT | What gives out first |+|---|---:|---:|---|+| Windows | 16 384 | 120 s | the port pool — the whole desktop stalls |+| Linux | 28 232 | 60 s | usually `RLIMIT_NOFILE` (1024 on a slim image) — the process dies instead |+| macOS | 16 384 | 15 s | the port pool, but it recovers fastest |++Windows is the hard case: smallest pool, longest linger. Code that stays inside+the budget there is safe everywhere.++**Guard.** `tests/contract/test_socket_budget.py` fakes each host in turn —+including a `netsh` that answers in German and in Japanese, because `netsh`+translates its own field names and a label-matching parser reports "no limit" on+most of the install base. `tests/unit/core/test_socket_budget_watchdog.py` pins+when the watchdog speaks and what it says.+`jarvis/ui/web/frontend/src/lib/connectBudget.test.ts` pins that twenty clients+waking together are paced to the budget and that none of them is dropped.+`tests/unit/ui/test_desktop_health_probe.py` gained the boot-wait case.+`paneSocket.test.ts` now pins that waking does NOT connect in the handler's own+tick, and does connect inside the fast window.++**Verification.** Before: ~165 sockets in TIME_WAIT on an idle machine and 32+Ollama requests a minute. The watchdog's own first census on the live box named+the three destinations the forensics had found by hand (`127.0.0.1:47821` ×44,+`127.0.0.1:47921` ×30, `::1:11434` ×20). After the build, re-read both numbers+and hide/show the app windows twenty times while watching the TIME_WAIT count.++**Related.** AP-33 (the new anti-pattern), AP-26 (nothing initialises on the+boot critical path — the watchdog's first census is a full interval away), AP-30+(the probe's silence is explained by its documented "None means not ready"+contract), BUG-113 (the pane-reconnect storm whose symptom was fixed in 2026-07+while this cause was left standing), `docs/os-parity.md` (the ceilings table).diff --git a/docs/os-parity.md b/docs/os-parity.mdindex a1a208fb67..3c975e1475 100644--- a/docs/os-parity.md+++ b/docs/os-parity.md@@ -298,6 +298,7 @@ shortcuts it never created. Covered by| P-40 | Low | JarvisBar | The Prompt Mode switch (`[dictation].prompt_mode`, every dictation comes out as a written prompt) is a control on the native bar since 2026-08-28: a sparkle in the resting pill's left slot, drawn ONLY while the SETTING is on (visible without a hover — the pill opens as it does for mute). A click PAUSES the rewriting rather than changing the setting: the mark goes red with a slash through it, jarvis.toml is untouched, and the same click brings it back. The pause is runtime-only and clears on restart or on any write of the setting. With the setting off nothing is drawn there and the spot starts a session as before. The glyph mirrors the mic's inset rather than the close-X's, which is left untouched on a live bar. The switch has ONE writer (`jarvis/dictation/prompt_mode_switch.py`) and one broadcast (`DictationPromptModeChanged`) that the bar, the front-page pill and the settings card all redraw from | `jarvis/ui/jarvisbar/renderer.py::_draw_sparkle`, `interaction.py::resolve_click` (`prompt_mode_toggle`), `overlay.py` / `qt_overlay.py` / `subprocess_overlay.py` (`set_prompt_mode`, `set_on_prompt_mode_toggle`), `host.py` (op `set_prompt_mode`, event `prompt_mode_toggle`), `ui/orb/bus_bridge.py` | Windows/Linux (Tk bar, in-process) and macOS (Qt bar in the companion host, over the existing IPC protocol): the same sparkle, the same click, the same event. Headless / `orb_style = none`: the NullOverlay accepts both methods as no-ops; the pill on the front page and the settings card remain the switch. Guards: `tests/unit/ui/jarvisbar/test_prompt_mode_button.py`, `test_prompt_mode_bridge.py`, `test_surface_contract.py`, `test_host_protocol.py` || P-41 | Medium | Dictation final pass | The dictation lane reads its final pass on the machine, in front of any configured cloud provider, through the same out-of-process worker the live preview uses (`jarvis.dictation.preview_worker`) with a stronger checkpoint (`[dictation].local_model`, default `large-v3-turbo`, `int8_float16`, beam 5). 2026-09-02 forensics: 220 of 220 dictations had run on a cloud recognizer the settings did not name, because the VOICE lane had been moved there to keep CUDA out of the desktop process, and the dictation lane inherited the move — round-trips as lag, near-empty answers on 25 s windows as "truncated window" repairs. The provider declines itself with a CROSSABLE failure (HTTP-503-shaped, `LocalEngineUnavailable`) so the configured provider is one step behind on every press and never a restart away: too little free accelerator memory (`[dictation].local_min_free_gb`, default 1.5; an UNKNOWN reading never blocks), a worker that only comes up on the CPU, a missing local runtime, a worker that died. A call abandoned at its ceiling leaves its answer in the pipe, so the next call kills that worker instead of reading it (AP-24) | `jarvis/dictation/local_final.py::LocalFinalSTT`, `jarvis/speech/pipeline.py::_dictation_local_final`, `_dictation_stt`, `jarvis/dictation/local_preview.py::_spawn_worker_model` (`compute`), `preview_worker.py` (`beam_size`, `segments`), `jarvis/hardware/detection.py::free_accelerator_gb` | Windows/Linux with an NVIDIA card and the local runtime: the final pass runs locally when ≥ 1.5 GB is free, else the cloud chain. macOS: no NVIDIA reading → the memory gate passes as unknown, and the worker's own device ladder decides; a CPU-only worker is declined for the final pass (a beam decode of a 25 s window on a CPU is slower than the cloud it would replace), so Apple Silicon stays on the configured provider until a Metal-backed runtime exists. Headless `python:3.11-slim` (no `faster-whisper`): the provider is never built and the chain is exactly what it was. Switch: `[dictation].local_engine = false`. Guards: `tests/unit/dictation/test_local_final.py`, `test_preview_worker.py` || P-41 | Low | CLI install / connect terminal | Pressing **Install** or **Connect** in the CLI section opens a REAL terminal window and runs the command there, because an interactive OAuth login belongs in a shell the user can see and type into. That spawn was Windows-only until 2026-08-28 (`wt` -> `pwsh` -> `powershell`), so on macOS and Linux the two buttons reported "No external terminal available" and no CLI could be installed from the app at all. macOS now goes through `osascript` to Terminal.app, Linux through the first of seven emulators that exists (Debian's `x-terminal-emulator` alternative first, so the user's own default wins), and the window is held open afterwards the way `-NoExit` holds it on Windows. One install click now carries all the way to signed in: the terminal runs the install, refreshes PATH in that same shell, and continues into the CLI's login command, stopping at the first step that fails. The chain operator differs per OS on purpose — Windows PowerShell 5.1 is a possible fallback shell and `&&` is a parser error there, so the Windows spelling nests `if ($?)` instead | `jarvis/clis/external_terminal.py` (`_spawn_windows`, `_spawn_macos`, `_spawn_linux`, `chain_commands`, `path_refresh_command`), `jarvis/ui/web/cli_routes.py::spawn_external`, `tests/unit/clis/test_install_path.py` | Windows: verified live (wt path, composed install+login command). macOS/Linux: same code path, the branch itself unverified from this machine. A box with no screen has no window to open, so there the endpoint falls back to the in-app streaming install job and the UI says so instead of claiming a terminal appeared — a headless server can still install a CLI, it just cannot run an interactive login |+| P-42 | High | Outbound sockets (every lane) | Every connection borrows an ephemeral port from a pool the OS owns, and a closed one keeps that port through TIME_WAIT. The ceiling and the linger differ per OS, and so does what gives out first: **Windows** 16 384 ports / 120 s — the tightest by far, and the port pool is what empties, which stops EVERY process on the machine from connecting (the desktop appears frozen, then heals itself when TIME_WAIT expires); **Linux** 28 232 ports / 60 s, but `RLIMIT_NOFILE` is normally the tighter of the two (1024 on a stock `python:3.11-slim`), so the backend dies of EMFILE while the desktop around it is fine; **macOS** 16 384 ports / 15 s, so it recovers fastest. BUG-215 was the Windows shape: an unjittered wake storm (one socket per pane, per window, per instance, all on one `visibilitychange`) on top of a standing per-call-client burn. The rule that follows is to cap the burst, never to widen a limit — code that fits the Windows budget fits everywhere | `jarvis/ui/web/frontend/src/lib/connectBudget.ts` (`jitteredDelay`, `spreadDelay`, `requestConnect`), `jarvis/core/http_pool.py` (`HttpClientPool`, `SyncHttpClientPool`, `default_limits`), `jarvis/core/socket_budget.py` (`read_platform_limits`, `take_census`, `SocketBudgetWatchdog`), `jarvis/realtime/local_server/supervisor.py::_port_open` (`SO_LINGER`) | Identical on all three: one shared connect budget in the frontend, pooled keep-alive clients in Python, and a watchdog armed by BOTH entry points (desktop shell and `--headless`) that reads the ceiling per platform — `netsh` on Windows (parsed as integers, because it translates its own field names), `/proc/sys/net/ipv4/ip_local_port_range` plus `RLIMIT_NOFILE` on Linux, `sysctl net.inet.ip.portrange.*` on macOS. A host that answers none of them falls back to the documented default and says the number was assumed; a host with no ceiling at all logs one line and throttles nothing, because a watchdog that cannot see must never be what stops the app working. macOS additionally refuses the system-wide socket table to an unprivileged process, so the census there counts this process only and labels the number a floor. Guards: `tests/contract/test_socket_budget.py` (all three hosts faked, plus a German and a Japanese `netsh`), `tests/unit/core/test_socket_budget_watchdog.py`, `jarvis/ui/web/frontend/src/lib/connectBudget.test.ts` |## Maintenancediff --git a/jarvis/brain/ollama_inventory.py b/jarvis/brain/ollama_inventory.pyindex 5b0bc325f6..9640e22429 100644--- a/jarvis/brain/ollama_inventory.py+++ b/jarvis/brain/ollama_inventory.py@@ -39,11 +39,14 @@ import asyncioimport loggingimport reimport time+from collections.abc import AsyncIterator+from contextlib import asynccontextmanagerfrom dataclasses import dataclassfrom typing import Anyimport httpx+from jarvis.core.http_pool import HttpClientPool, default_limitsfrom jarvis.plugins.brain.ollama import CLIENT_TIMEOUT, normalize_server_rootlog = logging.getLogger(__name__)@@ -187,8 +190,47 @@ def is_hidden_alias(name: str) -> bool:def _make_client(transport: httpx.AsyncBaseTransport | None = None) -> httpx.AsyncClient:- """One client per call; tests inject a fake transport here."""- return httpx.AsyncClient(timeout=CLIENT_TIMEOUT, transport=transport)+ """Build one client. The single construction point; tests replace THIS.++ Called with a transport for a throwaway client, and without one by the+ shared pool below. Both paths land here so a monkeypatched factory reaches+ the pooled client too.+ """+ return httpx.AsyncClient(timeout=CLIENT_TIMEOUT, transport=transport, limits=default_limits())+++#: The pooled client every real sweep shares.+#:+#: A sweep is ``/api/tags`` plus one ``/api/show`` per installed download --+#: sixteen requests on the maintainer's machine. Built per call, that was+#: sixteen fresh TCP connections to Ollama every time, each holding an+#: ephemeral port for two minutes afterwards on Windows, for a panel nobody was+#: looking at. Measured 2026-09-03 while the app sat idle: 32 requests a+#: minute, 421 of the last 549 of them ``/api/show``. Pooled, one sweep rides+#: one connection (BUG-215, AP-33).+_pool = HttpClientPool(timeout_s=CLIENT_TIMEOUT, factory=lambda: _make_client())+++@asynccontextmanager+async def _client(+ transport: httpx.AsyncBaseTransport | None = None,+) -> AsyncIterator[httpx.AsyncClient]:+ """The client for one sweep -- pooled normally, throwaway under a fake.++ A test that injects a transport gets its own client and gets it closed, so+ nothing leaks between tests. Everything else borrows the shared one and+ must NOT close it: the next sweep wants those connections still warm.+ """+ if transport is not None:+ async with _make_client(transport) as client:+ yield client+ return+ yield _pool.client()+++async def aclose_pool() -> None:+ """Close the shared client. Called on shutdown; safe to call repeatedly."""+ await _pool.aclose()def _unreachable(root: str, exc: Exception) -> OllamaServerError:@@ -316,7 +358,7 @@ async def list_models(``/api/show`` only leaves that row without capabilities."""root = normalize_server_root(root)- async with _make_client(transport) as client:+ async with _client(transport) as client:probed, _names = await _sweep(client, root, include_hidden=include_hidden)return probed@@ -364,7 +406,7 @@ async def running_models("""What ``/api/ps`` says is loaded right now (aliases included — they DOoccupy memory, and the caller maps them back to their base by name)."""root = normalize_server_root(root)- async with _make_client(transport) as client:+ async with _client(transport) as client:payload = await _get_json(client, root, "/api/ps")rows = payload.get("models") if isinstance(payload, dict) else Noneif not isinstance(rows, list):@@ -403,7 +445,7 @@ async def unload_model(name = name.strip()if not name:raise OllamaServerError("A model name is required to unload it.")- async with _make_client(transport) as client:+ async with _client(transport) as client:try:resp = await client.post(f"{root}/api/generate", json={"model": name, "keep_alive": 0})except httpx.HTTPError as exc:@@ -430,7 +472,7 @@ async def delete_model(name = name.strip()if not name:raise OllamaServerError("A model name is required to delete it.")- async with _make_client(transport) as client:+ async with _client(transport) as client:try:resp = await client.request("DELETE", f"{root}/api/delete", json={"model": name})except httpx.HTTPError as exc:@@ -456,10 +498,19 @@ async def disk_usage(root: str, *, transport: httpx.AsyncBaseTransport | None =# ── Shared snapshot ──────────────────────────────────────────────────────-#: How long one sweep serves every reader. Five seconds covers one paint of-#: the section (inventory, roles, server and shortlist ask within the same-#: second) without hiding a model that a voice turn just loaded for long.-DEFAULT_SNAPSHOT_MAX_AGE_S = 5.0+#: How long one sweep serves every reader.+#:+#: This has to be LONGER than the interval that polls it, or the cache never+#: hits and every poll is a full sweep. It was five seconds against a fifteen+#: second poll, which is exactly that failure: measured 2026-09-03, an idle app+#: swept Ollama 32 times a minute for a panel nobody had open (BUG-215).+#:+#: A minute is safe because staleness is not what this window guards. The three+#: moments the inventory actually changes because of Jarvis -- an unload, a+#: delete, a finished pull -- all call :func:`invalidate_snapshot`, so the only+#: thing a longer window can hide is a model pulled from an outside terminal,+#: which nobody is waiting on a spinner for.+DEFAULT_SNAPSHOT_MAX_AGE_S = 60.0#: Per-root memo: the last good snapshot, or the last failure (so an offline#: server is not asked five times per paint either — the sentence is the same).@@ -505,10 +556,16 @@ def invalidate_snapshot(root: str) -> None:def _reset_for_tests() -> None:- """Drop every memoised snapshot, failure and lock (tests only)."""+ """Drop every memoised snapshot, failure, lock and client (tests only).++ The client matters as much as the snapshots: a test that replaces+ :func:`_make_client` with a fake gets nowhere if the pool is still holding+ the one the previous test built.+ """_snapshots.clear()_failures.clear()_locks.clear()+ _pool.forget()async def cached_snapshot(@@ -535,7 +592,7 @@ async def cached_snapshot(if failed is not None and time.time() - failed[0] < max_age_s:raise failed[1]try:- async with _make_client(transport) as client:+ async with _client(transport) as client:models, names = await _sweep(client, root, include_hidden=False)try:running = await running_models(root, transport=transport)@@ -577,7 +634,7 @@ async def embed_probe(:class:`OllamaServerError` when the server refused or is unreachable."""root = normalize_server_root(root)- async with _make_client(transport) as client:+ async with _client(transport) as client:try:resp = await client.post(f"{root}/api/embed", json={"model": model, "input": "ping"})except httpx.HTTPError as exc:diff --git a/jarvis/core/http_pool.py b/jarvis/core/http_pool.pynew file mode 100644index 0000000000..ef56f555e8--- /dev/null+++ b/jarvis/core/http_pool.py@@ -0,0 +1,206 @@+"""One HTTP client per caller, kept warm — instead of one per request.++Opening a fresh ``httpx`` client per call costs a new TCP handshake (and a new+TLS handshake, and a fresh ``ssl.create_default_context`` that parses the whole+CA bundle) every single time. On a hot path that is measurable latency; on a+POLLING path it is something worse.++Every connection borrows an ephemeral port from a pool the operating system+owns, and a closed connection keeps that port through TIME_WAIT — 120 seconds+on Windows. A loop that opens one connection per tick and never reuses it turns+a harmless 5-second poll into a permanent standing population of dead sockets,+and several such loops together can empty the pool. When that happens NOTHING+on the machine can connect: not this app, not the file manager, not the+browser. That is BUG-215, and one of the loops behind it was probing+``127.0.0.1/api/health`` every 250 ms with a brand-new client each time.++So: build the client once, let it keep its connections alive, and let the+caller stop thinking about sockets. See :mod:`jarvis.core.socket_budget` for+the watchdog that notices when something still gets this wrong.++Both pools rebind whenever the running event loop changes (async) or the client+is closed, so a cached client is never reused across loops — each+``pytest-asyncio`` test runs in its own loop, and a pool reused across loops+would otherwise raise ``RuntimeError: Event loop is closed``.++Usage::++ from jarvis.core.http_pool import HttpClientPool, SyncHttpClientPool++ self._pool = HttpClientPool(timeout_s=10.0)+ resp = await self._pool.client().get(url)++ self._probe = SyncHttpClientPool(timeout_s=1.0)+ resp = self._probe.client().get("http://127.0.0.1:47821/api/health")+"""++from __future__ import annotations++import threading+from collections.abc import Callable+from typing import TYPE_CHECKING, Any, Final++if TYPE_CHECKING: # pragma: no cover - typing only+ import httpx++#: Keep-alive ceiling for a pooled client. Ten idle connections per client is+#: httpx's own default and plenty for the handful of hosts any one caller talks+#: to; the value is pinned here so the intent survives an upstream change.+#:+#: ``keepalive_expiry`` is the number that matters for the port pool: a+#: connection parked longer than this is closed and its port starts its+#: TIME_WAIT. Thirty seconds comfortably outlives the polling intervals in this+#: app (1-15 s), so a poller reuses one connection forever instead of leaving a+#: corpse behind every tick.+MAX_KEEPALIVE_CONNECTIONS: Final[int] = 10+MAX_CONNECTIONS: Final[int] = 50+KEEPALIVE_EXPIRY_S: Final[float] = 30.0+++def default_limits() -> httpx.Limits:+ """The connection limits every pooled client in this app is built with."""+ import httpx++ return httpx.Limits(+ max_keepalive_connections=MAX_KEEPALIVE_CONNECTIONS,+ max_connections=MAX_CONNECTIONS,+ keepalive_expiry=KEEPALIVE_EXPIRY_S,+ )+++class HttpClientPool:+ """Lazily create and reuse one ``httpx.AsyncClient`` (keep-alive pool).++ Args:+ timeout_s: Request timeout. Defaults to the shared REST budget.+ transport: Forwarded verbatim so tests can inject an+ ``httpx.MockTransport`` exactly as they did with per-request clients.+ client_kwargs: Extra keyword arguments for the client — e.g. the+ ``event_hooks`` from :func:`jarvis.core.http_guard.https_only_async`.+ factory: Builds the client instead of this class, when a module already+ has its own construction point that tests replace. Resolved on+ every build, so a monkeypatched factory is honoured — pair it with+ :meth:`forget` in the test reset so each test builds afresh.+ """++ def __init__(+ self,+ *,+ timeout_s: float | None = None,+ transport: Any | None = None,+ client_kwargs: dict[str, Any] | None = None,+ factory: Callable[[], Any] | None = None,+ ) -> None:+ if timeout_s is None:+ # The shared voice tool budget (5 s): a round trip not back by then+ # cannot make the spoken turn any more — fail it, free the+ # connection, log the cause. A tool that knows better passes its own.+ from jarvis.core.tool_budget import REST_REQUEST_TIMEOUT_S++ timeout_s = REST_REQUEST_TIMEOUT_S+ self._timeout_s = timeout_s+ self._transport = transport+ self._client_kwargs = dict(client_kwargs or {})+ self._factory = factory+ self._client: Any | None = None+ self._loop: Any | None = None++ def client(self) -> Any:+ """Return the pooled client, (re)binding it to the current loop."""+ import asyncio++ loop = asyncio.get_running_loop()+ client = self._client+ if client is None or self._loop is not loop or client.is_closed:+ client = self._build()+ self._client = client+ self._loop = loop+ return client++ def _build(self) -> Any:+ if self._factory is not None:+ return self._factory()+ import httpx++ return httpx.AsyncClient(+ timeout=self._timeout_s,+ transport=self._transport,+ limits=default_limits(),+ **self._client_kwargs,+ )++ def forget(self) -> None:+ """Drop the cached client WITHOUT closing it. Tests only.++ A test reset runs synchronously and cannot await ``aclose``; its+ clients ride a ``MockTransport`` and own no socket, so dropping the+ reference costs nothing. Production shutdown uses :meth:`aclose`.+ """+ self._client = None+ self._loop = None++ async def aclose(self) -> None:+ """Close the pooled client (best-effort; safe to call repeatedly)."""+ client = self._client+ self._client = None+ self._loop = None+ if client is not None and not client.is_closed:+ await client.aclose()+++class SyncHttpClientPool:+ """The same idea for ``httpx.Client``, for probes that are not on a loop.++ The readiness probes this replaces run on plain threads — the desktop+ shell's backend wait, the local-runtime monitor — so they cannot use the+ async pool. Thread-safe: several watchdog threads may share one probe.++ Args:+ timeout_s: Request timeout. Required; a probe's budget is never the+ REST default.+ transport: Injected for tests, as above.+ """++ def __init__(self, *, timeout_s: float, transport: Any | None = None) -> None:+ self._timeout_s = float(timeout_s)+ self._transport = transport+ self._client: Any | None = None+ self._lock = threading.Lock()++ def client(self) -> Any:+ """Return the pooled client, rebuilding it if it was closed."""+ import httpx++ with self._lock:+ client = self._client+ if client is None or client.is_closed:+ client = httpx.Client(+ timeout=self._timeout_s,+ transport=self._transport,+ limits=default_limits(),+ )+ self._client = client+ return client++ def close(self) -> None:+ """Close the pooled client (best-effort; safe to call repeatedly)."""+ with self._lock:+ client = self._client+ self._client = None+ if client is not None and not client.is_closed:+ try:+ client.close()+ except Exception as exc: # noqa: BLE001+ from loguru import logger++ logger.debug("HTTP pool: closing a probe client failed ({}).", exc)+++__all__ = [+ "KEEPALIVE_EXPIRY_S",+ "MAX_CONNECTIONS",+ "MAX_KEEPALIVE_CONNECTIONS",+ "HttpClientPool",+ "SyncHttpClientPool",+ "default_limits",+]diff --git a/jarvis/core/socket_budget.py b/jarvis/core/socket_budget.pynew file mode 100644index 0000000000..e59734154f--- /dev/null+++ b/jarvis/core/socket_budget.py@@ -0,0 +1,563 @@+"""Notice when the machine is running out of sockets — and say who took them.++Every outbound connection borrows an *ephemeral port* from a pool the operating+system owns, and hands it back only after the close lingers through TIME_WAIT.+Drain that pool and nothing on the machine can connect any more: not this app,+not the file manager, not the start menu, not the browser. The user does not+report "the socket pool is empty". They report that the whole computer went+unusable for a few minutes and then fixed itself — because TIME_WAIT expires on+its own and the pool refills with nobody having learned anything.++That is BUG-215, and it had been happening on the maintainer's machine almost+daily since 2026-08-04 without a single line anywhere naming a cause. Windows+logs event 4231 when the pool runs dry, but it throttles that to roughly one an+hour, so the entry rarely lands in the minute the freeze did — and it never+says which process was responsible.++So this watchdog lives OFF the event loop, in a plain daemon thread, and+answers the question the operating system will not: how much of the pool is+gone, and *to where*. When the pool gets tight it writes a census — states,+the busiest destinations, the busiest processes — into the log, which is where+the next person to hit this will actually look.++Three operating systems, three different ceilings, one shared failure:++=========== ================== ========== =====================================+System Ephemeral ports TIME_WAIT What gives out first+=========== ================== ========== =====================================+Windows 16 384 (default) 120 s the port pool — the whole desktop stalls+Linux 28 232 (default) 60 s usually ``RLIMIT_NOFILE`` first, which+ kills the process instead of the desktop+macOS 16 384 (default) 15 s the port pool, but it recovers fastest+=========== ================== ========== =====================================++Windows is the hard case: the smallest pool and the longest linger. Code that+stays inside the budget there is safe everywhere, which is why the thresholds+below are expressed as a FRACTION of whatever the host actually reports rather+than as a fixed number of sockets.++Usage — the app arms it once and forgets it::++ from jarvis.core.socket_budget import SocketBudgetWatchdog++ watchdog = SocketBudgetWatchdog()+ watchdog.start()++Anything that is about to open a connection it could just as well open later+asks first::++ from jarvis.core.socket_budget import should_defer_optional_io++ if should_defer_optional_io():+ return cached_answer # the pool is tight; a poll can wait a tick+"""++from __future__ import annotations++import sys+import threading+import time+from collections import Counter+from collections.abc import Callable+from dataclasses import dataclass, field+from typing import Final++#: How often the pool is counted. The census walks the host's whole socket+#: table, so it is not free; 30 s is often enough to catch a storm that lasts+#: minutes and rare enough to disappear into the noise.+DEFAULT_INTERVAL_S: Final[float] = 30.0++#: Fraction of the pool in use above which the census goes into the log. Two+#: thirds gone is not yet a failure, but it is already far outside anything a+#: healthy idle machine does, and the warning has to arrive while the culprit+#: is still connecting — a report written after exhaustion names TIME_WAIT and+#: nothing else.+DEFAULT_WARN_AT: Final[float] = 0.60++#: Fraction above which optional traffic stands down. Polls, inventory sweeps+#: and readiness probes can all wait a tick; a voice turn cannot. Degrade+#: honestly, never fail hard.+DEFAULT_DEFER_AT: Final[float] = 0.80++#: An ongoing squeeze is re-reported at this interval rather than every tick,+#: so ten minutes of pressure leave a readable trail instead of 20 identical+#: censuses.+DEFAULT_REPEAT_S: Final[float] = 300.0++#: Documented defaults, used when the host refuses to say. Better a slightly+#: wrong ceiling than no watchdog at all — the fraction still moves, and the+#: log line says the number was assumed.+_ASSUMED_PORTS: Final[dict[str, int]] = {"win32": 16384, "darwin": 16384, "linux": 28232}++#: How many rows of each breakdown the census keeps. Enough to name a culprit,+#: short enough that the log line stays readable.+_TOP_N: Final[int] = 5+++@dataclass(frozen=True, slots=True)+class PlatformLimits:+ """What this host says its socket ceiling is.++ Args:+ ephemeral_ports: Size of the outbound port pool, or ``None`` when the+ host would not say.+ fd_limit: Per-process open-file ceiling, or ``None`` on hosts without+ one. On Linux this is routinely the tighter of the two — a stock+ ``python:3.11-slim`` allows 1024 open files, well under the 28 232+ ports it advertises.+ source: Where the numbers came from, quoted verbatim in the log so a+ surprising ceiling can be traced instead of doubted.+ """++ ephemeral_ports: int | None+ fd_limit: int | None+ source: str++ @property+ def ceiling(self) -> int | None:+ """The limit that actually binds: the smaller of the two, if any."""+ candidates = [n for n in (self.ephemeral_ports, self.fd_limit) if n]+ return min(candidates) if candidates else None+++@dataclass(frozen=True, slots=True)+class SocketCensus:+ """One count of the host's sockets, with enough detail to blame someone.++ Args:+ total: Sockets seen, all states.+ lingering: Sockets in TIME_WAIT/CLOSE_WAIT — closed, but still holding+ a port. A pool emptied by churn is almost entirely this.+ by_state: Count per connection state.+ top_destinations: Busiest ``host:port`` targets, most first.+ top_processes: Busiest ``pid`` owners, most first. Empty on a host that+ will not attribute sockets without elevation (macOS).+ partial: True when the host refused the full table and only this+ process's own sockets were counted, so ``total`` is a floor.+ """++ total: int+ lingering: int+ by_state: dict[str, int] = field(default_factory=dict)+ top_destinations: list[tuple[str, int]] = field(default_factory=list)+ top_processes: list[tuple[int, int]] = field(default_factory=list)+ partial: bool = False++ def summary(self) -> str:+ """One multi-line block naming the pressure and its likeliest cause."""+ states = ", ".join(f"{name}={count}" for name, count in sorted(self.by_state.items()))+ destinations = (+ "\n".join(f" {target} x{count}" for target, count in self.top_destinations)+ or " (none)"+ )+ processes = (+ "\n".join(f" pid {pid} x{count}" for pid, count in self.top_processes)+ # A closed socket keeps its port but loses its owner, so the busiest+ # destination is the only handle on a pool emptied by pure churn.+ or " (unattributed — closed sockets keep the port, not the owner)"+ )+ scope = " (own process only — the host refused the full table)" if self.partial else ""+ return (+ f" sockets: {self.total} total, {self.lingering} lingering{scope}\n"+ f" states: {states}\n"+ f" busiest destinations:\n{destinations}\n"+ f" busiest processes:\n{processes}"+ )+++# ----------------------------------------------------------------------+# Reading the host's ceiling+# ----------------------------------------------------------------------+_limits_cache: PlatformLimits | None = None+_limits_lock = threading.Lock()+++def read_platform_limits(*, refresh: bool = False) -> PlatformLimits:+ """The host's socket ceiling, probed once and remembered.++ The pool size does not change while the process runs — and on Windows+ reading it costs a ``netsh`` spawn — so this is cached. ``refresh`` is for+ the tests, which need each fake host read fresh.+ """+ global _limits_cache+ with _limits_lock:+ if _limits_cache is not None and not refresh:+ return _limits_cache+ limits = _probe_platform_limits()+ _limits_cache = limits+ return limits+++def _probe_platform_limits() -> PlatformLimits:+ ports, source = _probe_ephemeral_ports()+ fd_limit = _probe_fd_limit()+ if ports is None:+ assumed = _ASSUMED_PORTS.get(sys.platform)+ if assumed is not None:+ ports = assumed+ source = f"{source}; port pool assumed to be the {sys.platform} default"+ return PlatformLimits(ephemeral_ports=ports, fd_limit=fd_limit, source=source)+++def _probe_ephemeral_ports() -> tuple[int | None, str]:+ """Size of the outbound port pool, per platform. ``None`` when unreadable."""+ if sys.platform == "win32":+ return _windows_port_pool()+ if sys.platform == "darwin":+ return _macos_port_pool()+ if sys.platform.startswith("linux"):+ return _linux_port_pool()+ return None, f"no port-pool probe for {sys.platform}"+++def _windows_port_pool() -> tuple[int | None, str]:+ """Read the dynamic port range out of ``netsh``.++ Parsed as "the first two integers in the output" rather than by label:+ ``netsh`` translates its own field names, so a German host answers+ ``Startport`` / ``Anzahl von Ports`` and a label-matching parser reports no+ limit at all on every non-English Windows in the install base.+ """+ import re+ import subprocess++ from jarvis.core.process_utils import NO_WINDOW_CREATIONFLAGS++ try:+ completed = subprocess.run( # noqa: S603 - fixed argv, no shell, no user input+ ["netsh", "int", "ipv4", "show", "dynamicport", "tcp"],+ capture_output=True,+ text=True,+ encoding="utf-8",+ errors="replace",+ timeout=5.0,+ creationflags=NO_WINDOW_CREATIONFLAGS,+ )+ except Exception as exc: # noqa: BLE001 - a missing/blocked netsh is not fatal+ return None, f"netsh unavailable ({type(exc).__name__})"+ numbers = [int(match) for match in re.findall(r"\d+", completed.stdout)]+ if len(numbers) < 2:+ return None, "netsh answered without a start/count pair"+ # Order is start, count — the same on every localisation.+ return numbers[1], f"netsh dynamicport tcp (start {numbers[0]})"+++def _linux_port_pool() -> tuple[int | None, str]:+ from pathlib import Path++ path = Path("/proc/sys/net/ipv4/ip_local_port_range")+ try:+ low, high = (int(part) for part in path.read_text(encoding="utf-8").split()[:2])+ except Exception as exc: # noqa: BLE001 - a container without /proc is fine+ return None, f"{path} unreadable ({type(exc).__name__})"+ return max(0, high - low + 1), str(path)+++def _macos_port_pool() -> tuple[int | None, str]:+ import subprocess++ try:+ completed = subprocess.run( # noqa: S603 - fixed argv, no shell, no user input+ ["sysctl", "-n", "net.inet.ip.portrange.first", "net.inet.ip.portrange.last"],+ capture_output=True,+ text=True,+ encoding="utf-8",+ errors="replace",+ timeout=5.0,+ )+ low, high = (int(line) for line in completed.stdout.split()[:2])+ except Exception as exc: # noqa: BLE001 - sysctl absent or renamed+ return None, f"sysctl net.inet.ip.portrange unreadable ({type(exc).__name__})"+ return max(0, high - low + 1), "sysctl net.inet.ip.portrange"+++def _probe_fd_limit() -> int | None:+ """The per-process open-file ceiling, where the platform has one.++ Windows has no ``RLIMIT_NOFILE``; its handle ceiling is far above anything+ a socket storm reaches, so there is nothing to report there.+ """+ try:+ import resource+ except ImportError:+ return None # Windows — no such limit, and none worth inventing.+ try:+ soft, _hard = resource.getrlimit(resource.RLIMIT_NOFILE)+ except Exception as exc: # noqa: BLE001 - an exotic host without the limit+ from loguru import logger++ logger.debug("Socket budget: RLIMIT_NOFILE unreadable ({}).", exc)+ return None+ if soft == resource.RLIM_INFINITY:+ return None+ return int(soft)+++# ----------------------------------------------------------------------+# Counting what is in use+# ----------------------------------------------------------------------+#: States that no longer carry traffic but still own their port. These are what+#: a churn storm leaves behind, and what makes the pool refill on its own.+_LINGERING: Final[frozenset[str]] = frozenset({"TIME_WAIT", "CLOSE_WAIT", "FIN_WAIT2", "LAST_ACK"})+++def take_census() -> SocketCensus | None:+ """Count the host's sockets. ``None`` when psutil cannot look at all."""+ try:+ import psutil # lazy (AP-26): nothing on the boot critical path+ except ImportError:+ from loguru import logger++ logger.debug("Socket budget: psutil not installed; the pool goes uncounted.")+ return None++ partial = False+ try:+ connections = psutil.net_connections(kind="inet")+ except (psutil.AccessDenied, PermissionError):+ # macOS refuses the system-wide table to an unprivileged process. Our+ # own sockets are still a real signal — this app is the one whose+ # behaviour we can change — so count those and say the number is a floor.+ partial = True+ try:+ connections = psutil.Process().net_connections(kind="inet")+ except Exception as exc: # noqa: BLE001 - nothing left to fall back to+ from loguru import logger++ logger.debug("Socket budget: no socket table available ({}).", exc)+ return None+ except Exception as exc: # noqa: BLE001 - a census must never take the app down+ from loguru import logger++ logger.debug("Socket budget: census failed ({}).", exc)+ return None++ states: Counter[str] = Counter()+ destinations: Counter[str] = Counter()+ processes: Counter[int] = Counter()+ lingering = 0+ for conn in connections:+ state = getattr(conn, "status", None) or "NONE"+ states[state] += 1+ if state in _LINGERING:+ lingering += 1+ remote = getattr(conn, "raddr", None)+ if remote:+ destinations[f"{remote[0]}:{remote[1]}"] += 1+ pid = getattr(conn, "pid", None)+ if pid:+ processes[int(pid)] += 1+ return SocketCensus(+ total=len(connections),+ lingering=lingering,+ by_state=dict(states),+ top_destinations=destinations.most_common(_TOP_N),+ top_processes=processes.most_common(_TOP_N),+ partial=partial,+ )+++# ----------------------------------------------------------------------+# The pressure everyone else reads+# ----------------------------------------------------------------------+_pressure = 0.0+_pressure_lock = threading.Lock()+++def current_pressure() -> float:+ """Fraction of the socket ceiling in use, 0.0-1.0, as last measured.++ Zero until the first census lands, and zero on a host whose ceiling could+ not be read — a number nobody can compute must never throttle anything.+ """+ with _pressure_lock:+ return _pressure+++def set_pressure_for_tests(value: float) -> None:+ """Force the reading. Tests only; nothing in the app calls this."""+ global _pressure+ with _pressure_lock:+ _pressure = max(0.0, float(value))+++def should_defer_optional_io(*, threshold: float = DEFAULT_DEFER_AT) -> bool:+ """Is the pool too tight for traffic that could just as well wait?++ Asked by pollers, inventory sweeps and readiness probes — never by a voice+ turn, a brain call or anything the user is waiting on. The default answer+ on an unmeasured host is False: a watchdog that cannot see must not be the+ thing that stops the app from working.+ """+ return current_pressure() >= threshold+++class SocketBudgetWatchdog:+ """Sample the socket pool from a thread nothing else can block.++ Args:+ interval_s: Seconds between censuses.+ warn_at: Fraction of the ceiling above which the census is logged.+ repeat_s: How often ongoing pressure is re-reported.+ on_pressure: Receives ``(fraction, census, limits)``. Defaults to a+ loguru warning. Injected so the behaviour is testable without+ asserting on log output.+ census: Injected census function, for the same reason.+ """++ def __init__(+ self,+ *,+ interval_s: float = DEFAULT_INTERVAL_S,+ warn_at: float = DEFAULT_WARN_AT,+ repeat_s: float = DEFAULT_REPEAT_S,+ on_pressure: Callable[[float, SocketCensus, PlatformLimits], None] | None = None,+ census: Callable[[], SocketCensus | None] | None = None,+ ) -> None:+ self._interval_s = max(1.0, float(interval_s))+ self._warn_at = max(0.0, min(1.0, float(warn_at)))+ self._repeat_s = max(self._interval_s, float(repeat_s))+ self._on_pressure = on_pressure or _log_pressure+ self._census = census or take_census+ self._stop = threading.Event()+ self._thread: threading.Thread | None = None+ self._reported_at: float | None = None++ def start(self) -> None:+ """Begin sampling. Idempotent."""+ if self._thread is not None:+ return+ self._stop.clear()+ self._thread = threading.Thread(+ target=self._run,+ name="jarvis-socket-budget",+ daemon=True,+ )+ self._thread.start()++ def stop(self, *, timeout: float = 2.0) -> None:+ """Stop sampling and wait briefly for the thread to unwind."""+ self._stop.set()+ thread = self._thread+ self._thread = None+ if thread is not None and thread.is_alive():+ thread.join(timeout=timeout)++ def tick(self) -> float:+ """One census. Returns the pressure it measured; 0.0 when unmeasurable.++ Separate from the loop so a test can drive a single sample without a+ thread and without waiting out an interval.+ """+ limits = read_platform_limits()+ ceiling = limits.ceiling+ census = self._census()+ if census is None or not ceiling:+ return 0.0+ fraction = census.total / float(ceiling)+ global _pressure+ with _pressure_lock:+ _pressure = fraction+ if fraction < self._warn_at:+ self._reported_at = None+ return fraction+ now = time.monotonic()+ if self._reported_at is not None and now - self._reported_at < self._repeat_s:+ return fraction+ self._reported_at = now+ try:+ self._on_pressure(fraction, census, limits)+ except Exception: # noqa: BLE001, S110 - reporting never kills the watchdog+ pass+ return fraction++ def _run(self) -> None:+ while not self._stop.is_set():+ try:+ self.tick()+ except Exception: # noqa: BLE001 - a watchdog never takes the app down+ from loguru import logger++ logger.opt(exception=True).debug("Socket-budget tick failed.")+ if self._stop.wait(self._interval_s):+ return+++#: The one watchdog per process, so every entry point arms the same thread.+_default: SocketBudgetWatchdog | None = None+_default_lock = threading.Lock()+++def start_default_watchdog() -> SocketBudgetWatchdog | None:+ """Arm the process-wide watchdog. Idempotent; never fatal.++ Called from both entry points, because both can be the process that empties+ the pool: the desktop shell (which drives the windows whose sockets storm)+ and a headless run (which polls exactly the same loops on a server with a+ far tighter file-descriptor ceiling).++ Nothing is measured here — the first census is a full interval away — so+ this stays off the boot critical path (AP-26).+ """+ global _default+ with _default_lock:+ if _default is not None:+ return _default+ try:+ watchdog = SocketBudgetWatchdog()+ watchdog.start()+ except Exception: # noqa: BLE001 — a diagnostic must never break boot+ from loguru import logger++ logger.opt(exception=True).debug("Socket-budget watchdog not armed.")+ return None+ _default = watchdog+ return watchdog+++def stop_default_watchdog() -> None:+ """Retire the process-wide watchdog, if one was armed."""+ global _default+ with _default_lock:+ watchdog = _default+ _default = None+ if watchdog is not None:+ try:+ watchdog.stop()+ except Exception: # noqa: BLE001, S110 — teardown of a diagnostic+ pass+++def _log_pressure(fraction: float, census: SocketCensus, limits: PlatformLimits) -> None:+ """Default reporter: the number, the ceiling it is measured against, the census."""+ from loguru import logger++ logger.warning(+ "Socket pool {:.0%} used ({} of ~{}, per {}). Empty this and NOTHING on "+ "the machine can connect until the closed ones time out, which reads "+ "to the user as the whole computer freezing and then healing itself "+ "(BUG-215). Census:\n{}",+ fraction,+ census.total,+ limits.ceiling,+ limits.source,+ census.summary(),+ )+++__all__ = [+ "DEFAULT_DEFER_AT",+ "DEFAULT_INTERVAL_S",+ "DEFAULT_REPEAT_S",+ "DEFAULT_WARN_AT",+ "PlatformLimits",+ "SocketBudgetWatchdog",+ "SocketCensus",+ "current_pressure",+ "read_platform_limits",+ "set_pressure_for_tests",+ "should_defer_optional_io",+ "start_default_watchdog",+ "stop_default_watchdog",+ "take_census",+]diff --git a/jarvis/plugins/tool/_http_pool.py b/jarvis/plugins/tool/_http_pool.pyindex 2009f7d456..49044aea8a 100644--- a/jarvis/plugins/tool/_http_pool.py+++ b/jarvis/plugins/tool/_http_pool.py@@ -1,61 +1,14 @@"""Shared connection-pool helper for the REST-backed router tools.-Opening a fresh ``httpx.AsyncClient`` per request forces a new TCP + TLS-handshake to the same host on every call. The REST plugins (gmail, vercel)-issue several such calls per user turn and across a session, so that repeated-handshake is a real, avoidable slice of the "plugin call takes ages" latency.-This helper hands back ONE client per tool instance and keeps its connections-warm (HTTP keep-alive) across calls.--The client is rebound whenever the running event loop changes, so a cached-client is never reused across loops — each ``pytest-asyncio`` test runs in its-own loop, and a tool instance reused across loops would otherwise raise-``RuntimeError: Event loop is closed``.+The implementation moved to :mod:`jarvis.core.http_pool` when BUG-215 turned+"a fresh client per request is slow" into "a fresh connection per request can+empty the machine's socket pool" — at which point it stopped being a plugin+convenience and became a core rule. This module stays as the import the REST+plugins (gmail, vercel) already use; there is only one implementation."""from __future__ import annotations-from typing import Any---class HttpClientPool:- """Lazily create and reuse one ``httpx.AsyncClient`` (keep-alive pool).-- ``transport`` is forwarded verbatim so tests can inject an- ``httpx.MockTransport`` exactly as they did with the per-request clients.- """-- def __init__(self, *, timeout_s: float | None = None, transport: Any | None = None) -> None:- if timeout_s is None:- # The shared voice tool budget (5 s): a round trip not back by then- # cannot make the spoken turn any more — fail it, free the- # connection, log the cause. A tool that knows better passes its own.- from jarvis.core.tool_budget import REST_REQUEST_TIMEOUT_S-- timeout_s = REST_REQUEST_TIMEOUT_S- self._timeout_s = timeout_s- self._transport = transport- self._client: Any | None = None- self._loop: Any | None = None-- def client(self) -> Any:- """Return the pooled client, (re)binding it to the current loop."""- import asyncio-- import httpx-- loop = asyncio.get_running_loop()- client = self._client- if client is None or self._loop is not loop or client.is_closed:- client = httpx.AsyncClient(timeout=self._timeout_s, transport=self._transport)- self._client = client- self._loop = loop- return client+from jarvis.core.http_pool import HttpClientPool- async def aclose(self) -> None:- """Close the pooled client (best-effort; safe to call repeatedly)."""- client = self._client- self._client = None- self._loop = None- if client is not None and not client.is_closed:- await client.aclose()+__all__ = ["HttpClientPool"]diff --git a/jarvis/realtime/local_server/supervisor.py b/jarvis/realtime/local_server/supervisor.pyindex 1ebb8bc8d3..cd1ec07f08 100644--- a/jarvis/realtime/local_server/supervisor.py+++ b/jarvis/realtime/local_server/supervisor.py@@ -32,6 +32,7 @@ import loggingimport osimport reimport socket+import structimport subprocessimport threadingimport time@@ -86,6 +87,11 @@ RUNTIME_MONITOR_UNREADY_GRACE_S = 30.0_MAX_POOL_RESPONSE_BYTES = 64 * 1024+#: Fallback timeout on the shared probe client. Every caller passes its own+#: per-request timeout (0.25 s for a decision, 0.75 s for the monitor); this is+#: only what the client is built with.+RUNTIME_PROBE_TIMEOUT_S = 0.75+#: Rotate the shared server log once it passes this size. Large enough that a#: whole boot transcript plus a long session's noise stays in one file for the#: crash tail, small enough that an always-on host cannot silently fill a disk.@@ -252,8 +258,19 @@ def _is_loopback(host: str) -> bool:def _port_open(port: int, timeout: float = 1.0) -> bool:+ """Is anything accepting on this loopback port right now?++ Closed with ``SO_LINGER = 0`` on purpose, which makes the kernel send RST+ instead of FIN and skip TIME_WAIT entirely. An ordinary close would park+ the ephemeral port for two minutes on Windows, and this probe runs on every+ status poll and every spawn decision — enough of them together drained the+ machine's port pool and froze the whole desktop (BUG-215, AP-33). Nothing+ here wants a graceful shutdown; the question was only whether someone+ accepts, and the answer arrives before the handshake is even finished.+ """try:- with socket.create_connection(("127.0.0.1", port), timeout=timeout):+ with socket.create_connection(("127.0.0.1", port), timeout=timeout) as sock:+ sock.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))return Trueexcept OSError:# Connection refused/timed out — the port simply is not open yet.@@ -282,6 +299,37 @@ def _pool_url(base_url: str) -> str:return urlunsplit((parsed.scheme, netloc, f"{path}/pool", "", ""))+#: One keep-alive client for every readiness probe this module makes.+#:+#: ``_runtime_monitor`` probes every 5 s for as long as a local voice server+#: lives, and ``wait_until_ready`` probes twice a second while one boots. Built+#: per call, each of those left a dead loopback socket holding an ephemeral+#: port for two minutes (Windows), and enough such loops together emptied the+#: machine's port pool — at which point nothing on the computer can connect+#: (BUG-215, AP-33). Pooled, the whole monitor costs one connection.+_probe_pool: Any | None = None+_probe_pool_lock = threading.Lock()+++def _probe_client() -> Any | None:+ """The shared probe client, built once. ``None`` when httpx is absent."""+ global _probe_pool+ with _probe_pool_lock:+ if _probe_pool is None:+ try:+ from jarvis.core.http_pool import SyncHttpClientPool+ except Exception as exc: # noqa: BLE001 — a build without httpx+ log.debug("local-realtime probe: httpx unavailable (%s)", exc)+ return None+ _probe_pool = SyncHttpClientPool(timeout_s=RUNTIME_PROBE_TIMEOUT_S)+ pool = _probe_pool+ try:+ return pool.client()+ except Exception as exc: # noqa: BLE001 — httpx missing at call time+ log.debug("local-realtime probe: no client (%s)", exc)+ return None++def probe_runtime(base_url: str, timeout: float = 0.75) -> dict[str, int] | None:"""Return a sanitized live pool snapshot, or ``None`` when not ready.@@ -295,33 +343,37 @@ def probe_runtime(base_url: str, timeout: float = 0.75) -> dict[str, int] | Nonehost, _port = _host_port(base_url)if parsed.scheme not in {"http", "https"} or not _is_loopback(host):return None- import http.client-url = _pool_url(base_url)- target = urlsplit(url)- target_host = target.hostname- if target_host is None:+ if urlsplit(url).hostname is None:return None- connection_type = (- http.client.HTTPSConnection if target.scheme == "https" else http.client.HTTPConnection- )- connection = connection_type(- target_host,- target.port,- timeout=max(0.05, timeout),- )- try:- connection.request("GET", target.path, headers={"Accept": "application/json"})- response = connection.getresponse()- if response.status != 200:- return None- raw = response.read(_MAX_POOL_RESPONSE_BYTES + 1)- except (http.client.HTTPException, OSError, TimeoutError, ValueError):- # Reported through the documented "None means not ready" return contract above.+ client = _probe_client()+ if client is None:return None- finally:- connection.close()- if len(raw) > _MAX_POOL_RESPONSE_BYTES:+ try:+ # Streamed so the response size stays capped: the body is read in+ # chunks and abandoned the moment it exceeds the cap, instead of+ # trusting the far end to be small.+ with client.stream(+ "GET",+ url,+ headers={"Accept": "application/json"},+ timeout=max(0.05, timeout),+ ) as response:+ if response.status_code != 200:+ return None+ chunks: list[bytes] = []+ size = 0+ for chunk in response.iter_bytes():+ size += len(chunk)+ if size > _MAX_POOL_RESPONSE_BYTES:+ return None+ chunks.append(chunk)+ raw = b"".join(chunks)+ except Exception: # noqa: BLE001+ # Reported through the documented "None means not ready" return contract+ # above — a refused, slow or malformed answer all mean the same thing+ # to every caller, and this runs on a 5 s monitor loop where logging+ # each miss would bury the log.return Nonetry:payload = json.loads(raw.decode("utf-8"))diff --git a/jarvis/ui/desktop_app.py b/jarvis/ui/desktop_app.pyindex 2f65e96450..e147e7d77c 100644--- a/jarvis/ui/desktop_app.py+++ b/jarvis/ui/desktop_app.py@@ -2779,6 +2779,13 @@ class DesktopApp:# and a sampling profiler attached to the live process was the only# way to learn it was a TOML parse). Costs one sleeping thread.self._start_loop_watchdog(loop)+ # And watch the OTHER shared resource a stall can be blamed on.+ # A loop can be perfectly healthy while the machine has run out of+ # ephemeral ports, at which point nothing anywhere can connect and+ # the user reports the whole computer freezing for a few minutes+ # and then healing itself. Off-loop, one census every 30 s, and it+ # names the destination the ports went to (BUG-215).+ self._start_socket_budget_watchdog()self._backend_serving_since = time.monotonic()# run_forever() has exactly one legitimate way out: shutdown()# calling loop.stop(). Every OTHER way out — a BaseException from@@ -2801,6 +2808,7 @@ class DesktopApp:self._note_backend_stopped("the loop was stopped")finally:self._stop_loop_watchdog()+ self._stop_socket_budget_watchdog()try:loop.close()except Exception: # noqa: BLE001, S110@@ -2845,6 +2853,20 @@ class DesktopApp:except Exception: # noqa: BLE001, S110 — teardown of a diagnosticpass+ # ---- Socket-budget watchdog ---------------------------------------------++ def _start_socket_budget_watchdog(self) -> None:+ """Arm the ephemeral-port watchdog. Never fatal — it is a diagnostic."""+ from jarvis.core.socket_budget import start_default_watchdog++ start_default_watchdog()++ def _stop_socket_budget_watchdog(self) -> None:+ """Retire it with the backend."""+ from jarvis.core.socket_budget import stop_default_watchdog++ stop_default_watchdog()+# ---- Backend death -------------------------------------------------------def _note_backend_stopped(self, reason: str) -> None:@@ -4834,8 +4856,6 @@ class DesktopApp:alive the wait continues past ``timeout_s`` (with one log line) up to``hard_timeout_s``; only a dead thread or the hard cap gives up."""- import httpx-url = f"http://127.0.0.1:{self.cfg.ui.admin_api_port}/api/health"start = time.monotonic()slow_boot_logged = False@@ -4858,7 +4878,10 @@ class DesktopApp:hard_timeout_s,)try:- r = httpx.get(url, timeout=0.5)+ client = self._health_probe_client()+ if client is None:+ return False+ r = client.get(url, timeout=0.5)if r.status_code == 200:return Trueexcept Exception: # noqa: BLE001, S110@@ -4881,12 +4904,19 @@ class DesktopApp:was **54 % of all backend CPU time**, by a wide margin the singlehottest thing the process did while idle. Built once and reused, it ispaid once per run.++ Since BUG-215 the CPU is the smaller half of the argument. A client per+ call is also a TCP connection per call, and a closed loopback+ connection holds its ephemeral port for 120 s on Windows. The boot wait+ below polls four times a second; unpooled, that alone parks ~480 dead+ sockets in a pool of 16 384 for as long as the boot takes. Pooled, it+ holds exactly one."""existing = getattr(self, "_health_http", None)if existing is not None:return existingtry:- import httpx+ from jarvis.core.http_pool import SyncHttpClientPoolexcept Exception as exc: # noqa: BLE001from loguru import logger@@ -4894,7 +4924,13 @@ class DesktopApp:return None# Annotated loosely on purpose: httpx is an optional import here, so# the attribute must also be allowed to go back to None on shutdown.- client: Any = httpx.Client(timeout=1.0)+ try:+ client: Any = SyncHttpClientPool(timeout_s=1.0).client()+ except Exception as exc: # noqa: BLE001+ from loguru import logger++ logger.debug("Health probe: httpx unavailable ({})", exc)+ return Noneself._health_http = client_silence_health_probe_log()return clientdiff --git a/jarvis/ui/web/frontend/src/components/agentic/paneSocket.test.ts b/jarvis/ui/web/frontend/src/components/agentic/paneSocket.test.tsindex be7afc7bb2..cd69af3302 100644--- a/jarvis/ui/web/frontend/src/components/agentic/paneSocket.test.ts+++ b/jarvis/ui/web/frontend/src/components/agentic/paneSocket.test.ts@@ -1,5 +1,6 @@import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";import { openPaneSocket } from "./paneSocket";+import { MIN_BACKOFF_MS, resetConnectBudgetForTests } from "../../lib/connectBudget";/** Minimal WebSocket stand-in, mirroring the one in __tests__/ws.test.ts. */class MockWebSocket {@@ -78,6 +79,14 @@ describe("openPaneSocket", () => {MockWebSocket.last = null;MockWebSocket.opened = [];vi.useFakeTimers();+ // Reconnect delays are jittered and paid for out of a shared per-tab+ // budget (BUG-215, AP-33). Neither is what these tests are about — the+ // spread has its own tests in `lib/connectBudget.test.ts` — so pin the+ // draw to the middle of its range and start each test with a full bucket.+ // A pinned 0.5 makes `spreadDelay(d)` exactly `d`, which is what the+ // "every half minute" assertions below mean by half a minute.+ vi.spyOn(Math, "random").mockReturnValue(0.5);+ resetConnectBudgetForTests();});afterEach(() => {@@ -278,7 +287,7 @@ describe("openPaneSocket", () => {socket.close();});- it("retries at once when the window comes back, without waiting out the clamp", async () => {+ it("retries promptly when the window comes back, without waiting out the clamp", async () => {/** The live 2026-07-27 20:11 failure. A hidden document has its timers* clamped to about once a second, and to once a minute after a few minutes@@ -287,8 +296,16 @@ describe("openPaneSocket", () => {* opening, and surfaced their agents around two minutes later. The prompts* had gone out on time; only the screen was late.*- * Being looked at ends the wait. No timer is advanced in this test on- * purpose: that is the whole point — the retry must not depend on one.+ * Being looked at ends that wait: the pane abandons the stretched timer+ * and comes back inside the fast window instead of minutes later.+ *+ * It must NOT come back in the same tick, and that is pinned below.+ * `visibilitychange` fires for every pane of every window simultaneously,+ * so reconnecting straight out of the handler put hundreds of sockets on+ * the wire in one millisecond and emptied the machine's ephemeral-port+ * pool — at which point nothing on the computer could connect for two+ * minutes (BUG-215, AP-33). A few hundred milliseconds is invisible to+ * the user; the freeze was not.*/const cb = handlers();const socket = openPaneSocket({ name: "Mika", cols: 80, rows: 24 }, cb);@@ -304,6 +321,11 @@ describe("openPaneSocket", () => {hidden(false);document.dispatchEvent(new Event("visibilitychange"));+ // Not in the handler's own tick — that is the storm.+ expect(MockWebSocket.opened.length).toBe(spent);++ // But well inside the fast window, not out at the clamped delay.+ await vi.advanceTimersByTimeAsync(MIN_BACKOFF_MS);expect(MockWebSocket.opened.length).toBe(spent + 1);MockWebSocket.last!.fire("open");@@ -405,6 +427,14 @@ describe("openPaneSocket delivery receipts", () => {MockWebSocket.last = null;MockWebSocket.opened = [];vi.useFakeTimers();+ // Reconnect delays are jittered and paid for out of a shared per-tab+ // budget (BUG-215, AP-33). Neither is what these tests are about — the+ // spread has its own tests in `lib/connectBudget.test.ts` — so pin the+ // draw to the middle of its range and start each test with a full bucket.+ // A pinned 0.5 makes `spreadDelay(d)` exactly `d`, which is what the+ // "every half minute" assertions below mean by half a minute.+ vi.spyOn(Math, "random").mockReturnValue(0.5);+ resetConnectBudgetForTests();});afterEach(() => {@@ -570,6 +600,14 @@ describe("openPaneSocket geometry reconciliation", () => {MockWebSocket.last = null;MockWebSocket.opened = [];vi.useFakeTimers();+ // Reconnect delays are jittered and paid for out of a shared per-tab+ // budget (BUG-215, AP-33). Neither is what these tests are about — the+ // spread has its own tests in `lib/connectBudget.test.ts` — so pin the+ // draw to the middle of its range and start each test with a full bucket.+ // A pinned 0.5 makes `spreadDelay(d)` exactly `d`, which is what the+ // "every half minute" assertions below mean by half a minute.+ vi.spyOn(Math, "random").mockReturnValue(0.5);+ resetConnectBudgetForTests();});afterEach(() => {diff --git a/jarvis/ui/web/frontend/src/components/agentic/paneSocket.ts b/jarvis/ui/web/frontend/src/components/agentic/paneSocket.tsindex e06cf540b1..11c086ce43 100644--- a/jarvis/ui/web/frontend/src/components/agentic/paneSocket.ts+++ b/jarvis/ui/web/frontend/src/components/agentic/paneSocket.ts@@ -48,6 +48,11 @@*/import { mintWsTicket } from "@/lib/ws";+import {+ jitteredDelay,+ requestConnect,+ spreadDelay,+} from "../../lib/connectBudget";/** Server close code: the handshake credential was missing or invalid. */const CLOSE_UNAUTHORIZED = 4401;@@ -275,7 +280,8 @@ export function openPaneSocket(handlers: PaneSocketHandlers,): PaneSocket {let ws: WebSocket | null = null;- let timer: ReturnType<typeof setTimeout> | null = null;+ /** Cancels the reconnect waiting in the shared connect budget. */+ let cancelPending: (() => void) | null = null;/** Set once nothing further should happen on this pane, for any reason. */let stopped = false;/** The agent reported its own exit — a later close is not a lost wire. */@@ -297,14 +303,29 @@ export function openPaneSocket(*/let serverReason = "";+ /**+ * The one place a pane reopens its socket, and the one that pays the budget.+ *+ * Every delay handed in here is already jittered by its caller: a workspace+ * of twenty panes settles into the same schedule at the same instant, so an+ * exact delay means twenty sockets in the same millisecond. On top of that,+ * `requestConnect` caps how many any client may open per second — the panes+ * of one window are only ever a fraction of what wakes at once (BUG-215).+ */const schedule = (delay: number) => {- if (timer !== null || stopped) return;- timer = setTimeout(() => {- timer = null;+ if (cancelPending !== null || stopped) return;+ cancelPending = requestConnect(() => {+ cancelPending = null;if (!stopped) open();}, delay);};+ /** Drop a reconnect waiting in the budget, if there is one. */+ const unschedule = () => {+ cancelPending?.();+ cancelPending = null;+ };+/*** The window came back — stop waiting and try now.*@@ -320,12 +341,17 @@ export function openPaneSocket(*/const onDocumentVisible = () => {if (stopped || typeof document === "undefined" || document.hidden) return;- if (timer === null) return;- clearTimeout(timer);- timer = null;+ if (cancelPending === null) return;+ unschedule();attempts = 0;waits = 0;- open();+ // Not `open()`. This handler fires for every pane of every window in the+ // same millisecond, and calling straight through here is what turned+ // coming back to the machine into a connection storm big enough to empty+ // the OS socket pool. Re-queue at the fast end instead: the pane is live+ // again within a few hundred milliseconds, and the budget keeps the total+ // survivable (BUG-215, AP-33).+ schedule(jitteredDelay(0, MIN_BACKOFF, MIN_BACKOFF));};if (typeof document !== "undefined") {document.addEventListener("visibilitychange", onDocumentVisible);@@ -344,11 +370,11 @@ export function openPaneSocket(// keeps knocking every half minute, so a backend that comes back finds// its terminals waiting rather than needing every one of them reopened.handlers.onTrouble(message, false);- schedule(IDLE_BACKOFF);+ schedule(spreadDelay(IDLE_BACKOFF));return;}handlers.onTrouble(message, true);- schedule(Math.min(MAX_BACKOFF, MIN_BACKOFF * 2 ** (attempts - 1)));+ schedule(jitteredDelay(attempts - 1, MIN_BACKOFF, MAX_BACKOFF));};const authRetry = async (): Promise<void> => {@@ -362,7 +388,7 @@ export function openPaneSocket(handlers.onTrouble("Authorizing the terminal…", true);// Fixed short delay: the very next attempt carries a fresh credential,// so escalating the backoff would only postpone a working terminal.- schedule(MIN_BACKOFF);+ schedule(jitteredDelay(0, MIN_BACKOFF, MIN_BACKOFF));return;}retryLater("Terminal authorization failed — retrying.");@@ -417,8 +443,8 @@ export function openPaneSocket(if (waits === MAX_ATTEMPTS + 1) askForFreshState("waited-out");schedule(waits > MAX_ATTEMPTS- ? IDLE_BACKOFF- : Math.min(MAX_BACKOFF, MIN_BACKOFF * 2 ** (waits - 1)),+ ? spreadDelay(IDLE_BACKOFF)+ : jitteredDelay(waits - 1, MIN_BACKOFF, MAX_BACKOFF),);return;}@@ -577,10 +603,7 @@ export function openPaneSocket(if (typeof document !== "undefined") {document.removeEventListener("visibilitychange", onDocumentVisible);}- if (timer !== null) {- clearTimeout(timer);- timer = null;- }+ unschedule();try {ws?.close();} catch {diff --git a/jarvis/ui/web/frontend/src/lib/connectBudget.test.ts b/jarvis/ui/web/frontend/src/lib/connectBudget.test.tsnew file mode 100644index 0000000000..3bd44f2238--- /dev/null+++ b/jarvis/ui/web/frontend/src/lib/connectBudget.test.ts@@ -0,0 +1,182 @@+/**+ * The connect budget is the thing standing between a wake and a frozen PC.+ *+ * BUG-215: coming back to the machine after a while made every socket client+ * in every window reconnect in the same millisecond, which emptied the+ * operating system's ephemeral-port pool and stopped EVERYTHING on the+ * computer from connecting for a couple of minutes.+ *+ * Two properties have to hold, and only the pair of them is enough:+ *+ * 1. Retries are jittered, so clients that started together do not stay+ * together.+ * 2. A hard ceiling on connections per second, so that even when they do+ * collide the total stays survivable.+ */++import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";++import {+ BURST,+ CONNECTS_PER_SECOND,+ jitteredDelay,+ requestConnect,+ resetConnectBudgetForTests,+ spreadDelay,+} from "./connectBudget";++beforeEach(() => {+ vi.useFakeTimers();+ resetConnectBudgetForTests();+});++afterEach(() => {+ resetConnectBudgetForTests();+ vi.useRealTimers();+});++describe("jitteredDelay", () => {+ it("picks from the whole range below the exponential ceiling", () => {+ // Full jitter, not half: the low end must be reachable, or clients keep a+ // shared floor and so keep a (thinner) grid.+ expect(jitteredDelay(0, 500, 10_000, () => 0)).toBe(0);+ expect(jitteredDelay(0, 500, 10_000, () => 0.999)).toBe(499);+ });++ it("doubles the ceiling per attempt, up to the cap", () => {+ const top = (attempt: number) => jitteredDelay(attempt, 500, 10_000, () => 0.999999);+ expect(top(0)).toBe(499);+ expect(top(1)).toBe(999);+ expect(top(2)).toBe(1999);+ expect(top(20)).toBe(9999);+ });++ it("survives an absurd attempt count instead of returning NaN", () => {+ // 2 ** 1024 is Infinity and Infinity * base is NaN — a NaN delay makes+ // setTimeout fire immediately, which is the storm this file prevents.+ const delay = jitteredDelay(5000, 500, 10_000, () => 0.5);+ expect(Number.isFinite(delay)).toBe(true);+ expect(delay).toBeLessThanOrEqual(10_000);+ });++ it("treats a negative attempt as the first one", () => {+ expect(jitteredDelay(-3, 500, 10_000, () => 0.999)).toBe(499);+ });+});++describe("spreadDelay", () => {+ it("keeps the average interval and destroys the phase", () => {+ // A slow knock must stay slow. Full jitter would halve it.+ expect(spreadDelay(30_000, () => 0)).toBe(15_000);+ expect(spreadDelay(30_000, () => 0.5)).toBe(30_000);+ expect(spreadDelay(30_000, () => 0.999999)).toBe(44_999);+ });+});++describe("requestConnect", () => {+ it("lets an ordinary page load through without pacing it", () => {+ const opened: number[] = [];+ for (let i = 0; i < BURST; i += 1) {+ requestConnect(() => opened.push(i), 0);+ }++ vi.advanceTimersByTime(0);++ expect(opened).toHaveLength(BURST);+ });++ it("caps a wake storm at the budget instead of letting it through", () => {+ // The BUG-215 shape: twenty clients, no delay, all in the same tick.+ const opened: number[] = [];+ for (let i = 0; i < 20; i += 1) {+ requestConnect(() => opened.push(i), 0);+ }++ vi.advanceTimersByTime(0);+ expect(opened).toHaveLength(BURST);++ // The rest arrive at the refill rate, not all at once.+ vi.advanceTimersByTime(1000);+ expect(opened.length).toBeLessThanOrEqual(BURST + CONNECTS_PER_SECOND + 1);++ // ...and none of them is dropped: the budget delays, it never refuses.+ vi.advanceTimersByTime(5000);+ expect(opened).toHaveLength(20);+ });++ it("preserves the order clients asked in", () => {+ const opened: number[] = [];+ for (let i = 0; i < 20; i += 1) {+ requestConnect(() => opened.push(i), 0);+ }++ vi.advanceTimersByTime(10_000);++ expect(opened).toEqual([...Array(20).keys()]);+ });++ it("cancels a pending connect", () => {+ const opened: string[] = [];+ const cancel = requestConnect(() => opened.push("a"), 1000);+ requestConnect(() => opened.push("b"), 1000);++ cancel();+ vi.advanceTimersByTime(5000);++ expect(opened).toEqual(["b"]);+ });++ it("cancels a connect already queued behind the budget", () => {+ const opened: number[] = [];+ const cancels: Array<() => void> = [];+ for (let i = 0; i < 20; i += 1) {+ cancels.push(requestConnect(() => opened.push(i), 0));+ }+ vi.advanceTimersByTime(0);+ // Everything past the burst is waiting for a token; drop it all.+ for (const cancel of cancels) cancel();++ vi.advanceTimersByTime(10_000);++ expect(opened).toHaveLength(BURST);+ });++ it("survives a client that throws on connect", () => {+ // One broken client must not stall every socket behind it in the queue.+ const opened: string[] = [];+ requestConnect(() => {+ throw new Error("WebSocket constructor blew up");+ }, 0);+ requestConnect(() => opened.push("after"), 0);++ vi.advanceTimersByTime(0);++ expect(opened).toEqual(["after"]);+ });++ it("cancelling twice is safe", () => {+ const cancel = requestConnect(() => undefined, 1000);+ cancel();+ expect(() => cancel()).not.toThrow();+ });++ it("jittered retries from one wake do not land in the same millisecond", () => {+ // The end-to-end property: twenty panes wake together, each schedules its+ // own retry through `jitteredDelay`, and the arrivals spread out.+ const arrivals: number[] = [];+ let seed = 0;+ const random = () => {+ seed += 1;+ return (seed % 17) / 17;+ };+ for (let i = 0; i < 20; i += 1) {+ const delay = jitteredDelay(0, 500, 10_000, random);+ requestConnect(() => arrivals.push(delay), delay);+ }++ vi.advanceTimersByTime(10_000);++ expect(arrivals).toHaveLength(20);+ expect(new Set(arrivals).size).toBeGreaterThan(10);+ });+});diff --git a/jarvis/ui/web/frontend/src/lib/connectBudget.ts b/jarvis/ui/web/frontend/src/lib/connectBudget.tsnew file mode 100644index 0000000000..567ac83b92--- /dev/null+++ b/jarvis/ui/web/frontend/src/lib/connectBudget.ts@@ -0,0 +1,210 @@+/**+ * One shared speed limit for every socket this tab opens.+ *+ * Every connection borrows an ephemeral port from a pool the operating system+ * owns, and a closed connection keeps that port through TIME_WAIT — 120+ * seconds on Windows, against a pool of 16 384. Empty that pool and NOTHING on+ * the machine can connect any more: not this app, not the file manager, not+ * the start menu, not the browser. The user does not report "the socket pool+ * is empty"; they report that the whole computer went unusable for a few+ * minutes after they came back to it, and then fixed itself. That is BUG-215.+ *+ * The shape of the storm that empties it:+ *+ * 1. Nobody touches the machine for a while. Windows are hidden, timers are+ * clamped, and the open sockets quietly die without a `close` event — the+ * peer is gone but `readyState` still says OPEN.+ * 2. The user comes back. `visibilitychange` and `online` fire in the same+ * millisecond in every window.+ * 3. Every socket client reacts at once and with no delay: the app socket, one+ * socket per terminal pane, the agent chat, the realtime transport —+ * multiplied by the number of panes, the number of windows, and the number+ * of running app instances.+ * 4. Whatever fails retries on a fixed grid, so the second wave lands together+ * too, and the third.+ *+ * Each piece of that is individually reasonable. Together they are a burst of+ * hundreds of connections with nothing anywhere that can see the total, which+ * is the point: no single client can fix this, because no single client is the+ * problem. So the limit lives here, above all of them.+ *+ * Two mechanisms, and both are needed:+ *+ * - {@link jitteredDelay} spreads retries that would otherwise stay in+ * lockstep. Exponential backoff alone does not help when twenty clients+ * start their backoff in the same millisecond — they just collide at 0.5 s,+ * then at 1 s, then at 2 s. Full jitter picks uniformly from `[0, delay]`,+ * which is what actually breaks the grid. (Measured in this app on+ * 2026-07-27: five panes asked for 0.5/1/2/4 s and knocked at 10.174,+ * 11.174, 12.162, 13.170 — a flat one-second grid. See `paneSocket.ts`.)+ *+ * - {@link requestConnect} is a token bucket every client passes through.+ * Jitter alone still allows twenty connections inside one second; the bucket+ * is the hard ceiling that does not care how many callers there are.+ *+ * Nothing here is a queue you can starve: a caller that waits for a token+ * keeps its place and fires as soon as one frees. The budget delays+ * connections, it never drops them.+ */++/**+ * New connections per second, across every client in this tab.+ *+ * Six is far above what normal use needs — a window opens its app socket, its+ * panes and its transports once — and far below the hundreds a wake storm+ * produced. A reconnect that has to wait for a token is late by milliseconds,+ * which nobody can perceive; the freeze it prevents lasted minutes.+ */+export const CONNECTS_PER_SECOND = 6;++/** Burst allowance, so an ordinary page load is not paced at all. */+export const BURST = 8;++/** Default floor for a first retry, in ms. */+export const MIN_BACKOFF_MS = 500;++/** Default ceiling for a retry wait, in ms. */+export const MAX_BACKOFF_MS = 10_000;++/**+ * Full jitter: a uniform pick from `[0, min(cap, base * 2 ** attempt)]`.+ *+ * "Full" rather than "equal" or "decorrelated" on purpose. The failure here is+ * many clients starting together, and only full jitter puts them at genuinely+ * independent points; the half-range variants keep a floor that they all share+ * and so keep a (thinner) grid.+ *+ * @param attempt Zero-based retry number.+ * @param base Delay for attempt 0, before jitter.+ * @param cap Ceiling on the pre-jitter delay.+ * @param random Injected for tests; defaults to `Math.random`.+ */+export function jitteredDelay(+ attempt: number,+ base: number = MIN_BACKOFF_MS,+ cap: number = MAX_BACKOFF_MS,+ random: () => number = Math.random,+): number {+ const exponent = Math.max(0, Math.floor(attempt));+ // 2 ** 1024 is Infinity, and Infinity * base is NaN — clamp before the shift+ // so a client that somehow reached attempt 2000 still gets `cap`.+ const grown = exponent > 40 ? cap : base * 2 ** exponent;+ const ceiling = Math.min(cap, grown);+ return Math.floor(random() * ceiling);+}++/**+ * Spread a steady interval around itself: a uniform pick from `[d/2, 3d/2]`.+ *+ * For a repeating knock rather than a backoff. Twenty panes that all settled+ * into "try again every 30 s" knock together forever otherwise, because they+ * settled together — the interval is right, the phase is the problem. Full+ * jitter would be wrong here: it halves the average interval and makes a slow+ * knock twice as fast. This keeps the mean and destroys the phase.+ *+ * @param delayMs The interval to keep on average.+ * @param random Injected for tests; defaults to `Math.random`.+ */+export function spreadDelay(delayMs: number, random: () => number = Math.random): number {+ return Math.floor(delayMs / 2 + random() * delayMs);+}++type Waiter = { readonly run: () => void; cancelled: boolean };++/**+ * The token bucket, module-scoped so every importer shares one.+ *+ * Refills continuously rather than on a timer: tokens are computed from the+ * time since the last take, so a tab whose timers were clamped while hidden+ * comes back with a full bucket instead of an empty one.+ */+let tokens = BURST;+let lastRefillAt = now();+const waiting: Waiter[] = [];+let drainTimer: ReturnType<typeof setTimeout> | null = null;++function now(): number {+ return typeof performance !== "undefined" && typeof performance.now === "function"+ ? performance.now()+ : Date.now();+}++function refill(): void {+ const at = now();+ const elapsedS = Math.max(0, (at - lastRefillAt) / 1000);+ if (elapsedS <= 0) return;+ tokens = Math.min(BURST, tokens + elapsedS * CONNECTS_PER_SECOND);+ lastRefillAt = at;+}++/** ms until the bucket holds at least one token. */+function msUntilToken(): number {+ refill();+ if (tokens >= 1) return 0;+ return Math.ceil(((1 - tokens) / CONNECTS_PER_SECOND) * 1000);+}++function drain(): void {+ drainTimer = null;+ while (waiting.length > 0) {+ const head = waiting[0];+ if (head.cancelled) {+ waiting.shift();+ continue;+ }+ const wait = msUntilToken();+ if (wait > 0) {+ drainTimer = setTimeout(drain, wait);+ return;+ }+ waiting.shift();+ tokens -= 1;+ try {+ head.run();+ } catch {+ // A caller that throws on connect must not stall everyone behind it. The+ // client owns its own error reporting; the budget only owes it a turn.+ }+ }+}++/**+ * Ask to connect after `delayMs`, then as soon as the shared budget allows.+ *+ * @param open What to run when the turn comes. Should start ONE connection.+ * @param delayMs How long to wait first — pass {@link jitteredDelay} for a+ * retry, or 0 for a first attempt (which the burst allowance absorbs).+ * @returns A cancel function. Safe to call more than once, and safe to call+ * after `open` has already run.+ */+export function requestConnect(open: () => void, delayMs = 0): () => void {+ const waiter: Waiter = { run: open, cancelled: false };+ let timer: ReturnType<typeof setTimeout> | null = setTimeout(+ () => {+ timer = null;+ if (waiter.cancelled) return;+ waiting.push(waiter);+ if (drainTimer === null) drain();+ },+ Math.max(0, delayMs),+ );+ return () => {+ waiter.cancelled = true;+ if (timer !== null) {+ clearTimeout(timer);+ timer = null;+ }+ };+}++/** Drop every pending request and refill the bucket. Tests only. */+export function resetConnectBudgetForTests(): void {+ for (const waiter of waiting) waiter.cancelled = true;+ waiting.length = 0;+ if (drainTimer !== null) {+ clearTimeout(drainTimer);+ drainTimer = null;+ }+ tokens = BURST;+ lastRefillAt = now();+}diff --git a/jarvis/ui/web/frontend/src/lib/realtimeTransportBroker.ts b/jarvis/ui/web/frontend/src/lib/realtimeTransportBroker.tsindex ccf7dacdb9..168b582286 100644--- a/jarvis/ui/web/frontend/src/lib/realtimeTransportBroker.ts+++ b/jarvis/ui/web/frontend/src/lib/realtimeTransportBroker.ts@@ -5,6 +5,7 @@ import {} from "./realtimeTransportIssue";import { fetchRealtimeOfferRequirement } from "./voiceApi";import { mintWsTicket } from "./ws";+import { jitteredDelay } from "./connectBudget";declare global {interface Window {@@ -294,10 +295,13 @@ export class RealtimeTransportBroker {private scheduleReconnect(): void {if (this.stopped || this.reconnectTimer !== null) return;- // Exponential, capped. A constant 1.5 s loop against a socket that answers- // 4401/1008 is a spin against a backend that has already said no.- const delay = Math.min(- RECONNECT_DELAY_MS * 2 ** Math.max(0, this.failures - 1),+ // Exponential, capped and jittered. A constant 1.5 s loop against a socket+ // that answers 4401/1008 is a spin against a backend that has already said+ // no; an unjittered one is worse still, because every window retries in+ // the same millisecond after a wake (BUG-215, AP-33).+ const delay = jitteredDelay(+ Math.max(0, this.failures - 1),+ RECONNECT_DELAY_MS,RECONNECT_DELAY_MAX_MS,);this.reconnectTimer = this.deps.schedule(() => {diff --git a/jarvis/ui/web/frontend/src/lib/ws.ts b/jarvis/ui/web/frontend/src/lib/ws.tsindex 28cdc363bf..0494a95a71 100644--- a/jarvis/ui/web/frontend/src/lib/ws.ts+++ b/jarvis/ui/web/frontend/src/lib/ws.ts@@ -11,6 +11,13 @@* the TerminalView to receive raw frames including the terminal.spawned reply.*/+import {+ MAX_BACKOFF_MS,+ MIN_BACKOFF_MS,+ jitteredDelay,+ requestConnect,+} from "./connectBudget";+export type WSHandler = (data: unknown) => void;export interface WSCloseInfo {@@ -57,8 +64,8 @@ export async function mintWsTicket(): Promise<string | null> {}}-const MIN_BACKOFF = 500;-const MAX_BACKOFF = 10_000;+const MIN_BACKOFF = MIN_BACKOFF_MS;+const MAX_BACKOFF = MAX_BACKOFF_MS;const PING_INTERVAL = 30_000;/**@@ -85,9 +92,16 @@ const SILENCE_LIMIT = PING_INTERVAL * 2.5;export class WSClient {private ws: WebSocket | null = null;- private backoff = MIN_BACKOFF;private pingTimer: ReturnType<typeof setInterval> | null = null;- private reconnectTimer: ReturnType<typeof setTimeout> | null = null;+ /**+ * Cancels the reconnect waiting in the shared connect budget, or null+ * when none is pending. Not a bare `setTimeout` handle any more: every+ * reconnect in the app now queues behind one speed limit, so a wake+ * storm cannot empty the machine's socket pool (BUG-215, AP-33).+ */+ private cancelReconnect: (() => void) | null = null;+ /** Consecutive failed attempts - the exponent under the jitter. */+ private reconnectAttempt = 0;private stopped = false;private lastCloseCode: number | undefined;private pendingTicket: string | null = null;@@ -170,12 +184,15 @@ export class WSClient {// A wait that was scheduled while the window was hidden has been stretched// by an unknown amount and may still be minutes out. Being looked at is// fresh evidence: start over at the fast end of the backoff, now.- if (this.reconnectTimer) {- clearTimeout(this.reconnectTimer);- this.reconnectTimer = null;- }- this.backoff = MIN_BACKOFF;- this.openSocket();+ // A wait scheduled while the window was hidden has been stretched by an+ // unknown amount and may still be minutes out. Being looked at is fresh+ // evidence: start over at the fast end - but still through the budget.+ // Connecting straight from here is what made waking a storm: this handler+ // runs in every window, for every pane, in the same millisecond.+ this.cancelReconnect?.();+ this.cancelReconnect = null;+ this.reconnectAttempt = 0;+ this.queueReconnect(jitteredDelay(0, MIN_BACKOFF, MIN_BACKOFF));};/**@@ -196,7 +213,7 @@ export class WSClient {/* already gone */}this.onClose?.(undefined);- this.backoff = MIN_BACKOFF;+ this.reconnectAttempt = 0;this.scheduleReconnect();}@@ -228,7 +245,7 @@ export class WSClient {socket.addEventListener("open", () => {if (!current()) return;- this.backoff = MIN_BACKOFF;+ this.reconnectAttempt = 0;this.lastFrameAt = Date.now();this.startPing();this.onOpen?.();@@ -293,11 +310,8 @@ export class WSClient {if (retrying) {// Fixed short retry, like the 1013 warming path: the very next attempt// carries a fresh credential, so backoff escalation would only delay it.- if (this.reconnectTimer) return;- this.reconnectTimer = setTimeout(() => {- this.reconnectTimer = null;- this.openSocket();- }, MIN_BACKOFF);+ if (this.cancelReconnect) return;+ this.queueReconnect(jitteredDelay(0, MIN_BACKOFF, MIN_BACKOFF));return;}// The session is gone, the backend vanished mid-mint, or fresh tickets@@ -333,18 +347,32 @@ export class WSClient {}private scheduleReconnect(): void {- if (this.reconnectTimer) return;+ if (this.cancelReconnect) return;// The fast-boot bootstrap closes a warming WS with code 1013 ("try again- // later"): the backend is still booting, this is NOT a failure. Retry at a- // fixed short interval instead of escalating the backoff, so the window- // reconnects within ~1s of the real app becoming ready.+ // later"): the backend is still booting, this is NOT a failure. Keep the+ // attempt counter where it is so the retry stays at the fast end and the+ // window reconnects within ~1s of the real app becoming ready.const warming = this.lastCloseCode === 1013;- const delay = warming ? MIN_BACKOFF : this.backoff;- if (!warming) this.backoff = Math.min(MAX_BACKOFF, this.backoff * 2);- this.reconnectTimer = setTimeout(() => {- this.reconnectTimer = null;+ const attempt = warming ? 0 : this.reconnectAttempt++;+ const cap = warming ? MIN_BACKOFF : MAX_BACKOFF;+ // Full jitter, not a plain doubling. Every window and every pane starts+ // its backoff in the same millisecond when the machine wakes, so an+ // unjittered schedule keeps them in lockstep and they collide at 0.5s,+ // then 1s, then 2s - measured in this app on 2026-07-27.+ this.queueReconnect(jitteredDelay(attempt, MIN_BACKOFF, cap));+ }++ /**+ * Hand one reconnect to the shared budget.+ *+ * Everything that reopens this socket goes through here, so the tab has+ * exactly one place that knows how many connections are in flight.+ */+ private queueReconnect(delayMs: number): void {+ this.cancelReconnect = requestConnect(() => {+ this.cancelReconnect = null;this.openSocket();- }, delay);+ }, delayMs);}send(payload: unknown): void {@@ -363,10 +391,8 @@ export class WSClient {this.stopped = true;this.stopPing();this.stopListeningForWake();- if (this.reconnectTimer) {- clearTimeout(this.reconnectTimer);- this.reconnectTimer = null;- }+ this.cancelReconnect?.();+ this.cancelReconnect = null;this.ws?.close();this.ws = null;}diff --git a/jarvis/ui/web/frontend/src/store/agentChat.ts b/jarvis/ui/web/frontend/src/store/agentChat.tsindex 6c5a6a1293..4209cd5976 100644--- a/jarvis/ui/web/frontend/src/store/agentChat.ts+++ b/jarvis/ui/web/frontend/src/store/agentChat.ts@@ -27,6 +27,7 @@ import {type ProviderHealth,} from "@/lib/agentChatApi";import { EMPTY_TIMELINE, reduceEvent, reduceEvents, type Timeline } from "@/components/agentchat/reduce";+import { jitteredDelay, requestConnect } from "../lib/connectBudget";/*** The agent chat's store — one per SURFACE (`createAgentChatStore`).@@ -66,7 +67,10 @@ export function draftKey(surface: AgentChatSurface): string {// folder, the ladder's default — instead of on a coding session's picks.return surface === "jarvis" ? "jarvis.agentChat.draft.v2" : `jarvis.agentChat.draft.${surface}.v1`;}+/** First retry delay; each further failure doubles it under full jitter. */const RECONNECT_MS = 1500;+/** Ceiling on the retry wait — a flat 1.5 s forever is a spin, not a retry. */+const RECONNECT_MAX_MS = 30_000;/** How many sessions the list holds — the "All chats" archive shows them all. */const SESSION_LIST_LIMIT = 500;@@ -266,13 +270,14 @@ export function createAgentChatStore(surface: AgentChatSurface) {let socket: WebSocket | null = null;let socketSession: string | null = null;- let reconnectTimer: number | null = null;+ /** Cancels the reconnect waiting in the shared connect budget. */+ let cancelReconnect: (() => void) | null = null;+ /** Consecutive failed reconnects — the exponent under the jitter. */+ let reconnectAttempt = 0;function closeSocket(): void {- if (reconnectTimer !== null) {- window.clearTimeout(reconnectTimer);- reconnectTimer = null;- }+ cancelReconnect?.();+ cancelReconnect = null;const s = socket;socket = null;socketSession = null;@@ -298,7 +303,11 @@ export function createAgentChatStore(surface: AgentChatSurface) {const ws = new WebSocket(agentChatSocketUrl(sessionId, afterSeq));socket = ws;ws.onopen = () => {- if (socket === ws) set({ socketState: "open" });+ if (socket !== ws) return;+ // A socket that opened is proof the backend is back: the next outage+ // starts its backoff from the fast end again.+ reconnectAttempt = 0;+ set({ socketState: "open" });};ws.onmessage = (msg) => {if (socket !== ws) return;@@ -330,13 +339,22 @@ export function createAgentChatStore(surface: AgentChatSurface) {set({ socketState: "closed" });// Reconnect while this session is still the open one; the backend// replays what was missed from the last seq.- reconnectTimer = window.setTimeout(() => {- reconnectTimer = null;- const st = get();- if (st.activeSessionId === sessionId && socketSession === sessionId) {- connect(sessionId, st.timeline.lastSeq);- }- }, RECONNECT_MS);+ // Jittered, escalating and paid for out of the shared connect budget.+ // It used to be a flat 1.5 s with no cap and no spread: every window+ // holding this session retried in the same millisecond, forever, which+ // is one of the loops that emptied the machine's socket pool when the+ // backend went away for a while (BUG-215, AP-33).+ reconnectAttempt += 1;+ cancelReconnect = requestConnect(+ () => {+ cancelReconnect = null;+ const st = get();+ if (st.activeSessionId === sessionId && socketSession === sessionId) {+ connect(sessionId, st.timeline.lastSeq);+ }+ },+ jitteredDelay(reconnectAttempt - 1, RECONNECT_MS, RECONNECT_MAX_MS),+ );};ws.onerror = () => {/* onclose follows and schedules the reconnect */diff --git a/jarvis/ui/web/launcher.py b/jarvis/ui/web/launcher.pyindex 3bad1382cc..42c70aecd8 100644--- a/jarvis/ui/web/launcher.py+++ b/jarvis/ui/web/launcher.py@@ -392,6 +392,19 @@ async def _run_headless(args) -> int:# behaviour we had yesterday; a boot that fails is not.pass+ # Watch the machine's socket pool from off the loop. A headless host is the+ # case this matters MOST for: it runs the same polling loops as the desktop+ # and typically has a far tighter file-descriptor ceiling (1024 on a stock+ # slim image against 28 232 ports), so it hits EMFILE and dies where a+ # desktop merely freezes (BUG-215). Costs one sleeping thread; the first+ # census is a full interval away, so nothing lands on the boot path (AP-26).+ try:+ from jarvis.core.socket_budget import start_default_watchdog++ start_default_watchdog()+ except Exception: # noqa: BLE001,S110 - a diagnostic never breaks a boot.+ pass+# The single-instance lock (and its heavy ``desktop_app`` import — pywebview +# win32, ~420 ms) is acquired in the deferred section below, OFF the# time-to-serving path. It only needs to set JARVIS_PRIMARY_INSTANCE beforediff --git a/scripts/jarvis-config-drift-guard.ps1 b/scripts/jarvis-config-drift-guard.ps1index 4492168042..131c5c0a39 100644--- a/scripts/jarvis-config-drift-guard.ps1+++ b/scripts/jarvis-config-drift-guard.ps1@@ -74,6 +74,31 @@ $logFile = Join-Path $logDir "config-drift-guard.log"if (-not (Test-Path $logDir)) {New-Item -ItemType Directory -Path $logDir -Force | Out-Null}++# ---------- Log rotation ---------------------------------------------------+#+# This guard runs every five minutes for as long as the machine is on, and it+# used to append forever: the log reached 28 MB and 276 000 lines before anyone+# looked (2026-09-03). One generation is enough — the interesting window is the+# last few days, and a 4 MB file still holds weeks of it.+#+# Rotated HERE, at the very start of a run, because that is the one moment no+# writer holds the file open on Windows (the same reason the local-server+# supervisor rotates at its spawn boundary).+$maxLogBytes = 4MB+if (Test-Path $logFile) {+ $existing = Get-Item $logFile+ if ($existing.Length -ge $maxLogBytes) {+ $rolled = "$logFile.1"+ try {+ if (Test-Path $rolled) { Remove-Item $rolled -Force }+ Move-Item -Path $logFile -Destination $rolled -Force+ } catch {+ # Another run holds it open: keep appending rather than lose the+ # run. The next run rotates it.+ }+ }+}if (-not (Test-Path $tomlFile)) {Write-Host "FATAL: jarvis.toml was not found at $tomlFile"exit 2@@ -152,6 +177,9 @@ function Test-IsScalarConfigValue {# Metadata keys use an underscore prefix and are intentionally ignored.$drifts = @()+# Desired values for keys jarvis.toml does not spell out. Reported as one line+# after the sweep instead of one line each, every five minutes, forever.+$absentKeys = @()foreach ($section in $desired.PSObject.Properties) {if ($section.Name.StartsWith("_")) { continue }@@ -179,7 +207,12 @@ foreach ($section in $desired.PSObject.Properties) {$keyPattern = "(?m)^\s*$key\s*=\s*(?:`"([^`"]*)`"|([^\s#`r`n]+))"$keyMatch = [regex]::Match($sectionBody, $keyPattern)if (-not $keyMatch.Success) {- Write-Log "WARN" "Key '$key' in [$sectionName] was not found; desired '$desiredValue' was skipped"+ # Collected, not logged one-per-key. These are the SAME keys every+ # run (a desired value for a setting jarvis.toml does not spell+ # out, because it still holds its default), and at seventeen of+ # them every five minutes they were ~5 800 lines a day of identical+ # text. One summary line below keeps every name without the flood.+ $absentKeys += "$sectionName.$key"continue}if ($keyMatch.Groups[1].Success) {@@ -202,6 +235,14 @@ foreach ($section in $desired.PSObject.Properties) {}}+if ($absentKeys.Count -gt 0) {+ # Still a WARN, and still every name: a desired value that cannot be+ # enforced because the key is absent from jarvis.toml is worth knowing+ # about. It is just one line instead of seventeen.+ $absentList = ($absentKeys | Sort-Object) -join ", "+ Write-Log "WARN" "$($absentKeys.Count) desired keys are absent from jarvis.toml and were skipped: $absentList"+}+# ---------- Environment comparison ----------------------------------------function Format-EnvLiteral {diff --git a/tests/contract/test_socket_budget.py b/tests/contract/test_socket_budget.pynew file mode 100644index 0000000000..901e5d06e8--- /dev/null+++ b/tests/contract/test_socket_budget.py@@ -0,0 +1,219 @@+"""The socket ceiling must be readable on every OS Jarvis ships to.++BUG-215 was an outage of the whole machine caused by running the operating+system out of ephemeral ports. The watchdog that catches it can only do so if+it knows how many there are — and every platform answers that question through+a different door, in a different unit, with a different thing that actually+runs out first:++=========== ========================================== =======================+System Where the number lives What binds first+=========== ========================================== =======================+Windows ``netsh int ipv4 show dynamicport tcp`` the port pool+Linux ``/proc/sys/net/ipv4/ip_local_port_range`` usually RLIMIT_NOFILE+macOS ``sysctl net.inet.ip.portrange.first/last`` the port pool+=========== ========================================== =======================++These tests fake each host in turn, because CI runs on one of them at a time+and the two it is not running on are exactly where a silent regression hides.++The Windows case has its own trap and its own test: ``netsh`` translates its+own output. A parser that looks for the words "Start Port" reports "no limit"+on every non-English Windows in the install base — which is most of them, and+which was the maintainer's own machine.+"""++from __future__ import annotations++import subprocess+import sys+from typing import Any++import pytest++from jarvis.core import socket_budget++[email protected](autouse=True)+def _fresh_limits() -> Any:+ """Each test gets an unprobed module.++ Both the ceiling and the last-measured pressure are process-wide on+ purpose — every caller reads one number — so a test that leaves either+ behind is a test that decides the next one's answer.+ """+ socket_budget.read_platform_limits(refresh=True)+ socket_budget.set_pressure_for_tests(0.0)+ yield+ socket_budget.read_platform_limits(refresh=True)+ socket_budget.set_pressure_for_tests(0.0)+++def _fake_run(stdout: str) -> Any:+ def _run(*_args: Any, **_kwargs: Any) -> Any:+ return subprocess.CompletedProcess(args=[], returncode=0, stdout=stdout, stderr="")++ return _run+++# ── Windows ──────────────────────────────────────────────────────────────+_NETSH_EN = """+Protocol tcp Dynamic Port Range+---------------------------------+Start Port : 49152+Number of Ports : 16384+"""++_NETSH_DE = """+Protokoll tcp Dynamischer Portbereich+---------------------------------+Startport : 49152+Anzahl von Ports : 16384+"""++_NETSH_JA = """+プロトコル tcp 動的ポート範囲+---------------------------------+開始ポート : 49152+ポート数 : 16384+"""+++ ("name", "output"),+ [("english", _NETSH_EN), ("german", _NETSH_DE), ("japanese", _NETSH_JA)],+)+def test_windows_port_pool_is_read_in_any_language(+ name: str, output: str, monkeypatch: pytest.MonkeyPatch+) -> None:+ """The pool size is the second integer, whatever the labels say.++ This is the whole reason the parser reads numbers instead of field names.+ """+ monkeypatch.setattr(sys, "platform", "win32")+ monkeypatch.setattr(subprocess, "run", _fake_run(output))++ limits = socket_budget.read_platform_limits(refresh=True)++ assert limits.ephemeral_ports == 16384, f"{name} netsh output was misread"+ assert "netsh" in limits.source+++def test_windows_without_netsh_falls_back_to_the_documented_default(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A blocked netsh must not silence the watchdog entirely."""+ monkeypatch.setattr(sys, "platform", "win32")++ def _boom(*_a: Any, **_kw: Any) -> Any:+ raise FileNotFoundError("netsh")++ monkeypatch.setattr(subprocess, "run", _boom)++ limits = socket_budget.read_platform_limits(refresh=True)++ assert limits.ephemeral_ports == 16384+ assert "assumed" in limits.source+++def test_windows_has_no_file_descriptor_ceiling(monkeypatch: pytest.MonkeyPatch) -> None:+ """There is no RLIMIT_NOFILE there, and inventing one would be a lie."""+ monkeypatch.setattr(sys, "platform", "win32")+ monkeypatch.setattr(subprocess, "run", _fake_run(_NETSH_EN))+ monkeypatch.setitem(sys.modules, "resource", None)++ limits = socket_budget.read_platform_limits(refresh=True)++ assert limits.fd_limit is None+ assert limits.ceiling == 16384+++# ── Linux ────────────────────────────────────────────────────────────────+def test_linux_reads_the_proc_range(monkeypatch: pytest.MonkeyPatch, tmp_path: Any) -> None:+ monkeypatch.setattr(sys, "platform", "linux")+ proc = tmp_path / "ip_local_port_range"+ proc.write_text("32768\t60999\n", encoding="utf-8")+ monkeypatch.setattr(socket_budget, "_linux_port_pool", lambda: (28232, str(proc)))++ limits = socket_budget.read_platform_limits(refresh=True)++ assert limits.ephemeral_ports == 28232+++def test_the_tighter_of_the_two_limits_is_the_one_that_binds(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """A slim container has 28 232 ports and 1024 file descriptors.++ The ports are irrelevant there: the process dies of EMFILE first. A+ watchdog measuring against the larger number would report 4 % pressure+ while the backend was already refusing connections.+ """+ monkeypatch.setattr(sys, "platform", "linux")+ monkeypatch.setattr(socket_budget, "_probe_ephemeral_ports", lambda: (28232, "fake /proc"))+ monkeypatch.setattr(socket_budget, "_probe_fd_limit", lambda: 1024)++ limits = socket_budget.read_platform_limits(refresh=True)++ assert limits.ceiling == 1024+++def test_an_unlimited_descriptor_ceiling_does_not_bind(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setattr(sys, "platform", "linux")+ monkeypatch.setattr(socket_budget, "_probe_ephemeral_ports", lambda: (28232, "fake /proc"))+ monkeypatch.setattr(socket_budget, "_probe_fd_limit", lambda: None)++ assert socket_budget.read_platform_limits(refresh=True).ceiling == 28232+++# ── macOS ────────────────────────────────────────────────────────────────+def test_macos_reads_the_sysctl_range(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(sys, "platform", "darwin")+ monkeypatch.setattr(subprocess, "run", _fake_run("49152\n65535\n"))+ monkeypatch.setattr(socket_budget, "_probe_fd_limit", lambda: None)++ limits = socket_budget.read_platform_limits(refresh=True)++ assert limits.ephemeral_ports == 16384+ assert "sysctl" in limits.source+++def test_macos_without_sysctl_still_reports_a_ceiling(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(sys, "platform", "darwin")++ def _boom(*_a: Any, **_kw: Any) -> Any:+ raise FileNotFoundError("sysctl")++ monkeypatch.setattr(subprocess, "run", _boom)+ monkeypatch.setattr(socket_budget, "_probe_fd_limit", lambda: None)++ assert socket_budget.read_platform_limits(refresh=True).ephemeral_ports == 16384+++# ── An OS nobody planned for ─────────────────────────────────────────────+def test_an_unknown_platform_degrades_quietly(monkeypatch: pytest.MonkeyPatch) -> None:+ """No ceiling means no watchdog — never a crash, and never a false alarm."""+ monkeypatch.setattr(sys, "platform", "sunos5")+ monkeypatch.setattr(socket_budget, "_probe_fd_limit", lambda: None)++ limits = socket_budget.read_platform_limits(refresh=True)++ assert limits.ceiling is None+ assert "sunos5" in limits.source+++def test_no_ceiling_never_throttles_anything(monkeypatch: pytest.MonkeyPatch) -> None:+ """A watchdog that cannot see must not be what stops the app working."""+ monkeypatch.setattr(sys, "platform", "sunos5")+ monkeypatch.setattr(socket_budget, "_probe_fd_limit", lambda: None)+ # The ceiling is cached for the life of the process (reading it costs a+ # netsh spawn on Windows), so the fake host has to be probed explicitly.+ assert socket_budget.read_platform_limits(refresh=True).ceiling is None+ watchdog = socket_budget.SocketBudgetWatchdog(+ census=lambda: socket_budget.SocketCensus(total=99_999, lingering=99_999)+ )++ assert watchdog.tick() == 0.0+ assert socket_budget.should_defer_optional_io() is Falsediff --git a/tests/unit/core/test_socket_budget_watchdog.py b/tests/unit/core/test_socket_budget_watchdog.pynew file mode 100644index 0000000000..f80b15b066--- /dev/null+++ b/tests/unit/core/test_socket_budget_watchdog.py@@ -0,0 +1,175 @@+"""The socket watchdog must warn early, name a culprit, and never cry wolf.++The failure it exists for (BUG-215) is invisible after the fact: the machine+runs out of ephemeral ports, everything on it stops connecting, and two minutes+later TIME_WAIT expires and the evidence is gone. Windows logs its own event at+most once an hour, so the entry usually lands in the wrong minute and never+names a process.++So the value of this watchdog is entirely in WHEN it speaks and WHAT it says.+These tests pin both.+"""++from __future__ import annotations++import sys+from typing import Any++import pytest++from jarvis.core import socket_budget+from jarvis.core.socket_budget import SocketBudgetWatchdog, SocketCensus++[email protected](autouse=True)+def _fake_ceiling(monkeypatch: pytest.MonkeyPatch) -> Any:+ """A round 1000-socket ceiling, so a count reads as a percentage."""+ monkeypatch.setattr(sys, "platform", "linux")+ monkeypatch.setattr(socket_budget, "_probe_ephemeral_ports", lambda: (1000, "fake"))+ monkeypatch.setattr(socket_budget, "_probe_fd_limit", lambda: None)+ socket_budget.read_platform_limits(refresh=True)+ socket_budget.set_pressure_for_tests(0.0)+ yield+ socket_budget.read_platform_limits(refresh=True)+ socket_budget.set_pressure_for_tests(0.0)+++def _census(total: int, **kwargs: Any) -> SocketCensus:+ kwargs.setdefault("lingering", total)+ return SocketCensus(total=total, **kwargs)+++def _watchdog(total: int, reports: list[tuple[float, SocketCensus, Any]]) -> SocketBudgetWatchdog:+ return SocketBudgetWatchdog(+ census=lambda: _census(total),+ on_pressure=lambda fraction, census, limits: reports.append((fraction, census, limits)),+ )+++def test_a_quiet_machine_says_nothing() -> None:+ """A watchdog that reports normal traffic gets muted, and then it is useless."""+ reports: list[Any] = []++ assert _watchdog(120, reports).tick() == pytest.approx(0.12)+ assert reports == []+++def test_it_warns_before_the_pool_is_empty_not_after() -> None:+ """60 %, not 99 %.++ The report has to arrive while the culprit is still connecting. Written+ after exhaustion it names TIME_WAIT and nothing else, which is exactly the+ useless evidence this replaces.+ """+ reports: list[Any] = []++ fraction = _watchdog(650, reports).tick()++ assert fraction == pytest.approx(0.65)+ assert len(reports) == 1+++def test_ongoing_pressure_is_re_reported_on_an_interval_not_every_tick() -> None:+ """Ten minutes of squeeze must leave a trail, not 20 identical blocks."""+ reports: list[Any] = []+ watchdog = _watchdog(900, reports)++ for _ in range(5):+ watchdog.tick()++ assert len(reports) == 1+++def test_recovering_re_arms_the_warning() -> None:+ """A second squeeze after a calm spell is news again."""+ reports: list[Any] = []+ counts = iter([900, 100, 900])+ watchdog = SocketBudgetWatchdog(+ census=lambda: _census(next(counts)),+ on_pressure=lambda *args: reports.append(args),+ )++ watchdog.tick()+ watchdog.tick()+ watchdog.tick()++ assert len(reports) == 2+++def test_optional_traffic_stands_down_only_once_it_is_really_tight() -> None:+ """Polls wait at 80 %; a voice turn never asks and is never refused."""+ reports: list[Any] = []++ _watchdog(700, reports).tick()+ assert socket_budget.should_defer_optional_io() is False++ _watchdog(850, reports).tick()+ assert socket_budget.should_defer_optional_io() is True+++def test_the_report_names_where_the_sockets_went() -> None:+ """Without a destination the report is a number nobody can act on."""+ census = SocketCensus(+ total=900,+ lingering=880,+ by_state={"TIME_WAIT": 880, "ESTABLISHED": 20},+ top_destinations=[("127.0.0.1:47821", 400), ("::1:11434", 300)],+ top_processes=[(24364, 500)],+ )++ summary = census.summary()++ assert "127.0.0.1:47821" in summary+ assert "::1:11434" in summary+ assert "pid 24364" in summary+ assert "TIME_WAIT=880" in summary+++def test_a_census_with_no_owners_says_so_instead_of_looking_empty() -> None:+ """Closed sockets keep the port and lose the owner — that is the answer.++ An empty "busiest processes" list reads as "nothing to see"; the truth is+ that the pool was emptied by churn, which is a different and more useful+ thing to know.+ """+ summary = SocketCensus(total=900, lingering=900).summary()++ assert "unattributed" in summary+++def test_a_partial_census_admits_it_is_a_floor() -> None:+ """macOS refuses the system-wide table; a floor must not read as a total."""+ summary = SocketCensus(total=40, lingering=10, partial=True).summary()++ assert "own process only" in summary+++def test_an_unreadable_census_is_not_pressure() -> None:+ """No psutil, no permission, no numbers — and so no throttling."""+ watchdog = SocketBudgetWatchdog(census=lambda: None)++ assert watchdog.tick() == 0.0+ assert socket_budget.should_defer_optional_io() is False+++def test_a_reporter_that_throws_does_not_kill_the_watchdog() -> None:+ """The diagnostic must never become the outage."""++ def _boom(*_args: Any) -> None:+ raise RuntimeError("the log sink is gone")++ watchdog = SocketBudgetWatchdog(census=lambda: _census(900), on_pressure=_boom)++ assert watchdog.tick() == pytest.approx(0.9)+++def test_start_and_stop_are_idempotent() -> None:+ """Arming twice must not leave a second thread sampling forever."""+ watchdog = SocketBudgetWatchdog(interval_s=1.0, census=lambda: _census(10))++ watchdog.start()+ watchdog.start()+ watchdog.stop()+ watchdog.stop()++ assert watchdog._thread is Nonediff --git a/tests/unit/ui/test_desktop_health_probe.py b/tests/unit/ui/test_desktop_health_probe.pyindex 68a8d2cde0..b9703271dd 100644--- a/tests/unit/ui/test_desktop_health_probe.py+++ b/tests/unit/ui/test_desktop_health_probe.py@@ -11,9 +11,17 @@ Profiled on the maintainer's laptop 2026-08-21 while the app sat idle,margin the hottest thing the process did, spent entirely on certificates fora connection to itself.-These tests pin the client to one build per run, and pin the two behaviours-that must survive that change: an unreachable server still answers "no", and-a missing httpx still answers "no" instead of raising.+BUG-215 added the larger reason. A client per call is also a TCP connection+per call, and a closed loopback connection holds its ephemeral port for 120 s+on Windows. ``_wait_for_backend`` polls four times a second for the whole boot;+unpooled it parks hundreds of dead sockets in a pool of 16 384, and several+such loops together empty it — at which point nothing on the machine can+connect at all. So the boot wait is pinned to the same one client here.++These tests pin the client to one build per run, and pin the three behaviours+that must survive that change: an unreachable server still answers "no", a+missing httpx still answers "no" instead of raising, and the boot wait shares+the probe's client rather than building its own."""from __future__ import annotations@@ -43,9 +51,13 @@ class _FakeApp:def __init__(self, port: int = 47821) -> None:self.cfg = _Cfg(port)+ #: ``_wait_for_backend`` gives up early on a dead thread; None means+ #: "no thread to judge", which is the keep-waiting path under test.+ self._backend_thread = None_health_probe_client = DesktopApp._health_probe_client_backend_healthy = DesktopApp._backend_healthy+ _wait_for_backend = DesktopApp._wait_for_backendclass _Response:@@ -83,6 +95,14 @@ def fake_httpx(monkeypatch: pytest.MonkeyPatch) -> types.ModuleType:module = types.ModuleType("httpx")module.Client = _FakeClient # type: ignore[attr-defined]+ class _Limits:+ """``httpx.Limits`` stand-in — the pool passes one to every client."""++ def __init__(self, **kwargs: Any) -> None:+ self.kwargs = kwargs++ module.Limits = _Limits # type: ignore[attr-defined]+def _forbidden(*_a: Any, **_kw: Any) -> None:raise AssertionError("httpx.get() builds a client per call — use the reused one")@@ -164,3 +184,61 @@ def test_timeout_is_carried_by_the_client(fake_httpx: types.ModuleType) -> None:app._backend_healthy()assert app._health_http.kwargs.get("timeout") == 1.0+++def test_boot_wait_shares_the_probe_client(+ fake_httpx: types.ModuleType, monkeypatch: pytest.MonkeyPatch+) -> None:+ """The 250 ms boot poll must reuse the client, not build one per tick.++ This is the BUG-215 half. Unpooled, a slow boot leaves one dead loopback+ socket per tick sitting in TIME_WAIT for two minutes; pooled, the whole+ wait costs one connection.+ """+ import jarvis.ui.desktop_app as desktop_app++ monkeypatch.setattr(desktop_app.time, "sleep", lambda _s: None)+ app = _FakeApp()+ # Refuse the first two polls, answer the third — the ordinary slow boot.+ client_holder: dict[str, Any] = {}++ def _answer_on_third(url: str, **_: Any) -> _Response:+ client = client_holder["client"]+ client.calls.append(url)+ if len(client.calls) < 3:+ raise OSError("connection refused")+ return _Response(200)++ assert app._health_probe_client() is not None+ client_holder["client"] = app._health_http+ app._health_http.get = _answer_on_third # type: ignore[method-assign]++ assert app._wait_for_backend(timeout_s=5.0, hard_timeout_s=5.0) is True+ assert _FakeClient.built == 1, "one client for the whole boot wait"+ assert len(app._health_http.calls) == 3+++def test_boot_wait_without_httpx_gives_up_instead_of_spinning(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """No httpx means the wait can never succeed — say so at once.++ Spinning the full ``hard_timeout_s`` would hold the splash screen for+ three minutes to reach the same answer.+ """+ import builtins++ import jarvis.ui.desktop_app as desktop_app++ real_import = builtins.__import__++ def _no_httpx(name: str, *args: Any, **kwargs: Any) -> Any:+ if name == "httpx":+ raise ImportError("no httpx in this build")+ return real_import(name, *args, **kwargs)++ monkeypatch.setattr(builtins, "__import__", _no_httpx)+ monkeypatch.setattr(desktop_app.time, "sleep", lambda _s: None)+ app = _FakeApp()++ assert app._wait_for_backend(timeout_s=1.0, hard_timeout_s=1.0) is False