Skip to content

Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.

fix(validation): keep sccache scratch valid across run cleanup (#204)

aias00fb73037d828d
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.md
index 43d234c1..8feac124 100644
--- a/docs/code-map/testing.md
+++ b/docs/code-map/testing.md
@@ -242,6 +242,11 @@ and [Gateway preparation](../../tools/e2e-harness/src/p0/production/candidate.rs
the exact receipts consumed by admission; same-byte caches from another recipe do
not replace them. Failed admission retains independent results and missing coverage.
+Local/SSH cache admission in [remote-rust.py](../../scripts/remote-rust.py) connects/starts
+sccache under the existing capacity lease and probes effective scratch with a
+unique cache-miss compilation; [local lifecycle coverage](../../scripts/test-local-rust.py)
+includes isolated real-cache cleanup and obsolete-server refusal.
+
Ordinary local/SSH totals and module reports share Cargo-target attribution in
[validation-report.py](../../scripts/validation-report.py). The
[runner regressions](../../scripts/test-remote-rust.py) exercise worker results and
diff --git a/docs/local-rust-validation.md b/docs/local-rust-validation.md
index 9bc36912..f15a7160 100644
--- a/docs/local-rust-validation.md
+++ b/docs/local-rust-validation.md
@@ -21,7 +21,7 @@ python3 scripts/local-rust.py run \
By default the runner fetches the branch from this repository's `origin` and verifies the SHA
is reachable. It creates a disposable detached worktree under
`~/.cache/hiroute/local-rust/checkouts/`, uses its default `target/`, jobs=2 (override with `--jobs`),
-incremental=0 and existing sccache. It does not restart or empty sccache.
+incremental=0 and shared sccache. It never restarts or empties the shared server.
It holds `~/.cache/hiroute/mac-rust-validation.lock` on Mac, or
`~/.cache/hiroute/local-rust-validation.lock` on Linux, plus a checkout lock; a busy
lock rejects the run rather than starting another local compiler by default. With
@@ -44,6 +44,24 @@ configured local backend selects this mode automatically. Remote Desktop keeps
the origin-fetch default. The independent SSH Linux workbench has its own locks;
do not run both runners on one host/account to bypass its capacity limits.
+Before creating a run TMPDIR, cache admission holds the existing compiler lease,
+compiles a unique tiny Rust library through sccache with owner-only stable
+`sccache-tmp/` under the runner root. The request connects to an existing server
+or starts one using that stable environment, with a 30-second bound. A cache miss
+exercises the
+server's effective scratch; environment metadata and `--show-stats` alone do not
+prove that the server started or uses that directory. An already-running server
+is admitted only if the probe succeeds. No server is automatically stopped.
+Admission failure prevents Cargo execution and retains `cache-admission.json` and
+`cache-probe.log` beside the ordinary result. The same command monitor retains
+compiler and checkout locks until the probe exits if the runner crashes; cache
+clients and servers never inherit those lock descriptors. A timed-out probe's
+client process group is reaped before releasing capacity. After all
+compiler owners are idle, an operator may hold the host's exclusive capacity
+locks, preserve statistics and explicitly stop/start the server using the recorded
+stable scratch, `TMP`/`TEMP`, idle timeout zero and existing cache configuration.
+Never recreate an obsolete run directory or clear compiler objects to recover.
+
TMPDIR is private and canonical. If macOS's per-user temporary path is too long for
Unix sockets, the runner resolves the OS `/tmp` directory and creates a short
private child there. No host IP or user-specific absolute directory is in Git.
@@ -184,8 +202,8 @@ product functionality**. A real Mac Cargo invocation still requires the relevant
exact pushed candidate and its platform environment.
An opt-in integration test uses a tiny dependency-free crate pushed to a disposable
-local bare origin, real Cargo/sccache and the shared Mac lock. It tests this runner's
-fetch/build/evidence/reclamation path without compiling the HiRoute workspace:
+local bare origin, real Cargo and an isolated sccache endpoint/cache under the shared host lock. It tests this runner's
+fetch/build/evidence/reclamation path, two compilations across run cleanup, stale-server refusal and failed-test retention without compiling the HiRoute workspace:
```sh
HIROUTE_REAL_CARGO_TEST=1 python3 scripts/test-local-rust.py Cleanup.test_real_cargo_pushed_fixture_lifecycle
diff --git a/docs/remote-rust-validation.md b/docs/remote-rust-validation.md
index 9f6e45a9..82c17558 100644
--- a/docs/remote-rust-validation.md
+++ b/docs/remote-rust-validation.md
@@ -40,8 +40,19 @@ locks used by older controllers. A separate repository does not add compiler slo
runner does not restart a shared server or reset its statistics.
Shared sccache scratch is an owner-only `sccache-tmp/` under that workbench root,
-not any run's disposable `TMPDIR`. Before invoking Cargo the runner queries/starts
-sccache with that stable directory and `SCCACHE_IDLE_TIMEOUT=0`; before/after stats
+not any run's disposable `TMPDIR`. Before invoking Cargo, under the capacity lease,
+the runner requires a unique cache-miss Rust compilation through sccache with
+that stable directory and `SCCACHE_IDLE_TIMEOUT=0`. This connects to the existing
+server or starts one using stable scratch, with a 30-second bound; statistics
+alone do not prove startup or scratch health. An
+existing server is never stopped automatically. Failed admission blocks Cargo and
+retains `cache-admission.json` and `cache-probe.log`.
+The command monitor retains capacity and checkout locks across a worker crash
+until the probe exits, without passing them to cache clients or servers. A
+timed-out probe's client process group is reaped before releasing capacity. Recovery
+requires all compiler owners idle, exclusive account-wide capacity locks, retained
+statistics and an explicit operator stop/start with stable scratch; never recreate
+a deleted run directory or clear cache objects. Before/after stats
use the same server environment. Cargo and tests still receive their own short
private temporary directory. Run cleanup never removes shared server scratch.
An already-running server keeps its original environment: if it inherited a deleted
diff --git a/scripts/local-rust.py b/scripts/local-rust.py
index e5d951b1..f65c00f7 100644
--- a/scripts/local-rust.py
+++ b/scripts/local-rust.py
@@ -324,8 +324,7 @@ def run(store, args, prepare=None, on_record=None):
row["checkout_finished_at"] = time.time()
row["status"] = "running"
store.save(row)
- # Start/cache-query before locking. The command monitor keeps lock FDs away
- # from Cargo and any sccache server it starts later.
+ # Cache admission and Cargo share the existing exclusive compiler lease.
env = dict(os.environ, CARGO_BUILD_JOBS=str(jobs), CARGO_INCREMENTAL="0", RUSTC_WRAPPER="sccache")
process = None
try:
@@ -334,13 +333,15 @@ def run(store, args, prepare=None, on_record=None):
"temporary_directory": server_env["TMPDIR"],
"idle_timeout": "0",
}
- before = remote.stats(store.record_path(key).parent, "cache-before", server_env)
row["queued_at"] = time.time()
store.save(row)
with store.locked(checkout, wait=row["wait_for_lock"]) as handles:
row["capacity_acquired_at"] = time.time()
if row["wait_for_lock"] and shutil.disk_usage(store.root).free < 30 * GIB:
raise ValueError("Less than 30 GiB free after waiting; build not started")
+ row["cache_admission"] = remote.admit_cache_server(
+ store.record_path(key).parent, server_env, checkout, handles=handles)
+ before = remote.stats(store.record_path(key).parent, "cache-before", server_env)
info = private_temp()
row["temporary_directory"] = info
env["TMPDIR"] = info["path"]
diff --git a/scripts/remote-rust.py b/scripts/remote-rust.py
index b99edbf9..2f1b91f7 100644
--- a/scripts/remote-rust.py
+++ b/scripts/remote-rust.py
@@ -179,12 +179,58 @@ def capacity(root, exclusive, deadline=None, slots=None):
def stats(run, name, server_env):
- # show-stats can start the server. Never give that long-lived process a run's TMPDIR.
+ # Statistics do not establish server startup or effective scratch health.
data = json.loads(command(["sccache", "--show-stats", "--stats-format=json"], env=server_env))
save(run / (name + ".json"), data)
return data
+def admit_cache_server(run, server_env, checkout=None, *, handles):
+ """Connect/start through a compile request and exercise effective server scratch.
+
+ --show-stats can succeed with zero statistics when no server exists. A unique
+ Rust crate forces a cache miss, so a warm cache cannot conceal obsolete scratch.
+ The caller holds its normal compiler-capacity lease throughout admission.
+ """
+ record = {"state": "checking"}
+ save(run / "cache-admission.json", record)
+ try:
+ # A compilation connects to the existing server or starts one with this
+ # environment. Never use --start-server blindly: Unix socket startup can
+ # unlink an existing endpoint. File output avoids inherited pipe writers.
+ with tempfile.TemporaryDirectory(prefix="probe-", dir=server_env["TMPDIR"]) as directory:
+ directory = Path(directory)
+ source = directory / "probe.rs"
+ source.write_text("pub const ID: &str = " + json.dumps(uuid.uuid4().hex) + ";\n")
+ artifact = directory / "libhiroute_cache_probe.rlib"
+ with (run / "cache-probe.log").open("w") as output:
+ probe = start_locked_command(
+ ["sccache", shutil.which("rustc") or "rustc", "--crate-name", "hiroute_cache_probe",
+ "--crate-type", "lib", "--emit=dep-info,link", str(source), "--out-dir", str(directory)],
+ checkout, server_env, output, handles)
+ try:
+ record["probe_exit"] = probe.wait(timeout=30)
+ finally:
+ if probe.poll() is None:
+ os.killpg(probe.pid, signal.SIGKILL)
+ probe.wait()
+ if record["probe_exit"] or not artifact.is_file():
+ raise RuntimeError("cache-miss compilation did not produce a library")
+ record["state"] = "ready"
+ except (OSError, subprocess.SubprocessError, RuntimeError) as error:
+ record.update(state="blocked", error=str(error))
+ raise RunBlocked(
+ "cache_server_unhealthy",
+ "sccache preflight failed; see cache-admission.json and cache-probe.log. "
+ "The existing server was not stopped. After all compiler owners are idle, acquire the "
+ "host's exclusive compiler-capacity locks, preserve server statistics, and explicitly "
+ "stop/start sccache with the recorded stable scratch environment before retrying. "
+ "Do not recreate deleted run directories or clear the disk cache.") from error
+ finally:
+ save(run / "cache-admission.json", record)
+ return record
+
+
def cache_server_environment(root):
"""Keep server scratch independent of disposable per-run test directories."""
directory = root.resolve(strict=True) / "sccache-tmp"
@@ -632,7 +678,8 @@ def worker(request_path):
result["cache_server_environment"] = {
"temporary_directory": server_env["TMPDIR"], "idle_timeout": "0",
}
- # Start/query before Cargo, using stable scratch. Never restart an existing server.
+ result["cache_admission"] = admit_cache_server(
+ run, server_env, checkout, handles=[*slots, reused_lease])
before = stats(run, "cache-before", server_env)
if before.get("max_cache_size") != 30 * 1024**3:
raise RuntimeError("sccache server must be configured with a 30 GiB cache; do not restart a busy shared server")
diff --git a/scripts/test-console-assets.py b/scripts/test-console-assets.py
index 42b95afa..4d7e3aca 100644
--- a/scripts/test-console-assets.py
+++ b/scripts/test-console-assets.py
@@ -193,6 +193,7 @@ print('test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered ou
patch.object(runner.sys, 'platform', 'linux'), \
patch.object(runner.tempfile, 'gettempdir', return_value=str(self.root)), \
patch.object(runner, 'MIN_FREE_BYTES', 0), \
+ patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \
patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}) as stats:
runner.worker(run / 'request.json')
return json.loads((run / 'result.json').read_text()), stats
diff --git a/scripts/test-local-rust.py b/scripts/test-local-rust.py
index f7528b8d..44885341 100644
--- a/scripts/test-local-rust.py
+++ b/scripts/test-local-rust.py
@@ -214,6 +214,10 @@ store=module.Store(sys.argv[2],sys.argv[3])
store.ensure_space=lambda: None
store.gc=lambda *a,**k: (_ for _ in ()).throw(AssertionError('GC must not run inside the acquired lock'))
module.remote.stats=lambda *a,**k: {'cache_hits':1}
+def cache_admission(*args,**kwargs):
+ with Path(sys.argv[4]).with_name('cache-admitted').open('w') as out: out.write('entered')
+ return {'state':'ready'}
+module.remote.admit_cache_server=cache_admission
module.shutil.disk_usage=lambda path: types.SimpleNamespace(free=int(Path(sys.argv[4]).read_text()))
flock=module.remote.fcntl.flock
first=True
@@ -286,6 +290,7 @@ print(json.dumps(result),flush=True)
self.assertIn('temporarily unavailable', row['error'])
self.assertNotIn('capacity_acquired_at', row)
self.assertFalse(calls.exists())
+ self.assertFalse((self.base / 'cache-admitted').exists())
self.assertIsNone(holder.poll(), 'the test must leave the competing owner active')
def test_interrupted_waiting_run_has_no_execution_or_lock_leak(self):
@@ -554,6 +559,7 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
source=source, jobs=jobs)
with patch.dict(os.environ, {'PATH': str(binary) + os.pathsep + os.environ['PATH']}), \
patch.object(m.remote, 'stats', return_value={'cache_hits': 1}), \
+ patch.object(m.remote, 'admit_cache_server', return_value={'state': 'ready'}), \
patch.object(self.store, 'ensure_space'):
result = m.run(self.store, args)
if result.get('temporary_directory'):
@@ -639,6 +645,21 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
@unittest.skipUnless(os.environ.get('HIROUTE_REAL_CARGO_TEST') == '1',
'Opt-in tiny platform-runner integration, not HiRoute product tests')
def test_real_cargo_pushed_fixture_lifecycle(self):
+ # Own a short endpoint/scratch/cache: never touch the daily server. macOS
+ # daemon-start notification sockets also need room below Unix SUN_LEN.
+ endpoint = tempfile.TemporaryDirectory(prefix='h202-', dir='/tmp')
+ self.addCleanup(endpoint.cleanup)
+ self.store = m.Store(Path(endpoint.name) / 'state', m.HOST_LOCK)
+ if os.environ.get('HIROUTE_RUNNER_EVIDENCE'):
+ destination = Path(os.environ['HIROUTE_RUNNER_EVIDENCE'])
+ self.addCleanup(lambda: m.shutil.copytree(self.store.root / 'runs', destination))
+ server_env = dict(os.environ, SCCACHE_SERVER_UDS=str(Path(endpoint.name) / 's'),
+ SCCACHE_DIR=str(self.base / 'cache'), SCCACHE_IDLE_TIMEOUT='0')
+ environment = patch.dict(os.environ, server_env)
+ environment.start()
+ self.addCleanup(environment.stop)
+ self.addCleanup(lambda: subprocess.run(['sccache', '--stop-server'], env=server_env,
+ capture_output=True, timeout=30))
(self.repo / 'Cargo.toml').write_text(
'[package]\nname="retention-fixture"\nversion="0.1.0"\nedition="2021"\n')
(self.repo / 'src').mkdir()
@@ -653,6 +674,7 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
branch = m.git(self.repo, 'symbolic-ref', 'HEAD')
m.git(self.repo, 'push', 'origin', branch)
self.store.global_lock = m.HOST_LOCK
+ self.store.global_lock.parent.mkdir(parents=True, exist_ok=True)
args = types.SimpleNamespace(repo=str(self.repo), ref=branch,
sha=m.git(self.repo, 'rev-parse', 'HEAD'),
command=['cargo', 'test', '--locked', '--offline'],
@@ -671,6 +693,30 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
} for phase in ('cache_before', 'cache_after')}
print(json.dumps(values))
report(result)
+ self.assertFalse(Path(result['temporary_directory']['path']).exists())
+ second = m.run(self.store, args)
+ self.assertEqual(second['scenario'], 'green', second)
+ self.assertTrue(second['removed'])
+ self.assertEqual(second['cache_admission']['state'], 'ready')
+ self.assertGreater(second['cache_before']['stats']['compile_requests'],
+ result['cache_after']['stats']['compile_requests']) # Existing server retained.
+ report(second)
+ # An obsolete pre-existing server is refused, never silently restarted.
+ subprocess.run(['sccache', '--stop-server'], env=server_env, check=True, capture_output=True)
+ obsolete = Path(endpoint.name) / 'obsolete-scratch'
+ obsolete.mkdir()
+ obsolete_env = dict(server_env, TMPDIR=str(obsolete), TMP=str(obsolete), TEMP=str(obsolete))
+ subprocess.run(['sccache', '--start-server'], env=obsolete_env, check=True, capture_output=True)
+ obsolete.rmdir()
+ blocked = m.run(self.store, args)
+ self.assertEqual(blocked['scenario'], 'red', blocked)
+ self.assertIsNone(blocked['process_exit'])
+ evidence = self.store.record_path(blocked['id']).parent
+ self.assertEqual(json.loads((evidence / 'cache-admission.json').read_text())['state'], 'blocked')
+ self.assertTrue((evidence / 'cache-probe.log').stat().st_size)
+ self.assertFalse(obsolete.exists())
+ # Explicit owner recovery is only for this test's isolated, idle endpoint.
+ subprocess.run(['sccache', '--stop-server'], env=server_env, check=True, capture_output=True)
(self.repo / 'src/lib.rs').write_text('#[test] fn intended_failure() { panic!("fixture"); }\n')
m.git(self.repo, 'add', 'src/lib.rs')
m.git(self.repo, 'commit', '-qm', 'intentional diagnostic retention fixture')
diff --git a/scripts/test-pilot-builds.py b/scripts/test-pilot-builds.py
index 850efd75..0e3bb873 100644
--- a/scripts/test-pilot-builds.py
+++ b/scripts/test-pilot-builds.py
@@ -100,6 +100,9 @@ print('fixture build completed')
stats = patch.object(m.local.remote, 'stats', return_value={'cache_hits': 0})
stats.start()
self.addCleanup(stats.stop)
+ admission = patch.object(m.local.remote, 'admit_cache_server', return_value={'state': 'ready'})
+ admission.start()
+ self.addCleanup(admission.stop)
self.store = m.local.Store(self.root / 'state', self.root / 'host.lock')
self.manager = m.Builds(self.store)
diff --git a/scripts/test-remote-rust.py b/scripts/test-remote-rust.py
index c800f72d..90540fde 100644
--- a/scripts/test-remote-rust.py
+++ b/scripts/test-remote-rust.py
@@ -9,7 +9,7 @@ import sys
import tempfile
import time
import unittest
-from unittest.mock import patch
+from unittest.mock import Mock, patch
import hashlib
import io
import json
@@ -249,6 +249,7 @@ sys.exit(101 if bad else 0)
patch.object(runner.tempfile, 'gettempdir', return_value=str(base.resolve())), \
patch.object(runner, 'console_assets') as console, \
patch.object(runner, 'MIN_FREE_BYTES', 0), \
+ patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \
patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}):
console.return_value.preflight.return_value = dict(state='not_required', cases=[])
runner.worker(path)
@@ -309,6 +310,105 @@ sys.exit(101 if bad else 0)
self.assertRaisesRegex(RuntimeError, 'immutable report helper mismatch'):
exec(runner.BOOTSTRAP, {})
+ def test_cache_admission_requires_real_output_and_retains_failure(self):
+ with tempfile.TemporaryDirectory() as parent:
+ root = Path(parent)
+ env = runner.cache_server_environment(root)
+ for failure in (False, True):
+ calls = []
+ def execute(argv, checkout, environment, output, handles):
+ calls.append(argv)
+ self.assertEqual(environment, env)
+ self.assertEqual(handles, [])
+ if not failure:
+ (Path(argv[-1]) / 'libhiroute_cache_probe.rlib').write_bytes(b'library')
+ return Mock(wait=Mock(return_value=0), poll=Mock(return_value=0))
+ with patch.object(runner, 'start_locked_command', side_effect=execute):
+ if failure:
+ with self.assertRaisesRegex(runner.RunBlocked, 'not stopped'):
+ runner.admit_cache_server(root, env, handles=[])
+ else:
+ self.assertEqual(runner.admit_cache_server(root, env, handles=[])['state'], 'ready')
+ self.assertEqual(len(calls), 1)
+ self.assertFalse(any('--stop-server' in call for call in calls))
+ record = json.loads((root / 'cache-admission.json').read_text())
+ self.assertEqual(record['state'], 'blocked' if failure else 'ready')
+ self.assertTrue((root / 'cache-probe.log').exists())
+ self.assertEqual(list(Path(env['TMPDIR']).iterdir()), [])
+
+ def test_cache_admission_bounds_probe_timeout(self):
+ with tempfile.TemporaryDirectory() as parent:
+ root = Path(parent)
+ env = runner.cache_server_environment(root)
+ process = Mock(pid=123, wait=Mock(side_effect=[subprocess.TimeoutExpired('sccache', 30), -9]),
+ poll=Mock(return_value=None))
+ with patch.object(runner, 'start_locked_command', return_value=process), \
+ patch.object(runner.os, 'killpg') as kill, \
+ self.assertRaises(runner.RunBlocked):
+ runner.admit_cache_server(root, env, handles=[])
+ self.assertEqual(process.wait.call_args_list[0].kwargs, {'timeout': 30})
+ self.assertEqual(process.wait.call_count, 2)
+ kill.assert_called_once_with(123, signal.SIGKILL)
+ self.assertEqual(json.loads((root / 'cache-admission.json').read_text())['state'], 'blocked')
+
+ def test_cache_admission_retains_capacity_after_runner_crash(self):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory)
+ binary = root / 'bin'
+ binary.mkdir()
+ probe = binary / 'sccache'
+ probe.write_text('#!' + sys.executable + '''
+import json,os,sys,time
+from pathlib import Path
+root=Path(os.environ['PROBE_ROOT'])
+try:
+ inherited=os.fstat(int(os.environ['LOCK_FD'])).st_ino==(root/'lock').stat().st_ino
+except OSError:
+ inherited=False
+(root/'ready').write_text(json.dumps({'pid':os.getpid(),'inherited':inherited}))
+while not (root/'release').exists(): time.sleep(.01)
+(Path(sys.argv[-1])/'libhiroute_cache_probe.rlib').write_bytes(b'library')
+''')
+ probe.chmod(0o700)
+ code = '''import fcntl,importlib.util,sys
+from pathlib import Path
+spec=importlib.util.spec_from_file_location('runner',sys.argv[1])
+runner=importlib.util.module_from_spec(spec); spec.loader.exec_module(runner)
+root=Path(sys.argv[2]); handle=(root/'lock').open('a'); fcntl.flock(handle,fcntl.LOCK_EX)
+env=runner.cache_server_environment(root); env['LOCK_FD']=str(handle.fileno())
+runner.admit_cache_server(root,env,handles=[handle])
+'''
+ env = dict(os.environ, PATH=str(binary) + os.pathsep + os.environ['PATH'], PROBE_ROOT=str(root))
+ owner = subprocess.Popen([sys.executable, '-c', code, runner.__file__, str(root)],
+ env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ deadline = time.monotonic() + 5
+ while not (root / 'ready').exists() and owner.poll() is None and time.monotonic() < deadline:
+ time.sleep(.01)
+ state = json.loads((root / 'ready').read_text())
+ self.assertFalse(state['inherited'], 'cache processes must not retain compiler locks')
+ owner.kill()
+ owner.wait(timeout=5)
+ os.kill(state['pid'], 0)
+ with (root / 'lock').open('a') as next_owner:
+ with self.assertRaises(BlockingIOError):
+ fcntl.flock(next_owner, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ (root / 'release').touch()
+ deadline = time.monotonic() + 5
+ while True:
+ try:
+ fcntl.flock(next_owner, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ break
+ except BlockingIOError:
+ if time.monotonic() >= deadline:
+ self.fail('probe completion did not release capacity')
+ time.sleep(.01)
+ finally:
+ (root / 'release').touch()
+ if owner.poll() is None:
+ owner.kill()
+ owner.wait(timeout=5)
+
def test_cache_server_temp_outlives_runs_and_does_not_change_test_environment(self):
with tempfile.TemporaryDirectory() as parent:
root = Path(parent) / "workbench"
@@ -731,6 +831,7 @@ sys.exit(int((home/'exit-code').read_text()))
patch.dict(os.environ, {}, clear=False), contextlib.redirect_stdout(io.StringIO()), \
patch.object(runner.tempfile, 'gettempdir', return_value=str(root.resolve())), \
patch.object(runner, 'MIN_FREE_BYTES', 0), \
+ patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \
patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}):
runner.worker(run / 'request.json')
result = json.loads((run / 'result.json').read_text())