Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(model-trellis): recycle an instance before VRAM exhaustion kills it mid-job, not after

nirholas9bbea4c095c5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b116878a1..c04cc259d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,10 @@
Public history for [three.ws](https://three.ws), newest first. New pages come from `added` dates in data/pages.json; everything else is curated in data/changelog.json. Also available as [JSON](https://three.ws/changelog.json) and [RSS](https://three.ws/changelog.xml), live at [three.ws/changelog](https://three.ws/changelog).
+## 2026-10-25
+
+- **Image-to-3D Forge instances recycle before a VRAM leak crashes a job**: The trellis lane's GPU worker already restarted itself between jobs once host memory ratcheted up, but had no equivalent check for GPU memory: a rasterizer dependency allocates outside torch's own memory pool, so an instance's usable VRAM only ever shrank, and the instance's own health check kept passing right up until a job crashed or Cloud Run killed it mid-generation. The worker now also tracks GPU memory after every job and restarts cleanly between jobs once it has ratcheted far enough, the same way the existing memory guard already does, so Forge image-to-3D jobs on this lane fail or stall less often. (`/image-to-3d`) `[fix]`
+
## 2026-10-24
- **Agent Predictions** (`/agent-predictions`): See what your agent's prediction strategy would have earned before it trades: shadow mode records every decision at the live price and marks it at resolution, beside a live fee-aware no-arbitrage scanner for Solana event contracts.
diff --git a/data/changelog.json b/data/changelog.json
index 73b736759..12a2f0510 100644
--- a/data/changelog.json
+++ b/data/changelog.json
@@ -1,6 +1,15 @@
{
"$comment": "Curated changelog entries for three.ws, the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security, desktop (desktop entries become the desktop app's release notes). `link` is optional and must be a live three.ws path.",
"entries": [
+ {
+ "date": "2026-10-25",
+ "title": "Image-to-3D Forge instances recycle before a VRAM leak crashes a job",
+ "summary": "The trellis lane's GPU worker already restarted itself between jobs once host memory ratcheted up, but had no equivalent check for GPU memory: a rasterizer dependency allocates outside torch's own memory pool, so an instance's usable VRAM only ever shrank, and the instance's own health check kept passing right up until a job crashed or Cloud Run killed it mid-generation. The worker now also tracks GPU memory after every job and restarts cleanly between jobs once it has ratcheted far enough, the same way the existing memory guard already does, so Forge image-to-3D jobs on this lane fail or stall less often.",
+ "tags": [
+ "fix"
+ ],
+ "link": "/image-to-3d"
+ },
{
"date": "2026-10-24",
"title": "GET /api/pump/coin degrades to on-chain data plus first-party metadata instead of a bare 502",
diff --git a/public/changelog-recent.json b/public/changelog-recent.json
index 0bf9c7625..8ea4d169d 100644
--- a/public/changelog-recent.json
+++ b/public/changelog-recent.json
@@ -1,14 +1,24 @@
{
- "generated_at": "2026-10-24T23:45:10.921Z",
+ "generated_at": "2026-10-25T01:03:00.076Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json",
"site": {
"name": "three.ws",
"url": "https://three.ws"
},
- "total": 4192,
+ "total": 4193,
"complete": false,
"full_feed": "/changelog.json",
"entries": [
+ {
+ "date": "2026-10-25",
+ "type": "update",
+ "title": "Image-to-3D Forge instances recycle before a VRAM leak crashes a job",
+ "summary": "The trellis lane's GPU worker already restarted itself between jobs once host memory ratcheted up, but had no equivalent check for GPU memory: a rasterizer dependency allocates outside torch's own memory pool, so an instance's usable VRAM only ever shrank, and the instance's own health check kept passing right up until a job crashed or Cloud Run killed it mid-generation. The worker now also tracks GPU memory after every job and restarts cleanly between jobs once it has ratcheted far enough, the same way the existing memory guard already does, so Forge image-to-3D jobs on this lane fail or stall less often.",
+ "link": "/image-to-3d",
+ "tags": [
+ "fix"
+ ]
+ },
{
"date": "2026-10-24",
"type": "launch",
@@ -1552,17 +1562,6 @@
"feature",
"fix"
]
- },
- {
- "date": "2026-10-12",
- "type": "update",
- "title": "Companion presence: sustained mood, emoji-driven expression, an LLM gesture tool, active-listening nods, and real multi-language lip-sync",
- "summary": "Agent avatars carry more of what makes a companion feel present. setMood(valence, arousal) sets a sustained resting bias (a soft smile or a worried brow that drifts in over about a second) on top of the existing per-utterance emotion blend, and a reply that is nothing but emoji (no English sentiment words at all) now drives a real expression change instead of staying neutral, because the built-in sentiment scorer reads emoji as direct valence/arousal signal. The LLM brain gets a general-purpose `gesture` tool covering all 19 body-language slots (nod, shake, shrug, bow, point, think, and the rest), so it can react physically mid-turn instead of only through a skill or an emotion threshold. While a real VAD/STT signal says the user is talking, the avatar nods on a fixed cadence to show it is listening, and stops the moment they stop. ElevenLabs text-to-speech now drives lip-sync from the provider's real per-character timestamps (any language the voice speaks), falling back to the energy-based analyser only if that call fails, so mouth shapes track what is actually being said instead of approximating it from amplitude alone.",
- "link": "/docs/agent-system",
- "tags": [
- "feature",
- "improvement"
- ]
}
]
}
diff --git a/public/changelog.json b/public/changelog.json
index 490b67ce3..dcf6c97d6 100644
--- a/public/changelog.json
+++ b/public/changelog.json
@@ -1,11 +1,21 @@
{
- "generated_at": "2026-10-24T23:45:10.896Z",
+ "generated_at": "2026-10-25T01:03:00.051Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json",
"site": {
"name": "three.ws",
"url": "https://three.ws"
},
"entries": [
+ {
+ "date": "2026-10-25",
+ "type": "update",
+ "title": "Image-to-3D Forge instances recycle before a VRAM leak crashes a job",
+ "summary": "The trellis lane's GPU worker already restarted itself between jobs once host memory ratcheted up, but had no equivalent check for GPU memory: a rasterizer dependency allocates outside torch's own memory pool, so an instance's usable VRAM only ever shrank, and the instance's own health check kept passing right up until a job crashed or Cloud Run killed it mid-generation. The worker now also tracks GPU memory after every job and restarts cleanly between jobs once it has ratcheted far enough, the same way the existing memory guard already does, so Forge image-to-3D jobs on this lane fail or stall less often.",
+ "link": "/image-to-3d",
+ "tags": [
+ "fix"
+ ]
+ },
{
"date": "2026-10-24",
"type": "launch",
diff --git a/public/changelog.xml b/public/changelog.xml
index cddce0d9d..8953825fd 100644
--- a/public/changelog.xml
+++ b/public/changelog.xml
@@ -6,6 +6,14 @@
<atom:link href="https://three.ws/changelog.xml" rel="self" type="application/rss+xml"/>
<description>What's new on three.ws: features, improvements, and releases, newest first.</description>
<language>en</language>
+ <item>
+ <title>Image-to-3D Forge instances recycle before a VRAM leak crashes a job</title>
+ <link>https://three.ws/image-to-3d</link>
+ <guid isPermaLink="false">2026-10-25:Image-to-3D Forge instances recycle before a VRAM leak crashes a job</guid>
+ <pubDate>Sun, 25 Oct 2026 12:00:00 GMT</pubDate>
+ <category>fix</category>
+ <description>The trellis lane's GPU worker already restarted itself between jobs once host memory ratcheted up, but had no equivalent check for GPU memory: a rasterizer dependency allocates outside torch's own memory pool, so an instance's usable VRAM only ever shrank, and the instance's own health check kept passing right up until a job crashed or Cloud Run killed it mid-generation. The worker now also tracks GPU memory after every job and restarts cleanly between jobs once it has ratcheted far enough, the same way the existing memory guard already does, so Forge image-to-3D jobs on this lane fail or stall less often.</description>
+ </item>
<item>
<title>Agent Predictions</title>
<link>https://three.ws/agent-predictions</link>
diff --git a/public/features.json b/public/features.json
index 1c7fdadd0..87b2bc58a 100644
--- a/public/features.json
+++ b/public/features.json
@@ -1,5 +1,5 @@
{
- "generated_at": "2026-10-24T23:45:10.879Z",
+ "generated_at": "2026-10-25T01:03:00.034Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json",
"site": {
"name": "three.ws",
diff --git a/workers/model-trellis/conftest.py b/workers/model-trellis/conftest.py
new file mode 100644
index 000000000..529b84323
--- /dev/null
+++ b/workers/model-trellis/conftest.py
@@ -0,0 +1,57 @@
+"""Make `import torch` resolvable on a dev box that has no CUDA wheel at all.
+
+Production runs this service in a GPU image where torch is real; this sandbox
+has no GPU and no torch package, which is a stricter version of the same
+"no GPU available" case the unit tests must already tolerate (see
+instance_health.py's own pure-stdlib design). A real torch import, if one is
+already installed, is left completely alone.
+"""
+
+from __future__ import annotations
+
+import sys
+import types
+
+
+def _install_torch_stub() -> None:
+ try:
+ import torch # noqa: F401
+
+ return
+ except ImportError:
+ pass
+
+ torch_stub = types.ModuleType("torch")
+
+ class _CudaStub:
+ @staticmethod
+ def is_available() -> bool:
+ return False
+
+ @staticmethod
+ def mem_get_info():
+ raise RuntimeError("no GPU device in this environment")
+
+ @staticmethod
+ def empty_cache() -> None:
+ pass
+
+ @staticmethod
+ def ipc_collect() -> None:
+ pass
+
+ @staticmethod
+ def get_device_name(index: int = 0) -> str:
+ return "stub"
+
+ hub_stub = types.ModuleType("torch.hub")
+ hub_stub.load = lambda *args, **kwargs: None
+
+ torch_stub.cuda = _CudaStub()
+ torch_stub.hub = hub_stub
+
+ sys.modules["torch"] = torch_stub
+ sys.modules["torch.hub"] = hub_stub
+
+
+_install_torch_stub()
diff --git a/workers/model-trellis/instance_health.py b/workers/model-trellis/instance_health.py
index 124dbc40a..1c7494e11 100644
--- a/workers/model-trellis/instance_health.py
+++ b/workers/model-trellis/instance_health.py
@@ -26,6 +26,10 @@ The fixes are split by what they guard against:
logs its footprint and the recycle guard has a number.
should_recycle() decides when an instance has ratcheted far enough that
it should restart BETWEEN jobs rather than die inside one.
+ vram_sample() shapes a torch.cuda.mem_get_info() reading into the same
+ shape, so should_recycle() also catches a VRAM leak that
+ host memory can never see (main.py owns the torch call;
+ see its own mem_get_info() comment, 2026-10-24).
task_is_orphaned() lets a heartbeat-refreshed record expire in minutes, not
half an hour, once its runner is gone.
"""
@@ -53,6 +57,15 @@ RECYCLE_FRACTION = float(os.environ.get("MEMORY_RECYCLE_FRACTION", "0.80"))
# into a restart after every single job.
RECYCLE_MIN_GROWTH = float(os.environ.get("MEMORY_RECYCLE_MIN_GROWTH", "0.15"))
+# Same two-condition recycle rule, applied to GPU memory instead of host RAM.
+# Tighter than the host-RAM pair because a VRAM OOM has no graceful fallback (it
+# crashes the job outright), whereas host RAM pressure degrades first. See
+# vram_sample() below: nvdiffrast's rasterizer allocates through its own
+# cudaMalloc outside torch's caching allocator, so empty_cache() cannot reach
+# it and an instance's usable VRAM only ever ratchets down.
+VRAM_RECYCLE_FRACTION = float(os.environ.get("VRAM_RECYCLE_FRACTION", "0.90"))
+VRAM_RECYCLE_MIN_GROWTH = float(os.environ.get("VRAM_RECYCLE_MIN_GROWTH", "0.10"))
+
_CGROUP_V2 = "/sys/fs/cgroup"
_CGROUP_V1 = "/sys/fs/cgroup/memory"
_MEMINFO = "/proc/meminfo"
@@ -158,6 +171,20 @@ def read_memory(
return None
+def vram_sample(free_bytes: int, total_bytes: int) -> Optional[MemorySample]:
+ """Shape a torch.cuda.mem_get_info() (free, total) pair into a MemorySample.
+
+ This module stays pure stdlib (no torch import), so the caller owns the
+ `torch.cuda.is_available()` guard and the `mem_get_info()` call itself and
+ passes in the two numbers it already has. Reuses MemorySample's shape
+ (used_bytes, limit_bytes, source) rather than a divergent VRAM type, so
+ should_recycle() applies unchanged.
+ """
+ if not total_bytes:
+ return None
+ return MemorySample(total_bytes - free_bytes, total_bytes, "cuda")
+
+
def should_recycle(
current: Optional[MemorySample],
baseline: Optional[MemorySample],
diff --git a/workers/model-trellis/main.py b/workers/model-trellis/main.py
index a25446c0a..21f103934 100644
--- a/workers/model-trellis/main.py
+++ b/workers/model-trellis/main.py
@@ -110,11 +110,14 @@ from worker_security import (
from storage_backend import LocalStorage
from instance_health import (
HEARTBEAT_SECS,
+ VRAM_RECYCLE_FRACTION,
+ VRAM_RECYCLE_MIN_GROWTH,
MemorySample,
read_memory,
should_recycle,
task_is_orphaned,
trim_heap,
+ vram_sample,
)
logging.basicConfig(
@@ -190,6 +193,8 @@ _live_tasks: set[str] = set()
# Memory held right after the model finished loading: the reference the recycle
# guard measures growth against (see instance_health.should_recycle).
_memory_baseline: Optional[MemorySample] = None
+# Same, for GPU memory. None on a CPU-only host, or if the device read fails.
+_vram_baseline: Optional[MemorySample] = None
# Set once this instance has decided to restart between jobs; /infer then refuses
# new work with a 503 so the caller fails over instead of queueing behind it.
_draining = False
@@ -278,8 +283,21 @@ def _pin_dinov2_to_local_checkout() -> bool:
return True
+def _read_vram() -> Optional[MemorySample]:
+ """Current GPU memory held by this instance, or None off a CPU-only host.
+
+ The only place this module calls torch.cuda.mem_get_info(); every caller
+ (load-time baseline, post-job recycle check) goes through here so the
+ is_available() guard lives in exactly one spot.
+ """
+ if not torch.cuda.is_available():
+ return None
+ free_bytes, total_bytes = torch.cuda.mem_get_info()
+ return vram_sample(free_bytes, total_bytes)
+
+
def _load_pipeline():
- global _pipeline, _memory_baseline
+ global _pipeline, _memory_baseline, _vram_baseline
from trellis.pipelines import TrellisImageTo3DPipeline
weights_path = _stage_weights_local() or WEIGHTS_DIR
@@ -294,6 +312,9 @@ def _load_pipeline():
_memory_baseline = read_memory()
if _memory_baseline is not None:
log.info("memory after load: %s", _memory_baseline.describe())
+ _vram_baseline = _read_vram()
+ if _vram_baseline is not None:
+ log.info("vram after load: %s", _vram_baseline.describe())
def _release_staged_weights() -> None:
@@ -639,7 +660,7 @@ async def _run_inference(task_id: str, *args, **kwargs) -> None:
def _recycle_if_bloated(task_id: str) -> None:
- """Restart an idle instance whose memory has ratcheted toward its ceiling.
+ """Restart an idle instance whose memory or VRAM has ratcheted toward its ceiling.
Even with the heap trimmed after every job, the native libraries in the
postprocess (xatlas, nvdiffrast, open3d) keep some of what they allocate,
@@ -647,6 +668,14 @@ def _recycle_if_bloated(task_id: str) -> None:
tips it over is what used to happen: the OOM kill took that job and the one
queued behind it. Restarting while idle costs one model load instead.
+ The same is true of GPU memory, and independently of host RAM:
+ nvdiffrast's rasterizer allocates through its own cudaMalloc outside
+ torch's caching allocator (see _release_gpu_memory's comment), so VRAM
+ only ever ratchets down even though host RAM can look perfectly healthy.
+ Left unchecked, the instance's own /health liveness probe eventually
+ times out mid-inference and Cloud Run kills it, instead of a clean
+ restart between jobs.
+
Once it decides, the instance drains: /infer refuses new work with a 503 so
the caller fails over, jobs already queued here still run, and the restart
happens when the last of them finishes. Runs on the event loop with no
@@ -658,14 +687,25 @@ def _recycle_if_bloated(task_id: str) -> None:
sample = read_memory()
if sample is not None:
log.info("[%s] memory after job: %s", task_id, sample.describe())
+ vram = _read_vram()
+ if vram is not None:
+ log.info("[%s] vram after job: %s", task_id, vram.describe())
if not _draining:
- if not should_recycle(sample, _memory_baseline):
+ ram_bloated = should_recycle(sample, _memory_baseline)
+ vram_bloated = should_recycle(
+ vram, _vram_baseline, VRAM_RECYCLE_FRACTION, VRAM_RECYCLE_MIN_GROWTH
+ )
+ if not (ram_bloated or vram_bloated):
return
_draining = True
log.warning(
- "recycling instance: memory %s, up from %s after load; draining %d queued job(s) first",
- sample.describe(),
+ "recycling instance (%s bloated): memory %s, up from %s after load; "
+ "vram %s, up from %s after load; draining %d queued job(s) first",
+ "ram" if ram_bloated else "vram",
+ sample.describe() if sample else "unknown",
_memory_baseline.describe() if _memory_baseline else "unknown",
+ vram.describe() if vram else "unknown",
+ _vram_baseline.describe() if _vram_baseline else "unknown",
len(_live_tasks),
)
if _live_tasks or _restart_signalled:
diff --git a/workers/model-trellis/test_instance_health.py b/workers/model-trellis/test_instance_health.py
index 1fa83a9ae..26790de0c 100644
--- a/workers/model-trellis/test_instance_health.py
+++ b/workers/model-trellis/test_instance_health.py
@@ -15,11 +15,14 @@ from pathlib import Path
from instance_health import (
HEARTBEAT_SECS,
ORPHAN_AFTER_SECS,
+ VRAM_RECYCLE_FRACTION,
+ VRAM_RECYCLE_MIN_GROWTH,
MemorySample,
read_memory,
should_recycle,
task_is_orphaned,
trim_heap,
+ vram_sample,
)
GIB = 1024 ** 3
@@ -105,6 +108,61 @@ def test_unknown_memory_never_recycles():
assert not should_recycle(_at(0.95), None)
+# ---------------------------------------------------------------- vram_sample
+
+
+def test_vram_sample_shapes_mem_get_info_into_used_and_limit():
+ # torch.cuda.mem_get_info() returns (free_bytes, total_bytes).
+ sample = vram_sample(free_bytes=12 * GIB, total_bytes=24 * GIB)
+ assert sample == MemorySample(12 * GIB, 24 * GIB, "cuda")
+ assert abs(sample.fraction - 0.5) < 1e-9
+
+
+def test_vram_sample_with_no_device_is_unknown_not_an_error():
+ assert vram_sample(free_bytes=0, total_bytes=0) is None
+
+
+# ------------------------------------------------------------- should_recycle (VRAM)
+
+
+def _vram_at(fraction: float, total_bytes: int = 24 * GIB) -> MemorySample:
+ free = int((1 - fraction) * total_bytes)
+ return vram_sample(free_bytes=free, total_bytes=total_bytes)
+
+
+def test_recycles_a_ratcheted_vram_instance():
+ assert should_recycle(
+ _vram_at(0.95), _vram_at(0.70), VRAM_RECYCLE_FRACTION, VRAM_RECYCLE_MIN_GROWTH
+ )
+
+
+def test_leaves_a_healthy_vram_instance_alone():
+ assert not should_recycle(
+ _vram_at(0.80), _vram_at(0.70), VRAM_RECYCLE_FRACTION, VRAM_RECYCLE_MIN_GROWTH
+ )
+
+
+def test_high_vram_baseline_alone_never_recycles():
+ # A model that is simply large must not restart the instance after every
+ # single job just because its baseline already reads near the ceiling.
+ assert not should_recycle(
+ _vram_at(0.92), _vram_at(0.90), VRAM_RECYCLE_FRACTION, VRAM_RECYCLE_MIN_GROWTH
+ )
+
+
+def test_unknown_vram_never_recycles():
+ assert not should_recycle(None, _vram_at(0.40), VRAM_RECYCLE_FRACTION, VRAM_RECYCLE_MIN_GROWTH)
+ assert not should_recycle(_vram_at(0.95), None, VRAM_RECYCLE_FRACTION, VRAM_RECYCLE_MIN_GROWTH)
+
+
+def test_vram_threshold_is_tighter_than_host_memory():
+ # A VRAM OOM crashes the job outright with no graceful degradation, unlike
+ # host RAM pressure, so the ceiling this guard fires at is tighter.
+ from instance_health import RECYCLE_FRACTION
+
+ assert VRAM_RECYCLE_FRACTION > RECYCLE_FRACTION
+
+
# ---------------------------------------------------------------- task_is_orphaned
diff --git a/workers/model-trellis/test_vram_recycle.py b/workers/model-trellis/test_vram_recycle.py
new file mode 100644
index 000000000..1f840a9ff
--- /dev/null
+++ b/workers/model-trellis/test_vram_recycle.py
@@ -0,0 +1,117 @@
+"""Unit tests for the VRAM recycle guard wired into main.py's _recycle_if_bloated.
+
+Mocks torch.cuda.is_available()/mem_get_info() exactly as the pre-answered
+blocker for this order prescribes, so this runs on any host, including a
+CPU-only dev box with no torch package at all (see conftest.py's stub). It
+proves the gap main.py:829-845 already diagnoses in its own `finally` block
+comment: nvdiffrast's rasterizer allocates through its own cudaMalloc outside
+torch's caching allocator, so an instance's usable VRAM only ever ratchets
+down, invisible to the host-RAM recycle guard instance_health.py already had.
+
+ cd workers/model-trellis && python3 -m pytest test_vram_recycle.py -q
+"""
+
+from __future__ import annotations
+
+import os
+
+os.environ.setdefault("API_KEY", "vram-test-key")
+os.environ.setdefault("GCS_BUCKET", "vram-test-bucket")
+
+import main as app_module
+from instance_health import MemorySample
+
+GIB = 1024 ** 3
+VRAM_TOTAL = 24 * GIB
+
+
+def _reset_recycle_state() -> None:
+ app_module._draining = False
+ app_module._restart_signalled = False
+ app_module._live_tasks.clear()
+ app_module._memory_baseline = None
+ app_module._vram_baseline = None
+
+
+def setup_function() -> None:
+ _reset_recycle_state()
+
+
+def teardown_function() -> None:
+ _reset_recycle_state()
+
+
+def _healthy_ram(monkeypatch) -> None:
+ monkeypatch.setattr(
+ app_module, "read_memory", lambda: MemorySample(4 * GIB, 32 * GIB, "cgroup2")
+ )
+ app_module._memory_baseline = MemorySample(3 * GIB, 32 * GIB, "cgroup2")
+
+
+def test_vram_alone_drains_the_instance_even_with_healthy_ram(monkeypatch):
+ _healthy_ram(monkeypatch)
+
+ # VRAM: ratcheted from a 40% baseline to 96%, past both the VRAM fraction
+ # and growth floors, with torch.cuda available.
+ app_module._vram_baseline = MemorySample(int(0.40 * VRAM_TOTAL), VRAM_TOTAL, "cuda")
+ monkeypatch.setattr(app_module.torch.cuda, "is_available", lambda: True)
+ monkeypatch.setattr(
+ app_module.torch.cuda, "mem_get_info", lambda: (int(0.04 * VRAM_TOTAL), VRAM_TOTAL)
+ )
+
+ # A job is still queued behind this one, so the decision must be provable
+ # without actually reaching signal.raise_signal against the test process.
+ app_module._live_tasks.add("other-queued-job")
+
+ app_module._recycle_if_bloated("finished-job")
+
+ assert app_module._draining is True
+
+
+def test_a_healthy_instance_on_both_axes_is_left_alone(monkeypatch):
+ _healthy_ram(monkeypatch)
+ app_module._vram_baseline = MemorySample(int(0.40 * VRAM_TOTAL), VRAM_TOTAL, "cuda")
+ monkeypatch.setattr(app_module.torch.cuda, "is_available", lambda: True)
+ monkeypatch.setattr(
+ app_module.torch.cuda, "mem_get_info", lambda: (int(0.45 * VRAM_TOTAL), VRAM_TOTAL)
+ )
+
+ app_module._recycle_if_bloated("finished-job")
+
+ assert app_module._draining is False
+
+
+def test_a_vram_baseline_that_already_reads_high_alone_never_recycles(monkeypatch):
+ # A genuinely large model must not restart the instance after every job
+ # just because its post-load baseline already reads near the ceiling.
+ _healthy_ram(monkeypatch)
+ app_module._vram_baseline = MemorySample(int(0.89 * VRAM_TOTAL), VRAM_TOTAL, "cuda")
+ monkeypatch.setattr(app_module.torch.cuda, "is_available", lambda: True)
+ monkeypatch.setattr(
+ app_module.torch.cuda, "mem_get_info", lambda: (int(0.08 * VRAM_TOTAL), VRAM_TOTAL)
+ )
+
+ app_module._recycle_if_bloated("finished-job")
+
+ assert app_module._draining is False
+
+
+def test_read_vram_skips_mem_get_info_without_a_gpu(monkeypatch):
+ monkeypatch.setattr(app_module.torch.cuda, "is_available", lambda: False)
+
+ def _must_not_be_called():
+ raise AssertionError("mem_get_info must not be called without a GPU")
+
+ monkeypatch.setattr(app_module.torch.cuda, "mem_get_info", _must_not_be_called)
+
+ assert app_module._read_vram() is None
+
+
+def test_load_pipeline_captures_a_vram_baseline(monkeypatch):
+ free_bytes = int(0.70 * VRAM_TOTAL)
+ monkeypatch.setattr(app_module.torch.cuda, "is_available", lambda: True)
+ monkeypatch.setattr(app_module.torch.cuda, "mem_get_info", lambda: (free_bytes, VRAM_TOTAL))
+
+ app_module._vram_baseline = app_module._read_vram()
+
+ assert app_module._vram_baseline == MemorySample(VRAM_TOTAL - free_bytes, VRAM_TOTAL, "cuda")