Skip to content

armgit / avibe

Public Imported from GitHub
0

Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.

fix(codex): bound retained Model Hub catalog generations (#2024)

Call White09b2d41e294f
diff --git a/docs/plans/issue-1791-codex-catalog-retention.md b/docs/plans/issue-1791-codex-catalog-retention.md
new file mode 100644
index 000000000..ae4928cb0
--- /dev/null
+++ b/docs/plans/issue-1791-codex-catalog-retention.md
@@ -0,0 +1,65 @@
+# Bounded Codex Model Hub catalogs
+
+## Contract and cause
+
+Content-addressed publication has no retirement owner. The Agent caches a path,
+but each per-working-directory app-server continues to consume that path after
+the cache is invalidated. Export failure must never select an older generation.
+
+Codex 0.154.0 (`openai/codex`, commit
+`6b9826e3aa83b1a5947db50f4332cb9c65f1b340`) calls
+`ConfigManager::load_with_overrides` in `thread_start_task`; Config construction
+calls `load_model_catalog`, which reads the file again. The hermetic native
+consumer test removes the catalog after initialization and observes
+`thread/start` fail to load configuration. Handshake-only protection is unsafe.
+
+## Ownership and retention
+
+- Agent cache, pending launch and live transport acquire independent pins on
+ the immutable file and explicitly close them at their lifecycle boundaries.
+ Invalidating the cache does not release the transport's pin. Finalization is
+ only a fallback; captured exception tracebacks cannot delay ordinary release.
+- A short cross-process publication lock covers creating/reusing the immutable
+ path, acquiring its pin and sweeping. Identical bytes preserve the inode.
+ Its storage import is deferred to those operations, preserving storage-free
+ catalog and session-handler imports (`MEMORY-INDEP-009`).
+- POSIX uses shared file locks. Windows open handles deny deletion. The native
+ child inherits the pin, including when its Avibe parent exits first.
+- The pinned catalog path is the final app-server override, so backend extra
+ arguments cannot silently select a different, unprotected generation.
+- Retain all protected generations plus at most one unprotected history file,
+ selected deterministically by filename. Age and mtime never prove inactivity.
+- Only ordinary, singly linked files matching
+ `standard-responses-[0-9a-f]{16}.json` in the Codex Hub directory are eligible.
+ Unrelated names, directories, symlinks and hard links are untouched.
+- Cleanup is best effort at publication and release. Failure logs preserve the
+ path and exception; publication and existing consumers remain usable. The
+ bound is restored by a successful later sweep.
+
+## Scope and validation
+
+Changes stay in catalog publication and the Codex Agent/transport lifecycle.
+No database, daemon, shared lifecycle rewrite or production-state cleanup.
+Validation covers unchanged/changed content, startup and active pins,
+invalidation, failed/cancelled launch, actual exit, export and cleanup failures,
+cross-process races, and the actual native consumer in isolated homes with
+loopback-only egress. The inheritance probe chooses a consumer catalog that
+cannot occupy the single unpinned spare; disabling only child inheritance
+deletes it and makes `thread/start` fail, while inheritance preserves both.
+After actual child exit, the next publication reclaims that generation.
+
+The native inheritance and configuration-reread evidence is for Codex 0.154.0 on
+macOS. Custom launcher wrappers must preserve inherited handles for protection
+after the Avibe parent dies; normal parent-owned lifecycle protection does not
+depend on child inheritance. Pinned descendants can conservatively extend a
+generation's lifetime. This change does not manage manually launched processes
+that bypass Avibe's pin-acquisition protocol.
+
+## Progress
+
+- [x] Read current issue, default branch and consumer lifecycle.
+- [x] Reproduce post-initialize catalog reread with native Codex 0.154.0.
+- [x] Implement and test ownership, retention and failure boundaries.
+- [x] Complete focused suites and changed-file Ruff: 539 passed, 8 opt-in native
+ prompt tests skipped; three additional native catalog cases passed on 0.154.0.
+- Exact-head Codex review and GitHub CI are tracked in the implementation PR.
diff --git a/modules/agents/codex/agent.py b/modules/agents/codex/agent.py
index 82bc2412b..ca2a1ba32 100644
--- a/modules/agents/codex/agent.py
+++ b/modules/agents/codex/agent.py
@@ -8,6 +8,7 @@ import logging
import os
import shlex
import time
+from contextlib import AsyncExitStack
from pathlib import Path
from typing import TYPE_CHECKING, Any, Awaitable, Callable, Dict, Optional
@@ -63,6 +64,7 @@ logger = logging.getLogger(__name__)
if TYPE_CHECKING:
from modules.agents.model_hub import ModelHubLaunch
+ from vibe.backend_model_catalog import CodexHubCatalog
_CODEX_MANAGED_PROVIDER_IDS = frozenset((MANAGED_PROVIDER_ID, *LEGACY_MANAGED_PROVIDER_IDS))
_CODEX_MODEL_HUB_PROVIDER_ID = "avibe_model_hub"
@@ -142,7 +144,7 @@ class CodexAgent(BaseAgent):
super().__init__(controller)
self.codex_config = codex_config
self._registered_runtime = registered_runtime
- self._model_hub_catalog_path: Path | None = None
+ self._model_hub_catalog: CodexHubCatalog | None = None
self._model_hub_catalog_lock = asyncio.Lock()
self._model_hub_catalog_generation = 0
@@ -856,22 +858,23 @@ class CodexAgent(BaseAgent):
"""Reload persisted runtime config before respawning app-server transports."""
self.codex_config = codex_config
self.controller.config.codex = codex_config
- self._model_hub_catalog_generation += 1
- self._model_hub_catalog_path = None
+ await self.invalidate_model_hub_runtime()
await self.refresh_auth_state()
async def invalidate_model_hub_runtime(self) -> None:
"""Make the next Hub launch rebuild its catalog without touching Direct transports."""
self._model_hub_catalog_generation += 1
- self._model_hub_catalog_path = None
+ if self._model_hub_catalog is not None:
+ self._model_hub_catalog.close()
+ self._model_hub_catalog = None
- async def prepare_model_hub_runtime(self) -> Path:
+ async def prepare_model_hub_runtime(self) -> CodexHubCatalog:
"""Bind Hub metadata to this Agent's exact configured Codex binary."""
from vibe import backend_model_catalog
async with self._model_hub_catalog_lock:
- if self._model_hub_catalog_path is not None:
- return self._model_hub_catalog_path
+ if self._model_hub_catalog is not None:
+ return self._model_hub_catalog
generation = self._model_hub_catalog_generation
binary = self.codex_config.binary
configured_models = None
@@ -882,23 +885,39 @@ class CodexAgent(BaseAgent):
model.to_payload()
for model in store.load().agents["codex"].models
]
- try:
- path = await asyncio.to_thread(
+ preparation = asyncio.create_task(
+ asyncio.to_thread(
backend_model_catalog.prepare_codex_hub_catalog,
binary,
None,
configured_models,
)
+ )
+ try:
+ catalog = await asyncio.shield(preparation)
+ except asyncio.CancelledError:
+ # The export runs in a thread and cannot be cancelled. Its
+ # result must still release its pin after the caller leaves.
+ preparation.add_done_callback(self._discard_model_hub_catalog)
+ raise
except Exception as exc:
raise CodexModelHubCatalogUnavailableError(
"Codex Model Hub catalog preparation failed"
) from exc
if self._model_hub_catalog_generation != generation:
+ catalog.close()
raise CodexModelHubCatalogUnavailableError(
"Codex Model Hub catalog generation changed during preparation"
)
- self._model_hub_catalog_path = path
- return path
+ self._model_hub_catalog = catalog
+ return catalog
+
+ @staticmethod
+ def _discard_model_hub_catalog(preparation: asyncio.Task) -> None:
+ try:
+ preparation.result().close()
+ except Exception:
+ logger.warning("Cancelled Codex catalog preparation failed", exc_info=True)
async def prepare_resume_binding(
self,
@@ -949,6 +968,7 @@ class CodexAgent(BaseAgent):
async def shutdown_runtime(self) -> None:
"""Stop all app-server transports during vibe-remote shutdown."""
+ await self.invalidate_model_hub_runtime()
if not hasattr(self, "_transport_last_activity"):
self._transport_last_activity = {}
if not hasattr(self, "_transport_locks"):
@@ -1674,7 +1694,7 @@ class CodexAgent(BaseAgent):
while True:
wait_for_active_turns = False
- async with self._transport_locks[cwd]:
+ async with self._transport_locks[cwd], AsyncExitStack() as catalog_pins:
# Double-check after acquiring lock
existing = self._transports.get(cwd)
existing_dead = bool(
@@ -1716,19 +1736,21 @@ class CodexAgent(BaseAgent):
runtime_args: list[str] = []
runtime_env: dict[str, str] | None = None
runtime_fingerprint = "direct"
+ catalog = None
if launch is not None:
from modules.agents.model_hub import build_codex_hub_launch
- if (
- launch.channel == "hub"
- and self._model_hub_catalog_path is None
- ):
- await self.prepare_model_hub_runtime()
+ if launch.channel == "hub":
+ # Capture the reference before any later await.
+ # Invalidation may drop the cache while we retire
+ # an old transport or spawn/initialize its successor.
+ catalog = (await self.prepare_model_hub_runtime()).retain()
+ catalog_pins.callback(catalog.close)
runtime_args, runtime_env = build_codex_hub_launch(
[],
os.environ.copy(),
launch,
- model_catalog_path=self._model_hub_catalog_path,
+ model_catalog_path=catalog.path if catalog is not None else None,
)
runtime_fingerprint = launch.fingerprint
@@ -1794,6 +1816,7 @@ class CodexAgent(BaseAgent):
runtime_args=runtime_args,
runtime_env=runtime_env,
runtime_fingerprint=runtime_fingerprint,
+ model_hub_catalog=catalog,
)
# Wire up callbacks
diff --git a/modules/agents/codex/transport.py b/modules/agents/codex/transport.py
index cc21df2b1..6953adaf9 100644
--- a/modules/agents/codex/transport.py
+++ b/modules/agents/codex/transport.py
@@ -8,13 +8,18 @@ import logging
import os
import signal
from asyncio.subprocess import Process
-from typing import Any, Awaitable, Callable, Optional
+from contextlib import nullcontext
+from typing import TYPE_CHECKING, Any, Awaitable, Callable, Optional
from core.process_diagnostics import log_process_snapshot, process_identity
from core.process_isolation import KILL_SIGNAL, isolated_subprocess_kwargs, signal_process_tree
+from vibe.codex_config import format_toml_basic_string
logger = logging.getLogger(__name__)
+if TYPE_CHECKING:
+ from vibe.backend_model_catalog import CodexHubCatalog
+
STREAM_BUFFER_LIMIT = 128 * 1024 * 1024 # 128 MB
@@ -97,6 +102,7 @@ class CodexTransport:
runtime_args: list[str] | None = None,
runtime_env: dict[str, str] | None = None,
runtime_fingerprint: str = "direct",
+ model_hub_catalog: CodexHubCatalog | None = None,
) -> None:
self._binary = binary
self._cwd = cwd
@@ -104,6 +110,9 @@ class CodexTransport:
self._runtime_args = runtime_args or []
self._runtime_env = runtime_env
self.runtime_fingerprint = runtime_fingerprint
+ self._model_hub_catalog = model_hub_catalog.retain() if model_hub_catalog is not None else None
+ self._catalog_required = model_hub_catalog is not None
+ self._catalog_exit_task: asyncio.Task[None] | None = None
self._process: Optional[Process] = None
self._request_id: int = 0
self._pending: dict[int | str, asyncio.Future[dict[str, Any]]] = {}
@@ -129,9 +138,19 @@ class CodexTransport:
async def start(self) -> None:
"""Launch the app-server and perform the ``initialize`` handshake."""
+ try:
+ await self._start()
+ except BaseException:
+ if self._process is None or self._process.returncode is not None:
+ self._release_model_hub_catalog()
+ raise
+
+ async def _start(self) -> None:
if self._process and self._process.returncode is None:
logger.warning("CodexTransport.start() called but process is already running")
return
+ if self._catalog_required and self._model_hub_catalog is None:
+ raise RuntimeError("A stopped Codex Hub transport needs a newly prepared catalog")
self._closed_event.clear()
cmd = (
@@ -141,6 +160,10 @@ class CodexTransport:
+ self._extra_args
+ _avibe_app_server_config_args()
)
+ if self._model_hub_catalog is not None:
+ # The path actually consumed must be the one we pin, even if a
+ # backend extra argument tries to select a different generation.
+ cmd += ["-c", f"model_catalog_json={format_toml_basic_string(str(self._model_hub_catalog.path))}"]
logger.info("Launching Codex app-server: %s (cwd=%s)", " ".join(cmd), self._cwd)
if not os.path.exists(self._cwd):
@@ -156,7 +179,24 @@ class CodexTransport:
}
if self._runtime_env is not None:
subprocess_kwargs["env"] = self._runtime_env
- self._process = await asyncio.create_subprocess_exec(*cmd, **subprocess_kwargs)
+ catalog = self._model_hub_catalog
+ inheritance = catalog.inherited_subprocess_kwargs() if catalog is not None else nullcontext({})
+ with inheritance as catalog_kwargs:
+ spawn = asyncio.create_task(
+ asyncio.create_subprocess_exec(*cmd, **subprocess_kwargs, **catalog_kwargs)
+ )
+ try:
+ self._process = await asyncio.shield(spawn)
+ except asyncio.CancelledError:
+ # Cancellation can race the OS spawn. Settle it before closing
+ # inherited handles or losing the only reference to this child.
+ self._process = await spawn
+ await self.stop()
+ raise
+ if catalog is not None:
+ self._catalog_exit_task = asyncio.create_task(
+ self._release_catalog_on_exit(self._process, catalog)
+ )
identity = process_identity(self._process.pid)
logger.info(
"Codex app-server started (pid=%s pgid=%s sid=%s service_pgid=%s)",
@@ -218,6 +258,7 @@ class CodexTransport:
self._initialized = False
proc = self._process
if not proc or proc.returncode is not None:
+ self._release_model_hub_catalog()
self._closed_event.set()
self._cleanup_tasks()
return
@@ -243,6 +284,7 @@ class CodexTransport:
pass
self._cleanup_tasks()
+ self._release_model_hub_catalog()
self._closed_event.set()
# Fail all pending futures
for fut in self._pending.values():
@@ -251,6 +293,18 @@ class CodexTransport:
self._pending.clear()
logger.info("Codex app-server stopped")
+ async def _release_catalog_on_exit(self, proc: Process, catalog: CodexHubCatalog) -> None:
+ """Stdout EOF is not process exit; keep the pin until wait confirms it."""
+ await proc.wait()
+ catalog.close()
+ if self._model_hub_catalog is catalog:
+ self._model_hub_catalog = None
+
+ def _release_model_hub_catalog(self) -> None:
+ if self._model_hub_catalog is not None:
+ self._model_hub_catalog.close()
+ self._model_hub_catalog = None
+
def _cleanup_tasks(self) -> None:
for task in (self._reader_task, self._stderr_task, self._notify_task):
if task and not task.done():
diff --git a/tests/e2e/test_model_hub_catalog_consumer.py b/tests/e2e/test_model_hub_catalog_consumer.py
index a09b21fa7..15e3dde9b 100644
--- a/tests/e2e/test_model_hub_catalog_consumer.py
+++ b/tests/e2e/test_model_hub_catalog_consumer.py
@@ -9,12 +9,14 @@ import shlex
import shutil
import socket
import subprocess
+from contextlib import nullcontext
from pathlib import Path
import pytest
from config.v2_config import ModelHubBackendModelConfig
from modules.agents.codex.transport import CodexTransport
+from modules.agents.codex.transport import CodexRPCError
from modules.agents.model_hub import (
ModelHubLaunch,
build_claude_hub_env,
@@ -24,7 +26,12 @@ from modules.agents.model_hub import (
)
from tests.e2e.drivers.model_hub_app import ModelHubTestApp
from tests.e2e.drivers.mock_llm_upstream import MockLLMUpstream
-from vibe.backend_model_catalog import _codex_hub_catalog_bytes
+from vibe.backend_model_catalog import (
+ CodexHubCatalog,
+ _codex_hub_catalog_bytes,
+ _codex_hub_catalog_path,
+ _publish_codex_hub_catalog,
+)
pytestmark = pytest.mark.e2e_model_hub
@@ -155,6 +162,94 @@ async def _codex_turn(binary, runtime, gateway, catalog, *, effort, native_confi
return bodies, models, notifications, config["config"]
+def test_codex_reloads_catalog_after_initialize(codex_catalog_runtime):
+ """MH-PROTOCOL-004: a live app-server rereads its catalog for a new thread."""
+ binary, runtime, raw_catalog = codex_catalog_runtime
+ catalog_path = runtime.home / "reload-catalog.json"
+ catalog_path.write_bytes(_codex_hub_catalog_bytes(raw_catalog))
+
+ async def probe():
+ transport = CodexTransport(
+ binary=binary,
+ cwd=str(runtime.home),
+ runtime_args=["-c", f"model_catalog_json={json.dumps(str(catalog_path))}"],
+ runtime_env=runtime.env,
+ )
+ try:
+ await transport.start()
+ catalog_path.unlink()
+ with pytest.raises(CodexRPCError, match="failed to load configuration"):
+ await transport.send_request(
+ "thread/start",
+ {"cwd": str(runtime.home), "ephemeral": True},
+ )
+ finally:
+ await transport.stop()
+
+ asyncio.run(probe())
+
+
[email protected]("inherit", [True, False])
+def test_codex_inherits_catalog_pin(codex_catalog_runtime, monkeypatch, inherit):
+ """MH-PROTOCOL-004: only a child pin, not the history spare, saves its catalog."""
+ binary, runtime, raw_catalog = codex_catalog_runtime
+ # Preserve native model behavior while producing distinct valid catalogs.
+ # Choose the consumer's path after sorting, so it cannot win the history
+ # spare by chance, regardless of which catalog this Codex version bundles.
+ variants = []
+ for index in range(5):
+ payload = json.loads(raw_catalog)
+ payload["models"][0]["display_name"] = f"Retention fixture {index}"
+ variants.append(json.dumps(payload).encode())
+ variants.sort(key=lambda raw: _codex_hub_catalog_path(_codex_hub_catalog_bytes(raw)).name)
+ catalog = _publish_codex_hub_catalog(variants[-1])
+ path = catalog.path
+ if not inherit:
+ monkeypatch.setattr(CodexHubCatalog, "inherited_subprocess_kwargs", lambda self: nullcontext({}))
+
+ async def probe():
+ transport = CodexTransport(
+ binary=binary,
+ cwd=str(runtime.home),
+ runtime_args=["-c", f"model_catalog_json={json.dumps(str(path))}"],
+ runtime_env=runtime.env,
+ model_hub_catalog=catalog,
+ )
+ try:
+ await transport.start()
+ # Simulate loss of the parent's descriptor without killing pytest.
+ catalog.close()
+ transport._model_hub_catalog.close()
+ for raw in variants[:-1]:
+ with _publish_codex_hub_catalog(raw) as successor:
+ assert successor.path.name < path.name
+ assert path.is_file() is inherit
+ if inherit:
+ thread = await transport.send_request(
+ "thread/start",
+ {"cwd": str(runtime.home), "ephemeral": True},
+ )
+ assert thread["thread"]["id"]
+ else:
+ with pytest.raises(CodexRPCError, match="failed to load configuration"):
+ await transport.send_request(
+ "thread/start",
+ {"cwd": str(runtime.home), "ephemeral": True},
+ )
+ finally:
+ catalog.close()
+ await transport.stop()
+ assert transport._process.returncode is not None
+ # Simulated parent loss already closed every parent pin. The next
+ # publication must reclaim the child's catalog once it really exits.
+ with _publish_codex_hub_catalog(variants[0]):
+ pass
+ assert not path.exists()
+ assert len(list(path.parent.glob("standard-responses-*.json"))) == 1
+
+ asyncio.run(probe())
+
+
@pytest.mark.parametrize("effort", [None, "none", "minimal", "low", "medium", "high", "xhigh", "max"])
def test_codex_unknown_catalog_preserves_supplied_image_and_effort(codex_catalog_runtime, effort):
"""MH-PROTOCOL-004: absence must survive the real catalog/turn consumer."""
diff --git a/tests/scenarios/model_hub/catalog.yaml b/tests/scenarios/model_hub/catalog.yaml
index b96c97e87..197765cb0 100644
--- a/tests/scenarios/model_hub/catalog.yaml
+++ b/tests/scenarios/model_hub/catalog.yaml
@@ -390,6 +390,8 @@ scenarios:
- tests/e2e/test_model_hub_catalog_consumer.py::test_codex_explicit_catalog_choices_reach_the_consumer
- tests/e2e/test_model_hub_catalog_consumer.py::test_codex_native_context_override_wins_over_catalog_planning
- tests/e2e/test_model_hub_catalog_consumer.py::test_codex_auto_compaction_consumes_resolved_limits
+ - tests/e2e/test_model_hub_catalog_consumer.py::test_codex_reloads_catalog_after_initialize
+ - tests/e2e/test_model_hub_catalog_consumer.py::test_codex_inherits_catalog_pin
- id: MH-CLAUDE-LAUNCH-001
name: Claude Hub turns preserve the configured Source and model despite native connection settings
status: covered
diff --git a/tests/test_agent_auth_service.py b/tests/test_agent_auth_service.py
index 499c72a98..049cfafd2 100644
--- a/tests/test_agent_auth_service.py
+++ b/tests/test_agent_auth_service.py
@@ -2236,7 +2236,7 @@ class AgentAuthServiceTests(_IsolatedClaudeConfigDirMixin, unittest.IsolatedAsyn
new_config = CodexCompatConfig(enabled=True, binary="/new/codex", extra_args=[])
agent = CodexAgent.__new__(CodexAgent)
agent.codex_config = old_config
- agent._model_hub_catalog_path = Path("/runtime/codex-old.json")
+ agent._model_hub_catalog = SimpleNamespace(path=Path("/runtime/codex-old.json"), close=Mock())
agent._model_hub_catalog_lock = asyncio.Lock()
agent._model_hub_catalog_generation = 0
agent.controller = SimpleNamespace(config=SimpleNamespace(codex=old_config))
@@ -2251,7 +2251,7 @@ class AgentAuthServiceTests(_IsolatedClaudeConfigDirMixin, unittest.IsolatedAsyn
prepare_catalog.assert_not_called()
self.assertIs(agent.codex_config, new_config)
self.assertIs(agent.controller.config.codex, new_config)
- self.assertIsNone(agent._model_hub_catalog_path)
+ self.assertIsNone(agent._model_hub_catalog)
self.assertEqual(agent._model_hub_catalog_generation, 1)
agent.refresh_auth_state.assert_awaited_once()
diff --git a/tests/test_backend_model_catalog.py b/tests/test_backend_model_catalog.py
index 88a8d52c6..f112a0ef4 100644
--- a/tests/test_backend_model_catalog.py
+++ b/tests/test_backend_model_catalog.py
@@ -181,8 +181,8 @@ def test_codex_hub_catalog_projects_complete_catalog(monkeypatch, tmp_path):
).encode()
)
- assert path.name.startswith("standard-responses-")
- payload = json.loads(path.read_text(encoding="utf-8"))
+ assert path.path.name.startswith("standard-responses-")
+ payload = json.loads(path.path.read_text(encoding="utf-8"))
assert payload["client_version"] == "0.149.1"
assert payload["models"][0] == {
"slug": "gpt-5.6-luna",
@@ -663,9 +663,9 @@ def test_codex_hub_catalog_preparation_exports_current_binary(monkeypatch, tmp_p
exported = backend_model_catalog.prepare_codex_hub_catalog("codex")
- assert exported.name.startswith("standard-responses-")
+ assert exported.path.name.startswith("standard-responses-")
assert calls == [("codex", None)]
- assert json.loads(exported.read_text(encoding="utf-8"))["models"][0]["use_responses_lite"] is False
+ assert json.loads(exported.path.read_text(encoding="utf-8"))["models"][0]["use_responses_lite"] is False
def test_codex_hub_catalog_failure_cannot_select_a_previous_generation(monkeypatch, tmp_path):
@@ -681,8 +681,8 @@ def test_codex_hub_catalog_failure_cannot_select_a_previous_generation(monkeypat
with pytest.raises(RuntimeError, match="export failed"):
backend_model_catalog.prepare_codex_hub_catalog("/opt/codex")
- assert previous.exists()
- assert list(previous.parent.glob("standard-responses-*.json")) == [previous]
+ assert previous.path.exists()
+ assert list(previous.path.parent.glob("standard-responses-*.json")) == [previous.path]
def test_codex_hub_catalog_generations_are_content_addressed(monkeypatch, tmp_path):
@@ -692,9 +692,9 @@ def test_codex_hub_catalog_generations_are_content_addressed(monkeypatch, tmp_pa
first = backend_model_catalog._publish_codex_hub_catalog(b'{"models":[{"slug":"gpt-first"}]}')
second = backend_model_catalog._publish_codex_hub_catalog(b'{"models":[{"slug":"gpt-second"}]}')
- assert first != second
- assert json.loads(first.read_text(encoding="utf-8"))["models"][0]["slug"] == "gpt-first"
- assert json.loads(second.read_text(encoding="utf-8"))["models"][0]["slug"] == "gpt-second"
+ assert first.path != second.path
+ assert json.loads(first.path.read_text(encoding="utf-8"))["models"][0]["slug"] == "gpt-first"
+ assert json.loads(second.path.read_text(encoding="utf-8"))["models"][0]["slug"] == "gpt-second"
def test_merge_sources_applies_tombstones_and_fills_missing_metadata():
diff --git a/tests/test_codex_agent.py b/tests/test_codex_agent.py
index f294a235f..8724eb83e 100644
--- a/tests/test_codex_agent.py
+++ b/tests/test_codex_agent.py
@@ -22,6 +22,12 @@ from modules.agents.codex.transport import CodexRPCError
_AGENT_PATH = Path(__file__).resolve().parents[1] / "modules/agents/codex/agent.py"
+
+def _catalog_reference(path):
+ catalog = SimpleNamespace(path=Path(path), close=Mock())
+ catalog.retain = Mock(return_value=catalog)
+ return catalog
+
_modules_pkg = types.ModuleType("modules")
_agents_pkg = types.ModuleType("modules.agents")
_codex_pkg = types.ModuleType("modules.agents.codex")
@@ -594,6 +600,8 @@ class CodexAgentConnectionProbeTests(unittest.IsolatedAsyncioTestCase):
)
self.assertIsNone(agent._attach_transport_activation(cwd, transport))
+ agent._model_hub_catalog_generation = 0
+ agent._model_hub_catalog = None
await agent.shutdown_runtime()
transport.stop.assert_awaited_once_with()
@@ -897,6 +905,8 @@ class CodexAgentStopTests(unittest.IsolatedAsyncioTestCase):
transport.stop = stop_transport
+ agent._model_hub_catalog_generation = 0
+ agent._model_hub_catalog = None
await agent.shutdown_runtime()
late_commit.assert_not_called()
@@ -5413,7 +5423,7 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
agent._session_locks = {}
agent._session_mgr = SimpleNamespace(sessions_for_cwd=lambda cwd: [])
agent.codex_config = SimpleNamespace(binary="codex", extra_args=[])
- agent._model_hub_catalog_path = None
+ agent._model_hub_catalog = None
agent._model_hub_catalog_lock = asyncio.Lock()
agent._model_hub_catalog_generation = 0
agent.controller = SimpleNamespace(config=SimpleNamespace(codex=agent.codex_config))
@@ -5698,7 +5708,7 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
gateway_base_url="http://127.0.0.1:8317",
gateway_token="ephemeral-token",
)
- agent._model_hub_catalog_path = Path(cwd) / "codex-hub-catalog.json"
+ agent._model_hub_catalog = _catalog_reference(Path(cwd) / "codex-hub-catalog.json")
with patch(
"vibe.backend_model_catalog.prepare_codex_hub_catalog",
@@ -5712,8 +5722,8 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
async def test_runtime_config_switches_binary_without_catalog_export(self):
agent = self._agent()
- previous_catalog = Path("/runtime/codex-old.json")
- agent._model_hub_catalog_path = previous_catalog
+ previous_catalog = _catalog_reference("/runtime/codex-old.json")
+ agent._model_hub_catalog = previous_catalog
next_config = SimpleNamespace(binary="/opt/codex-next", extra_args=[])
agent.refresh_auth_state = AsyncMock()
@@ -5726,14 +5736,14 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
prepare_catalog.assert_not_called()
self.assertIs(agent.codex_config, next_config)
self.assertIs(agent.controller.config.codex, next_config)
- self.assertIsNone(agent._model_hub_catalog_path)
+ self.assertIsNone(agent._model_hub_catalog)
self.assertEqual(agent._model_hub_catalog_generation, 1)
agent.refresh_auth_state.assert_awaited_once_with()
async def test_runtime_config_same_binary_invalidates_prepared_catalog(self):
agent = self._agent()
- previous_catalog = Path("/runtime/codex-old.json")
- agent._model_hub_catalog_path = previous_catalog
+ previous_catalog = _catalog_reference("/runtime/codex-old.json")
+ agent._model_hub_catalog = previous_catalog
agent.refresh_auth_state = AsyncMock()
next_config = SimpleNamespace(binary=agent.codex_config.binary, extra_args=["--next"])
@@ -5745,7 +5755,7 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
prepare_catalog.assert_not_called()
self.assertIs(agent.codex_config, next_config)
- self.assertIsNone(agent._model_hub_catalog_path)
+ self.assertIsNone(agent._model_hub_catalog)
self.assertEqual(agent._model_hub_catalog_generation, 1)
agent.refresh_auth_state.assert_awaited_once_with()
@@ -5753,19 +5763,19 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
agent = self._agent()
transport = SimpleNamespace(stop=AsyncMock())
agent._transports["/repo"] = transport
- agent._model_hub_catalog_path = Path("/runtime/codex-old.json")
+ agent._model_hub_catalog = _catalog_reference("/runtime/codex-old.json")
await agent.invalidate_model_hub_runtime()
- self.assertIsNone(agent._model_hub_catalog_path)
+ self.assertIsNone(agent._model_hub_catalog)
self.assertEqual(agent._model_hub_catalog_generation, 1)
transport.stop.assert_not_awaited()
async def test_startup_catalog_preparation_cannot_overwrite_new_runtime_generation(self):
agent = self._agent()
previous_config = agent.codex_config
- previous_catalog = Path("/runtime/codex-old.json")
- next_catalog = Path("/runtime/codex-new.json")
+ previous_catalog = _catalog_reference("/runtime/codex-old.json")
+ next_catalog = _catalog_reference("/runtime/codex-new.json")
next_config = SimpleNamespace(binary="/opt/codex-next", extra_args=[])
agent.refresh_auth_state = AsyncMock()
previous_started = threading.Event()
@@ -5788,7 +5798,7 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
self.assertTrue(await asyncio.to_thread(previous_started.wait, 1))
await agent.refresh_runtime_config(next_config)
self.assertIs(agent.codex_config, next_config)
- self.assertIsNone(agent._model_hub_catalog_path)
+ self.assertIsNone(agent._model_hub_catalog)
release_previous.set()
with self.assertRaises(_MODULE.CodexModelHubCatalogUnavailableError):
await startup
@@ -5802,12 +5812,12 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
],
)
self.assertIs(agent.codex_config, next_config)
- self.assertEqual(agent._model_hub_catalog_path, next_catalog)
+ self.assertEqual(agent._model_hub_catalog, next_catalog)
self.assertEqual(recovered, next_catalog)
async def test_model_hub_catalog_preparation_retries_after_transient_failure(self):
agent = self._agent()
- catalog = Path("/runtime/codex-recovered.json")
+ catalog = _catalog_reference("/runtime/codex-recovered.json")
with patch(
"vibe.backend_model_catalog.prepare_codex_hub_catalog",
@@ -5815,11 +5825,11 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):
) as prepare_catalog:
with self.assertRaises(_MODULE.CodexModelHubCatalogUnavailableError):
await agent.prepare_model_hub_runtime()
- self.assertIsNone(agent._model_hub_catalog_path)
+ self.assertIsNone(agent._model_hub_catalog)
recovered = await agent.prepare_model_hub_runtime()
self.assertEqual(recovered, catalog)
- self.assertEqual(agent._model_hub_catalog_path, catalog)
+ self.assertEqual(agent._model_hub_catalog, catalog)
self.assertEqual(prepare_catalog.call_count, 2)
async def test_missing_prepared_hub_catalog_preserves_existing_transport_and_threads(self):
diff --git a/tests/test_codex_catalog_retention.py b/tests/test_codex_catalog_retention.py
new file mode 100644
index 000000000..0e86a6d6e
--- /dev/null
+++ b/tests/test_codex_catalog_retention.py
@@ -0,0 +1,466 @@
+"""Catalog ownership tests use only the autouse test-owned Avibe home."""
+
+from __future__ import annotations
+
+import asyncio
+import gc
+import json
+import os
+import subprocess
+import sys
+import threading
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+
+from config import paths
+from modules.agents.codex.agent import CodexAgent, CodexModelHubCatalogUnavailableError
+from modules.agents.codex.transport import CodexTransport
+from vibe import backend_model_catalog as catalogs
+
+
+def publish(slug):
+ return catalogs._publish_codex_hub_catalog(json.dumps({"models": [{"slug": slug}]}).encode())
+
+
+def retained():
+ return set((paths.get_runtime_dir() / "model-hub" / "codex").glob("standard-responses-*.json"))
+
+
+def churn(count=6):
+ for index in range(count):
+ with publish(f"churn-{index}"):
+ pass
+ gc.collect()
+
+
+def agent():
+ value = CodexAgent.__new__(CodexAgent)
+ value._model_hub_catalog = None
+ value._model_hub_catalog_lock = asyncio.Lock()
+ value._model_hub_catalog_generation = 0
+ value.codex_config = SimpleNamespace(binary="fixture-codex", extra_args=[])
+ value.controller = SimpleNamespace()
+ return value
+
+
+def test_same_content_reuses_inode_and_each_reader_protects_it():
+ first = publish("中文")
+ path = first.path
+ inode = path.stat().st_ino
+ second = publish("中文")
+ assert first.path == second.path
+ assert second.path.stat().st_ino == inode
+ first.close()
+ churn()
+ assert path in retained()
+ assert len(retained() - {path}) <= catalogs.CODEX_HUB_CATALOG_HISTORY_LIMIT
+ second.close()
+ churn()
+ assert len(retained()) <= catalogs.CODEX_HUB_CATALOG_HISTORY_LIMIT
+
+
+def test_many_active_generations_are_separate_from_bounded_history():
+ active = [publish(f"active-{index}") for index in range(4)]
+ protected = {value.path for value in active}
+ # Deliberately ancient timestamps cannot turn a live reader into history.
+ for path in protected:
+ os.utime(path, (1, 1))
+ churn(20)
+ assert protected <= retained()
+ assert len(retained() - protected) <= catalogs.CODEX_HUB_CATALOG_HISTORY_LIMIT
+ for value in active:
+ value.close()
+ active.clear()
+ churn()
+ assert len(retained()) <= catalogs.CODEX_HUB_CATALOG_HISTORY_LIMIT
+
+
+def test_cleanup_preserves_foreign_names_symlinks_directories_and_hardlinks(tmp_path):
+ active = publish("active")
+ directory = active.path.parent
+ foreign = directory / "user-catalog.json"
+ foreign.write_text("user content")
+ malformed = directory / "standard-responses-old.json"
+ malformed.write_text("released name prefix is insufficient")
+ subdir = directory / ("standard-responses-" + "a" * 16 + ".json")
+ subdir.mkdir()
+ outside = tmp_path / "outside.json"
+ outside.write_text("outside")
+ symlink = directory / ("standard-responses-" + "b" * 16 + ".json")
+ symlink.symlink_to(outside)
+ hardlink = directory / ("standard-responses-" + "c" * 16 + ".json")
+ os.link(outside, hardlink)
+ churn()
+ assert foreign.read_text() == "user content"
+ assert malformed.is_file() and subdir.is_dir() and symlink.is_symlink()
+ assert hardlink.read_text() == outside.read_text() == "outside"
+
+
+def test_cleanup_failure_does_not_break_publication_or_existing_reader(monkeypatch, caplog):
+ active = publish("active")
+ path = active.path
+ real_unlink = Path.unlink
+
+ def denied(value, *args, **kwargs):
+ if catalogs._CODEX_HUB_CATALOG_NAME.fullmatch(value.name):
+ raise PermissionError("fixture cleanup denial")
+ return real_unlink(value, *args, **kwargs)
+
+ with monkeypatch.context() as patch:
+ patch.setattr(Path, "unlink", denied)
+ churn()
+ latest = publish("new-success")
+ assert path.is_file()
+ assert json.loads(latest.path.read_bytes())["models"][0]["slug"] == "new-success"
+ assert "fixture cleanup denial" in caplog.text
+ assert "Could not reclaim Codex Model Hub catalog" in caplog.text
+ latest.close()
+ churn()
+ assert len(retained() - {path}) <= 1
+
+
+def test_failed_publication_or_export_cannot_select_old_catalog(monkeypatch):
+ active = publish("old")
+ monkeypatch.setattr(catalogs, "write_atomic", lambda *_: (_ for _ in ()).throw(OSError("disk full")))
+ with pytest.raises(OSError, match="disk full"):
+ publish("new")
+ assert active.path.is_file()
+ monkeypatch.setattr(
+ catalogs, "_export_codex_bundled_catalog",
+ lambda *_: (_ for _ in ()).throw(RuntimeError("export failed")),
+ )
+ with pytest.raises(RuntimeError, match="export failed"):
+ catalogs.prepare_codex_hub_catalog("fixture-codex")
+ assert retained() == {active.path}
+
+
+def test_concurrent_publication_and_cleanup_preserve_every_returned_pin():
+ barrier = threading.Barrier(8)
+
+ def launch(index):
+ barrier.wait(timeout=5)
+ value = publish(f"concurrent-{index % 3}")
+ catalogs.prune_codex_hub_catalogs(value.path.parent)
+ assert value.path.is_file()
+ return value
+
+ with ThreadPoolExecutor(max_workers=8) as workers:
+ values = list(workers.map(launch, range(8)))
+ protected = {value.path for value in values}
+ churn()
+ assert protected <= retained()
+ for value in values:
+ value.close()
+ values.clear()
+ gc.collect()
+ churn()
+ assert len(retained()) <= 1
+
+
+def test_cleaner_cannot_enter_between_file_publication_and_pin(monkeypatch):
+ written = threading.Event()
+ finish = threading.Event()
+ write = catalogs.write_atomic
+ published_paths = []
+
+ def paused_write(path, data):
+ write(path, data)
+ published_paths.append(path)
+ written.set()
+ assert finish.wait(timeout=5)
+
+ monkeypatch.setattr(catalogs, "write_atomic", paused_write)
+ with ThreadPoolExecutor(max_workers=1) as worker:
+ preparation = worker.submit(publish, "paused-publication")
+ try:
+ assert written.wait(timeout=2)
+ path = published_paths[0]
+ catalogs.prune_codex_hub_catalogs(path.parent)
+ assert path.is_file()
+ finally:
+ finish.set()
+ catalog = preparation.result(timeout=2)
+ monkeypatch.setattr(catalogs, "write_atomic", write)
+ churn()
+ assert catalog.path in retained()
+ catalog.close()
+ churn()
+ assert len(retained()) <= 1
+
+
+def test_separate_process_cannot_clean_starting_or_cached_generation():
+ active = publish("parent")
+ code = (
+ "from vibe.backend_model_catalog import _publish_codex_hub_catalog\n"
+ "for i in range(8):\n"
+ " with _publish_codex_hub_catalog(('"
+ '{"models":[{"slug":"child-%s"}]}' + "' % i).encode()): pass\n"
+ )
+ result = subprocess.run(
+ [sys.executable, "-c", code], capture_output=True, text=True, timeout=15,
+ env=os.environ.copy(),
+ )
+ assert result.returncode == 0, result.stderr
+ assert active.path in retained()
+ assert len(retained() - {active.path}) <= 1
+
+
+async def test_cache_invalidation_during_pending_launch_keeps_local_reference(monkeypatch):
+ value = agent()
+ monkeypatch.setattr(catalogs, "_export_codex_bundled_catalog", lambda *_: b'{"models":[{"slug":"first"}]}')
+ starting = (await value.prepare_model_hub_runtime()).retain()
+ path = starting.path
+ await value.invalidate_model_hub_runtime()
+ churn()
+ assert value._model_hub_catalog is None and path in retained()
+ starting.close()
+ churn()
+ assert len(retained()) <= 1
+
+
+async def test_invalidated_export_releases_unpublished_pin(monkeypatch):
+ value = agent()
+ entered = threading.Event()
+ released = threading.Event()
+
+ def export(*_):
+ entered.set()
+ assert released.wait(timeout=5)
+ return b'{"models":[{"slug":"raced"}]}'
+
+ monkeypatch.setattr(catalogs, "_export_codex_bundled_catalog", export)
+ preparation = asyncio.create_task(value.prepare_model_hub_runtime())
+ assert await asyncio.to_thread(entered.wait, 2)
+ await value.invalidate_model_hub_runtime()
+ released.set()
+ with pytest.raises(CodexModelHubCatalogUnavailableError, match="generation changed"):
+ await preparation
+ assert value._model_hub_catalog is None
+ del preparation
+ churn()
+ assert len(retained()) <= 1
+
+
+async def test_cancelled_export_closes_its_eventual_result(monkeypatch):
+ value = agent()
+ entered = threading.Event()
+ finish = threading.Event()
+ closed = asyncio.Event()
+ loop = asyncio.get_running_loop()
+ release = catalogs._release_codex_hub_catalog
+
+ def export(*_):
+ entered.set()
+ assert finish.wait(timeout=5)
+ return b'{"models":[{"slug":"cancelled"}]}'
+
+ def observe_close(path, descriptor):
+ release(path, descriptor)
+ loop.call_soon_threadsafe(closed.set)
+
+ monkeypatch.setattr(catalogs, "_export_codex_bundled_catalog", export)
+ monkeypatch.setattr(catalogs, "_release_codex_hub_catalog", observe_close)
+ preparation = asyncio.create_task(value.prepare_model_hub_runtime())
+ assert await asyncio.to_thread(entered.wait, 2)
+ preparation.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await preparation
+ finish.set()
+ await asyncio.wait_for(closed.wait(), 5)
+ assert value._model_hub_catalog is None
+ churn()
+ assert len(retained()) <= 1
+
+
+_STDIO_CONSUMER = """
+import json, os, sys
+for line in sys.stdin:
+ request = json.loads(line)
+ if request.get("method") == "test/exit":
+ break
+ if "id" in request:
+ print(json.dumps({"id": request["id"], "result": {}}), flush=True)
+ if request.get("method") == "test/closeStdout":
+ os.close(1)
+"""
+
+
+async def test_agent_launch_keeps_pin_across_invalidation_and_reuses_transport(monkeypatch, tmp_path):
+ value = agent()
+ value._transports = {}
+ value._transport_locks = {}
+ value._transport_cwd_inodes = {}
+ initialized = asyncio.Event()
+ finish = asyncio.Event()
+ spawn = asyncio.create_subprocess_exec
+ argv = []
+ monkeypatch.setattr(catalogs, "_export_codex_bundled_catalog", lambda *_: b'{"models":[{"slug":"agent"}]}')
+
+ async def fixture_spawn(*args, **kwargs):
+ argv.extend(args)
+ return await spawn(sys.executable, "-u", "-c", _STDIO_CONSUMER, **kwargs)
+
+ async def pause_governor_start(self):
+ await original_start(self)
+ initialized.set()
+ await finish.wait()
+
+ original_start = CodexTransport.start
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", fixture_spawn)
+ monkeypatch.setattr(CodexTransport, "start", pause_governor_start)
+ # An extra argument must not override the path whose lifetime we protect.
+ value.codex_config.extra_args = ["-c", 'model_catalog_json="/obsolete/catalog.json"']
+ launch = SimpleNamespace(
+ channel="hub", fingerprint="hub:test",
+ gateway_base_url="http://127.0.0.1:1", gateway_token="fixture-only",
+ )
+ starting = asyncio.create_task(value._get_or_create_transport(str(tmp_path), launch))
+ try:
+ await asyncio.wait_for(initialized.wait(), 5)
+ path = value._model_hub_catalog.path
+ assert argv[-1] == f"model_catalog_json={json.dumps(str(path))}"
+ await value.invalidate_model_hub_runtime()
+ churn()
+ assert path in retained()
+ finish.set()
+ transport = await starting
+ assert (await value._get_or_create_transport(str(tmp_path), launch)) is transport
+ assert path in retained()
+ await transport.stop()
+ await transport._catalog_exit_task
+ churn()
+ assert len(retained()) <= 1
+ finally:
+ finish.set()
+ transport = await starting
+ await transport.stop()
+
+
+async def test_stdout_eof_keeps_pin_until_real_process_exit(monkeypatch, tmp_path):
+ spawn = asyncio.create_subprocess_exec
+
+ async def fixture_spawn(*_args, **kwargs):
+ return await spawn(sys.executable, "-u", "-c", _STDIO_CONSUMER, **kwargs)
+
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", fixture_spawn)
+ catalog = publish("stdout-live")
+ path = catalog.path
+ transport = CodexTransport("fixture", str(tmp_path), model_hub_catalog=catalog)
+ catalog.close()
+ try:
+ await transport.start()
+ await transport.send_request("test/closeStdout", {})
+ await asyncio.wait_for(transport.wait_closed(), 5)
+ assert transport._process.returncode is None
+ churn()
+ assert path.is_file()
+ transport._process.stdin.close()
+ await asyncio.wait_for(transport._catalog_exit_task, 5)
+ assert transport._process.returncode == 0
+ assert transport._model_hub_catalog is None
+ churn()
+ assert len(retained()) <= 1
+ finally:
+ await transport.stop()
+
+
+async def test_cancel_during_real_spawn_keeps_pin_until_child_is_reaped(monkeypatch, tmp_path):
+ spawn = asyncio.create_subprocess_exec
+ created = asyncio.Event()
+ finish_spawn = asyncio.Event()
+ processes = []
+
+ async def fixture_spawn(*_args, **kwargs):
+ proc = await spawn(sys.executable, "-u", "-c", _STDIO_CONSUMER, **kwargs)
+ processes.append(proc)
+ created.set()
+ await finish_spawn.wait()
+ return proc
+
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", fixture_spawn)
+ catalog = publish("cancel-spawn")
+ path = catalog.path
+ transport = CodexTransport("fixture", str(tmp_path), model_hub_catalog=catalog)
+ catalog.close()
+ task = asyncio.create_task(transport.start())
+ try:
+ await asyncio.wait_for(created.wait(), 5)
+ task.cancel()
+ churn()
+ assert processes[0].returncode is None and path.is_file()
+ finish_spawn.set()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert processes[0].returncode == 0
+ assert transport._model_hub_catalog is None
+ churn()
+ assert len(retained()) <= 1
+ finally:
+ finish_spawn.set()
+ await transport.stop()
+
+
[email protected]("failure", ["spawn", "handshake", "cancel-handshake"])
+async def test_failed_start_releases_transport_reference(monkeypatch, tmp_path, failure):
+ catalog = publish(failure)
+ path = catalog.path
+ transport = CodexTransport("fixture-codex", str(tmp_path), model_hub_catalog=catalog)
+ catalog.close()
+ exited = asyncio.Event()
+ entered = asyncio.Event()
+ process = SimpleNamespace(pid=123, returncode=None, stdin=None, stdout=None, stderr=None)
+
+ async def wait():
+ await exited.wait()
+ return 1
+
+ async def spawn(*_, **__):
+ if failure == "spawn":
+ raise OSError("fixture spawn failed")
+ return process
+
+ async def handshake(*_):
+ entered.set()
+ if failure == "cancel-handshake":
+ await asyncio.Event().wait()
+ raise RuntimeError("fixture handshake failed")
+
+ async def stop():
+ process.returncode = 1
+ exited.set()
+ transport._release_model_hub_catalog()
+ transport._cleanup_tasks()
+
+ process.wait = wait
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", spawn)
+ monkeypatch.setattr(transport, "send_request", handshake)
+ monkeypatch.setattr(transport, "stop", stop)
+ monkeypatch.setattr(transport, "_reader_loop", AsyncMock())
+ monkeypatch.setattr(transport, "_stderr_reader", AsyncMock())
+ monkeypatch.setattr("modules.agents.codex.transport.process_identity", lambda *_: {})
+ monkeypatch.setattr("modules.agents.codex.transport.log_process_snapshot", lambda *_args, **_kw: None)
+ task = asyncio.create_task(transport.start())
+ if failure == "cancel-handshake":
+ await entered.wait()
+ churn()
+ assert path.is_file()
+ task.cancel()
+ with pytest.raises((OSError, RuntimeError, asyncio.CancelledError)):
+ await task
+ if transport._catalog_exit_task is not None:
+ await transport._catalog_exit_task
+ assert transport._model_hub_catalog is None
+ del task
+ churn()
+ assert len(retained()) <= 1
diff --git a/vibe/backend_model_catalog.py b/vibe/backend_model_catalog.py
index 07cbb7654..bf2a3acfb 100644
--- a/vibe/backend_model_catalog.py
+++ b/vibe/backend_model_catalog.py
@@ -5,13 +5,18 @@ import hashlib
import json
import logging
import os
+import re
+import stat
+import subprocess
import threading
import time
import urllib.error
import urllib.request
+import weakref
+from contextlib import contextmanager
from pathlib import Path
from types import MappingProxyType
-from typing import Any, Callable, Final, Iterable, Mapping, Sequence
+from typing import TYPE_CHECKING, Any, Callable, Final, Iterable, Mapping, Sequence
from config import paths
from config.atomic_io import write_atomic
@@ -23,6 +28,9 @@ from vibe.claude_model_catalog import (
)
from vibe.codex_config import get_codex_home
+if TYPE_CHECKING:
+ from storage.lock import MigrationFileLock
+
logger = logging.getLogger(__name__)
@@ -38,6 +46,8 @@ REMOTE_CATALOG_USER_AGENT = "avibe/backend-model-catalog"
REMOTE_CATALOG_CACHE_VERSION = 2
CODEX_HUB_CATALOG_TIMEOUT_SECONDS = 15.0
CODEX_HUB_CATALOG_MAX_BYTES = 8 * 1024 * 1024
+CODEX_HUB_CATALOG_HISTORY_LIMIT = 1
+_CODEX_HUB_CATALOG_NAME = re.compile(r"standard-responses-[0-9a-f]{16}\.json")
_HIDDEN_VISIBILITIES = {"hide", "hidden"}
_VISIBLE_VISIBILITIES = {"visible", "list"}
@@ -131,6 +141,166 @@ def _codex_hub_catalog_path(catalog: bytes) -> Path:
return paths.get_runtime_dir() / "model-hub" / "codex" / f"standard-responses-{digest}.json"
+class CodexHubCatalog:
+ """One owner's pin on an immutable catalog generation.
+
+ Caches, pending launches and transports acquire independent pins and close
+ them at their lifecycle boundary. Finalization is only an abandoned-owner
+ fallback: exception tracebacks must not delay ordinary release.
+ """
+
+ def __init__(self, path: Path, descriptor: int) -> None:
+ self.path = path
+ self.descriptor = descriptor
+ self._release = weakref.finalize(self, _release_codex_hub_catalog, path, descriptor)
+
+ def retain(self) -> CodexHubCatalog:
+ """Acquire an independent pin before giving up this owner's pin."""
+ if not self._release.alive:
+ raise RuntimeError("Cannot retain a released Codex catalog")
+ with _codex_hub_catalog_lock(self.path.parent):
+ return _pin_codex_hub_catalog(self.path)
+
+ def close(self) -> None:
+ self._release()
+
+ def __enter__(self) -> CodexHubCatalog:
+ return self
+
+ def __exit__(self, *_args: object) -> None:
+ self.close()
+
+ @contextmanager
+ def inherited_subprocess_kwargs(self):
+ """Keep the file pinned in the child even if its Avibe parent exits."""
+ if not self._release.alive:
+ raise RuntimeError("Cannot launch with a released Codex catalog")
+ if os.name != "nt":
+ yield {"pass_fds": (self.descriptor,)}
+ return
+ import msvcrt
+
+ # Each spawn gets its own temporary inheritable handle, so simultaneous
+ # launches cannot reset one another's inheritance flags.
+ descriptor = os.dup(self.descriptor)
+ try:
+ os.set_inheritable(descriptor, True)
+ startupinfo = subprocess.STARTUPINFO()
+ startupinfo.lpAttributeList = {"handle_list": [msvcrt.get_osfhandle(descriptor)]}
+ yield {"startupinfo": startupinfo, "close_fds": True}
+ finally:
+ os.close(descriptor)
+
+
+def _codex_hub_catalog_lock(directory: Path, *, timeout_seconds: float = 30.0) -> MigrationFileLock:
+ # Lightweight catalog/handler imports must not initialize SQLite storage.
+ from storage.lock import MigrationFileLock
+
+ return MigrationFileLock(directory / ".catalog.lock", timeout_seconds=timeout_seconds)
+
+
+def _open_codex_hub_catalog(path: Path) -> int:
+ """Open only an ordinary file without following a candidate symlink."""
+ flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) | getattr(os, "O_NOFOLLOW", 0)
+ flags |= getattr(os, "O_NONBLOCK", 0)
+ descriptor = os.open(path, flags)
+ try:
+ opened = os.fstat(descriptor)
+ named = path.lstat()
+ if (
+ not stat.S_ISREG(named.st_mode)
+ or not stat.S_ISREG(opened.st_mode)
+ or opened.st_nlink != 1
+ or (opened.st_dev, opened.st_ino) != (named.st_dev, named.st_ino)
+ ):
+ raise OSError(f"Codex catalog is not an exclusive regular file: {path}")
+ return descriptor
+ except BaseException:
+ os.close(descriptor)
+ raise
+
+
+def _pin_codex_hub_catalog(path: Path) -> CodexHubCatalog:
+ descriptor = _open_codex_hub_catalog(path)
+ try:
+ if os.name != "nt":
+ import fcntl
+
+ fcntl.flock(descriptor, fcntl.LOCK_SH | fcntl.LOCK_NB)
+ return CodexHubCatalog(path, descriptor)
+ except BaseException:
+ os.close(descriptor)
+ raise
+
+
+def _prune_codex_hub_catalogs_locked(directory: Path) -> None:
+ """Keep every pinned generation and at most one unpinned history file.
+
+ Caller holds the publication lock. Name order only chooses the spare;
+ neither timestamps nor age establish that a generation is unused.
+ """
+ spare = 0
+ for path in sorted(directory.iterdir()):
+ if not _CODEX_HUB_CATALOG_NAME.fullmatch(path.name):
+ continue
+ descriptor = None
+ try:
+ if not stat.S_ISREG(path.lstat().st_mode) or path.lstat().st_nlink != 1:
+ continue
+ descriptor = _open_codex_hub_catalog(path)
+ if os.name != "nt":
+ import fcntl
+
+ try:
+ fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except BlockingIOError:
+ continue
+ # On Windows the deletion itself is the non-blocking pin probe:
+ # any reader's open handle denies deletion. Close our probe first,
+ # while the publication lock still excludes a new reader.
+ if os.name == "nt":
+ os.close(descriptor)
+ descriptor = None
+ if spare < CODEX_HUB_CATALOG_HISTORY_LIMIT:
+ spare += 1
+ continue
+ path.unlink()
+ except FileNotFoundError:
+ continue
+ except OSError as exc:
+ if os.name == "nt" and getattr(exc, "winerror", None) == 32:
+ # ERROR_SHARING_VIOLATION: a cache/child still owns its pin.
+ continue
+ logger.warning("Could not reclaim Codex Model Hub catalog %s", path, exc_info=True)
+ finally:
+ if descriptor is not None:
+ os.close(descriptor)
+
+
+def prune_codex_hub_catalogs(directory: Path) -> None:
+ """Best-effort reclamation at publication and reference-release boundaries."""
+ from storage.lock import MigrationLockTimeout
+
+ try:
+ if not directory.is_dir():
+ return
+ with _codex_hub_catalog_lock(directory, timeout_seconds=0):
+ _prune_codex_hub_catalogs_locked(directory)
+ except MigrationLockTimeout:
+ # A publisher/cleaner already owns this directory. Its sweep, or the
+ # next lifecycle boundary, retries reclamation.
+ logger.debug("Codex catalog cleanup deferred while publication is in progress")
+ except OSError:
+ logger.warning("Could not clean Codex Model Hub catalogs in %s", directory, exc_info=True)
+
+
+def _release_codex_hub_catalog(path: Path, descriptor: int) -> None:
+ # Close, do not LOCK_UN: a spawned app-server inherits the same open file
+ # description and must keep its lock even if the Avibe parent goes away.
+ os.close(descriptor)
+ prune_codex_hub_catalogs(path.parent)
+
+
def _codex_hub_catalog_bytes(
raw_catalog: bytes,
configured_models: Sequence[Mapping[str, Any]] | None = None,
@@ -278,11 +448,26 @@ def _codex_hub_catalog_bytes(
def _publish_codex_hub_catalog(
raw_catalog: bytes,
configured_models: Sequence[Mapping[str, Any]] | None = None,
-) -> Path:
+) -> CodexHubCatalog:
catalog = _codex_hub_catalog_bytes(raw_catalog, configured_models)
path = _codex_hub_catalog_path(catalog)
- write_atomic(path, catalog)
- return path
+ with _codex_hub_catalog_lock(path.parent):
+ try:
+ path.lstat()
+ except FileNotFoundError:
+ write_atomic(path, catalog)
+ pinned = _pin_codex_hub_catalog(path)
+ try:
+ if os.read(pinned.descriptor, len(catalog) + 1) != catalog:
+ raise ValueError("Codex catalog digest path contains different bytes")
+ except BaseException:
+ pinned.close()
+ raise
+ try:
+ _prune_codex_hub_catalogs_locked(path.parent)
+ except OSError:
+ logger.warning("Could not clean Codex Model Hub catalogs in %s", path.parent, exc_info=True)
+ return pinned
def _export_codex_bundled_catalog(
@@ -331,7 +516,7 @@ def prepare_codex_hub_catalog(
binary: str,
base_env: dict[str, str] | None = None,
configured_models: Sequence[Mapping[str, Any]] | None = None,
-) -> Path:
+) -> CodexHubCatalog:
"""Prepare the exact binary's catalog immediately before a Hub launch."""
return _publish_codex_hub_catalog(