Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Python agentagent-osaiai-agentsai-partnerchatopsclaudeclaude-codecodexdevtoolsdiscord-botlark-botllmlocal-firstlong-term-memoryopencodeslack-botvibe-codingvibeywechat
fix(codex): bound retained Model Hub catalog generations (#2024)
Call White09b2d41e294f
diff --git a/docs/plans/issue-1791-codex-catalog-retention.md b/docs/plans/issue-1791-codex-catalog-retention.mdnew file mode 100644index 000000000..ae4928cb0--- /dev/null+++ b/docs/plans/issue-1791-codex-catalog-retention.md@@ -0,0 +1,65 @@+# Bounded Codex Model Hub catalogs++## Contract and cause++Content-addressed publication has no retirement owner. The Agent caches a path,+but each per-working-directory app-server continues to consume that path after+the cache is invalidated. Export failure must never select an older generation.++Codex 0.154.0 (`openai/codex`, commit+`6b9826e3aa83b1a5947db50f4332cb9c65f1b340`) calls+`ConfigManager::load_with_overrides` in `thread_start_task`; Config construction+calls `load_model_catalog`, which reads the file again. The hermetic native+consumer test removes the catalog after initialization and observes+`thread/start` fail to load configuration. Handshake-only protection is unsafe.++## Ownership and retention++- Agent cache, pending launch and live transport acquire independent pins on+ the immutable file and explicitly close them at their lifecycle boundaries.+ Invalidating the cache does not release the transport's pin. Finalization is+ only a fallback; captured exception tracebacks cannot delay ordinary release.+- A short cross-process publication lock covers creating/reusing the immutable+ path, acquiring its pin and sweeping. Identical bytes preserve the inode.+ Its storage import is deferred to those operations, preserving storage-free+ catalog and session-handler imports (`MEMORY-INDEP-009`).+- POSIX uses shared file locks. Windows open handles deny deletion. The native+ child inherits the pin, including when its Avibe parent exits first.+- The pinned catalog path is the final app-server override, so backend extra+ arguments cannot silently select a different, unprotected generation.+- Retain all protected generations plus at most one unprotected history file,+ selected deterministically by filename. Age and mtime never prove inactivity.+- Only ordinary, singly linked files matching+ `standard-responses-[0-9a-f]{16}.json` in the Codex Hub directory are eligible.+ Unrelated names, directories, symlinks and hard links are untouched.+- Cleanup is best effort at publication and release. Failure logs preserve the+ path and exception; publication and existing consumers remain usable. The+ bound is restored by a successful later sweep.++## Scope and validation++Changes stay in catalog publication and the Codex Agent/transport lifecycle.+No database, daemon, shared lifecycle rewrite or production-state cleanup.+Validation covers unchanged/changed content, startup and active pins,+invalidation, failed/cancelled launch, actual exit, export and cleanup failures,+cross-process races, and the actual native consumer in isolated homes with+loopback-only egress. The inheritance probe chooses a consumer catalog that+cannot occupy the single unpinned spare; disabling only child inheritance+deletes it and makes `thread/start` fail, while inheritance preserves both.+After actual child exit, the next publication reclaims that generation.++The native inheritance and configuration-reread evidence is for Codex 0.154.0 on+macOS. Custom launcher wrappers must preserve inherited handles for protection+after the Avibe parent dies; normal parent-owned lifecycle protection does not+depend on child inheritance. Pinned descendants can conservatively extend a+generation's lifetime. This change does not manage manually launched processes+that bypass Avibe's pin-acquisition protocol.++## Progress++- [x] Read current issue, default branch and consumer lifecycle.+- [x] Reproduce post-initialize catalog reread with native Codex 0.154.0.+- [x] Implement and test ownership, retention and failure boundaries.+- [x] Complete focused suites and changed-file Ruff: 539 passed, 8 opt-in native+ prompt tests skipped; three additional native catalog cases passed on 0.154.0.+- Exact-head Codex review and GitHub CI are tracked in the implementation PR.diff --git a/modules/agents/codex/agent.py b/modules/agents/codex/agent.pyindex 82bc2412b..ca2a1ba32 100644--- a/modules/agents/codex/agent.py+++ b/modules/agents/codex/agent.py@@ -8,6 +8,7 @@ import loggingimport osimport shleximport time+from contextlib import AsyncExitStackfrom pathlib import Pathfrom typing import TYPE_CHECKING, Any, Awaitable, Callable, Dict, Optional@@ -63,6 +64,7 @@ logger = logging.getLogger(__name__)if TYPE_CHECKING:from modules.agents.model_hub import ModelHubLaunch+ from vibe.backend_model_catalog import CodexHubCatalog_CODEX_MANAGED_PROVIDER_IDS = frozenset((MANAGED_PROVIDER_ID, *LEGACY_MANAGED_PROVIDER_IDS))_CODEX_MODEL_HUB_PROVIDER_ID = "avibe_model_hub"@@ -142,7 +144,7 @@ class CodexAgent(BaseAgent):super().__init__(controller)self.codex_config = codex_configself._registered_runtime = registered_runtime- self._model_hub_catalog_path: Path | None = None+ self._model_hub_catalog: CodexHubCatalog | None = Noneself._model_hub_catalog_lock = asyncio.Lock()self._model_hub_catalog_generation = 0@@ -856,22 +858,23 @@ class CodexAgent(BaseAgent):"""Reload persisted runtime config before respawning app-server transports."""self.codex_config = codex_configself.controller.config.codex = codex_config- self._model_hub_catalog_generation += 1- self._model_hub_catalog_path = None+ await self.invalidate_model_hub_runtime()await self.refresh_auth_state()async def invalidate_model_hub_runtime(self) -> None:"""Make the next Hub launch rebuild its catalog without touching Direct transports."""self._model_hub_catalog_generation += 1- self._model_hub_catalog_path = None+ if self._model_hub_catalog is not None:+ self._model_hub_catalog.close()+ self._model_hub_catalog = None- async def prepare_model_hub_runtime(self) -> Path:+ async def prepare_model_hub_runtime(self) -> CodexHubCatalog:"""Bind Hub metadata to this Agent's exact configured Codex binary."""from vibe import backend_model_catalogasync with self._model_hub_catalog_lock:- if self._model_hub_catalog_path is not None:- return self._model_hub_catalog_path+ if self._model_hub_catalog is not None:+ return self._model_hub_cataloggeneration = self._model_hub_catalog_generationbinary = self.codex_config.binaryconfigured_models = None@@ -882,23 +885,39 @@ class CodexAgent(BaseAgent):model.to_payload()for model in store.load().agents["codex"].models]- try:- path = await asyncio.to_thread(+ preparation = asyncio.create_task(+ asyncio.to_thread(backend_model_catalog.prepare_codex_hub_catalog,binary,None,configured_models,)+ )+ try:+ catalog = await asyncio.shield(preparation)+ except asyncio.CancelledError:+ # The export runs in a thread and cannot be cancelled. Its+ # result must still release its pin after the caller leaves.+ preparation.add_done_callback(self._discard_model_hub_catalog)+ raiseexcept Exception as exc:raise CodexModelHubCatalogUnavailableError("Codex Model Hub catalog preparation failed") from excif self._model_hub_catalog_generation != generation:+ catalog.close()raise CodexModelHubCatalogUnavailableError("Codex Model Hub catalog generation changed during preparation")- self._model_hub_catalog_path = path- return path+ self._model_hub_catalog = catalog+ return catalog++ @staticmethod+ def _discard_model_hub_catalog(preparation: asyncio.Task) -> None:+ try:+ preparation.result().close()+ except Exception:+ logger.warning("Cancelled Codex catalog preparation failed", exc_info=True)async def prepare_resume_binding(self,@@ -949,6 +968,7 @@ class CodexAgent(BaseAgent):async def shutdown_runtime(self) -> None:"""Stop all app-server transports during vibe-remote shutdown."""+ await self.invalidate_model_hub_runtime()if not hasattr(self, "_transport_last_activity"):self._transport_last_activity = {}if not hasattr(self, "_transport_locks"):@@ -1674,7 +1694,7 @@ class CodexAgent(BaseAgent):while True:wait_for_active_turns = False- async with self._transport_locks[cwd]:+ async with self._transport_locks[cwd], AsyncExitStack() as catalog_pins:# Double-check after acquiring lockexisting = self._transports.get(cwd)existing_dead = bool(@@ -1716,19 +1736,21 @@ class CodexAgent(BaseAgent):runtime_args: list[str] = []runtime_env: dict[str, str] | None = Noneruntime_fingerprint = "direct"+ catalog = Noneif launch is not None:from modules.agents.model_hub import build_codex_hub_launch- if (- launch.channel == "hub"- and self._model_hub_catalog_path is None- ):- await self.prepare_model_hub_runtime()+ if launch.channel == "hub":+ # Capture the reference before any later await.+ # Invalidation may drop the cache while we retire+ # an old transport or spawn/initialize its successor.+ catalog = (await self.prepare_model_hub_runtime()).retain()+ catalog_pins.callback(catalog.close)runtime_args, runtime_env = build_codex_hub_launch([],os.environ.copy(),launch,- model_catalog_path=self._model_hub_catalog_path,+ model_catalog_path=catalog.path if catalog is not None else None,)runtime_fingerprint = launch.fingerprint@@ -1794,6 +1816,7 @@ class CodexAgent(BaseAgent):runtime_args=runtime_args,runtime_env=runtime_env,runtime_fingerprint=runtime_fingerprint,+ model_hub_catalog=catalog,)# Wire up callbacksdiff --git a/modules/agents/codex/transport.py b/modules/agents/codex/transport.pyindex cc21df2b1..6953adaf9 100644--- a/modules/agents/codex/transport.py+++ b/modules/agents/codex/transport.py@@ -8,13 +8,18 @@ import loggingimport osimport signalfrom asyncio.subprocess import Process-from typing import Any, Awaitable, Callable, Optional+from contextlib import nullcontext+from typing import TYPE_CHECKING, Any, Awaitable, Callable, Optionalfrom core.process_diagnostics import log_process_snapshot, process_identityfrom core.process_isolation import KILL_SIGNAL, isolated_subprocess_kwargs, signal_process_tree+from vibe.codex_config import format_toml_basic_stringlogger = logging.getLogger(__name__)+if TYPE_CHECKING:+ from vibe.backend_model_catalog import CodexHubCatalog+STREAM_BUFFER_LIMIT = 128 * 1024 * 1024 # 128 MB@@ -97,6 +102,7 @@ class CodexTransport:runtime_args: list[str] | None = None,runtime_env: dict[str, str] | None = None,runtime_fingerprint: str = "direct",+ model_hub_catalog: CodexHubCatalog | None = None,) -> None:self._binary = binaryself._cwd = cwd@@ -104,6 +110,9 @@ class CodexTransport:self._runtime_args = runtime_args or []self._runtime_env = runtime_envself.runtime_fingerprint = runtime_fingerprint+ self._model_hub_catalog = model_hub_catalog.retain() if model_hub_catalog is not None else None+ self._catalog_required = model_hub_catalog is not None+ self._catalog_exit_task: asyncio.Task[None] | None = Noneself._process: Optional[Process] = Noneself._request_id: int = 0self._pending: dict[int | str, asyncio.Future[dict[str, Any]]] = {}@@ -129,9 +138,19 @@ class CodexTransport:async def start(self) -> None:"""Launch the app-server and perform the ``initialize`` handshake."""+ try:+ await self._start()+ except BaseException:+ if self._process is None or self._process.returncode is not None:+ self._release_model_hub_catalog()+ raise++ async def _start(self) -> None:if self._process and self._process.returncode is None:logger.warning("CodexTransport.start() called but process is already running")return+ if self._catalog_required and self._model_hub_catalog is None:+ raise RuntimeError("A stopped Codex Hub transport needs a newly prepared catalog")self._closed_event.clear()cmd = (@@ -141,6 +160,10 @@ class CodexTransport:+ self._extra_args+ _avibe_app_server_config_args())+ if self._model_hub_catalog is not None:+ # The path actually consumed must be the one we pin, even if a+ # backend extra argument tries to select a different generation.+ cmd += ["-c", f"model_catalog_json={format_toml_basic_string(str(self._model_hub_catalog.path))}"]logger.info("Launching Codex app-server: %s (cwd=%s)", " ".join(cmd), self._cwd)if not os.path.exists(self._cwd):@@ -156,7 +179,24 @@ class CodexTransport:}if self._runtime_env is not None:subprocess_kwargs["env"] = self._runtime_env- self._process = await asyncio.create_subprocess_exec(*cmd, **subprocess_kwargs)+ catalog = self._model_hub_catalog+ inheritance = catalog.inherited_subprocess_kwargs() if catalog is not None else nullcontext({})+ with inheritance as catalog_kwargs:+ spawn = asyncio.create_task(+ asyncio.create_subprocess_exec(*cmd, **subprocess_kwargs, **catalog_kwargs)+ )+ try:+ self._process = await asyncio.shield(spawn)+ except asyncio.CancelledError:+ # Cancellation can race the OS spawn. Settle it before closing+ # inherited handles or losing the only reference to this child.+ self._process = await spawn+ await self.stop()+ raise+ if catalog is not None:+ self._catalog_exit_task = asyncio.create_task(+ self._release_catalog_on_exit(self._process, catalog)+ )identity = process_identity(self._process.pid)logger.info("Codex app-server started (pid=%s pgid=%s sid=%s service_pgid=%s)",@@ -218,6 +258,7 @@ class CodexTransport:self._initialized = Falseproc = self._processif not proc or proc.returncode is not None:+ self._release_model_hub_catalog()self._closed_event.set()self._cleanup_tasks()return@@ -243,6 +284,7 @@ class CodexTransport:passself._cleanup_tasks()+ self._release_model_hub_catalog()self._closed_event.set()# Fail all pending futuresfor fut in self._pending.values():@@ -251,6 +293,18 @@ class CodexTransport:self._pending.clear()logger.info("Codex app-server stopped")+ async def _release_catalog_on_exit(self, proc: Process, catalog: CodexHubCatalog) -> None:+ """Stdout EOF is not process exit; keep the pin until wait confirms it."""+ await proc.wait()+ catalog.close()+ if self._model_hub_catalog is catalog:+ self._model_hub_catalog = None++ def _release_model_hub_catalog(self) -> None:+ if self._model_hub_catalog is not None:+ self._model_hub_catalog.close()+ self._model_hub_catalog = None+def _cleanup_tasks(self) -> None:for task in (self._reader_task, self._stderr_task, self._notify_task):if task and not task.done():diff --git a/tests/e2e/test_model_hub_catalog_consumer.py b/tests/e2e/test_model_hub_catalog_consumer.pyindex a09b21fa7..15e3dde9b 100644--- a/tests/e2e/test_model_hub_catalog_consumer.py+++ b/tests/e2e/test_model_hub_catalog_consumer.py@@ -9,12 +9,14 @@ import shleximport shutilimport socketimport subprocess+from contextlib import nullcontextfrom pathlib import Pathimport pytestfrom config.v2_config import ModelHubBackendModelConfigfrom modules.agents.codex.transport import CodexTransport+from modules.agents.codex.transport import CodexRPCErrorfrom modules.agents.model_hub import (ModelHubLaunch,build_claude_hub_env,@@ -24,7 +26,12 @@ from modules.agents.model_hub import ()from tests.e2e.drivers.model_hub_app import ModelHubTestAppfrom tests.e2e.drivers.mock_llm_upstream import MockLLMUpstream-from vibe.backend_model_catalog import _codex_hub_catalog_bytes+from vibe.backend_model_catalog import (+ CodexHubCatalog,+ _codex_hub_catalog_bytes,+ _codex_hub_catalog_path,+ _publish_codex_hub_catalog,+)pytestmark = pytest.mark.e2e_model_hub@@ -155,6 +162,94 @@ async def _codex_turn(binary, runtime, gateway, catalog, *, effort, native_confireturn bodies, models, notifications, config["config"]+def test_codex_reloads_catalog_after_initialize(codex_catalog_runtime):+ """MH-PROTOCOL-004: a live app-server rereads its catalog for a new thread."""+ binary, runtime, raw_catalog = codex_catalog_runtime+ catalog_path = runtime.home / "reload-catalog.json"+ catalog_path.write_bytes(_codex_hub_catalog_bytes(raw_catalog))++ async def probe():+ transport = CodexTransport(+ binary=binary,+ cwd=str(runtime.home),+ runtime_args=["-c", f"model_catalog_json={json.dumps(str(catalog_path))}"],+ runtime_env=runtime.env,+ )+ try:+ await transport.start()+ catalog_path.unlink()+ with pytest.raises(CodexRPCError, match="failed to load configuration"):+ await transport.send_request(+ "thread/start",+ {"cwd": str(runtime.home), "ephemeral": True},+ )+ finally:+ await transport.stop()++ asyncio.run(probe())++[email protected]("inherit", [True, False])+def test_codex_inherits_catalog_pin(codex_catalog_runtime, monkeypatch, inherit):+ """MH-PROTOCOL-004: only a child pin, not the history spare, saves its catalog."""+ binary, runtime, raw_catalog = codex_catalog_runtime+ # Preserve native model behavior while producing distinct valid catalogs.+ # Choose the consumer's path after sorting, so it cannot win the history+ # spare by chance, regardless of which catalog this Codex version bundles.+ variants = []+ for index in range(5):+ payload = json.loads(raw_catalog)+ payload["models"][0]["display_name"] = f"Retention fixture {index}"+ variants.append(json.dumps(payload).encode())+ variants.sort(key=lambda raw: _codex_hub_catalog_path(_codex_hub_catalog_bytes(raw)).name)+ catalog = _publish_codex_hub_catalog(variants[-1])+ path = catalog.path+ if not inherit:+ monkeypatch.setattr(CodexHubCatalog, "inherited_subprocess_kwargs", lambda self: nullcontext({}))++ async def probe():+ transport = CodexTransport(+ binary=binary,+ cwd=str(runtime.home),+ runtime_args=["-c", f"model_catalog_json={json.dumps(str(path))}"],+ runtime_env=runtime.env,+ model_hub_catalog=catalog,+ )+ try:+ await transport.start()+ # Simulate loss of the parent's descriptor without killing pytest.+ catalog.close()+ transport._model_hub_catalog.close()+ for raw in variants[:-1]:+ with _publish_codex_hub_catalog(raw) as successor:+ assert successor.path.name < path.name+ assert path.is_file() is inherit+ if inherit:+ thread = await transport.send_request(+ "thread/start",+ {"cwd": str(runtime.home), "ephemeral": True},+ )+ assert thread["thread"]["id"]+ else:+ with pytest.raises(CodexRPCError, match="failed to load configuration"):+ await transport.send_request(+ "thread/start",+ {"cwd": str(runtime.home), "ephemeral": True},+ )+ finally:+ catalog.close()+ await transport.stop()+ assert transport._process.returncode is not None+ # Simulated parent loss already closed every parent pin. The next+ # publication must reclaim the child's catalog once it really exits.+ with _publish_codex_hub_catalog(variants[0]):+ pass+ assert not path.exists()+ assert len(list(path.parent.glob("standard-responses-*.json"))) == 1++ asyncio.run(probe())++@pytest.mark.parametrize("effort", [None, "none", "minimal", "low", "medium", "high", "xhigh", "max"])def test_codex_unknown_catalog_preserves_supplied_image_and_effort(codex_catalog_runtime, effort):"""MH-PROTOCOL-004: absence must survive the real catalog/turn consumer."""diff --git a/tests/scenarios/model_hub/catalog.yaml b/tests/scenarios/model_hub/catalog.yamlindex b96c97e87..197765cb0 100644--- a/tests/scenarios/model_hub/catalog.yaml+++ b/tests/scenarios/model_hub/catalog.yaml@@ -390,6 +390,8 @@ scenarios:- tests/e2e/test_model_hub_catalog_consumer.py::test_codex_explicit_catalog_choices_reach_the_consumer- tests/e2e/test_model_hub_catalog_consumer.py::test_codex_native_context_override_wins_over_catalog_planning- tests/e2e/test_model_hub_catalog_consumer.py::test_codex_auto_compaction_consumes_resolved_limits+ - tests/e2e/test_model_hub_catalog_consumer.py::test_codex_reloads_catalog_after_initialize+ - tests/e2e/test_model_hub_catalog_consumer.py::test_codex_inherits_catalog_pin- id: MH-CLAUDE-LAUNCH-001name: Claude Hub turns preserve the configured Source and model despite native connection settingsstatus: covereddiff --git a/tests/test_agent_auth_service.py b/tests/test_agent_auth_service.pyindex 499c72a98..049cfafd2 100644--- a/tests/test_agent_auth_service.py+++ b/tests/test_agent_auth_service.py@@ -2236,7 +2236,7 @@ class AgentAuthServiceTests(_IsolatedClaudeConfigDirMixin, unittest.IsolatedAsynnew_config = CodexCompatConfig(enabled=True, binary="/new/codex", extra_args=[])agent = CodexAgent.__new__(CodexAgent)agent.codex_config = old_config- agent._model_hub_catalog_path = Path("/runtime/codex-old.json")+ agent._model_hub_catalog = SimpleNamespace(path=Path("/runtime/codex-old.json"), close=Mock())agent._model_hub_catalog_lock = asyncio.Lock()agent._model_hub_catalog_generation = 0agent.controller = SimpleNamespace(config=SimpleNamespace(codex=old_config))@@ -2251,7 +2251,7 @@ class AgentAuthServiceTests(_IsolatedClaudeConfigDirMixin, unittest.IsolatedAsynprepare_catalog.assert_not_called()self.assertIs(agent.codex_config, new_config)self.assertIs(agent.controller.config.codex, new_config)- self.assertIsNone(agent._model_hub_catalog_path)+ self.assertIsNone(agent._model_hub_catalog)self.assertEqual(agent._model_hub_catalog_generation, 1)agent.refresh_auth_state.assert_awaited_once()diff --git a/tests/test_backend_model_catalog.py b/tests/test_backend_model_catalog.pyindex 88a8d52c6..f112a0ef4 100644--- a/tests/test_backend_model_catalog.py+++ b/tests/test_backend_model_catalog.py@@ -181,8 +181,8 @@ def test_codex_hub_catalog_projects_complete_catalog(monkeypatch, tmp_path):).encode())- assert path.name.startswith("standard-responses-")- payload = json.loads(path.read_text(encoding="utf-8"))+ assert path.path.name.startswith("standard-responses-")+ payload = json.loads(path.path.read_text(encoding="utf-8"))assert payload["client_version"] == "0.149.1"assert payload["models"][0] == {"slug": "gpt-5.6-luna",@@ -663,9 +663,9 @@ def test_codex_hub_catalog_preparation_exports_current_binary(monkeypatch, tmp_pexported = backend_model_catalog.prepare_codex_hub_catalog("codex")- assert exported.name.startswith("standard-responses-")+ assert exported.path.name.startswith("standard-responses-")assert calls == [("codex", None)]- assert json.loads(exported.read_text(encoding="utf-8"))["models"][0]["use_responses_lite"] is False+ assert json.loads(exported.path.read_text(encoding="utf-8"))["models"][0]["use_responses_lite"] is Falsedef test_codex_hub_catalog_failure_cannot_select_a_previous_generation(monkeypatch, tmp_path):@@ -681,8 +681,8 @@ def test_codex_hub_catalog_failure_cannot_select_a_previous_generation(monkeypatwith pytest.raises(RuntimeError, match="export failed"):backend_model_catalog.prepare_codex_hub_catalog("/opt/codex")- assert previous.exists()- assert list(previous.parent.glob("standard-responses-*.json")) == [previous]+ assert previous.path.exists()+ assert list(previous.path.parent.glob("standard-responses-*.json")) == [previous.path]def test_codex_hub_catalog_generations_are_content_addressed(monkeypatch, tmp_path):@@ -692,9 +692,9 @@ def test_codex_hub_catalog_generations_are_content_addressed(monkeypatch, tmp_pafirst = backend_model_catalog._publish_codex_hub_catalog(b'{"models":[{"slug":"gpt-first"}]}')second = backend_model_catalog._publish_codex_hub_catalog(b'{"models":[{"slug":"gpt-second"}]}')- assert first != second- assert json.loads(first.read_text(encoding="utf-8"))["models"][0]["slug"] == "gpt-first"- assert json.loads(second.read_text(encoding="utf-8"))["models"][0]["slug"] == "gpt-second"+ assert first.path != second.path+ assert json.loads(first.path.read_text(encoding="utf-8"))["models"][0]["slug"] == "gpt-first"+ assert json.loads(second.path.read_text(encoding="utf-8"))["models"][0]["slug"] == "gpt-second"def test_merge_sources_applies_tombstones_and_fills_missing_metadata():diff --git a/tests/test_codex_agent.py b/tests/test_codex_agent.pyindex f294a235f..8724eb83e 100644--- a/tests/test_codex_agent.py+++ b/tests/test_codex_agent.py@@ -22,6 +22,12 @@ from modules.agents.codex.transport import CodexRPCError_AGENT_PATH = Path(__file__).resolve().parents[1] / "modules/agents/codex/agent.py"++def _catalog_reference(path):+ catalog = SimpleNamespace(path=Path(path), close=Mock())+ catalog.retain = Mock(return_value=catalog)+ return catalog+_modules_pkg = types.ModuleType("modules")_agents_pkg = types.ModuleType("modules.agents")_codex_pkg = types.ModuleType("modules.agents.codex")@@ -594,6 +600,8 @@ class CodexAgentConnectionProbeTests(unittest.IsolatedAsyncioTestCase):)self.assertIsNone(agent._attach_transport_activation(cwd, transport))+ agent._model_hub_catalog_generation = 0+ agent._model_hub_catalog = Noneawait agent.shutdown_runtime()transport.stop.assert_awaited_once_with()@@ -897,6 +905,8 @@ class CodexAgentStopTests(unittest.IsolatedAsyncioTestCase):transport.stop = stop_transport+ agent._model_hub_catalog_generation = 0+ agent._model_hub_catalog = Noneawait agent.shutdown_runtime()late_commit.assert_not_called()@@ -5413,7 +5423,7 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):agent._session_locks = {}agent._session_mgr = SimpleNamespace(sessions_for_cwd=lambda cwd: [])agent.codex_config = SimpleNamespace(binary="codex", extra_args=[])- agent._model_hub_catalog_path = None+ agent._model_hub_catalog = Noneagent._model_hub_catalog_lock = asyncio.Lock()agent._model_hub_catalog_generation = 0agent.controller = SimpleNamespace(config=SimpleNamespace(codex=agent.codex_config))@@ -5698,7 +5708,7 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):gateway_base_url="http://127.0.0.1:8317",gateway_token="ephemeral-token",)- agent._model_hub_catalog_path = Path(cwd) / "codex-hub-catalog.json"+ agent._model_hub_catalog = _catalog_reference(Path(cwd) / "codex-hub-catalog.json")with patch("vibe.backend_model_catalog.prepare_codex_hub_catalog",@@ -5712,8 +5722,8 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):async def test_runtime_config_switches_binary_without_catalog_export(self):agent = self._agent()- previous_catalog = Path("/runtime/codex-old.json")- agent._model_hub_catalog_path = previous_catalog+ previous_catalog = _catalog_reference("/runtime/codex-old.json")+ agent._model_hub_catalog = previous_catalognext_config = SimpleNamespace(binary="/opt/codex-next", extra_args=[])agent.refresh_auth_state = AsyncMock()@@ -5726,14 +5736,14 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):prepare_catalog.assert_not_called()self.assertIs(agent.codex_config, next_config)self.assertIs(agent.controller.config.codex, next_config)- self.assertIsNone(agent._model_hub_catalog_path)+ self.assertIsNone(agent._model_hub_catalog)self.assertEqual(agent._model_hub_catalog_generation, 1)agent.refresh_auth_state.assert_awaited_once_with()async def test_runtime_config_same_binary_invalidates_prepared_catalog(self):agent = self._agent()- previous_catalog = Path("/runtime/codex-old.json")- agent._model_hub_catalog_path = previous_catalog+ previous_catalog = _catalog_reference("/runtime/codex-old.json")+ agent._model_hub_catalog = previous_catalogagent.refresh_auth_state = AsyncMock()next_config = SimpleNamespace(binary=agent.codex_config.binary, extra_args=["--next"])@@ -5745,7 +5755,7 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):prepare_catalog.assert_not_called()self.assertIs(agent.codex_config, next_config)- self.assertIsNone(agent._model_hub_catalog_path)+ self.assertIsNone(agent._model_hub_catalog)self.assertEqual(agent._model_hub_catalog_generation, 1)agent.refresh_auth_state.assert_awaited_once_with()@@ -5753,19 +5763,19 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):agent = self._agent()transport = SimpleNamespace(stop=AsyncMock())agent._transports["/repo"] = transport- agent._model_hub_catalog_path = Path("/runtime/codex-old.json")+ agent._model_hub_catalog = _catalog_reference("/runtime/codex-old.json")await agent.invalidate_model_hub_runtime()- self.assertIsNone(agent._model_hub_catalog_path)+ self.assertIsNone(agent._model_hub_catalog)self.assertEqual(agent._model_hub_catalog_generation, 1)transport.stop.assert_not_awaited()async def test_startup_catalog_preparation_cannot_overwrite_new_runtime_generation(self):agent = self._agent()previous_config = agent.codex_config- previous_catalog = Path("/runtime/codex-old.json")- next_catalog = Path("/runtime/codex-new.json")+ previous_catalog = _catalog_reference("/runtime/codex-old.json")+ next_catalog = _catalog_reference("/runtime/codex-new.json")next_config = SimpleNamespace(binary="/opt/codex-next", extra_args=[])agent.refresh_auth_state = AsyncMock()previous_started = threading.Event()@@ -5788,7 +5798,7 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):self.assertTrue(await asyncio.to_thread(previous_started.wait, 1))await agent.refresh_runtime_config(next_config)self.assertIs(agent.codex_config, next_config)- self.assertIsNone(agent._model_hub_catalog_path)+ self.assertIsNone(agent._model_hub_catalog)release_previous.set()with self.assertRaises(_MODULE.CodexModelHubCatalogUnavailableError):await startup@@ -5802,12 +5812,12 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):],)self.assertIs(agent.codex_config, next_config)- self.assertEqual(agent._model_hub_catalog_path, next_catalog)+ self.assertEqual(agent._model_hub_catalog, next_catalog)self.assertEqual(recovered, next_catalog)async def test_model_hub_catalog_preparation_retries_after_transient_failure(self):agent = self._agent()- catalog = Path("/runtime/codex-recovered.json")+ catalog = _catalog_reference("/runtime/codex-recovered.json")with patch("vibe.backend_model_catalog.prepare_codex_hub_catalog",@@ -5815,11 +5825,11 @@ class CodexTransportCwdStalenessTests(unittest.IsolatedAsyncioTestCase):) as prepare_catalog:with self.assertRaises(_MODULE.CodexModelHubCatalogUnavailableError):await agent.prepare_model_hub_runtime()- self.assertIsNone(agent._model_hub_catalog_path)+ self.assertIsNone(agent._model_hub_catalog)recovered = await agent.prepare_model_hub_runtime()self.assertEqual(recovered, catalog)- self.assertEqual(agent._model_hub_catalog_path, catalog)+ self.assertEqual(agent._model_hub_catalog, catalog)self.assertEqual(prepare_catalog.call_count, 2)async def test_missing_prepared_hub_catalog_preserves_existing_transport_and_threads(self):diff --git a/tests/test_codex_catalog_retention.py b/tests/test_codex_catalog_retention.pynew file mode 100644index 000000000..0e86a6d6e--- /dev/null+++ b/tests/test_codex_catalog_retention.py@@ -0,0 +1,466 @@+"""Catalog ownership tests use only the autouse test-owned Avibe home."""++from __future__ import annotations++import asyncio+import gc+import json+import os+import subprocess+import sys+import threading+from concurrent.futures import ThreadPoolExecutor+from pathlib import Path+from types import SimpleNamespace+from unittest.mock import AsyncMock++import pytest++from config import paths+from modules.agents.codex.agent import CodexAgent, CodexModelHubCatalogUnavailableError+from modules.agents.codex.transport import CodexTransport+from vibe import backend_model_catalog as catalogs+++def publish(slug):+ return catalogs._publish_codex_hub_catalog(json.dumps({"models": [{"slug": slug}]}).encode())+++def retained():+ return set((paths.get_runtime_dir() / "model-hub" / "codex").glob("standard-responses-*.json"))+++def churn(count=6):+ for index in range(count):+ with publish(f"churn-{index}"):+ pass+ gc.collect()+++def agent():+ value = CodexAgent.__new__(CodexAgent)+ value._model_hub_catalog = None+ value._model_hub_catalog_lock = asyncio.Lock()+ value._model_hub_catalog_generation = 0+ value.codex_config = SimpleNamespace(binary="fixture-codex", extra_args=[])+ value.controller = SimpleNamespace()+ return value+++def test_same_content_reuses_inode_and_each_reader_protects_it():+ first = publish("中文")+ path = first.path+ inode = path.stat().st_ino+ second = publish("中文")+ assert first.path == second.path+ assert second.path.stat().st_ino == inode+ first.close()+ churn()+ assert path in retained()+ assert len(retained() - {path}) <= catalogs.CODEX_HUB_CATALOG_HISTORY_LIMIT+ second.close()+ churn()+ assert len(retained()) <= catalogs.CODEX_HUB_CATALOG_HISTORY_LIMIT+++def test_many_active_generations_are_separate_from_bounded_history():+ active = [publish(f"active-{index}") for index in range(4)]+ protected = {value.path for value in active}+ # Deliberately ancient timestamps cannot turn a live reader into history.+ for path in protected:+ os.utime(path, (1, 1))+ churn(20)+ assert protected <= retained()+ assert len(retained() - protected) <= catalogs.CODEX_HUB_CATALOG_HISTORY_LIMIT+ for value in active:+ value.close()+ active.clear()+ churn()+ assert len(retained()) <= catalogs.CODEX_HUB_CATALOG_HISTORY_LIMIT+++def test_cleanup_preserves_foreign_names_symlinks_directories_and_hardlinks(tmp_path):+ active = publish("active")+ directory = active.path.parent+ foreign = directory / "user-catalog.json"+ foreign.write_text("user content")+ malformed = directory / "standard-responses-old.json"+ malformed.write_text("released name prefix is insufficient")+ subdir = directory / ("standard-responses-" + "a" * 16 + ".json")+ subdir.mkdir()+ outside = tmp_path / "outside.json"+ outside.write_text("outside")+ symlink = directory / ("standard-responses-" + "b" * 16 + ".json")+ symlink.symlink_to(outside)+ hardlink = directory / ("standard-responses-" + "c" * 16 + ".json")+ os.link(outside, hardlink)+ churn()+ assert foreign.read_text() == "user content"+ assert malformed.is_file() and subdir.is_dir() and symlink.is_symlink()+ assert hardlink.read_text() == outside.read_text() == "outside"+++def test_cleanup_failure_does_not_break_publication_or_existing_reader(monkeypatch, caplog):+ active = publish("active")+ path = active.path+ real_unlink = Path.unlink++ def denied(value, *args, **kwargs):+ if catalogs._CODEX_HUB_CATALOG_NAME.fullmatch(value.name):+ raise PermissionError("fixture cleanup denial")+ return real_unlink(value, *args, **kwargs)++ with monkeypatch.context() as patch:+ patch.setattr(Path, "unlink", denied)+ churn()+ latest = publish("new-success")+ assert path.is_file()+ assert json.loads(latest.path.read_bytes())["models"][0]["slug"] == "new-success"+ assert "fixture cleanup denial" in caplog.text+ assert "Could not reclaim Codex Model Hub catalog" in caplog.text+ latest.close()+ churn()+ assert len(retained() - {path}) <= 1+++def test_failed_publication_or_export_cannot_select_old_catalog(monkeypatch):+ active = publish("old")+ monkeypatch.setattr(catalogs, "write_atomic", lambda *_: (_ for _ in ()).throw(OSError("disk full")))+ with pytest.raises(OSError, match="disk full"):+ publish("new")+ assert active.path.is_file()+ monkeypatch.setattr(+ catalogs, "_export_codex_bundled_catalog",+ lambda *_: (_ for _ in ()).throw(RuntimeError("export failed")),+ )+ with pytest.raises(RuntimeError, match="export failed"):+ catalogs.prepare_codex_hub_catalog("fixture-codex")+ assert retained() == {active.path}+++def test_concurrent_publication_and_cleanup_preserve_every_returned_pin():+ barrier = threading.Barrier(8)++ def launch(index):+ barrier.wait(timeout=5)+ value = publish(f"concurrent-{index % 3}")+ catalogs.prune_codex_hub_catalogs(value.path.parent)+ assert value.path.is_file()+ return value++ with ThreadPoolExecutor(max_workers=8) as workers:+ values = list(workers.map(launch, range(8)))+ protected = {value.path for value in values}+ churn()+ assert protected <= retained()+ for value in values:+ value.close()+ values.clear()+ gc.collect()+ churn()+ assert len(retained()) <= 1+++def test_cleaner_cannot_enter_between_file_publication_and_pin(monkeypatch):+ written = threading.Event()+ finish = threading.Event()+ write = catalogs.write_atomic+ published_paths = []++ def paused_write(path, data):+ write(path, data)+ published_paths.append(path)+ written.set()+ assert finish.wait(timeout=5)++ monkeypatch.setattr(catalogs, "write_atomic", paused_write)+ with ThreadPoolExecutor(max_workers=1) as worker:+ preparation = worker.submit(publish, "paused-publication")+ try:+ assert written.wait(timeout=2)+ path = published_paths[0]+ catalogs.prune_codex_hub_catalogs(path.parent)+ assert path.is_file()+ finally:+ finish.set()+ catalog = preparation.result(timeout=2)+ monkeypatch.setattr(catalogs, "write_atomic", write)+ churn()+ assert catalog.path in retained()+ catalog.close()+ churn()+ assert len(retained()) <= 1+++def test_separate_process_cannot_clean_starting_or_cached_generation():+ active = publish("parent")+ code = (+ "from vibe.backend_model_catalog import _publish_codex_hub_catalog\n"+ "for i in range(8):\n"+ " with _publish_codex_hub_catalog(('"+ '{"models":[{"slug":"child-%s"}]}' + "' % i).encode()): pass\n"+ )+ result = subprocess.run(+ [sys.executable, "-c", code], capture_output=True, text=True, timeout=15,+ env=os.environ.copy(),+ )+ assert result.returncode == 0, result.stderr+ assert active.path in retained()+ assert len(retained() - {active.path}) <= 1+++async def test_cache_invalidation_during_pending_launch_keeps_local_reference(monkeypatch):+ value = agent()+ monkeypatch.setattr(catalogs, "_export_codex_bundled_catalog", lambda *_: b'{"models":[{"slug":"first"}]}')+ starting = (await value.prepare_model_hub_runtime()).retain()+ path = starting.path+ await value.invalidate_model_hub_runtime()+ churn()+ assert value._model_hub_catalog is None and path in retained()+ starting.close()+ churn()+ assert len(retained()) <= 1+++async def test_invalidated_export_releases_unpublished_pin(monkeypatch):+ value = agent()+ entered = threading.Event()+ released = threading.Event()++ def export(*_):+ entered.set()+ assert released.wait(timeout=5)+ return b'{"models":[{"slug":"raced"}]}'++ monkeypatch.setattr(catalogs, "_export_codex_bundled_catalog", export)+ preparation = asyncio.create_task(value.prepare_model_hub_runtime())+ assert await asyncio.to_thread(entered.wait, 2)+ await value.invalidate_model_hub_runtime()+ released.set()+ with pytest.raises(CodexModelHubCatalogUnavailableError, match="generation changed"):+ await preparation+ assert value._model_hub_catalog is None+ del preparation+ churn()+ assert len(retained()) <= 1+++async def test_cancelled_export_closes_its_eventual_result(monkeypatch):+ value = agent()+ entered = threading.Event()+ finish = threading.Event()+ closed = asyncio.Event()+ loop = asyncio.get_running_loop()+ release = catalogs._release_codex_hub_catalog++ def export(*_):+ entered.set()+ assert finish.wait(timeout=5)+ return b'{"models":[{"slug":"cancelled"}]}'++ def observe_close(path, descriptor):+ release(path, descriptor)+ loop.call_soon_threadsafe(closed.set)++ monkeypatch.setattr(catalogs, "_export_codex_bundled_catalog", export)+ monkeypatch.setattr(catalogs, "_release_codex_hub_catalog", observe_close)+ preparation = asyncio.create_task(value.prepare_model_hub_runtime())+ assert await asyncio.to_thread(entered.wait, 2)+ preparation.cancel()+ with pytest.raises(asyncio.CancelledError):+ await preparation+ finish.set()+ await asyncio.wait_for(closed.wait(), 5)+ assert value._model_hub_catalog is None+ churn()+ assert len(retained()) <= 1+++_STDIO_CONSUMER = """+import json, os, sys+for line in sys.stdin:+ request = json.loads(line)+ if request.get("method") == "test/exit":+ break+ if "id" in request:+ print(json.dumps({"id": request["id"], "result": {}}), flush=True)+ if request.get("method") == "test/closeStdout":+ os.close(1)+"""+++async def test_agent_launch_keeps_pin_across_invalidation_and_reuses_transport(monkeypatch, tmp_path):+ value = agent()+ value._transports = {}+ value._transport_locks = {}+ value._transport_cwd_inodes = {}+ initialized = asyncio.Event()+ finish = asyncio.Event()+ spawn = asyncio.create_subprocess_exec+ argv = []+ monkeypatch.setattr(catalogs, "_export_codex_bundled_catalog", lambda *_: b'{"models":[{"slug":"agent"}]}')++ async def fixture_spawn(*args, **kwargs):+ argv.extend(args)+ return await spawn(sys.executable, "-u", "-c", _STDIO_CONSUMER, **kwargs)++ async def pause_governor_start(self):+ await original_start(self)+ initialized.set()+ await finish.wait()++ original_start = CodexTransport.start+ monkeypatch.setattr(asyncio, "create_subprocess_exec", fixture_spawn)+ monkeypatch.setattr(CodexTransport, "start", pause_governor_start)+ # An extra argument must not override the path whose lifetime we protect.+ value.codex_config.extra_args = ["-c", 'model_catalog_json="/obsolete/catalog.json"']+ launch = SimpleNamespace(+ channel="hub", fingerprint="hub:test",+ gateway_base_url="http://127.0.0.1:1", gateway_token="fixture-only",+ )+ starting = asyncio.create_task(value._get_or_create_transport(str(tmp_path), launch))+ try:+ await asyncio.wait_for(initialized.wait(), 5)+ path = value._model_hub_catalog.path+ assert argv[-1] == f"model_catalog_json={json.dumps(str(path))}"+ await value.invalidate_model_hub_runtime()+ churn()+ assert path in retained()+ finish.set()+ transport = await starting+ assert (await value._get_or_create_transport(str(tmp_path), launch)) is transport+ assert path in retained()+ await transport.stop()+ await transport._catalog_exit_task+ churn()+ assert len(retained()) <= 1+ finally:+ finish.set()+ transport = await starting+ await transport.stop()+++async def test_stdout_eof_keeps_pin_until_real_process_exit(monkeypatch, tmp_path):+ spawn = asyncio.create_subprocess_exec++ async def fixture_spawn(*_args, **kwargs):+ return await spawn(sys.executable, "-u", "-c", _STDIO_CONSUMER, **kwargs)++ monkeypatch.setattr(asyncio, "create_subprocess_exec", fixture_spawn)+ catalog = publish("stdout-live")+ path = catalog.path+ transport = CodexTransport("fixture", str(tmp_path), model_hub_catalog=catalog)+ catalog.close()+ try:+ await transport.start()+ await transport.send_request("test/closeStdout", {})+ await asyncio.wait_for(transport.wait_closed(), 5)+ assert transport._process.returncode is None+ churn()+ assert path.is_file()+ transport._process.stdin.close()+ await asyncio.wait_for(transport._catalog_exit_task, 5)+ assert transport._process.returncode == 0+ assert transport._model_hub_catalog is None+ churn()+ assert len(retained()) <= 1+ finally:+ await transport.stop()+++async def test_cancel_during_real_spawn_keeps_pin_until_child_is_reaped(monkeypatch, tmp_path):+ spawn = asyncio.create_subprocess_exec+ created = asyncio.Event()+ finish_spawn = asyncio.Event()+ processes = []++ async def fixture_spawn(*_args, **kwargs):+ proc = await spawn(sys.executable, "-u", "-c", _STDIO_CONSUMER, **kwargs)+ processes.append(proc)+ created.set()+ await finish_spawn.wait()+ return proc++ monkeypatch.setattr(asyncio, "create_subprocess_exec", fixture_spawn)+ catalog = publish("cancel-spawn")+ path = catalog.path+ transport = CodexTransport("fixture", str(tmp_path), model_hub_catalog=catalog)+ catalog.close()+ task = asyncio.create_task(transport.start())+ try:+ await asyncio.wait_for(created.wait(), 5)+ task.cancel()+ churn()+ assert processes[0].returncode is None and path.is_file()+ finish_spawn.set()+ with pytest.raises(asyncio.CancelledError):+ await task+ assert processes[0].returncode == 0+ assert transport._model_hub_catalog is None+ churn()+ assert len(retained()) <= 1+ finally:+ finish_spawn.set()+ await transport.stop()++[email protected]("failure", ["spawn", "handshake", "cancel-handshake"])+async def test_failed_start_releases_transport_reference(monkeypatch, tmp_path, failure):+ catalog = publish(failure)+ path = catalog.path+ transport = CodexTransport("fixture-codex", str(tmp_path), model_hub_catalog=catalog)+ catalog.close()+ exited = asyncio.Event()+ entered = asyncio.Event()+ process = SimpleNamespace(pid=123, returncode=None, stdin=None, stdout=None, stderr=None)++ async def wait():+ await exited.wait()+ return 1++ async def spawn(*_, **__):+ if failure == "spawn":+ raise OSError("fixture spawn failed")+ return process++ async def handshake(*_):+ entered.set()+ if failure == "cancel-handshake":+ await asyncio.Event().wait()+ raise RuntimeError("fixture handshake failed")++ async def stop():+ process.returncode = 1+ exited.set()+ transport._release_model_hub_catalog()+ transport._cleanup_tasks()++ process.wait = wait+ monkeypatch.setattr(asyncio, "create_subprocess_exec", spawn)+ monkeypatch.setattr(transport, "send_request", handshake)+ monkeypatch.setattr(transport, "stop", stop)+ monkeypatch.setattr(transport, "_reader_loop", AsyncMock())+ monkeypatch.setattr(transport, "_stderr_reader", AsyncMock())+ monkeypatch.setattr("modules.agents.codex.transport.process_identity", lambda *_: {})+ monkeypatch.setattr("modules.agents.codex.transport.log_process_snapshot", lambda *_args, **_kw: None)+ task = asyncio.create_task(transport.start())+ if failure == "cancel-handshake":+ await entered.wait()+ churn()+ assert path.is_file()+ task.cancel()+ with pytest.raises((OSError, RuntimeError, asyncio.CancelledError)):+ await task+ if transport._catalog_exit_task is not None:+ await transport._catalog_exit_task+ assert transport._model_hub_catalog is None+ del task+ churn()+ assert len(retained()) <= 1diff --git a/vibe/backend_model_catalog.py b/vibe/backend_model_catalog.pyindex 07cbb7654..bf2a3acfb 100644--- a/vibe/backend_model_catalog.py+++ b/vibe/backend_model_catalog.py@@ -5,13 +5,18 @@ import hashlibimport jsonimport loggingimport os+import re+import stat+import subprocessimport threadingimport timeimport urllib.errorimport urllib.request+import weakref+from contextlib import contextmanagerfrom pathlib import Pathfrom types import MappingProxyType-from typing import Any, Callable, Final, Iterable, Mapping, Sequence+from typing import TYPE_CHECKING, Any, Callable, Final, Iterable, Mapping, Sequencefrom config import pathsfrom config.atomic_io import write_atomic@@ -23,6 +28,9 @@ from vibe.claude_model_catalog import ()from vibe.codex_config import get_codex_home+if TYPE_CHECKING:+ from storage.lock import MigrationFileLock+logger = logging.getLogger(__name__)@@ -38,6 +46,8 @@ REMOTE_CATALOG_USER_AGENT = "avibe/backend-model-catalog"REMOTE_CATALOG_CACHE_VERSION = 2CODEX_HUB_CATALOG_TIMEOUT_SECONDS = 15.0CODEX_HUB_CATALOG_MAX_BYTES = 8 * 1024 * 1024+CODEX_HUB_CATALOG_HISTORY_LIMIT = 1+_CODEX_HUB_CATALOG_NAME = re.compile(r"standard-responses-[0-9a-f]{16}\.json")_HIDDEN_VISIBILITIES = {"hide", "hidden"}_VISIBLE_VISIBILITIES = {"visible", "list"}@@ -131,6 +141,166 @@ def _codex_hub_catalog_path(catalog: bytes) -> Path:return paths.get_runtime_dir() / "model-hub" / "codex" / f"standard-responses-{digest}.json"+class CodexHubCatalog:+ """One owner's pin on an immutable catalog generation.++ Caches, pending launches and transports acquire independent pins and close+ them at their lifecycle boundary. Finalization is only an abandoned-owner+ fallback: exception tracebacks must not delay ordinary release.+ """++ def __init__(self, path: Path, descriptor: int) -> None:+ self.path = path+ self.descriptor = descriptor+ self._release = weakref.finalize(self, _release_codex_hub_catalog, path, descriptor)++ def retain(self) -> CodexHubCatalog:+ """Acquire an independent pin before giving up this owner's pin."""+ if not self._release.alive:+ raise RuntimeError("Cannot retain a released Codex catalog")+ with _codex_hub_catalog_lock(self.path.parent):+ return _pin_codex_hub_catalog(self.path)++ def close(self) -> None:+ self._release()++ def __enter__(self) -> CodexHubCatalog:+ return self++ def __exit__(self, *_args: object) -> None:+ self.close()++ @contextmanager+ def inherited_subprocess_kwargs(self):+ """Keep the file pinned in the child even if its Avibe parent exits."""+ if not self._release.alive:+ raise RuntimeError("Cannot launch with a released Codex catalog")+ if os.name != "nt":+ yield {"pass_fds": (self.descriptor,)}+ return+ import msvcrt++ # Each spawn gets its own temporary inheritable handle, so simultaneous+ # launches cannot reset one another's inheritance flags.+ descriptor = os.dup(self.descriptor)+ try:+ os.set_inheritable(descriptor, True)+ startupinfo = subprocess.STARTUPINFO()+ startupinfo.lpAttributeList = {"handle_list": [msvcrt.get_osfhandle(descriptor)]}+ yield {"startupinfo": startupinfo, "close_fds": True}+ finally:+ os.close(descriptor)+++def _codex_hub_catalog_lock(directory: Path, *, timeout_seconds: float = 30.0) -> MigrationFileLock:+ # Lightweight catalog/handler imports must not initialize SQLite storage.+ from storage.lock import MigrationFileLock++ return MigrationFileLock(directory / ".catalog.lock", timeout_seconds=timeout_seconds)+++def _open_codex_hub_catalog(path: Path) -> int:+ """Open only an ordinary file without following a candidate symlink."""+ flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) | getattr(os, "O_NOFOLLOW", 0)+ flags |= getattr(os, "O_NONBLOCK", 0)+ descriptor = os.open(path, flags)+ try:+ opened = os.fstat(descriptor)+ named = path.lstat()+ if (+ not stat.S_ISREG(named.st_mode)+ or not stat.S_ISREG(opened.st_mode)+ or opened.st_nlink != 1+ or (opened.st_dev, opened.st_ino) != (named.st_dev, named.st_ino)+ ):+ raise OSError(f"Codex catalog is not an exclusive regular file: {path}")+ return descriptor+ except BaseException:+ os.close(descriptor)+ raise+++def _pin_codex_hub_catalog(path: Path) -> CodexHubCatalog:+ descriptor = _open_codex_hub_catalog(path)+ try:+ if os.name != "nt":+ import fcntl++ fcntl.flock(descriptor, fcntl.LOCK_SH | fcntl.LOCK_NB)+ return CodexHubCatalog(path, descriptor)+ except BaseException:+ os.close(descriptor)+ raise+++def _prune_codex_hub_catalogs_locked(directory: Path) -> None:+ """Keep every pinned generation and at most one unpinned history file.++ Caller holds the publication lock. Name order only chooses the spare;+ neither timestamps nor age establish that a generation is unused.+ """+ spare = 0+ for path in sorted(directory.iterdir()):+ if not _CODEX_HUB_CATALOG_NAME.fullmatch(path.name):+ continue+ descriptor = None+ try:+ if not stat.S_ISREG(path.lstat().st_mode) or path.lstat().st_nlink != 1:+ continue+ descriptor = _open_codex_hub_catalog(path)+ if os.name != "nt":+ import fcntl++ try:+ fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)+ except BlockingIOError:+ continue+ # On Windows the deletion itself is the non-blocking pin probe:+ # any reader's open handle denies deletion. Close our probe first,+ # while the publication lock still excludes a new reader.+ if os.name == "nt":+ os.close(descriptor)+ descriptor = None+ if spare < CODEX_HUB_CATALOG_HISTORY_LIMIT:+ spare += 1+ continue+ path.unlink()+ except FileNotFoundError:+ continue+ except OSError as exc:+ if os.name == "nt" and getattr(exc, "winerror", None) == 32:+ # ERROR_SHARING_VIOLATION: a cache/child still owns its pin.+ continue+ logger.warning("Could not reclaim Codex Model Hub catalog %s", path, exc_info=True)+ finally:+ if descriptor is not None:+ os.close(descriptor)+++def prune_codex_hub_catalogs(directory: Path) -> None:+ """Best-effort reclamation at publication and reference-release boundaries."""+ from storage.lock import MigrationLockTimeout++ try:+ if not directory.is_dir():+ return+ with _codex_hub_catalog_lock(directory, timeout_seconds=0):+ _prune_codex_hub_catalogs_locked(directory)+ except MigrationLockTimeout:+ # A publisher/cleaner already owns this directory. Its sweep, or the+ # next lifecycle boundary, retries reclamation.+ logger.debug("Codex catalog cleanup deferred while publication is in progress")+ except OSError:+ logger.warning("Could not clean Codex Model Hub catalogs in %s", directory, exc_info=True)+++def _release_codex_hub_catalog(path: Path, descriptor: int) -> None:+ # Close, do not LOCK_UN: a spawned app-server inherits the same open file+ # description and must keep its lock even if the Avibe parent goes away.+ os.close(descriptor)+ prune_codex_hub_catalogs(path.parent)++def _codex_hub_catalog_bytes(raw_catalog: bytes,configured_models: Sequence[Mapping[str, Any]] | None = None,@@ -278,11 +448,26 @@ def _codex_hub_catalog_bytes(def _publish_codex_hub_catalog(raw_catalog: bytes,configured_models: Sequence[Mapping[str, Any]] | None = None,-) -> Path:+) -> CodexHubCatalog:catalog = _codex_hub_catalog_bytes(raw_catalog, configured_models)path = _codex_hub_catalog_path(catalog)- write_atomic(path, catalog)- return path+ with _codex_hub_catalog_lock(path.parent):+ try:+ path.lstat()+ except FileNotFoundError:+ write_atomic(path, catalog)+ pinned = _pin_codex_hub_catalog(path)+ try:+ if os.read(pinned.descriptor, len(catalog) + 1) != catalog:+ raise ValueError("Codex catalog digest path contains different bytes")+ except BaseException:+ pinned.close()+ raise+ try:+ _prune_codex_hub_catalogs_locked(path.parent)+ except OSError:+ logger.warning("Could not clean Codex Model Hub catalogs in %s", path.parent, exc_info=True)+ return pinneddef _export_codex_bundled_catalog(@@ -331,7 +516,7 @@ def prepare_codex_hub_catalog(binary: str,base_env: dict[str, str] | None = None,configured_models: Sequence[Mapping[str, Any]] | None = None,-) -> Path:+) -> CodexHubCatalog:"""Prepare the exact binary's catalog immediately before a Hub launch."""return _publish_codex_hub_catalog(