Skip to content

armgit / avibe

Public Imported from GitHub
0

Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.

fix(model-hub): hot-reload source saves instead of restarting the engine (#2138)

Call Whiteac979138f0fe
diff --git a/core/handlers/model_hub/service.py b/core/handlers/model_hub/service.py
index 6fb5947dd..c131964f7 100644
--- a/core/handlers/model_hub/service.py
+++ b/core/handlers/model_hub/service.py
@@ -1340,7 +1340,7 @@ class ModelHubService:
updated_bindings = self._bindings(updated)
# SourceBinding order is the engine-config serialization order. Agent
# Route target order is not part of this projection, so a pure chain reorder
- # compares equal and does not restart a healthy engine.
+ # compares equal and does not touch the engine.
if previous_bindings == updated_bindings:
self._save_config(updated)
return
diff --git a/docs/plans/model-hub-routing-modes.md b/docs/plans/model-hub-routing-modes.md
index f268c4354..02dd3e15a 100644
--- a/docs/plans/model-hub-routing-modes.md
+++ b/docs/plans/model-hub-routing-modes.md
@@ -369,23 +369,43 @@ transport is authorized by this amendment.
### Registration Synchronization Decision
-Pinned-engine experiments proved file/atomic/management configuration hot reload
-with unchanged PID and uninterrupted existing streams for all three protocols.
-However, neither the management acknowledgment nor `/v1/models` provides an
-atomic scheduler/config-generation completion barrier. The feature therefore
-retains the existing supervised restart transaction with explicit in-flight
-protection instead of introducing an unproven readiness heuristic.
-
-The adapter owns invocation leases and drains existing transport operations
-before a configuration-triggered engine restart. New invocations wait behind the
-same configuration barrier. Lease release belongs to transport completion,
-close, or cancellation, independently of service settlement and its mutation
-lock. A synchronization cancellation or exception cannot leak a barrier or
-lease. Do not cancel unrelated active requests just to save configuration.
-Pure reordering with unchanged registration remains restart-free, and ordinary
-invocations never trigger registration changes. Exercise buffered and streaming
-completion, early close, cancellation, sync failure/rollback, and concurrent Save
-to prove there is no deadlock or partial-output interruption.
+Saving configuration never restarts the engine and never waits for in-flight
+work. Pinned-engine experiments proved that CPA hot-reloads its config with an
+unchanged PID and uninterrupted existing streams for all three protocols. They
+also proved that CPA does not observe an atomic rename of its watched config
+file. A running engine therefore receives the rendered projection through the
+management `PUT /v0/management/config.yaml`, which validates it, writes the
+file in place and reloads.
+
+Neither the acknowledgment nor `/v1/models` is an atomic config-generation
+barrier, and a save may leave every routed ID unchanged (for example a
+reasoning-effort edit). Each rendered projection therefore carries a
+per-process generation in every API-key model's display name. Synchronization
+completes when CPA's Grok-client model listing, the one format that keeps both
+the routed ID and the display name, shows every expected route at this
+generation and no removed one, within a bounded wait. A rejection or timeout
+fails the save and restores the previous projection through the same reload
+path. A lost acknowledgment or a stalled management path never stops a live
+process, because neither proves its streams dead; only an engine that has
+already exited is left to its next start, which renders the projection.
+
+The adapter's routing lock orders a projection change against new admissions.
+A request admitted before a save keeps its projection until the engine has
+read the request body, so a removal cannot strand it; that wait is bounded by a
+local write, never by inference. CPA resolves the route on reading the body and
+applies a config write only after its reload debounce, so the written body
+orders the request first; its only later acknowledgment, response headers,
+waits for upstream output and would make a save wait on inference. Established transport leases are neither
+awaited nor cancelled, and a Save settles while a turn is streaming. Leases
+still gate the one remaining restart: an engine binary upgrade. Credential
+revocation follows an unbinding save, so a live engine drops the secret through
+the same verified hot reload, which rewrites its config; other instances'
+configs are cleared. An unverifiable reload fails the revocation instead of
+restarting, and the journaled pending revocation is retried. Pure reordering with
+unchanged registration is a no-op, and ordinary invocations never trigger
+registration changes. Exercise buffered and streaming calls across a Save,
+cancellation, reload failure/rollback, and concurrent Save, to prove there is
+no waiting, restart, or partial-output interruption.
Acceptance states properties, with fixtures covering the supported shapes:
diff --git a/tests/e2e/test_model_hub_routing.py b/tests/e2e/test_model_hub_routing.py
index 1599e6d83..d09ce5f8d 100644
--- a/tests/e2e/test_model_hub_routing.py
+++ b/tests/e2e/test_model_hub_routing.py
@@ -201,8 +201,8 @@ def test_mh_routing_012_persisted_dormant_opencode_override_reaches_exact_source
@pytest.mark.parametrize("protocol", ["openai_responses", "openai_chat", "anthropic"])
@pytest.mark.parametrize("disconnect", [False, True], ids=["complete", "client-close"])
-def test_mh_routing_009_save_drains_active_stream_without_truncation_or_deadlock(tmp_path, monkeypatch, protocol, disconnect):
- """MH-ROUTING-009: Save waits for real stream completion or client close, then serves the new target without deadlock."""
+def test_mh_routing_009_save_during_active_stream_hot_reloads_without_restart_or_truncation(tmp_path, monkeypatch, protocol, disconnect):
+ """MH-ROUTING-009: Save hot-reloads the live engine during active output; the stream is neither awaited nor truncated."""
async def exercise(adapter, mock):
service, menu, target, sources = _real_route_service(tmp_path, adapter, [mock], protocol, [])
await service.runtime_start()
@@ -234,16 +234,15 @@ def test_mh_routing_009_save_drains_active_stream_without_truncation_or_deadlock
save = asyncio.create_task(service.set_agent_chain("opencode", menu, {
"hops": [{"source_id": sources[0].id, "model_id": next_target}],
}))
- await asyncio.sleep(0.05)
- assert not save.done(), "Save must wait for the active transport to finish"
+ saved = await asyncio.wait_for(save, timeout=10)
assert adapter.supervisor.client_if_running().connection == old_connection
+ assert adapter._active_transports == 1
if disconnect:
response.close()
- saved = await asyncio.wait_for(save, timeout=15)
mock.configure(stream="healthy")
else:
mock.configure(stream="healthy")
- saved, tail = await asyncio.wait_for(asyncio.gather(save, response.read()), timeout=15)
+ tail = await asyncio.wait_for(response.read(), timeout=15)
assert terminal in prefix + tail
assert saved["chain"]["manual_override"] == {"hops": [{"source_id": sources[0].id, "model_id": next_target}]}
async with client.post(f"{base_url}/v1/{path}", json={**payload, "stream": False},
@@ -253,6 +252,7 @@ def test_mh_routing_009_save_drains_active_stream_without_truncation_or_deadlock
captured = [row["body"]["model"] for row in mock.requests() if row["path"] == f"/v1/{path}"]
assert captured == [target, next_target]
assert service.store.load().sources[0].models == []
+ assert adapter.supervisor.client_if_running().connection == old_connection
finally:
mock.configure(stream="healthy")
if save is not None and not save.done():
diff --git a/tests/scenarios/model_hub/catalog.yaml b/tests/scenarios/model_hub/catalog.yaml
index 6b60bc599..4f50a832b 100644
--- a/tests/scenarios/model_hub/catalog.yaml
+++ b/tests/scenarios/model_hub/catalog.yaml
@@ -356,11 +356,11 @@ scenarios:
layer: e2e
test: tests/e2e/test_model_hub_routing.py::test_mh_routing_008_real_gateway_cpa_exact_model_and_source
- id: MH-ROUTING-009
- name: Registration-changing Save waits for active output completion or client close without deadlock or truncation
+ name: Registration-changing Save hot-reloads the live engine during active output without waiting, restart, or truncation
status: assert
kind: concurrency
layer: e2e
- test: tests/e2e/test_model_hub_routing.py::test_mh_routing_009_save_drains_active_stream_without_truncation_or_deadlock
+ test: tests/e2e/test_model_hub_routing.py::test_mh_routing_009_save_during_active_stream_hot_reloads_without_restart_or_truncation
- id: MH-ROUTING-010
name: Real gateway model-not-found preserves source health and records exact attributed errors superseded by success
status: assert
diff --git a/tests/test_model_hub_migration_bearer.py b/tests/test_model_hub_migration_bearer.py
index a222fcf1f..91abc2ce5 100644
--- a/tests/test_model_hub_migration_bearer.py
+++ b/tests/test_model_hub_migration_bearer.py
@@ -30,6 +30,7 @@ def _adapter(tmp_path):
store = EngineStateStore(tmp_path / "engine")
supervisor = SimpleNamespace(
state_store=store, client_if_running=lambda: None, restart_if_running=lambda: None,
+ reload_config_if_running=lambda _previous=None: None,
invalidate_configs=store.clear_runtime_configs,
)
return CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
diff --git a/tests/test_model_hub_routing_modes.py b/tests/test_model_hub_routing_modes.py
index 53f7be4c8..14175f3d1 100644
--- a/tests/test_model_hub_routing_modes.py
+++ b/tests/test_model_hub_routing_modes.py
@@ -300,8 +300,8 @@ def test_engine_compiler_registers_inventory_union_without_invented_reasoning(tm
payload = {}
_append_source(payload, record, store)
assert payload[section][0]["models"] == [
- {"name": "listed", "alias": "listed", "thinking": {"levels": ["high", "low"]}},
- {"name": "unlisted", "alias": "unlisted"},
+ {"name": "listed", "alias": "listed", "display-name": "listed #0", "thinking": {"levels": ["high", "low"]}},
+ {"name": "unlisted", "alias": "unlisted", "display-name": "unlisted #0"},
]
store.sync_sources([replace(binding, model_ids=(), model_reasoning_efforts=())])
assert store.list_sources()[0].model_ids == ()
@@ -882,7 +882,7 @@ def test_retained_opencode_route_registration_matches_resolver_without_inventing
_append_source(compiled, engine_store.get_source(source.id), engine_store)
compiled_models = compiled["codex-api-key"][0]["models"]
assert {item["name"] for item in compiled_models} == set(binding.model_ids) | set(expected_targets)
- assert all(set(item) == {"name", "alias"} for item in compiled_models)
+ assert all(set(item) == {"name", "alias", "display-name"} for item in compiled_models)
if state == "unknown":
adapter.outcomes.append(
RawCallOutcome(
diff --git a/tests/test_model_hub_runtime.py b/tests/test_model_hub_runtime.py
index c36da64e4..c8dbaab7d 100644
--- a/tests/test_model_hub_runtime.py
+++ b/tests/test_model_hub_runtime.py
@@ -6,6 +6,7 @@ import io
import json
import logging
import subprocess
+import signal
import stat
import sys
import tarfile
@@ -48,6 +49,7 @@ from core.handlers.model_hub.stream_wire import (
from vibe.model_hub_runtime import adapter as runtime_adapter_module
from vibe.model_hub_runtime import client as client_module
from vibe.model_hub_runtime import installer as runtime_installer_module
+from vibe.model_hub_runtime import supervisor as supervisor_module
from vibe.model_hub_runtime.adapter import CLIProxyEngineAdapter
from vibe.model_hub_runtime.api_key_vendors import api_key_vendor_catalog
from vibe.model_hub_runtime.client import EngineClient, EngineClientError, EngineConnection
@@ -1626,6 +1628,7 @@ def test_config_generation_is_private_and_never_logs_secrets(
{
"name": "model-a",
"alias": "model-a",
+ "display-name": "model-a #0",
"thinking": {"levels": ["high", "low"]},
}
]
@@ -1712,7 +1715,7 @@ def test_mixed_anthropic_credentials_disable_cloak_only_for_api_key_entry(
"cloak": {"mode": "never"},
"rebuild-mid-system-message": False,
"models": [
- {"name": "claude-api-model", "alias": "claude-api-model"}
+ {"name": "claude-api-model", "alias": "claude-api-model", "display-name": "claude-api-model #0"}
],
}
]
@@ -2319,7 +2322,8 @@ def test_long_inventory_identity_reaches_runtime_config_and_http_consumer(
)
config = yaml.safe_load(config_path.read_text(encoding="utf-8"))
assert config[config_key][0]["models"] == [
- {"name": identity, "alias": identity} for identity in (*identities, route_only)
+ {"name": identity, "alias": identity, "display-name": f"{identity} #0"}
+ for identity in (*identities, route_only)
]
client = EngineClient(EngineConnection(base_url.removesuffix("/v1"), "management", "gateway"))
for identity in (*identities, route_only):
@@ -2612,10 +2616,13 @@ def _write_mock_engine(
startup_output_repeat: int = 1,
echo_runtime_secrets: bool = False,
exit_before_ready: int | None = None,
+ reload_delay: float = 0.0,
+ drop_reload_ack: bool = False,
) -> None:
script = f"""#!{sys.executable}
import json
import sys
+import threading
import time
from http.server import BaseHTTPRequestHandler, HTTPServer
@@ -2646,6 +2653,12 @@ if exit_before_ready is not None:
time.sleep({startup_delay!r})
health_surfaces = set()
+def configured_models():
+ for section in ('claude-api-key', 'codex-api-key', 'openai-compatibility'):
+ for entry in config.get(section) or ():
+ for model in entry.get('models') or ():
+ yield f"{{entry['prefix']}}/{{model['alias']}}", model.get('display-name')
+
def mark_health_surface(surface):
health_surfaces.add(surface)
if len(health_surfaces) == 2:
@@ -2667,7 +2680,7 @@ class Handler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == '/v1/models' and self.headers.get('Authorization') == f'Bearer {{gateway}}':
mark_health_surface('gateway')
- self._json(200, {{'object': 'list', 'data': []}})
+ self._json(200, {{'object': 'list', 'data': [{{'id': model, 'name': name}} for model, name in configured_models()]}})
return
if self.path == '/v0/management/config' and self.headers.get('X-Management-Key') == management:
mark_health_surface('management')
@@ -2678,6 +2691,29 @@ class Handler(BaseHTTPRequestHandler):
return
self._json(401, {{'error': {{'type': 'unauthorized'}}}})
+ def do_PUT(self):
+ global config
+ length = int(self.headers.get('Content-Length', '0'))
+ body = self.rfile.read(length)
+ if self.path != '/v0/management/config.yaml' or self.headers.get('X-Management-Key') != management:
+ self._json(401, {{'error': {{'type': 'unauthorized'}}}})
+ return
+ with open(config_path, 'wb') as handle:
+ handle.write(body)
+
+ def apply():
+ global config
+ config = yaml.safe_load(body)
+ with open('config-reloads', 'a', encoding='utf-8') as handle:
+ handle.write('reload\\n')
+
+ # CPA acknowledges the write and applies it from a debounced watcher.
+ threading.Timer({reload_delay!r}, apply).start()
+ if {drop_reload_ack!r}:
+ self.close_connection = True
+ return
+ self._json(200, {{'ok': True, 'changed': ['config']}})
+
def do_POST(self):
length = int(self.headers.get('Content-Length', '0'))
payload = json.loads(self.rfile.read(length))
@@ -2803,6 +2839,8 @@ def _fixture_supervisor(
startup_output_repeat: int = 1,
echo_runtime_secrets: bool = False,
exit_before_ready: int | None = None,
+ reload_delay: float = 0.0,
+ drop_reload_ack: bool = False,
) -> tuple[EngineSupervisor, EngineStateStore]:
tmp_path.mkdir(parents=True, exist_ok=True)
binary = tmp_path / "mock-engine"
@@ -2813,6 +2851,8 @@ def _fixture_supervisor(
startup_output_repeat=startup_output_repeat,
echo_runtime_secrets=echo_runtime_secrets,
exit_before_ready=exit_before_ready,
+ reload_delay=reload_delay,
+ drop_reload_ack=drop_reload_ack,
)
installer = _FixtureInstaller(binary, tmp_path / "versions" / "install-1")
store = EngineStateStore(tmp_path / "state")
@@ -2827,6 +2867,199 @@ def _fixture_supervisor(
)
+def test_source_sync_hot_reloads_a_running_engine_without_restart(tmp_path: Path) -> None:
+ supervisor, store = _fixture_supervisor(tmp_path)
+ credential_ref = store.store_api_key("upstream-secret", base_url="https://api.example.test/v1")
+ store.sync_sources([_binding(credential_ref)])
+ connection = supervisor.ensure_running()
+ process = supervisor._process
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ try:
+ asyncio.run(adapter.sync_sources([_binding(credential_ref, route_model_ids=("model-b",))]))
+ client = EngineClient(connection)
+ prefix = store.get_source("src_fixture123").prefix
+ assert supervisor._process is process
+ assert supervisor._connection == connection
+ assert client.list_model_ids() == {f"{prefix}/model-a", f"{prefix}/model-b"}
+ asyncio.run(adapter.sync_sources([_binding(credential_ref)]))
+ assert supervisor._process is process
+ assert client.list_model_ids() == {f"{prefix}/model-a"}
+ instance_dir = store.root / "instances" / "install-1"
+ assert (instance_dir / "config-reloads").read_text().count("reload") == 2
+ assert "model-b" not in (instance_dir / "config.yaml").read_text()
+ finally:
+ supervisor.stop()
+
+
+def test_source_sync_waits_for_an_id_preserving_reload_to_apply(tmp_path: Path) -> None:
+ supervisor, store = _fixture_supervisor(tmp_path, reload_delay=0.3)
+ credential_ref = store.store_api_key("upstream-secret", base_url="https://api.example.test/v1")
+ store.sync_sources([_binding(credential_ref)])
+ supervisor.ensure_running()
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ instance_dir = store.root / "instances" / "install-1"
+ try:
+ # A reasoning-only save renders new config yet leaves every routed ID
+ # listed; the save returns only once the engine has applied it.
+ asyncio.run(
+ adapter.sync_sources(
+ [_binding(credential_ref, model_reasoning_efforts=(("model-a", ("low", "high")),))]
+ )
+ )
+ assert (instance_dir / "config-reloads").read_text().count("reload") == 1
+ finally:
+ supervisor.stop()
+
+
+def test_source_sync_waits_for_a_reload_that_removes_every_model(tmp_path: Path) -> None:
+ supervisor, store = _fixture_supervisor(tmp_path, reload_delay=0.3)
+ credential_ref = store.store_api_key("upstream-secret", base_url="https://api.example.test/v1")
+ store.sync_sources([_binding(credential_ref)])
+ connection = supervisor.ensure_running()
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ try:
+ # No model is expected afterwards, so readiness is the removed
+ # model leaving the listing.
+ asyncio.run(adapter.sync_sources([_binding(credential_ref, model_ids=())]))
+ assert EngineClient(connection).list_model_ids() == frozenset()
+ finally:
+ supervisor.stop()
+
+
+def test_source_sync_never_restarts_a_live_engine_that_lost_the_reload_ack(tmp_path: Path) -> None:
+ supervisor, store = _fixture_supervisor(tmp_path, drop_reload_ack=True)
+ credential_ref = store.store_api_key("upstream-secret", base_url="https://api.example.test/v1")
+ store.sync_sources([_binding(credential_ref)])
+ connection = supervisor.ensure_running()
+ process = supervisor._process
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ try:
+ asyncio.run(adapter.sync_sources([_binding(credential_ref, route_model_ids=("model-b",))]))
+ prefix = store.get_source("src_fixture123").prefix
+ assert supervisor._process is process
+ assert EngineClient(connection).list_model_ids() == {f"{prefix}/model-a", f"{prefix}/model-b"}
+ finally:
+ supervisor.stop()
+
+
+def test_source_sync_fails_without_restarting_a_stalled_engine(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(supervisor_module, "MODEL_HUB_CONFIG_RELOAD_TIMEOUT_SECONDS", 0.5)
+ supervisor, store = _fixture_supervisor(tmp_path)
+ credential_ref = store.store_api_key("upstream-secret", base_url="https://api.example.test/v1")
+ store.sync_sources([_binding(credential_ref)])
+ supervisor.ensure_running()
+ process = supervisor._process
+ original = store.list_sources()
+ process.send_signal(signal.SIGSTOP)
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ try:
+ # A stalled management path does not prove its streams dead.
+ with pytest.raises(EngineStateError):
+ asyncio.run(adapter.sync_sources([_binding(credential_ref, route_model_ids=("model-b",))]))
+ assert supervisor._process is process
+ assert process.poll() is None
+ finally:
+ process.send_signal(signal.SIGCONT)
+ supervisor.stop()
+ assert store.list_sources() == original
+
+
+def test_source_sync_of_an_exited_engine_is_applied_at_next_start(tmp_path: Path) -> None:
+ supervisor, store = _fixture_supervisor(tmp_path)
+ credential_ref = store.store_api_key("upstream-secret", base_url="https://api.example.test/v1")
+ store.sync_sources([_binding(credential_ref)])
+ supervisor.ensure_running()
+ exited = supervisor._process
+ exited.kill()
+ exited.wait(timeout=3)
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ try:
+ asyncio.run(adapter.sync_sources([_binding(credential_ref, route_model_ids=("model-b",))]))
+ prefix = store.get_source("src_fixture123").prefix
+ connection = supervisor.ensure_running()
+ assert supervisor._process is not exited
+ assert EngineClient(connection).list_model_ids() == {f"{prefix}/model-a", f"{prefix}/model-b"}
+ finally:
+ supervisor.stop()
+
+
+def test_credential_revocation_hot_reloads_a_live_engine_without_restart(tmp_path: Path) -> None:
+ supervisor, store = _fixture_supervisor(tmp_path)
+ old_ref = store.store_api_key("old-upstream-secret", base_url="https://api.example.test/v1")
+ new_ref = store.store_api_key("new-upstream-secret", base_url="https://api.example.test/v1")
+ store.sync_sources([_binding(old_ref)])
+ connection = supervisor.ensure_running()
+ process = supervisor._process
+ stale_instance = store.root / "instances" / "install-0"
+ stale_instance.mkdir(mode=0o700)
+ stale_config = stale_instance / "config.yaml"
+ stale_config.write_text("api-key: old-upstream-secret\n", encoding="utf-8")
+ stale_config.chmod(0o600)
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ try:
+ # Replacing a key rebinds first, then revokes the old credential.
+ asyncio.run(adapter.sync_sources([_binding(new_ref)]))
+ asyncio.run(adapter.revoke_api_key_credential(old_ref))
+ assert supervisor._process is process
+ live_config = (store.root / "instances" / "install-1" / "config.yaml").read_text()
+ assert "old-upstream-secret" not in live_config
+ assert "new-upstream-secret" in live_config
+ assert not stale_config.exists()
+ prefix = store.get_source("src_fixture123").prefix
+ assert EngineClient(connection).list_model_ids() == {f"{prefix}/model-a"}
+ finally:
+ supervisor.stop()
+
+
+def test_credential_revocation_stays_pending_without_restarting_a_stalled_engine(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(supervisor_module, "MODEL_HUB_CONFIG_RELOAD_TIMEOUT_SECONDS", 0.5)
+ supervisor, store = _fixture_supervisor(tmp_path)
+ old_ref = store.store_api_key("old-upstream-secret", base_url="https://api.example.test/v1")
+ new_ref = store.store_api_key("new-upstream-secret", base_url="https://api.example.test/v1")
+ store.sync_sources([_binding(old_ref)])
+ supervisor.ensure_running()
+ process = supervisor._process
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ try:
+ asyncio.run(adapter.sync_sources([_binding(new_ref)]))
+ process.send_signal(signal.SIGSTOP)
+ try:
+ # An unverifiable reload must not truncate live streams; the
+ # caller keeps the revocation pending and retries it.
+ with pytest.raises(EngineUnavailableError):
+ asyncio.run(adapter.revoke_api_key_credential(old_ref))
+ assert supervisor._process is process
+ assert process.poll() is None
+ finally:
+ process.send_signal(signal.SIGCONT)
+ assert store.credential_metadata_if_present(old_ref) is not None
+ asyncio.run(adapter.revoke_api_key_credential(old_ref))
+ assert supervisor._process is process
+ assert store.credential_metadata_if_present(old_ref) is None
+ finally:
+ supervisor.stop()
+
+
+def test_source_sync_of_a_stopped_engine_is_applied_at_next_start(tmp_path: Path) -> None:
+ supervisor, store = _fixture_supervisor(tmp_path)
+ credential_ref = store.store_api_key("upstream-secret", base_url="https://api.example.test/v1")
+ adapter = CLIProxyEngineAdapter(supervisor=supervisor, state_store=store)
+ asyncio.run(adapter.sync_sources([_binding(credential_ref, route_model_ids=("model-b",))]))
+ assert supervisor._process is None
+ try:
+ connection = supervisor.ensure_running()
+ prefix = store.get_source("src_fixture123").prefix
+ assert EngineClient(connection).list_model_ids() == {f"{prefix}/model-a", f"{prefix}/model-b"}
+ finally:
+ supervisor.stop()
+
+
@pytest.mark.parametrize(
("installed", "start_attempted", "running", "healthy", "expected"),
[
@@ -3249,14 +3482,14 @@ def test_supervisor_reaps_an_engine_whose_pid_was_never_recorded(tmp_path: Path)
def test_supervisor_refresh_and_reap_stop_an_engine_a_previous_service_left_running(tmp_path: Path) -> None:
record = tmp_path / "state" / "engine-process.json"
- for operation in ("restart_if_running", "with_engine_excluded"):
+ for operation in ("restart_if_running", "reload_config_if_running", "with_engine_excluded"):
orphan = _orphan_engine(tmp_path)
second, _store = _fixture_supervisor(tmp_path)
if operation == "with_engine_excluded":
assert second.with_engine_excluded(lambda client: client) is None
else:
- second.restart_if_running()
+ getattr(second, operation)()
_wait_for(lambda: orphan.poll() is not None)
assert second._process is None
@@ -3295,7 +3528,11 @@ def test_supervisor_refresh_refuses_while_a_previous_engine_is_unconfirmed(
monkeypatch.setattr(supervisor_module, "reap_marked_processes", lambda *_a, **_k: "unconfirmed")
second, _store = _fixture_supervisor(tmp_path)
- for call in (second.restart_if_running, lambda: second.with_engine_excluded(pytest.fail)):
+ for call in (
+ second.restart_if_running,
+ second.reload_config_if_running,
+ lambda: second.with_engine_excluded(pytest.fail),
+ ):
with pytest.raises(EngineUnavailableError) as raised:
call()
assert raised.value.reason == "previous_engine_alive"
@@ -3865,6 +4102,7 @@ def test_adapter_uses_origin_protocol_for_engine_translation(
request_protocol=None,
request_headers=None,
on_transport_done=None,
+ on_request_sent=None,
):
self.request_protocol = request_protocol
self.request_headers = request_headers
@@ -3917,20 +4155,16 @@ def test_adapter_uses_origin_protocol_for_engine_translation(
asyncio.run(run())
-def test_adapter_restores_source_projection_when_restart_fails(tmp_path: Path) -> None:
+def test_adapter_restores_source_projection_when_reload_fails(tmp_path: Path) -> None:
class Supervisor:
def __init__(self) -> None:
self.restore_calls = 0
- def client_if_running(self):
- return object()
-
- def restart_if_running(self) -> None:
- raise EngineUnavailableError("models.engine.health_failed")
-
- def ensure_running(self):
+ def reload_config_if_running(self, _previous=None) -> None:
+ if self.restore_calls == 0:
+ self.restore_calls += 1
+ raise EngineUnavailableError("models.engine.health_failed")
self.restore_calls += 1
- return object()
async def run() -> None:
store = EngineStateStore(tmp_path / "state")
@@ -3956,7 +4190,7 @@ def test_adapter_restores_source_projection_when_restart_fails(tmp_path: Path) -
restored = store.get_source("src_fixture123")
assert restored is not None
assert restored.credential_ref == old_ref
- assert supervisor.restore_calls == 1
+ assert supervisor.restore_calls == 2
asyncio.run(run())
@@ -3977,6 +4211,7 @@ def test_adapter_serializes_source_sync_with_new_invocations(tmp_path: Path) ->
request_protocol=None,
request_headers=None,
on_transport_done=None,
+ on_request_sent=None,
):
invoked_refs.append(source.credential_ref)
on_transport_done()
@@ -3992,7 +4227,7 @@ def test_adapter_serializes_source_sync_with_new_invocations(tmp_path: Path) ->
def with_engine_excluded(self, operation):
return operation(self._client)
- def restart_if_running(self) -> None:
+ def reload_config_if_running(self, _previous=None) -> None:
restart_started.set()
assert allow_restart.wait(timeout=2)
diff --git a/tests/test_model_hub_takeover_integration.py b/tests/test_model_hub_takeover_integration.py
index 43d8d133a..8d5964bb2 100644
--- a/tests/test_model_hub_takeover_integration.py
+++ b/tests/test_model_hub_takeover_integration.py
@@ -45,6 +45,9 @@ class FixtureSupervisor:
def restart_if_running(self):
pass
+ def reload_config_if_running(self, _previous=None):
+ pass
+
def status(self):
return {"status": {
"health": "ok", "installed_version": "fixture", "verified": True,
diff --git a/tests/test_model_hub_transport_leases.py b/tests/test_model_hub_transport_leases.py
index 60a49880f..c52d68a4c 100644
--- a/tests/test_model_hub_transport_leases.py
+++ b/tests/test_model_hub_transport_leases.py
@@ -22,12 +22,11 @@ LAST = b'data: {"choices":[{"delta":{},"finish_reason":"stop"}]}\n\ndata: [DONE]
class LeaseSupervisor:
def __init__(self, origin):
self.connection = EngineConnection(origin, "fixture-management", "fixture-token")
- self.restart_started = threading.Event()
- self.allow_restart = threading.Event()
- self.allow_restart.set()
- self.restart_count = 0
- self.restore_count = 0
- self.fail_restart = False
+ self.reload_started = threading.Event()
+ self.allow_reload = threading.Event()
+ self.allow_reload.set()
+ self.reload_count = 0
+ self.fail_reload = False
def client(self):
return EngineClient(self.connection)
@@ -36,16 +35,16 @@ class LeaseSupervisor:
return self.client()
def restart_if_running(self):
- self.restart_count += 1
- self.restart_started.set()
- assert self.allow_restart.wait(3)
- if self.fail_restart:
+ raise AssertionError("a source save must hot-reload, never restart")
+
+ def reload_config_if_running(self, _previous=None):
+ self.reload_count += 1
+ self.reload_started.set()
+ assert self.allow_reload.wait(3)
+ if self.fail_reload:
+ self.fail_reload = False
raise EngineUnavailableError("models.engine.health_failed")
- def ensure_running(self):
- self.restore_count += 1
- return self.connection
-
@asynccontextmanager
async def _transport(tmp_path):
@@ -97,20 +96,14 @@ async def _transport(tmp_path):
yield adapter, supervisor, binding, received, finish, requests
finally:
finish.set()
- supervisor.allow_restart.set()
+ supervisor.allow_reload.set()
await runner.cleanup()
-async def _barrier_waiting(adapter, sync):
- async with asyncio.timeout(2):
- while not adapter._routing_lock.locked():
- assert not sync.done()
- await asyncio.sleep(0)
- assert not sync.done()
-
-
@pytest.mark.parametrize("stream", [False, True])
-def test_sync_waits_for_buffered_and_streaming_transport_not_service_settlement(tmp_path, stream):
+def test_sync_applies_during_buffered_and_streaming_transport_without_waiting(tmp_path, stream):
+ """A save must not wait for, restart under, or truncate an in-flight call."""
+
async def run():
async with _transport(tmp_path) as (adapter, supervisor, binding, received, finish, requests):
invoke = asyncio.create_task(adapter.invoke(binding.source_id, "unknown", {}, stream, "opencode"))
@@ -119,24 +112,23 @@ def test_sync_waits_for_buffered_and_streaming_transport_not_service_settlement(
if handle is not None:
assert await anext(handle.stream) == FIRST
changed = replace(binding, route_model_ids=("unknown", "new"))
- sync = asyncio.create_task(adapter.sync_sources([changed]))
- await _barrier_waiting(adapter, sync)
+ await asyncio.wait_for(adapter.sync_sources([changed]), 2)
+ assert supervisor.reload_count == 1
+ assert adapter._active_transports == 1
+ # The fixture holds every upstream response until ``finish``; the
+ # new route is admitted while the first call is still open.
later = asyncio.create_task(adapter.invoke(binding.source_id, "new", {}, False, "opencode"))
- await asyncio.sleep(0)
- assert not later.done()
- assert supervisor.restart_count == 0
- assert len(requests) == 1
- # A service mutation lock can remain held during this drain. No
- # outcome()/settlement call is needed to release the transport.
+ async with asyncio.timeout(2):
+ while len(requests) < 2:
+ await asyncio.sleep(0.01)
finish.set()
if handle is not None:
assert b"".join([part async for part in handle.stream]) == LAST
else:
handle = await asyncio.wait_for(invoke, 2)
- await asyncio.wait_for(sync, 2)
- assert supervisor.restart_count == 1
- assert (await handle.outcome()).kind is RawOutcomeKind.SUCCESS
later_handle = await asyncio.wait_for(later, 2)
+ assert (await handle.outcome()).kind is RawOutcomeKind.SUCCESS
+ assert (await later_handle.outcome()).kind is RawOutcomeKind.SUCCESS
await handle.close_stream()
await later_handle.close_stream()
assert adapter._active_transports == 0
@@ -145,76 +137,49 @@ def test_sync_waits_for_buffered_and_streaming_transport_not_service_settlement(
asyncio.run(run())
[email protected]("finish_kind", ["unstarted_close", "started_close", "stream_cancel", "invoke_cancel"])
-def test_close_and_cancellation_release_transport_without_waiting_for_outcome(tmp_path, finish_kind):
+def test_sync_keeps_an_admitted_request_routable_until_the_engine_has_read_it(tmp_path, monkeypatch):
+ """A removal saved after admission must not strand a request not yet sent."""
+
async def run():
- async with _transport(tmp_path) as (adapter, supervisor, binding, received, _finish, _requests):
- streaming = finish_kind != "invoke_cancel"
- invoke = asyncio.create_task(adapter.invoke(binding.source_id, "unknown", {}, streaming, "opencode"))
+ async with _transport(tmp_path) as (adapter, supervisor, binding, received, finish, requests):
+ connect = asyncio.Event()
+ original = EngineClient.invoke
+
+ async def delayed_invoke(self, *args, **kwargs):
+ await connect.wait()
+ return await original(self, *args, **kwargs)
+
+ monkeypatch.setattr(EngineClient, "invoke", delayed_invoke)
+ invoke = asyncio.create_task(adapter.invoke(binding.source_id, "unknown", {}, False, "opencode"))
+ async with asyncio.timeout(2):
+ while adapter._requests_sent.is_set():
+ await asyncio.sleep(0.01)
+ sync = asyncio.create_task(adapter.sync_sources([replace(binding, route_model_ids=())]))
+ await asyncio.sleep(0.05)
+ assert supervisor.reload_count == 0
+ connect.set()
await asyncio.wait_for(received.wait(), 2)
- handle = await invoke if streaming else None
- pending_read = None
- if finish_kind in {"started_close", "stream_cancel"}:
- assert await anext(handle.stream) == FIRST
- if finish_kind == "stream_cancel":
- pending_read = asyncio.create_task(anext(handle.stream))
- await asyncio.sleep(0)
- sync = asyncio.create_task(adapter.sync_sources([replace(binding, route_model_ids=("new",))]))
- await _barrier_waiting(adapter, sync)
- assert supervisor.restart_count == 0
- if handle is None:
- invoke.cancel()
- with pytest.raises(asyncio.CancelledError):
- await invoke
- elif pending_read is not None:
- pending_read.cancel()
- with pytest.raises(asyncio.CancelledError):
- await pending_read
- await handle.close_stream()
- else:
- await handle.close_stream()
- await handle.close_stream()
await asyncio.wait_for(sync, 2)
- assert supervisor.restart_count == 1
- assert adapter._active_transports == 0
- assert adapter._transports_idle.is_set()
-
- asyncio.run(run())
-
-
-def test_cancelled_drain_does_not_cancel_active_request_or_leak_barrier(tmp_path):
- async def run():
- async with _transport(tmp_path) as (adapter, supervisor, binding, _received, finish, _requests):
- handle = await adapter.invoke(binding.source_id, "unknown", {}, True, "opencode")
- assert await anext(handle.stream) == FIRST
- before = adapter.state_store.list_sources()
- sync = asyncio.create_task(adapter.sync_sources([replace(binding, route_model_ids=("new",))]))
- await _barrier_waiting(adapter, sync)
- sync.cancel()
- with pytest.raises(asyncio.CancelledError):
- await sync
- assert not adapter._routing_lock.locked()
- assert adapter.state_store.list_sources() == before
- assert adapter._active_transports == 1
- assert supervisor.restart_count == 0
+ assert supervisor.reload_count == 1
+ assert requests[0]["model"].endswith("/unknown")
finish.set()
- assert b"".join([part async for part in handle.stream]) == LAST
+ handle = await asyncio.wait_for(invoke, 2)
assert (await handle.outcome()).kind is RawOutcomeKind.SUCCESS
- await adapter.sync_sources([binding])
- assert adapter._active_transports == 0
+ await handle.close_stream()
+ assert adapter._unsent_requests == 0 and adapter._active_transports == 0
asyncio.run(run())
[email protected]("fail_restart", [False, True])
-def test_cancelled_restart_retains_barrier_until_commit_or_rollback(tmp_path, fail_restart):
[email protected]("fail_reload", [False, True])
+def test_cancelled_reload_holds_admission_until_commit_or_rollback(tmp_path, fail_reload):
async def run():
async with _transport(tmp_path) as (adapter, supervisor, binding, _received, finish, requests):
original = adapter.state_store.list_sources()
- supervisor.allow_restart.clear()
- supervisor.fail_restart = fail_restart
+ supervisor.allow_reload.clear()
+ supervisor.fail_reload = fail_reload
sync = asyncio.create_task(adapter.sync_sources([replace(binding, route_model_ids=("changed",))]))
- assert await asyncio.to_thread(supervisor.restart_started.wait, 2)
+ assert await asyncio.to_thread(supervisor.reload_started.wait, 2)
sync.cancel()
await asyncio.sleep(0)
assert not sync.done()
@@ -222,16 +187,16 @@ def test_cancelled_restart_retains_barrier_until_commit_or_rollback(tmp_path, fa
later = asyncio.create_task(adapter.invoke(binding.source_id, "listed", {}, False, "opencode"))
await asyncio.sleep(0)
assert requests == []
- supervisor.allow_restart.set()
+ supervisor.allow_reload.set()
with pytest.raises(asyncio.CancelledError):
await asyncio.wait_for(sync, 2)
finish.set()
handle = await asyncio.wait_for(later, 2)
assert (await handle.outcome()).kind is RawOutcomeKind.SUCCESS
await handle.close_stream()
- if fail_restart:
+ if fail_reload:
assert adapter.state_store.list_sources() == original
- assert supervisor.restore_count == 1
+ assert supervisor.reload_count == 2
else:
assert adapter.state_store.list_sources()[0].route_model_ids == ("changed",)
assert adapter._active_transports == 0
@@ -240,33 +205,33 @@ def test_cancelled_restart_retains_barrier_until_commit_or_rollback(tmp_path, fa
asyncio.run(run())
-def test_two_concurrent_saves_are_serial_and_restart_failure_restores_projection(tmp_path):
+def test_two_concurrent_saves_are_serial_and_reload_failure_restores_projection(tmp_path):
async def run():
async with _transport(tmp_path) as (adapter, supervisor, binding, _received, _finish, _requests):
original = adapter.state_store.list_sources()
- supervisor.fail_restart = True
+ supervisor.fail_reload = True
with pytest.raises(EngineUnavailableError):
await adapter.sync_sources([replace(binding, route_model_ids=("bad",))])
assert adapter.state_store.list_sources() == original
- assert supervisor.restore_count == 1
- supervisor.fail_restart = False
- supervisor.restart_started.clear()
- supervisor.allow_restart.clear()
+ assert supervisor.reload_count == 2
+ supervisor.reload_started.clear()
+ supervisor.allow_reload.clear()
first = asyncio.create_task(adapter.sync_sources([replace(binding, route_model_ids=("first",))]))
- assert await asyncio.to_thread(supervisor.restart_started.wait, 2)
+ assert await asyncio.to_thread(supervisor.reload_started.wait, 2)
second = asyncio.create_task(adapter.sync_sources([replace(binding, route_model_ids=("second",))]))
await asyncio.sleep(0)
assert adapter.state_store.list_sources()[0].route_model_ids == ("first",)
- supervisor.allow_restart.set()
+ supervisor.allow_reload.set()
await asyncio.wait_for(asyncio.gather(first, second), 2)
assert adapter.state_store.list_sources()[0].route_model_ids == ("second",)
- assert supervisor.restart_count == 3
+ assert supervisor.reload_count == 4
assert not adapter._routing_lock.locked()
asyncio.run(run())
-def test_service_save_drain_releases_before_settlement_can_acquire_mutation_lock(tmp_path):
+def test_service_save_completes_while_a_stream_is_active(tmp_path):
+ """The reported hang: a model-list save during an active turn must settle."""
from tests.test_model_hub_resolution import _service, _source
from tests.test_model_hub_routing_modes import MODEL, _sparse_config
@@ -280,27 +245,52 @@ def test_service_save_drain_releases_before_settlement_can_acquire_mutation_lock
adapter.state_store.sync_sources(service._bindings(config))
handle = await adapter.invoke(source.id, MODEL, {}, True, "opencode")
assert await anext(handle.stream) == FIRST
- save = asyncio.create_task(
- service.set_agent_chain("claude", MODEL, {"hops": [{"source_id": source.id, "model_id": "new"}]})
+ result = await asyncio.wait_for(
+ service.set_agent_chain("claude", MODEL, {"hops": [{"source_id": source.id, "model_id": "new"}]}),
+ 2,
)
- await _barrier_waiting(adapter, save)
- assert service._mutation_lock.locked()
-
- async def settle():
- async with service._mutation_lock:
- return await handle.outcome()
-
- settlement = asyncio.create_task(settle())
- await asyncio.sleep(0)
- assert not settlement.done()
- assert supervisor.restart_count == 0
- finish.set()
- assert b"".join([part async for part in handle.stream]) == LAST
- result, outcome = await asyncio.wait_for(asyncio.gather(save, settlement), 2)
assert result["chain"]["current"] == {"source_id": source.id, "model_id": "new"}
- assert outcome.kind is RawOutcomeKind.SUCCESS
assert store.config.agents["claude"].routes[MODEL].hops[0].model_id == "new"
- assert supervisor.restart_count == 1
+ assert supervisor.reload_count == 1
+ assert not service._mutation_lock.locked()
+ finish.set()
+ assert b"".join([part async for part in handle.stream]) == LAST
+ async with service._mutation_lock:
+ assert (await handle.outcome()).kind is RawOutcomeKind.SUCCESS
+ assert adapter._active_transports == 0
+
+ asyncio.run(run())
+
+
[email protected]("finish_kind", ["unstarted_close", "started_close", "stream_cancel", "invoke_cancel"])
+def test_close_and_cancellation_release_transport_without_waiting_for_outcome(tmp_path, finish_kind):
+ # Leases still gate an engine binary upgrade restart.
+ async def run():
+ async with _transport(tmp_path) as (adapter, _supervisor, binding, received, _finish, _requests):
+ streaming = finish_kind != "invoke_cancel"
+ invoke = asyncio.create_task(adapter.invoke(binding.source_id, "unknown", {}, streaming, "opencode"))
+ await asyncio.wait_for(received.wait(), 2)
+ handle = await invoke if streaming else None
+ pending_read = None
+ if finish_kind in {"started_close", "stream_cancel"}:
+ assert await anext(handle.stream) == FIRST
+ if finish_kind == "stream_cancel":
+ pending_read = asyncio.create_task(anext(handle.stream))
+ await asyncio.sleep(0)
+ assert not adapter._transports_idle.is_set()
+ if handle is None:
+ invoke.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await invoke
+ elif pending_read is not None:
+ pending_read.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await pending_read
+ await handle.close_stream()
+ else:
+ await handle.close_stream()
+ await handle.close_stream()
+ await asyncio.wait_for(adapter._transports_idle.wait(), 2)
assert adapter._active_transports == 0
asyncio.run(run())
diff --git a/vibe/model_hub_runtime/adapter.py b/vibe/model_hub_runtime/adapter.py
index eb04827ef..7ccda9e07 100644
--- a/vibe/model_hub_runtime/adapter.py
+++ b/vibe/model_hub_runtime/adapter.py
@@ -1247,6 +1247,9 @@ class CLIProxyEngineAdapter:
self._active_transports = 0
self._transports_idle = asyncio.Event()
self._transports_idle.set()
+ self._unsent_requests = 0
+ self._requests_sent = asyncio.Event()
+ self._requests_sent.set()
self._installation_lock = asyncio.Lock()
self._install_task: asyncio.Task[None] | None = None
self._install_admission: asyncio.Future[EngineStatus] | None = None
@@ -1682,9 +1685,14 @@ class CLIProxyEngineAdapter:
async def sync_sources(self, bindings: Sequence[SourceBinding]) -> None:
async with self._routing_lock:
- await self._transports_idle.wait()
- # Cancellation must retain the barrier until the finite transaction
- # has either restarted or restored the prior projection.
+ # The routing lock orders the projection against new admissions.
+ # An admitted request keeps the projection it was admitted under
+ # until the engine has read it; established streams keep their
+ # connection, since CPA hot-reloads the config, so a save neither
+ # waits for them nor restarts. Cancellation must retain the lock
+ # until the finite transaction has applied or restored the prior
+ # projection.
+ await self._requests_sent.wait()
await run_owned_in_thread(self._sync_sources_transaction, tuple(bindings))
def _sync_sources_transaction(self, bindings: Sequence[SourceBinding]) -> None:
@@ -1692,21 +1700,36 @@ class CLIProxyEngineAdapter:
previous = self.state_store.list_sources()
except EngineStateError:
previous = []
- was_running = self.supervisor.client_if_running() is not None
self.state_store.sync_sources(bindings)
try:
- self.supervisor.restart_if_running()
+ self.supervisor.reload_config_if_running(previous)
except Exception:
+ applied = self.state_store.list_sources()
self.state_store.replace_sources(previous)
- if was_running:
- try:
- self.supervisor.ensure_running()
- except Exception as restore_error:
- raise EngineStateError(
- "source sync failed and the previous engine state could not be restored"
- ) from restore_error
+ try:
+ self.supervisor.reload_config_if_running(applied)
+ except Exception as restore_error:
+ raise EngineStateError(
+ "source sync failed and the previous engine state could not be restored"
+ ) from restore_error
raise
+ def _hold_unsent_request(self) -> Callable[[], None]:
+ self._unsent_requests += 1
+ self._requests_sent.clear()
+ released = False
+
+ def release() -> None:
+ nonlocal released
+ if released:
+ return
+ released = True
+ self._unsent_requests -= 1
+ if self._unsent_requests == 0:
+ self._requests_sent.set()
+
+ return release
+
def _acquire_transport(self) -> Callable[[], None]:
self._active_transports += 1
self._transports_idle.clear()
@@ -2627,10 +2650,12 @@ class CLIProxyEngineAdapter:
)
)
release = self._acquire_transport()
+ request_sent = self._hold_unsent_request()
try:
if on_admitted is not None:
on_admitted()
except BaseException:
+ request_sent()
release()
raise
try:
@@ -2646,10 +2671,13 @@ class CLIProxyEngineAdapter:
request_protocol=request_protocol,
request_headers=getattr(request, "headers", None),
on_transport_done=release,
+ on_request_sent=request_sent,
)
except BaseException:
release()
raise
+ finally:
+ request_sent()
async def _complete_oauth(self, flow: _OAuthFlow, client: EngineClient) -> None:
flow.grant_write_possible = True
diff --git a/vibe/model_hub_runtime/client.py b/vibe/model_hub_runtime/client.py
index d7e5bcbc9..494d1c9fa 100644
--- a/vibe/model_hub_runtime/client.py
+++ b/vibe/model_hub_runtime/client.py
@@ -355,6 +355,56 @@ class EngineClient:
timeout=timeout,
)
+ def put_config_yaml(self, text: str, *, timeout: float | None = None) -> dict[str, Any]:
+ """Replace the running engine config through its hot-reload endpoint."""
+ return self._request_json_projection(
+ "PUT",
+ "/v0/management/config.yaml",
+ _load_json_object,
+ data=text.encode(),
+ content_type="application/yaml",
+ headers={"X-Management-Key": self.connection.management_key},
+ timeout=timeout,
+ )
+
+ def list_model_ids(self, *, timeout: float | None = None) -> frozenset[str]:
+ payload = self._request_json(
+ "GET",
+ "/v1/models",
+ headers={"Authorization": f"Bearer {self.connection.gateway_token}"},
+ timeout=timeout,
+ )
+ entries = payload.get("data")
+ if not isinstance(entries, list):
+ raise EngineClientError("engine API returned an invalid payload", error_type="invalid_json")
+ return frozenset(
+ str(entry["id"]) for entry in entries if isinstance(entry, dict) and isinstance(entry.get("id"), str)
+ )
+
+ def list_model_names(self, *, timeout: float | None = None) -> dict[str, str]:
+ """Routed model ID to registered display name.
+
+ CPA's default OpenAI listing drops display names; its Grok-client
+ listing keeps the routed ID and names every provider's registration.
+ """
+ payload = self._request_json(
+ "GET",
+ "/v1/models",
+ headers={
+ "Authorization": f"Bearer {self.connection.gateway_token}",
+ "User-Agent": "grok-shell",
+ },
+ timeout=timeout,
+ )
+ entries = payload.get("data")
+ if not isinstance(entries, list):
+ raise EngineClientError("engine API returned an invalid payload", error_type="invalid_json")
+ return {
+ str(entry["id"]): str(entry.get("name") or "")
+ for entry in entries
+ if isinstance(entry, dict) and isinstance(entry.get("id"), str)
+ }
+
async def invoke(
self,
source: SourceRecord,
@@ -365,6 +415,7 @@ class EngineClient:
request_protocol: str | None = None,
request_headers: Mapping[str, str] | None = None,
on_transport_done: Callable[[], None] | None = None,
+ on_request_sent: Callable[[], None] | None = None,
) -> EngineInvokeHandle:
request_protocol = request_protocol or source.protocol
endpoint = _endpoint_for_protocol(request_protocol)
@@ -393,7 +444,21 @@ class EngineClient:
# Connecting to the local engine is bounded. Once connected, headers
# and response bytes can wait on upstream inference for any duration;
# completion, transport failure, or owner cancellation ends that wait.
- session = aiohttp.ClientSession(timeout=timeout, trust_env=False)
+ trace_configs: list[aiohttp.TraceConfig] = []
+ if on_request_sent is not None:
+ # CPA resolves the route as soon as it reads the body, while a
+ # config write only applies after its 150ms reload debounce, so a
+ # body written to the loopback socket precedes any route removal.
+ # CPA sends no earlier acknowledgment: response headers wait for
+ # the upstream's first chunk, which a save must never wait on.
+ trace = aiohttp.TraceConfig()
+
+ async def request_sent(*_args: Any) -> None:
+ on_request_sent()
+
+ trace.on_request_chunk_sent.append(request_sent)
+ trace_configs.append(trace)
+ session = aiohttp.ClientSession(timeout=timeout, trust_env=False, trace_configs=trace_configs)
response: aiohttp.ClientResponse | None = None
retry_after: str | None = None
response_received_at: datetime | None = None
@@ -686,16 +751,19 @@ class EngineClient:
*,
query: Mapping[str, str] | None = None,
payload: Mapping[str, Any] | None = None,
+ data: bytes | None = None,
+ content_type: str = "application/json",
headers: Mapping[str, str] | None = None,
timeout: float | None = None,
) -> _ProjectedJSON:
url = self._url(path, query=query)
request_timeout = timeout or self.timeout
deadline = time.monotonic() + request_timeout
- data = None if payload is None else json.dumps(payload, separators=(",", ":")).encode()
+ if payload is not None:
+ data = json.dumps(payload, separators=(",", ":")).encode()
request_headers = dict(headers or {})
if data is not None:
- request_headers["Content-Type"] = "application/json"
+ request_headers["Content-Type"] = content_type
request = urllib.request.Request(url, data=data, headers=request_headers, method=method)
try:
opener = urllib.request.build_opener(
diff --git a/vibe/model_hub_runtime/config.py b/vibe/model_hub_runtime/config.py
index aaf1582ff..1ba2c0468 100644
--- a/vibe/model_hub_runtime/config.py
+++ b/vibe/model_hub_runtime/config.py
@@ -21,7 +21,39 @@ def write_engine_config(
runtime_secrets: RuntimeSecrets,
sources: Iterable[SourceRecord],
state_store: EngineStateStore,
+ generation: str = "0",
) -> None:
+ _secure_write_text(
+ path,
+ render_engine_config(
+ path,
+ host=host,
+ port=port,
+ auth_dir=auth_dir,
+ runtime_secrets=runtime_secrets,
+ sources=sources,
+ state_store=state_store,
+ generation=generation,
+ ),
+ )
+
+
+def render_engine_config(
+ path: Path,
+ *,
+ host: str,
+ port: int,
+ auth_dir: Path,
+ runtime_secrets: RuntimeSecrets,
+ sources: Iterable[SourceRecord],
+ state_store: EngineStateStore,
+ generation: str = "0",
+) -> str:
+ """Render the engine YAML that ``path`` holds, for a start or a hot reload.
+
+ ``generation`` marks every configured model's display name, so a reload is
+ observable as applied even when it leaves the routed model IDs unchanged.
+ """
if host != "127.0.0.1":
raise EngineStateError("model hub engine must bind to 127.0.0.1")
payload: dict[str, Any] = {
@@ -63,11 +95,16 @@ def write_engine_config(
"ws-auth": True,
}
for source in sources:
- _append_source(payload, source, state_store)
- _secure_write_yaml(path, payload)
+ _append_source(payload, source, state_store, generation)
+ return yaml.safe_dump(payload, sort_keys=False, default_flow_style=False)
-def _append_source(payload: dict[str, Any], source: SourceRecord, store: EngineStateStore) -> None:
+def _append_source(
+ payload: dict[str, Any],
+ source: SourceRecord,
+ store: EngineStateStore,
+ generation: str = "0",
+) -> None:
credential = store.credential_metadata(source.credential_ref)
if credential["kind"] == "oauth":
# OAuth credentials are engine auth files, not YAML credential values.
@@ -82,7 +119,11 @@ def _append_source(payload: dict[str, Any], source: SourceRecord, store: EngineS
reasoning_by_model = dict(source.model_reasoning_efforts)
models = []
for model in dict.fromkeys((*source.model_ids, *source.route_model_ids)):
- entry: dict[str, Any] = {"name": model, "alias": model}
+ entry: dict[str, Any] = {
+ "name": model,
+ "alias": model,
+ "display-name": reload_display_name(model, generation),
+ }
reasoning_efforts = reasoning_by_model.get(model, ())
if reasoning_efforts:
# CLIProxyAPI's measured model-registration shape is strongest-first.
@@ -145,9 +186,29 @@ def _append_source(payload: dict[str, Any], source: SourceRecord, store: EngineS
raise EngineStateError("unsupported source protocol")
-def _secure_write_yaml(path: Path, payload: dict[str, Any]) -> None:
+def expected_model_names(
+ sources: Iterable[SourceRecord],
+ store: EngineStateStore,
+ generation: str,
+) -> dict[str, str]:
+ """Routed ID to display name the rendered YAML registers; OAuth models come from auth files."""
+ expected: dict[str, str] = {}
+ for source in sources:
+ metadata = store.credential_metadata_if_present(source.credential_ref)
+ if metadata is None or metadata.get("kind") != "api_key":
+ continue
+ for model in (*source.model_ids, *source.route_model_ids):
+ expected[f"{source.prefix}/{model}"] = reload_display_name(model, generation)
+ return expected
+
+
+def reload_display_name(model: str, generation: str) -> str:
+ return f"{model} #{generation}"
+
+
+def _secure_write_text(path: Path, text: str) -> None:
# The file is 0600 by ``write_atomic``; the directory is this function's own
# concern, because the config it holds names an upstream API key.
path.parent.mkdir(parents=True, exist_ok=True)
path.parent.chmod(0o700)
- write_atomic(path, yaml.safe_dump(payload, sort_keys=False, default_flow_style=False))
+ write_atomic(path, text)
diff --git a/vibe/model_hub_runtime/state.py b/vibe/model_hub_runtime/state.py
index 148705294..dde02df3e 100644
--- a/vibe/model_hub_runtime/state.py
+++ b/vibe/model_hub_runtime/state.py
@@ -919,8 +919,12 @@ class EngineStateStore:
if path.parent in present_directories:
self._remove_private_file_if_present(path)
- def clear_runtime_configs(self) -> None:
- """Remove persisted engine configs after any credential is revoked."""
+ def clear_runtime_configs(self, *, keep: Path | None = None) -> None:
+ """Remove persisted engine configs after any credential is revoked.
+
+ ``keep`` names the live instance directory whose config was just
+ re-rendered without the revoked credential.
+ """
with self._lock:
self._ensure_private_dir(self.root)
instances_dir = self.root / "instances"
@@ -931,6 +935,8 @@ class EngineStateStore:
mode = instance_dir.lstat().st_mode
if not stat.S_ISDIR(mode):
raise EngineStateError("engine instance directory is unsafe")
+ if keep is not None and instance_dir == keep:
+ continue
config_path = instance_dir / "config.yaml"
try:
config_mode = config_path.lstat().st_mode
diff --git a/vibe/model_hub_runtime/supervisor.py b/vibe/model_hub_runtime/supervisor.py
index 85bdcb865..54b456d89 100644
--- a/vibe/model_hub_runtime/supervisor.py
+++ b/vibe/model_hub_runtime/supervisor.py
@@ -30,11 +30,11 @@ from core.process_isolation import (
serialize_process_identity,
signal_process_tree,
)
-from vibe.model_hub_runtime.client import EngineClient, EngineConnection
-from vibe.model_hub_runtime.config import write_engine_config
+from vibe.model_hub_runtime.client import EngineClient, EngineClientError, EngineConnection
+from vibe.model_hub_runtime.config import expected_model_names, render_engine_config, write_engine_config
from vibe.model_hub_runtime.environment import engine_subprocess_environment
from vibe.model_hub_runtime.installer import EngineRuntimeManager
-from vibe.model_hub_runtime.state import EngineStateStore
+from vibe.model_hub_runtime.state import EngineStateStore, SourceRecord
logger = logging.getLogger(__name__)
@@ -42,6 +42,8 @@ _T = TypeVar("_T")
MODEL_HUB_STARTUP_TIMEOUT_SECONDS = 30.0
+MODEL_HUB_CONFIG_RELOAD_TIMEOUT_SECONDS = 5.0
+_STARTUP_POLL_INTERVAL_SECONDS = 0.05
# Durable identity of the engine this state root last spawned. A service that
# dies without running ``atexit`` (SIGKILL, crash, forced restart) leaves its
# isolated engine group running; the next start reaps it by identity. The marker
@@ -53,7 +55,6 @@ _ENGINE_PROCESS_RECORD = "engine-process.json"
class _EngineRecord:
worker_fingerprint: str
identity: PersistedProcessIdentity | None = None
-_STARTUP_POLL_INTERVAL_SECONDS = 0.05
class EngineUnavailableError(RuntimeError):
@@ -89,6 +90,7 @@ class EngineSupervisor:
self._last_check: str | None = None
self._start_attempted = False
self._health_failure_signature: tuple[str, str, int | None] | None = None
+ self._config_generation = 0
def ensure_running(self) -> EngineConnection:
with self._lock:
@@ -139,6 +141,105 @@ class EngineSupervisor:
if not self._reap_recorded_engines_locked():
raise EngineUnavailableError("models.engine.stop_unconfirmed", reason="previous_engine_alive")
+ def reload_config_if_running(self, previous_sources: list[SourceRecord] | None = None) -> None:
+ """Hot-apply the current source projection to a live engine.
+
+ CPA reloads its config file on change, so a Source or model-list save
+ never restarts the process and never waits for in-flight streams. The
+ management PUT is used because CPA does not observe an atomic rename of
+ the watched file. A stopped engine renders the projection at next start.
+ """
+ with self._lock:
+ if not self._is_running_locked() or self._connection is None:
+ # A previous service may have left an engine serving the old
+ # projection: a save must not succeed beside it.
+ self._require_no_untracked_engine_locked()
+ return
+ self._reload_locked(previous_sources)
+
+ def _reload_locked(self, previous_sources: list[SourceRecord] | None) -> None:
+ assert self._connection is not None
+ connection = self._connection
+ sources = self.state_store.list_sources()
+ generation = self._next_generation_locked()
+ text = self._render_config_locked(int(connection.base_url.rsplit(":", 1)[1]), sources, generation)
+ client = EngineClient(connection, timeout=MODEL_HUB_CONFIG_RELOAD_TIMEOUT_SECONDS)
+ try:
+ client.put_config_yaml(text)
+ except EngineClientError as exc:
+ if exc.status_code is not None:
+ raise EngineUnavailableError("models.engine.health_failed") from exc
+ if not self._is_running_locked():
+ # An exited engine holds no stream; a start renders the
+ # current projection.
+ self._stop_locked()
+ self._start_locked()
+ return
+ # A lost acknowledgment may still have applied, and a stalled
+ # management path does not prove live streams dead. The
+ # generation check decides; the process is never killed here.
+ expected = expected_model_names(sources, self.state_store, generation)
+ stale = frozenset(expected_model_names(previous_sources or (), self.state_store, generation)) - set(expected)
+ self._await_models_locked(client, expected, stale)
+ self._last_check = _utc_now()
+
+ def _await_models_locked(
+ self,
+ client: EngineClient,
+ expected: dict[str, str],
+ stale: frozenset[str],
+ ) -> None:
+ # CPA applies the written file asynchronously (debounced watcher). Its
+ # model listing carrying this generation's display names is the first
+ # observable point at which the new projection admits, including a
+ # save that leaves every routed ID unchanged. A save that removes
+ # models waits for them to leave. With no API-key model before or
+ # after, the projection routes nothing either way, so there is nothing
+ # to confirm; CPA validated these exact bytes before acknowledging the
+ # PUT, and its watcher reloads the same file.
+ deadline = time.monotonic() + MODEL_HUB_CONFIG_RELOAD_TIMEOUT_SECONDS
+ while True:
+ try:
+ listed = client.list_model_names(timeout=1.0)
+ except EngineClientError:
+ listed = None
+ if (
+ listed is not None
+ and all(listed.get(model) == name for model, name in expected.items())
+ and not stale & listed.keys()
+ ):
+ return
+ if time.monotonic() >= deadline:
+ raise EngineUnavailableError("models.engine.health_failed")
+ time.sleep(_STARTUP_POLL_INTERVAL_SECONDS)
+
+ def _next_generation_locked(self) -> str:
+ self._config_generation += 1
+ return str(self._config_generation)
+
+ def _render_config_locked(self, port: int, sources: list[SourceRecord], generation: str) -> str:
+ _managed, _binary, instance_dir, runtime_secrets = self._prepare_instance_locked()
+ return render_engine_config(
+ instance_dir / "config.yaml",
+ host="127.0.0.1",
+ port=port,
+ auth_dir=self.state_store.auth_dir,
+ runtime_secrets=runtime_secrets,
+ sources=sources,
+ state_store=self.state_store,
+ generation=generation,
+ )
+
+ def _prepare_instance_locked(self):
+ managed = self.installer.status()
+ binary = self.installer.resolve_engine_path()
+ if binary is None:
+ reason = str(managed.get("reason") or "engine_not_installed")
+ raise EngineUnavailableError("models.engine.install_failed", reason=reason)
+ install_id = Path(str(managed.get("install_dir") or binary.parent)).name
+ instance_dir, runtime_secrets = self.state_store.prepare_instance(install_id, rotate=False)
+ return managed, binary, instance_dir, runtime_secrets
+
def note_installation_settled(self) -> None:
"""Expose a newly verified binary as lazy-started, not previously down."""
with self._lock:
@@ -146,19 +247,25 @@ class EngineSupervisor:
self._start_attempted = False
def invalidate_configs(self) -> None:
- """Remove secret-bearing configs and recreate one only for a live engine."""
+ """Remove secret-bearing configs without restarting a live engine.
+
+ A revoked credential is already unbound, so a live engine drops it by
+ hot-reloading the current projection, which also rewrites its config
+ without the secret; streams keep running. An unverifiable reload
+ raises instead of restarting, so the caller keeps the revocation
+ pending and retries it rather than truncating active streams.
+ """
with self._lock:
- should_restart = self._is_running_locked() and self._healthy_locked()
- # Always stop: with no local handle this still reaps a recorded engine a
- # previous service left running with the credential being revoked, and
+ if self._is_running_locked() and self._connection is not None:
+ _managed, _binary, instance_dir, _secrets = self._prepare_instance_locked()
+ self.state_store.clear_runtime_configs(keep=instance_dir)
+ self._reload_locked(None)
+ return
+ # With no local handle this still reaps a recorded engine a previous
+ # service left running with the credential being revoked, and
# revocation must not proceed while that engine may still hold it.
self._require_stopped_locked()
self.state_store.clear_runtime_configs()
- if should_restart:
- try:
- self._start_locked()
- except EngineUnavailableError:
- logger.warning("Model Hub engine remains stopped after credential revocation")
def status(self) -> dict[str, Any]:
with self._lock:
@@ -229,16 +336,7 @@ class EngineSupervisor:
# start retries the reap, so this clears once the survivor is confirmed gone.
if not self._reap_recorded_engines_locked():
raise EngineUnavailableError("models.engine.start_failed", reason="previous_engine_alive")
- managed = self.installer.status()
- binary = self.installer.resolve_engine_path()
- if binary is None:
- reason = str(managed.get("reason") or "engine_not_installed")
- raise EngineUnavailableError("models.engine.install_failed", reason=reason)
- install_id = Path(str(managed.get("install_dir") or binary.parent)).name
- instance_dir, runtime_secrets = self.state_store.prepare_instance(
- install_id,
- rotate=False,
- )
+ managed, binary, instance_dir, runtime_secrets = self._prepare_instance_locked()
port = self._port_allocator()
config_path = instance_dir / "config.yaml"
write_engine_config(
@@ -249,6 +347,7 @@ class EngineSupervisor:
runtime_secrets=runtime_secrets,
sources=self.state_store.list_sources(),
state_store=self.state_store,
+ generation=self._next_generation_locked(),
)
connection = EngineConnection(
base_url=f"http://127.0.0.1:{port}",