Skip to content

The agent that grows with you

fix(config-remote): review round 3 — intent-preserving CAS retry, private in-memory migration, poller roster snapshots, dotenv+managed bootstrap before the first read, deployment carried into routed children, remote-safe profile create/rename, backend-versioned roster cache, keyed TUI edits, RPC answers on backend refusal, stage2 uses the runtime's selection, refuse values a pending migration would rewrite

Ben Barclay6dd799b30a72
diff --git a/docker/stage2-hook.sh b/docker/stage2-hook.sh
index 78a1cad77b0..7b80140a8d7 100755
--- a/docker/stage2-hook.sh
+++ b/docker/stage2-hook.sh
@@ -640,16 +640,22 @@ fi
# config-schema migrations that `hermes update` runs for non-Docker installs,
# after first-boot seeding and before supervised gateway services start.
# Set HERMES_SKIP_CONFIG_MIGRATION=1 for controlled/manual migrations.
-# Remote config mode (HERMES_CONFIG_BACKEND=remote, in the container env or $HERMES_HOME/.env) has
-# no local config to migrate: the agent migrates the fetched document in memory and never writes
-# it back, so a leftover config.yaml must not be migrated or treated as live.
-_remote_config_mode=0
-if [ "${HERMES_CONFIG_BACKEND:-}" = "remote" ] \
- || grep -Eqs '^[[:space:]]*(export[[:space:]]+)?HERMES_CONFIG_BACKEND=["'"'"']?remote["'"'"']?[[:space:]]*$' "$HERMES_HOME/.env"; then
- _remote_config_mode=1
-fi
-if [ "$_remote_config_mode" = "1" ]; then
+# Remote config mode (HERMES_CONFIG_BACKEND=remote) has no local config to migrate: the agent
+# migrates the fetched document in memory and never writes it back, so a leftover config.yaml must
+# not be migrated or treated as live. The backend is decided by the agent's own selection (same
+# dotenv parser and precedence as at boot: container env, $HERMES_HOME/.env, managed .env), never
+# by a second grammar here. If it cannot be decided, nothing is migrated: the agent still decides
+# at boot, and a file-mode migration simply waits for the next boot.
+_config_backend=$(s6-setuidgid hermes "$INSTALL_DIR/.venv/bin/python" -c '
+import sys
+from pathlib import Path
+from hermes_cli.env_loader import selected_config_backend_name
+print(selected_config_backend_name(Path(sys.argv[1])))
+' "$HERMES_HOME" 2>/dev/null) || _config_backend=""
+if [ "$_config_backend" = "remote" ]; then
echo "[stage2] HERMES_CONFIG_BACKEND=remote: skipping docker_config_migrate.py"
+elif [ -z "$_config_backend" ]; then
+ echo "[stage2] Warning: could not determine the config backend; skipping docker_config_migrate.py"
elif [ -f "$HERMES_HOME/config.yaml" ]; then
s6-setuidgid hermes "$INSTALL_DIR/.venv/bin/python" "$INSTALL_DIR/scripts/docker_config_migrate.py" \
|| echo "[stage2] Warning: docker_config_migrate.py failed; continuing"
diff --git a/hermes_cli/config_backend.py b/hermes_cli/config_backend.py
index 7a2a80e9eb5..8bd660065d1 100644
--- a/hermes_cli/config_backend.py
+++ b/hermes_cli/config_backend.py
@@ -19,6 +19,7 @@ whatever user layer the backend returns.
from __future__ import annotations
import os
+import threading
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Mapping, Optional, Protocol, Union, runtime_checkable
@@ -92,6 +93,10 @@ class ConfigBackend(Protocol):
"""Env names no external secret source may supply (the plane credential, D32)."""
...
+ def deployment_env_names(self) -> frozenset:
+ """Non-secret env names that select and address this backend (see :func:`deployment_env_names`)."""
+ ...
+
def apply_in_memory(self, home: Path, doc: dict) -> None:
"""Replace the user layer in memory only (in-memory migration, D12). Only called for a
backend without file tooling; nothing is sent or written."""
@@ -211,6 +216,9 @@ class FileBackend:
def protected_env_names(self) -> frozenset:
return frozenset()
+ def deployment_env_names(self) -> frozenset:
+ return frozenset()
+
def apply_in_memory(self, home: Path, doc: dict) -> None:
raise NotImplementedError("the file backend persists migrations to config.yaml")
@@ -219,6 +227,33 @@ class FileBackend:
_FILE_BACKEND = FileBackend()
+_BOOTSTRAP_LOCK = threading.Lock()
+_BOOTSTRAPPED = False
+
+
+def _bootstrap_names() -> frozenset:
+ from plugins.config_backends.remote.backend import POLL_ENV
+ from plugins.config_backends.remote.credentials import PLANE_CREDENTIAL_ENV_NAMES
+ return PLANE_CREDENTIAL_ENV_NAMES | {POLL_ENV}
+
+
+def _ensure_bootstrap_env() -> None:
+ """Once per process, before the first selection: the deployment that selects, addresses and
+ authenticates the backend comes from the process env, the launch home's ``.env`` and the
+ managed ``.env`` (design §4.4 steps 1-2, D26/D32) — also when the first config read comes before
+ ``load_hermes_dotenv`` (``hermes_cli.config`` reads config at import time)."""
+ global _BOOTSTRAPPED
+ if _BOOTSTRAPPED:
+ return
+ with _BOOTSTRAP_LOCK:
+ if _BOOTSTRAPPED:
+ return
+ _BOOTSTRAPPED = True
+ try:
+ from hermes_cli.env_loader import apply_config_bootstrap_env
+ apply_config_bootstrap_env(_bootstrap_names) # names resolved only for a remote deployment
+ except ImportError: # no usable python-dotenv (a bare installer interpreter): no .env loads either
+ return
def get_config_backend() -> ConfigBackend:
@@ -227,6 +262,7 @@ def get_config_backend() -> ConfigBackend:
Read from the environment on every call, before any config read, so no config value can
select it (D11). Raises :class:`ConfigBackendUnavailable` for a backend this build lacks.
"""
+ _ensure_bootstrap_env()
kind = os.environ.get(BACKEND_ENV, "").strip().lower() or "file"
if kind == "file":
return _FILE_BACKEND
@@ -336,3 +372,28 @@ def write_config_key(config_path: PathLike, key_path: str, value: Any) -> None:
"""Set one dotted key (``None`` removes it) — ``atomic_roundtrip_yaml_update`` semantics."""
backend, home = _route(config_path)
backend.write_changes(home, Changes(unset=(key_path,)) if value is None else Changes(set={key_path: value}))
+
+
+
+def deployment_env_names() -> frozenset:
+ """The selected backend's deployment env names (which backend, which plane, which instance).
+
+ The deployment is a property of the instance, not of a profile, but it usually arrives in the
+ launch profile's ``.env``, which a child acting for ANOTHER profile drops as launch residue
+ (``strip_launch_profile_env``). Such a child re-reads only its own profile's ``.env`` and would
+ silently select the file backend, so these names are never residue. Credentials are not part
+ of it: the child resolves its own, and fails closed without them."""
+ return get_config_backend().deployment_env_names()
+
+
+def supports_file_tooling() -> bool:
+ """Gate for tools that copy, edit or back up the config FILE itself (§4.3, §4.7)."""
+ return get_config_backend().supports_file_tooling()
+
+
+def require_file_tooling(what: str) -> None:
+ """Refuse *what* (a file-copy tool) with a clear message when the backend has no config file."""
+ if not supports_file_tooling():
+ raise ConfigBackendUnavailable(
+ f"{what} copies the local config.yaml, and the {get_config_backend().name!r} config "
+ "backend has none. It is not available with this backend.")
diff --git a/hermes_cli/env_loader.py b/hermes_cli/env_loader.py
index 339cfba5f84..984f56970cf 100644
--- a/hermes_cli/env_loader.py
+++ b/hermes_cli/env_loader.py
@@ -439,6 +439,68 @@ def _load_dotenv_with_fallback(
_sanitize_loaded_credentials() # httpx encodes headers as ASCII
+def _bootstrap_env(home: Path | None = None) -> dict:
+ """The process env as ``load_hermes_dotenv`` leaves it for the names the home's ``.env`` and the
+ managed ``.env`` define: both override, managed last, ``${VAR}`` resolved, the loader's parser.
+ Read-only: nothing is published and no file is rewritten (sanitizing imports hermes_cli.config,
+ whose import-time config read must not run before the backend is decided)."""
+ from dotenv.variables import parse_variables
+
+ home = Path(home) if home is not None else _process_hermes_home()
+ env = dict(os.environ)
+ layers = [home / ".env"]
+ try:
+ from hermes_cli import managed_scope
+
+ managed_dir = managed_scope.get_managed_dir()
+ except Exception: # noqa: BLE001 — same fail-open as _apply_managed_env
+ managed_dir = None
+ if managed_dir is not None:
+ layers.append(managed_dir / ".env")
+ for path in layers:
+ try:
+ assignments = _parse_dotenv_file(path)
+ except OSError:
+ continue
+ resolved: dict[str, str | None] = {}
+ for name, value in assignments:
+ if value is not None:
+ value = "".join(atom.resolve({**env, **resolved}) for atom in parse_variables(value))
+ resolved[name] = value
+ env.update({k: v for k, v in resolved.items() if v is not None})
+ return env
+
+
+def selected_config_backend_name(home: Path | None = None) -> str:
+ """The config backend ``load_hermes_dotenv`` selects for *home*, decided without loading or
+ booting anything. For a caller that must decide before Hermes starts (Docker stage2 skips the
+ config-file migration in remote mode, D12)."""
+ from hermes_cli.config_backend import BACKEND_ENV
+
+ return (_bootstrap_env(home).get(BACKEND_ENV) or "").strip().lower() or "file"
+
+
+def apply_config_bootstrap_env(remote_names, home: Path | None = None) -> None:
+ """Publish ``remote_names()`` (the remote backend's deployment and plane credential, D26/D32) from the
+ launch home's ``.env`` and the managed ``.env``, with the values ``load_hermes_dotenv`` gives
+ them. Called before the first config read of a process, which can come before
+ ``load_hermes_dotenv`` (``hermes_cli.config`` reads config at import time): the backend is
+ selected and authenticated from the same deployment either way. A file deployment gets only its
+ selector (the file backend needs nothing else before boot); values the process env holds and
+ no file defines are left alone; ``load_hermes_dotenv`` later publishes the same values."""
+ from hermes_cli.config_backend import BACKEND_ENV
+
+ env = _bootstrap_env(home)
+ if (env.get(BACKEND_ENV) or "").strip().lower() == "remote":
+ names = remote_names()
+ else:
+ names = {BACKEND_ENV} # the selector itself, so a file deployment is not read as remote meanwhile
+ for name in names:
+ value = env.get(name)
+ if value is not None and os.environ.get(name) != value:
+ os.environ[name] = value
+
+
def _sanitize_env_file_if_needed(path: Path) -> None:
"""Pre-sanitize a .env file before python-dotenv reads it. Sniffs a leading BOM *before* any text
decode: UTF-16 (Notepad "Unicode") is rewritten as clean UTF-8; UTF-32 is refused (left untouched) so
@@ -597,6 +659,10 @@ def load_hermes_dotenv(
# remote backend fetches this home's config here, failing closed; the file backend does nothing.
from hermes_cli.config_backend import get_config_backend
+ # The administrator-managed .env is part of the deployment the backend boots from (it may be
+ # where the selector, plane URL or instance id live), so it applies before the boot as well as
+ # last below, where it keeps its precedence over secret-source results.
+ _apply_managed_env(load_pass=load_pass)
get_config_backend().boot(home_path)
# External sources are skipped for the updater (dotenv + managed env still load): ``update`` must not
diff --git a/hermes_cli/profiles.py b/hermes_cli/profiles.py
index f0bd12c6c91..8c51fa228b7 100644
--- a/hermes_cli/profiles.py
+++ b/hermes_cli/profiles.py
@@ -587,6 +587,9 @@ def remove_wrapper_script(name: str) -> bool:
def _migrate_profile_config_if_outdated(profile_dir: Path) -> None:
"""Migrate a copied config.yaml to the current schema (non-interactive, scoped to the new
profile); otherwise the first desktop/doctor view shows a scary ``v0 -> latest`` warning."""
+ from hermes_cli.config_backend import config_exists, supports_file_tooling
+ if not supports_file_tooling():
+ return # no local config.yaml to migrate: a remote backend migrates in memory on read (D12)
if not config_exists(profile_dir / "config.yaml"):
return
# Creation must not fail over an unmigratable old config; `hermes doctor --fix` surfaces
@@ -2279,6 +2282,18 @@ def _record_profile_rename(new_dir: Path, old_canon: str) -> None:
logger.debug("profile rename: could not record previous name %r in %s: %s", old_canon, new_dir, exc)
+def _refuse_rename_without_file_tooling(old_canon: str) -> None:
+ """A remote config backend keys each profile's settings by its canonical name, and the plane
+ has no rename: renaming the local home would leave the settings under the old name and read
+ another (possibly populated) level under the new one. Refused before any side effect."""
+ from hermes_cli.config_backend import get_config_backend
+ backend = get_config_backend()
+ if not backend.supports_file_tooling():
+ raise ValueError(
+ f"Cannot rename profile '{old_canon}': its config lives in the {backend.name!r} config "
+ "backend under that name, and renaming it there is not supported. Nothing was changed.")
+
+
def rename_profile(old_name: str, new_name: str) -> Path:
"""Rename a profile: directory, wrapper script, service, active_profile. The default
profile's home IS the installation root, so "renaming" it sets a presentation-only
@@ -2293,6 +2308,7 @@ def rename_profile(old_name: str, new_name: str) -> Path:
new_canon = _canon_valid(new_name)
if new_canon == "default":
raise ValueError("Cannot rename to 'default' — it is reserved.")
+ _refuse_rename_without_file_tooling(old_canon)
old_dir = get_profile_dir(old_canon)
new_dir = get_profile_dir(new_canon)
if not old_dir.is_dir():
diff --git a/plugins/config_backends/remote/backend.py b/plugins/config_backends/remote/backend.py
index 63b1817d9ae..52bb214d769 100644
--- a/plugins/config_backends/remote/backend.py
+++ b/plugins/config_backends/remote/backend.py
@@ -7,7 +7,9 @@ level with CAS (D10). Nothing is read from or written to a local ``config.yaml``
"""
from __future__ import annotations
+import contextvars
import copy
+import itertools
import logging
import os
import random
@@ -23,7 +25,7 @@ from hermes_cli.config_backend import (
from . import client
from .credentials import PLANE_CREDENTIAL_ENV_NAMES, PlaneCredentialError
-from .diff import diff, encode_changes, strip_locked, write_check
+from .diff import _get_path, diff, encode_changes, json_equal, strip_locked, write_check
from .paths import Path as KeyPath
from .paths import PathError, decode, encode, from_dotted
from .values import secret_literal_path, to_wire
@@ -70,13 +72,13 @@ class _ProfileState:
profile_version: int = 0
# The profile level's stored writerConfigVersion: None = stored null, _UNKNOWN = not reported.
profile_writer: Any = None
+ other_writers: List[int] = field(default_factory=list) # the other levels' int stamps (R6)
locks: List[Tuple[KeyPath, str]] = field(default_factory=list)
provenance: Dict[str, str] = field(default_factory=dict)
changed_ns: int = 0
gen: int = 0
installed: int = 0 # bumped by every _install; a GET started before a newer install is stale
base: Optional[Dict[str, Any]] = None # the migrated doc writes diff against (None = server_config)
- migrating: Optional[int] = None # the `installed` a running in-memory migration is for
postprocessed: bool = False
in_postprocess: bool = False
fetched_at: float = 0.0
@@ -87,6 +89,30 @@ class _ProfileState:
_UNKNOWN = object()
+
+@dataclass
+class _Private:
+ """An in-memory migration's working copy of one profile's document (D12).
+
+ The migration steps read and persist through ``hermes_cli.config`` like any reader and writer;
+ while one runs, THAT thread sees this copy instead of the published one (``_PRIVATE``), and
+ everyone else keeps seeing the published doc with its matching diff base. The migrated doc and
+ its base are published together, in one step, when the migration ends: a concurrent writer
+ never diffs a half-migrated document against the unmigrated base, or the reverse."""
+ st: "_ProfileState"
+ doc: Dict[str, Any]
+ serial: int
+ gen: int = 0
+
+
+_PRIVATE: "contextvars.ContextVar[Optional[_Private]]" = contextvars.ContextVar("remote_config_private", default=None)
+_PRIVATE_SERIAL = itertools.count(1)
+
+
+def _private_for(st: "_ProfileState") -> Optional[_Private]:
+ m = _PRIVATE.get()
+ return m if m is not None and m.st is st else None
+
# Plane refusals caused by the submitted value (contract §2.5, §11.1): surfaced to the caller as
# ConfigValueError with the plane's code. config_request_invalid is deliberately absent: a request
# the plane cannot parse is this client's bug, not the user's value, so it stays opaque.
@@ -144,7 +170,10 @@ class RemoteBackend:
key = self._key(home)
st = self._states.get(key)
if st is not None:
- if self._poller_pid != os.getpid(): # a forked child has no poller thread
+ poller = self._poller
+ # A forked child has no poller thread, and a poller that died must not leave every
+ # cached profile without updates until the next new profile happens to re-arm it.
+ if poller is None or self._poller_pid != os.getpid() or not poller.is_alive():
self._ensure_poller()
return st
with self._lock:
@@ -153,10 +182,15 @@ class RemoteBackend:
st = self._states.get(key)
if st is None:
st = self._initial_fetch(Path(home))
- self._states[key] = st
+ with self._lock: # the poller snapshots the roster under the same lock
+ self._states[key] = st
self._ensure_poller()
return st
+ def _roster(self) -> List[_ProfileState]:
+ with self._lock:
+ return list(self._states.values())
+
def _initial_fetch(self, home: Path) -> _ProfileState:
"""Boot fetch with bounded retry (contract §11.4); fails closed (D2)."""
profile = self._profile_for(home)
@@ -232,6 +266,9 @@ class RemoteBackend:
st.etag = etag or str(body.get("etag") or "")
st.profile_version = int(body["profileVersion"])
st.profile_writer = _profile_writer(body)
+ st.other_writers = [
+ lv["writerConfigVersion"] for lv in body.get("levels") or []
+ if isinstance(lv, dict) and lv.get("kind") != "profile" and isinstance(lv.get("writerConfigVersion"), int)]
st.locks = locks
st.provenance = dict(body.get("provenance") or {})
st.changed_ns = time.time_ns()
@@ -246,89 +283,117 @@ class RemoteBackend:
"""First read after a change: warn on unknown keys and migrate in memory (D12). Runs on a
read, not at fetch, because both need ``hermes_cli.config``, which may still be importing
when the boot fetch runs."""
- if st.postprocessed or st.in_postprocess:
- return
+ if _private_for(st) is not None:
+ return # a migration's own reads: never start another one
# The flag, not st.lock, guards the work: a migration reads through hermes_cli.config (its
# _CONFIG_LOCK), and holding st.lock across that could deadlock against a writer that holds
- # _CONFIG_LOCK and waits for st.lock. A concurrent reader meanwhile gets the unmigrated doc.
+ # _CONFIG_LOCK and waits for st.lock. A concurrent reader meanwhile gets the published
+ # (unmigrated) doc, and a concurrent writer diffs against its matching base.
with st.lock:
if st.postprocessed or st.in_postprocess:
return
st.in_postprocess = True
- # The doc's schema version and its generation are captured together: a poll may install
- # a newer doc at any point after this, and migrating that doc from THIS version would
- # run steps its data never needed (and, D12, drop settings it holds on purpose).
+ # The doc, its schema version and its generation are captured together: a poll may
+ # install a newer doc at any point after this, and migrating that doc from THIS version
+ # would run steps its data never needed (and, D12, drop settings it holds on purpose).
seen = st.installed
current = int(st.doc.get("_config_version") or 0)
- keys = set(st.doc)
+ doc = copy.deepcopy(st.doc)
try:
try:
from hermes_cli.config import _known_top_level_keys
- from hermes_cli.config_migrations import SUPPORT_FLOOR_VERSION, run_migrations
+ from hermes_cli.config_migrations import SUPPORT_FLOOR_VERSION
except ImportError:
return # still importing; the next read retries
- for key in sorted(keys - _known_top_level_keys() - {"_config_version"}):
+ for key in sorted(set(doc) - _known_top_level_keys() - {"_config_version"}):
if key not in self._unknown_warned:
self._unknown_warned.add(key)
logger.warning("Remote Config: unknown config key %r is ignored by this Hermes version", key)
latest = _latest_config_version()
if SUPPORT_FLOOR_VERSION <= current < latest:
- self._migrate_in_memory(st, seen, current, run_migrations)
+ self._migrate_in_memory(st, seen, current, doc)
elif current < SUPPORT_FLOOR_VERSION:
logger.warning("Remote Config: profile %r was written by config version %d, below the "
"migration floor %d; not migrated", st.profile, current, SUPPORT_FLOOR_VERSION)
- st.postprocessed = st.installed == seen # a doc installed meanwhile needs its own pass
+ with st.lock:
+ st.postprocessed = st.installed == seen # a doc installed meanwhile needs its own pass
finally:
st.in_postprocess = False
- def _migrate_in_memory(self, st: _ProfileState, started: int, current: int, run_migrations) -> None:
- """Migrate doc generation *started* (whose schema version is *current*) in memory (D12).
- The migrated doc also becomes the base later writes diff against: a reader was handed the
- migrated doc, so a write that leaves the migration's changes in place must not send them
- (D12: never write the migration back).
-
- Generation-checked at start, at each step (:meth:`apply_in_memory`) and at commit: once a
- poll or write has installed another doc, nothing of this migration lands on it."""
+ def _run_private(self, st: _ProfileState, doc: Dict[str, Any], current: int) -> Dict[str, Any]:
+ """Run the migrations from schema *current* over a private copy of *doc* and return it.
+ Nothing is published: only this thread sees the copy while the steps run (:class:`_Private`)."""
+ from hermes_cli.config_migrations import run_migrations
from hermes_constants import reset_hermes_home_override, set_hermes_home_override
- with st.lock:
- if st.installed != started:
- return # replaced before we began; the new doc gets its own pass
- st.migrating = started
- token = set_hermes_home_override(st.home)
+ m = _Private(st=st, doc=copy.deepcopy(doc), serial=next(_PRIVATE_SERIAL))
+ m.doc["_config_version"] = current
+ private_token = _PRIVATE.set(m)
+ home_token = set_hermes_home_override(st.home)
try:
run_migrations(current, {"env_added": [], "config_added": [], "warnings": []}, True)
finally:
- reset_hermes_home_override(token)
+ reset_hermes_home_override(home_token)
+ _PRIVATE.reset(private_token)
+ return m.doc
+
+ def _migrate_in_memory(self, st: _ProfileState, started: int, current: int, doc: Dict[str, Any]) -> None:
+ """Migrate doc generation *started* (*doc*, schema version *current*) in memory (D12).
+
+ The steps work on a private copy; the result is published with its diff base in one step,
+ and only while generation *started* is still the installed one: once a poll or write has
+ installed another doc, nothing of this migration lands on it (that doc gets its own pass).
+ The migrated doc becomes the base later writes diff against: a reader was handed the
+ migrated doc, so a write that leaves the migration's changes in place must not send them
+ (D12: never write the migration back)."""
with st.lock:
- st.migrating = None
if st.installed != started:
- return # a poll or write installed a newer doc mid-migration; its own pass migrates it
- doc = copy.deepcopy(st.doc)
- doc["_config_version"] = _latest_config_version()
- self._apply_in_memory_locked(st, doc)
- base = copy.deepcopy(doc)
- base.pop("_config_version", None)
- st.base = base
+ return # replaced before we began
+ migrated = self._run_private(st, doc, current)
+ migrated["_config_version"] = _latest_config_version()
+ base = copy.deepcopy(migrated)
+ base.pop("_config_version", None)
+ with st.lock:
+ if st.installed != started:
+ return # a poll or write installed a newer doc mid-migration
+ self._publish_locked(st, migrated, base)
logger.info("Remote Config: migrated profile %r in memory from config version %d", st.profile, current)
+ @staticmethod
+ def _publish_locked(st: _ProfileState, doc: Dict[str, Any], base: Dict[str, Any]) -> None:
+ st.doc = doc
+ st.base = base
+ st.changed_ns = time.time_ns()
+ st.gen += 1
+
# --- ConfigBackend: reads ---------------------------------------------------------------
def read_user_layer(self, home: Path) -> UserLayer:
st = self._state(home)
self._postprocess(st)
- return UserLayer(doc=copy.deepcopy(st.doc), version=self._version_of(st),
- locks={encode(p): level for p, level in st.locks},
+ with st.lock:
+ m = _private_for(st)
+ doc = copy.deepcopy(m.doc if m is not None else st.doc)
+ version = self._version_of(st)
+ locks = {encode(p): level for p, level in st.locks}
+ return UserLayer(doc=doc, version=version, locks=locks,
provenance=f"remote:{client.base_url()} profile={st.profile}")
def read_user_doc_readonly(self, home: Path) -> Any:
st = self._state(home)
self._postprocess(st)
- return st.doc
+ m = _private_for(st)
+ return m.doc if m is not None else st.doc
@staticmethod
def _version_of(st: _ProfileState) -> Tuple[Any, ...]:
# Leads with a nanosecond timestamp like the file backend's st_mtime_ns (the TUI's config
- # watcher reads element 0 as a time); the etag and local generation make it exact.
+ # watcher reads element 0 as a time); the etag and local generation make it exact. A
+ # migration's own reads see its private copy under a version of their own, so no config
+ # cache ever pairs the published version with a private doc, or the reverse. It must not
+ # START with the published version either: config caches match a signature as a prefix.
+ m = _private_for(st)
+ if m is not None:
+ return (-m.serial, m.gen, st.changed_ns, st.etag, st.gen)
return (st.changed_ns, st.etag, st.gen)
def version(self, home: Path) -> Tuple[Any, ...]:
@@ -346,37 +411,65 @@ class RemoteBackend:
# --- ConfigBackend: writes --------------------------------------------------------------
+ _MAX_PREPARE = 3 # re-preparations when a poll lands between preparing and sending
+
def write_changes(self, home: Path, changes: Changes) -> None:
+ """Send ``changes`` as one PATCH with CAS (contract §15.2), re-read and retry once on a
+ conflict (§15.3).
+
+ The edit is fixed once, against the doc the caller read (its *intent*: the key-level sets
+ and unsets that turn that doc into ``changes``). A retry re-diffs that intent against the
+ re-read doc, with the re-read locks; it never replays the caller's whole document, which
+ would turn every key another writer changed meanwhile into an edit of ours."""
st = self._state(home)
+ intent: Optional[Tuple[Dict[KeyPath, Any], List[KeyPath]]] = None
for attempt in (1, 2):
# Diff against the doc readers get, i.e. migrated in memory (D12) — also after the CAS
# re-read below. Outside st.lock: a migration reads through hermes_cli.config.
self._postprocess(st)
- with st.lock:
- built = self._build_patch(st, changes)
- if built is None:
- return
- body, unsets = built
- try:
- resp = client.request("PATCH", st.home, st.profile, body=body)
- except (PlaneCredentialError, client.TransportError) as exc:
- raise ConfigWriteError(f"Remote Config write failed: {exc}", code="config_plane_unreachable") from exc
- if resp.status == 200 and _effective_ok(resp.body):
- self._install(st, resp.body, resp.etag)
- self._warn_inherited(st, unsets)
- return
- if resp.status == 409 and attempt == 1:
- try: # contract §15.3: re-read, re-diff, retry once
- self._fetch_into(st, conditional=False)
- except _FetchFailed as exc:
- raise ConfigWriteError(f"Remote Config write conflict and re-read failed: {exc}",
- code="config_version_conflict") from exc
- continue
- raise self._write_error(resp)
+ for _ in range(self._MAX_PREPARE):
+ with st.lock:
+ seen = st.installed
+ if intent is None:
+ intent = self._intent(st, changes)
+ built = self._patch_body(st, *intent)
+ if built is None:
+ return
+ body, sets, unsets = built
+ check = self._migration_check_input(st, body, sets, unsets)
+ if check is not None: # outside st.lock, like any migration run
+ self._check_migration_keeps(st, sets, *check)
+ with st.lock:
+ if st.installed != seen:
+ continue # a poll landed meanwhile: prepare again against it
+ try:
+ resp = client.request("PATCH", st.home, st.profile, body=body)
+ except (PlaneCredentialError, client.TransportError) as exc:
+ raise ConfigWriteError(f"Remote Config write failed: {exc}", code="config_plane_unreachable") from exc
+ if resp.status == 200 and _effective_ok(resp.body):
+ self._install(st, resp.body, resp.etag)
+ self._warn_inherited(st, unsets)
+ return
+ if resp.status == 409 and attempt == 1:
+ try: # contract §15.3: re-read, re-diff, retry once
+ self._fetch_into(st, conditional=False)
+ except _FetchFailed as exc:
+ raise ConfigWriteError(f"Remote Config write conflict and re-read failed: {exc}",
+ code="config_version_conflict") from exc
+ break
+ raise self._write_error(resp)
+ else:
+ raise ConfigWriteError("Remote Config write not sent: the profile's config kept changing "
+ "while the write was prepared", code="config_version_conflict")
- def _build_patch(self, st: _ProfileState, changes: Changes) -> Optional[Tuple[Dict[str, Any], List[KeyPath]]]:
- """The PATCH body for ``changes`` against the last read (§15.2), or None for no change."""
+ def _diff_base(self, st: _ProfileState) -> Dict[str, Any]:
base = copy.deepcopy(st.base if st.base is not None else st.server_config)
+ base.pop("_config_version", None)
+ return base
+
+ def _intent(self, st: _ProfileState, changes: Changes) -> Tuple[Dict[KeyPath, Any], List[KeyPath]]:
+ """The key-level edit ``changes`` makes to the last read doc (§15.2 steps 1-5)."""
+ base = self._diff_base(st)
if changes.document is not None:
new = to_wire(copy.deepcopy(changes.document))
if not isinstance(new, dict):
@@ -388,7 +481,6 @@ class RemoteBackend:
_set_path(new, path, to_wire(value, path))
for key in changes.unset:
_pop_path(new, from_dotted(key))
- base.pop("_config_version", None)
new.pop("_config_version", None)
if changes.document is not None:
@@ -396,6 +488,17 @@ class RemoteBackend:
if dropped:
print(f"Note: {len(dropped)} setting(s) locked by Remote Config were not saved: "
f"{', '.join(sorted(encode(p) for p in dropped))}", file=sys.stderr)
+ return diff(base, new)
+
+ def _patch_body(self, st: _ProfileState, sets: Dict[KeyPath, Any], unsets: List[KeyPath]
+ ) -> Optional[Tuple[Dict[str, Any], Dict[KeyPath, Any], List[KeyPath]]]:
+ """The PATCH body applying the intent to the current doc, or None when that is no change."""
+ base = self._diff_base(st)
+ new = copy.deepcopy(base)
+ for p, v in sets.items():
+ _set_path(new, p, copy.deepcopy(v))
+ for p in unsets:
+ _pop_path(new, p)
sets, unsets = diff(base, new)
refused = write_check(sets, unsets, st.locks)
if refused is not None:
@@ -424,7 +527,54 @@ class RemoteBackend:
body["set"] = set_body
if unset_body:
body["unset"] = unset_body
- return body, unsets
+ return body, sets, unsets
+
+ def _migration_check_input(self, st: _ProfileState, body: Dict[str, Any], sets: Dict[KeyPath, Any],
+ unsets: List[KeyPath]) -> Optional[Tuple[Dict[str, Any], int]]:
+ """``(stored doc after this write, its schema version)`` when readers will migrate it in
+ memory after the write (R6: the oldest level's stamp), else None. Call under st.lock."""
+ if not sets:
+ return None
+ try:
+ from hermes_cli.config_migrations import SUPPORT_FLOOR_VERSION
+ except ImportError:
+ return None
+ latest = _latest_config_version()
+ profile = latest if "writerConfigVersion" in body else st.profile_writer
+ stamps = [*st.other_writers, *([profile] if isinstance(profile, int) else [])]
+ after = min(stamps) if stamps else latest
+ if not SUPPORT_FLOOR_VERSION <= after < latest:
+ return None
+ stored = copy.deepcopy(st.server_config)
+ stored.pop("_config_version", None)
+ for p, v in sets.items():
+ _set_path(stored, p, copy.deepcopy(v))
+ for p in unsets:
+ _pop_path(stored, p)
+ return stored, after
+
+ def _check_migration_keeps(self, st: _ProfileState, sets: Dict[KeyPath, Any],
+ stored: Dict[str, Any], after: int) -> None:
+ """Refuse a write whose values every later read would migrate away.
+
+ The plane stores the write as sent, but under a level stamp older than this agent's schema
+ (the stamp keeps describing the level's untouched legacy data, contract §5.3), so every
+ reader — this process and any later one — re-runs the migrations from that stamp over the
+ stored doc. A value one of those steps rewrites (an old default the user set on purpose)
+ would be acknowledged and then never read back. Refused instead, before anything is sent."""
+ migrated = self._run_private(st, stored, after)
+ latest = _latest_config_version()
+ for p, v in sets.items():
+ found, got = _get_path(migrated, p)
+ changed = _first_change(p, v, found, got)
+ if changed is not None:
+ raise ConfigValueError(
+ f"Remote Config: {encode(changed)} cannot be saved with that value. This profile's remote "
+ f"settings are stored at config schema v{after}, and every read migrates them to "
+ f"v{latest} in memory, which rewrites that value (migrations are never written back, "
+ "D12), so it would never be read back. Nothing was sent; choose another value, or "
+ "have the profile level re-saved at the current schema on the config plane.",
+ code="config_migration_conflict")
@staticmethod
def _write_error(resp: client.Response) -> ConfigWriteError:
@@ -461,16 +611,16 @@ class RemoteBackend:
def apply_in_memory(self, home: Path, doc: dict) -> None:
st = self._state(home)
- with st.lock:
- if st.migrating is not None and st.migrating != st.installed:
- return # a migration step of a doc a poll/write has since replaced: drop it
- self._apply_in_memory_locked(st, doc)
-
- @staticmethod
- def _apply_in_memory_locked(st: _ProfileState, doc: dict) -> None:
- st.doc = copy.deepcopy(doc)
- st.changed_ns = time.time_ns()
- st.gen += 1
+ m = _private_for(st)
+ if m is not None: # a migration step: lands in that migration's private copy only
+ m.doc = copy.deepcopy(doc)
+ m.gen += 1
+ return
+ with st.lock: # outside a migration: the doc and the base writes diff against move together
+ doc = copy.deepcopy(doc)
+ base = copy.deepcopy(doc)
+ base.pop("_config_version", None)
+ self._publish_locked(st, doc, base)
# --- ConfigBackend: capabilities and boot -----------------------------------------------
@@ -483,6 +633,10 @@ class RemoteBackend:
def protected_env_names(self) -> frozenset:
return PLANE_CREDENTIAL_ENV_NAMES
+ def deployment_env_names(self) -> frozenset:
+ from hermes_cli.config_backend import BACKEND_ENV
+ return frozenset({BACKEND_ENV, client.URL_ENV, client.INSTANCE_ENV, POLL_ENV})
+
def boot(self, home: Path) -> None:
"""Design §4.4 steps 2-4: fetch this home's layer (fail closed) and flag a managed file."""
self._state(home)
@@ -517,12 +671,16 @@ class RemoteBackend:
def _poll_loop(self) -> None:
while not self._stop.is_set():
- now = time.monotonic()
- states = list(self._states.values())
- due = [st for st in states if st.next_poll <= now]
- for st in due:
- self.poll_one(st)
- wake = min((st.next_poll for st in self._states.values()), default=now + poll_interval())
+ try:
+ now = time.monotonic()
+ roster = self._roster() # a snapshot: a lazily added profile must not break iteration
+ for st in roster:
+ if st.next_poll <= now:
+ self.poll_one(st)
+ wake = min((st.next_poll for st in self._roster()), default=now + poll_interval())
+ except Exception: # noqa: BLE001 — the only poller thread must survive anything
+ logger.warning("Remote Config: poll loop error; retrying", exc_info=True)
+ wake = time.monotonic() + MIN_POLL_SECONDS
self._stop.wait(max(1.0, wake - time.monotonic()))
def poll_one(self, st: _ProfileState) -> bool:
@@ -547,7 +705,7 @@ class RemoteBackend:
return changed
def poll_all(self) -> None:
- for st in list(self._states.values()):
+ for st in self._roster():
self.poll_one(st)
@@ -560,6 +718,22 @@ def managed_config_file() -> Optional[Path]:
return path if path.exists() else None # config-reader: ok — existence only, to warn it is ignored (D18)
+def _first_change(path: KeyPath, sent: Any, found: bool, got: Any) -> Optional[KeyPath]:
+ """The first path at or under *path* where *got* (as read back) differs from *sent*."""
+ if not found:
+ return path
+ if isinstance(sent, dict) and isinstance(got, dict):
+ for key, value in sent.items():
+ changed = _first_change(path + (key,), value, key in got, got.get(key))
+ if changed is not None:
+ return changed
+ return None
+ try:
+ return None if json_equal(to_wire(copy.deepcopy(got), path), sent) else path
+ except ConfigValueError:
+ return path
+
+
def _set_path(doc: Dict[str, Any], path: KeyPath, value: Any) -> None:
node = doc
for seg in path[:-1]:
diff --git a/tests/plugins/config_backends/test_remote_backend.py b/tests/plugins/config_backends/test_remote_backend.py
index 422bf3d2790..55c3a429a3d 100644
--- a/tests/plugins/config_backends/test_remote_backend.py
+++ b/tests/plugins/config_backends/test_remote_backend.py
@@ -922,3 +922,367 @@ def test_stub_plane_deep_merge_follows_contract_6_2():
assert deep_merge(upper, {"display": None}) == upper
assert deep_merge(upper, {"model": None})["model"] is None
assert deep_merge(upper, {"tags": [3]})["tags"] == [3]
+
+
+# --- review round 3 regressions -----------------------------------------------------------
+
+def _drop_process_deployment(plane, monkeypatch, tmp_path):
+ """Move the deployment out of the process env (as on a host where it lives in .env files);
+ returns the managed dir. Every name stays recorded by monkeypatch, so whatever a dotenv load
+ publishes is undone at teardown."""
+ from hermes_cli import config_backend
+ for name in remote_env(plane):
+ monkeypatch.delenv(name, raising=False)
+ managed = tmp_path / "managed"
+ managed.mkdir()
+ monkeypatch.setenv("HERMES_MANAGED_DIR", str(managed))
+ monkeypatch.setattr(config_backend, "_BOOTSTRAPPED", True) # this process is past its first read
+ return managed
+
+
+def test_cas_retry_of_a_document_write_replays_only_its_own_edit(plane, monkeypatch):
+ """F1: another writer commits between our read and our PATCH. The 409 retry re-applies OUR
+ edit to the re-read doc; it must not turn the other writer's changes into edits of ours."""
+ from hermes_cli.config import set_config_value
+ from plugins.config_backends.remote import client
+ plane.profile("default").update(values={"display": {"personality": "concise", "compact": False}}, version=1)
+ real_request = client.request
+ raced = []
+
+ def other_writer_lands_first(method, *args, **kwargs):
+ if method == "PATCH" and not raced:
+ raced.append(True)
+ prof = plane.profile("default")
+ prof["values"]["display"]["compact"] = True
+ prof["values"]["terminal"] = {"timeout": 321}
+ prof["version"] += 1
+ return real_request(method, *args, **kwargs)
+
+ monkeypatch.setattr(client, "request", other_writer_lands_first)
+
+ set_config_value("display.personality", "pirate")
+
+ assert plane.profile("default")["values"] == {
+ "display": {"personality": "pirate", "compact": True}, "terminal": {"timeout": 321}}
+ first, retry = plane.patches()
+ assert first["body"]["expectedVersion"] == 1 and retry["body"]["expectedVersion"] == 2
+ assert "unset" not in retry["body"] and "compact" not in json.dumps(retry["body"])
+
+
+def test_edit_during_an_in_memory_migration_never_sends_the_migration(plane, monkeypatch):
+ """F2: another thread's in-memory migration has run its steps but not finished when a public
+ edit reads and saves. The edit must diff against the doc it read, so the migration's removal
+ of compression.threshold_tokens is not sent (D12)."""
+ import threading
+
+ from hermes_cli import config_migrations
+ from hermes_cli.config import read_raw_config, set_config_value
+ from hermes_cli.config_backend import read_config_doc
+ assert backend_mod._latest_config_version() > 46
+ plane.profile("default").update(values={"compression": {"threshold_tokens": 256000}}, version=1, writer=46)
+ get_config_backend()._state(plane.home)
+ steps_ran, resume = threading.Event(), threading.Event()
+ real_run = config_migrations.run_migrations
+
+ def paused(*args, **kwargs):
+ real_run(*args, **kwargs)
+ if threading.current_thread().name == "migrator": # the edit's own checks run unpaused
+ steps_ran.set()
+ assert resume.wait(10), "the edit waited for the migration"
+
+ monkeypatch.setattr(config_migrations, "run_migrations", paused)
+ failures = []
+
+ def migrate(): # a lock-free reader (read_config_doc) starts the migration
+ try:
+ read_config_doc(plane.home / "config.yaml")
+ except BaseException as exc: # noqa: BLE001 — surfaced below
+ failures.append(exc)
+
+ migrator = threading.Thread(target=migrate, name="migrator")
+ migrator.start()
+ try:
+ assert steps_ran.wait(10)
+ set_config_value("display.personality", "pirate")
+ finally:
+ resume.set()
+ migrator.join(10)
+ assert not migrator.is_alive() and not failures, failures
+
+ (patch,) = plane.patches()
+ assert "unset" not in patch["body"] and "compression" not in json.dumps(patch["body"])
+ assert plane.profile("default")["values"]["compression"] == {"threshold_tokens": 256000}
+ later = read_raw_config()
+ assert later["display"]["personality"] == "pirate"
+ assert "threshold_tokens" not in (later.get("compression") or {}) # still migrated, in memory
+
+
+def test_profile_added_while_the_poller_iterates_does_not_kill_it(plane):
+ """F3: a first read of another profile inserts its state while the poll loop walks the
+ roster; the only poller thread must survive it."""
+ import dataclasses
+ import threading
+
+ backend = get_config_backend()
+ st = backend._state(plane.home)
+ work = plane.home / "profiles" / "work"
+ work.mkdir(parents=True)
+ inserted = threading.Event()
+
+ class AddsAProfileMidPass(backend_mod._ProfileState):
+ reads = 0
+
+ @property
+ def next_poll(self):
+ AddsAProfileMidPass.reads += 1
+ if AddsAProfileMidPass.reads == 2 and not inserted.is_set(): # mid-pass: the deadline scan
+ backend._state(work)
+ inserted.set()
+ return time.monotonic() + 1000
+
+ @next_poll.setter
+ def next_poll(self, value):
+ pass
+
+ import time
+ key = backend._key(plane.home)
+ backend._states[key] = AddsAProfileMidPass(**{
+ f.name: getattr(st, f.name) for f in dataclasses.fields(st) if f.name != "next_poll"})
+ loop = threading.Thread(target=backend._poll_loop, daemon=True)
+ loop.start()
+ assert inserted.wait(10)
+ loop.join(2)
+ assert loop.is_alive()
+ assert backend._key(work) in backend._states
+ backend._stop.set()
+
+
+def test_a_dead_poller_is_rearmed_by_a_cached_read(plane):
+ """F3: a poller that died must not leave cached profiles without updates."""
+ import threading
+
+ from hermes_cli.config import load_config
+ load_config()
+ backend = remote_pkg.get_remote_backend()
+ dead = threading.Thread(target=lambda: None)
+ dead.start()
+ dead.join()
+ backend._poller = dead
+ load_config()
+ assert backend._poller is not dead and backend._poller.is_alive()
+
+
+def test_instance_id_from_the_managed_dotenv_boots(plane, monkeypatch, tmp_path):
+ """F4: the deployment in the home's .env, the instance id only in the managed .env."""
+ import os
+
+ from hermes_cli.env_loader import load_hermes_dotenv
+ managed = _drop_process_deployment(plane, monkeypatch, tmp_path)
+ env = remote_env(plane)
+ (managed / ".env").write_text(f"HERMES_CONFIG_INSTANCE_ID={env.pop('HERMES_CONFIG_INSTANCE_ID')}\n")
+ (plane.home / ".env").write_text("".join(f"{k}={v}\n" for k, v in env.items()))
+ plane.upper = {"display": {"personality": "managed"}}
+
+ load_hermes_dotenv(hermes_home=plane.home)
+
+ gets = _gets(plane)
+ assert gets and gets[0]["instance"] == INSTANCE
+ assert os.environ["HERMES_CONFIG_INSTANCE_ID"] == INSTANCE
+
+
+def test_child_for_another_profile_keeps_the_remote_deployment(plane, monkeypatch, tmp_path):
+ """F5: the deployment came from the launch profile's .env. A Hermes child built for another
+ profile must still read that profile's config remotely, never fall back to its local file."""
+ import os
+ import subprocess
+ import sys
+
+ from hermes_cli.env_loader import load_hermes_dotenv
+ from tools.environments.local import served_profile_child_env
+ _drop_process_deployment(plane, monkeypatch, tmp_path)
+ env = remote_env(plane)
+ (plane.home / ".env").write_text("".join(f"{k}={v}\n" for k, v in env.items()))
+ load_hermes_dotenv(hermes_home=plane.home)
+ beta = plane.home / "profiles" / "beta"
+ beta.mkdir(parents=True)
+ (beta / "config.yaml").write_text("display:\n personality: local-child\n")
+ # beta's own plane credential (credentials are never carried into another profile's child)
+ (beta / ".env").write_text("".join(f"{k}={v}\n" for k, v in env.items() if k.startswith("GATEWAY_RELAY_IDP_")))
+ plane.profile("beta")["values"] = {"display": {"personality": "remote-child"}}
+
+ child_env = served_profile_child_env(target_home=beta, inherit_credentials=True)
+ child_env["PYTHONPATH"] = str(Path(__file__).resolve().parents[3])
+ code = ("import json\n"
+ "from hermes_cli.env_loader import load_hermes_dotenv\n"
+ "load_hermes_dotenv()\n"
+ "from hermes_cli.config_backend import get_config_backend\n"
+ "from hermes_cli.config import load_config\n"
+ "print('RESULT=' + json.dumps([get_config_backend().name, load_config()['display']['personality']]))\n")
+ proc = subprocess.run([sys.executable, "-c", code], env=child_env, capture_output=True, text=True,
+ timeout=120, stdin=subprocess.DEVNULL, cwd=str(tmp_path))
+ assert proc.returncode == 0, proc.stderr[-3000:]
+ line = [ln for ln in (proc.stdout + proc.stderr).splitlines() if ln.startswith("RESULT=")][-1]
+ assert json.loads(line[len("RESULT="):]) == ["remote", "remote-child"]
+ assert "beta" in {r["profile"] for r in _gets(plane)}
+ assert not any(k.startswith("GATEWAY_RELAY_IDP_") and v != env[k] for k, v in child_env.items())
+ assert os.environ["HERMES_CONFIG_BACKEND"] == "remote"
+
+
+def test_setting_a_value_that_old_schema_migration_rewrites_is_refused(plane, capsys):
+ """F8: on a level stamped 46 (option 1 keeps that stamp), 256000 is exactly what the 46->47
+ step removes on every read: acknowledging it would hide it. Refused, nothing sent. A value
+ the migration keeps is saved as usual."""
+ from hermes_cli.config import read_raw_config
+ assert backend_mod._latest_config_version() > 46
+ plane.profile("default").update(values={"display": {"personality": "old"}}, version=1, writer=46)
+ read_raw_config()
+
+ code, err = _config_cmd(capsys, "set", "compression.threshold_tokens", "256000")
+
+ assert code == 1 and "compression.threshold_tokens" in err and "schema v46" in err
+ assert plane.patches() == []
+ with pytest.raises(ConfigValueError) as exc:
+ write_config_key(plane.home / "config.yaml", "compression.threshold_tokens", 256000)
+ assert exc.value.code == "config_migration_conflict"
+
+ write_config_key(plane.home / "config.yaml", "compression.threshold_tokens", 300000)
+ assert plane.profile("default")["values"]["compression"] == {"threshold_tokens": 300000}
+ remote_pkg._reset_for_tests() # a fresh process reads it back
+ assert read_raw_config()["compression"]["threshold_tokens"] == 300000
+
+
+# --- profiles, TUI and RPC in remote mode -------------------------------------------------
+
+def profile_plane(tmp_path, monkeypatch):
+ """``plane`` with a real profiles root (``Path.home()/.hermes``)."""
+ home = tmp_path / ".hermes"
+ home.mkdir()
+ monkeypatch.setattr(Path, "home", lambda: tmp_path)
+ monkeypatch.setenv("HERMES_HOME", str(home))
+ with StubPlane() as p:
+ for k, v in remote_env(p).items():
+ monkeypatch.setenv(k, v)
+ monkeypatch.setattr(backend_mod, "BOOT_RETRY_DELAYS", (0.0, 0.0))
+ remote_pkg._reset_for_tests()
+ cred_mod._IDP_CACHE.clear()
+ p.home = home
+ get_config_backend()._state(home)
+ yield p
+ remote_pkg._reset_for_tests()
+ cred_mod._IDP_CACHE.clear()
+
+
+def test_fresh_profile_creation_succeeds_against_a_healthy_plane(profile_plane):
+ """F6: the staging home is never read through the remote backend."""
+ from hermes_cli.profiles import create_profile
+ before = len(profile_plane.requests)
+
+ path = create_profile("fresh", no_alias=True, no_skills=True)
+
+ assert path.is_dir() and (path / ".env").exists()
+ assert not list(path.parent.glob(".fresh.staging-*"))
+ assert not any(r["profile"].startswith(".") for r in profile_plane.requests[before:])
+
+
+def test_rename_is_refused_before_anything_moves(profile_plane, monkeypatch):
+ """F7: the plane keys settings by profile name and has no rename."""
+ from hermes_cli import profiles
+ for name in ("_check_gateway_running", "_cleanup_gateway_service", "_maybe_unregister_gateway_service",
+ "_stop_bot_desktop", "_live_default_multiplexer", "_maybe_register_gateway_service"):
+ monkeypatch.setattr(profiles, name, lambda *a, **k: False)
+ old = profile_plane.home / "profiles" / "old"
+ old.mkdir(parents=True)
+ profile_plane.profile("old").update(values={"model": {"default": "profile-model"}}, version=1)
+ profile_plane.profile("new").update(values={"model": {"default": "target-model"}}, version=1)
+
+ with pytest.raises(ValueError, match="not supported"):
+ profiles.rename_profile("old", "new")
+
+ assert old.is_dir() and not (profile_plane.home / "profiles" / "new").exists()
+ assert profile_plane.profile("old")["values"] == {"model": {"default": "profile-model"}}
+ assert profile_plane.patches() == []
+
+
+def test_roster_model_follows_the_remote_layer_not_the_local_file(profile_plane):
+ """F9: a poll changes the profile's model while an ignored local config.yaml stays put."""
+ from hermes_cli.profiles import list_profiles
+ named = profile_plane.home / "profiles" / "roster"
+ named.mkdir(parents=True)
+ (named / "config.yaml").write_text("model:\n default: ignored-local\n")
+ profile_plane.profile("roster").update(values={"model": {"default": "model-A", "provider": "nous"}}, version=1)
+
+ def roster():
+ return {p.name: (p.model, p.provider) for p in list_profiles()}["roster"]
+
+ assert roster() == ("model-A", "nous")
+ profile_plane.profile("roster").update(values={"model": {"default": "model-B", "provider": "openrouter"}}, version=2)
+ backend = get_config_backend()
+ assert backend.poll_one(backend._state(named))
+ assert roster() == ("model-B", "openrouter")
+
+
+def _tui_server(monkeypatch, home):
+ import hermes_cli.banner as banner
+ monkeypatch.setattr(banner, "prefetch_update_check", lambda: None)
+ from tui_gateway import server
+ monkeypatch.setattr(server, "_hermes_home", home)
+ server._cfg_cache = server._cfg_sig = server._cfg_path = None
+ return server
+
+
+def test_tui_config_set_of_a_locked_key_is_refused_not_reported_saved(plane, monkeypatch):
+ """F10: a keyed TUI edit under a lock answers with the lock, sends nothing, and the raw cache
+ keeps the accepted value."""
+ plane.upper = {"display": {"tui_theme": "dark"}}
+ plane.upper_locks = [{"path": "display.tui_theme", "level": "tenant"}]
+ server = _tui_server(monkeypatch, plane.home)
+
+ answer = server._methods["config.set"](1, {"key": "theme", "value": "light"})
+
+ assert answer["error"]["code"] == 4002 and "locked" in answer["error"]["message"]
+ assert plane.patches() == []
+ assert server._load_cfg_raw()["display"]["tui_theme"] == "dark"
+
+ plane.upper_locks = []
+ backend = get_config_backend()
+ assert backend.poll_one(backend._state(plane.home))
+ assert server._methods["config.set"](2, {"key": "theme", "value": "light"})["result"]["value"] == "light"
+ assert server._load_cfg_raw()["display"]["tui_theme"] == "light"
+ assert plane.profile("default")["values"] == {"display": {"tui_theme": "light"}}
+
+
+def test_rpc_refused_by_the_config_backend_still_gets_its_one_answer(plane, monkeypatch):
+ """F11: ConfigBackendUnavailable is a SystemExit; an admitted request (inline or pooled) still
+ answers once, with its own id."""
+ import copy
+ import threading
+
+ server = _tui_server(monkeypatch, plane.home)
+
+ def refused(rid, params):
+ raise ConfigBackendUnavailable("Remote Config: cannot load the config for profile 'lazy'")
+
+ monkeypatch.setitem(server._methods, "cc.refused", refused)
+ inline = server.handle_request({"jsonrpc": "2.0", "id": "inline-1", "method": "cc.refused", "params": {}})
+ assert inline["id"] == "inline-1" and "cannot load the config" in inline["error"]["message"]
+
+ monkeypatch.setattr(server, "_LONG_HANDLERS", server._LONG_HANDLERS | {"cc.refused"})
+ written = threading.Event()
+
+ class Recorder:
+ frames = []
+
+ def write(self, obj):
+ self.frames.append(copy.deepcopy(obj))
+ written.set()
+ return True
+
+ def close(self):
+ pass
+
+ transport = Recorder()
+ assert server.dispatch({"jsonrpc": "2.0", "id": "pooled-1", "method": "cc.refused", "params": {}}, transport) is None
+ assert written.wait(10)
+ (frame,) = transport.frames
+ assert frame["id"] == "pooled-1" and "cannot load the config" in frame["error"]["message"]
diff --git a/tests/plugins/config_backends/test_remote_multiprocess.py b/tests/plugins/config_backends/test_remote_multiprocess.py
index 7ae0b49904c..02cd6f803c9 100644
--- a/tests/plugins/config_backends/test_remote_multiprocess.py
+++ b/tests/plugins/config_backends/test_remote_multiprocess.py
@@ -75,3 +75,28 @@ def test_process_exits_non_zero_when_plane_is_down(plane_env, monkeypatch):
assert proc.returncode != 0
assert "Hermes does not start without its remote config" in proc.stderr
assert "concise" not in proc.stdout # no default or cached value was served
+
+
+def test_first_config_read_before_dotenv_uses_the_dotenv_deployment(plane_env, tmp_path):
+ """F4: ``hermes_cli.config`` reads config at import time, before ``load_hermes_dotenv`` runs
+ (main.py imports it first). The selector is in the process env; the plane URL and credential
+ are in the home's .env and the instance id only in the managed .env. That first read must
+ already select, address and authenticate from that whole deployment."""
+ plane, env, home = plane_env
+ deployment = remote_env(plane)
+ managed = tmp_path / "managed"
+ managed.mkdir()
+ (managed / ".env").write_text(f"HERMES_CONFIG_INSTANCE_ID={deployment.pop('HERMES_CONFIG_INSTANCE_ID')}\n")
+ (home / ".env").write_text("".join(f"{k}={v}\n" for k, v in deployment.items() if k != "HERMES_CONFIG_BACKEND"))
+ for name in remote_env(plane):
+ env.pop(name, None)
+ env.update(HERMES_CONFIG_BACKEND="remote", HERMES_MANAGED_DIR=str(managed))
+ code = ("import json\n"
+ "import hermes_cli.config\n"
+ "from hermes_cli.env_loader import load_hermes_dotenv\n"
+ "load_hermes_dotenv()\n"
+ "from hermes_cli.config import load_config\n"
+ "print('RESULT=' + json.dumps(load_config()['display']['personality']))\n")
+ assert _result(_run([sys.executable, "-c", code], env)) == "concise"
+ gets = [r for r in plane.requests if r["method"] == "GET"]
+ assert gets and {r["instance"] for r in gets} == {INSTANCE}
diff --git a/tests/tools/test_stage2_hook_remote_config_skip.py b/tests/tools/test_stage2_hook_remote_config_skip.py
index adcf681fde9..b8c92b12a55 100644
--- a/tests/tools/test_stage2_hook_remote_config_skip.py
+++ b/tests/tools/test_stage2_hook_remote_config_skip.py
@@ -1,49 +1,95 @@
"""stage2 skips ``docker_config_migrate.py`` in remote config mode (config-config P4, D12): the
agent migrates the fetched document in memory, so a leftover local config.yaml must not be
-migrated. The mode comes from the container env or from ``$HERMES_HOME/.env``."""
+migrated. stage2 decides with the agent's own backend selection, so every dotenv form the runtime
+accepts (comments, spaces around ``=``, quotes, ``export``) and the runtime's precedence (the home's
+``.env`` over the container env) decide the same way at both places."""
from __future__ import annotations
+import os
import shutil
import subprocess
+import sys
from pathlib import Path
import pytest
-STAGE2_HOOK = Path(__file__).resolve().parents[2] / "docker" / "stage2-hook.sh"
+REPO = Path(__file__).resolve().parents[2]
+STAGE2_HOOK = REPO / "docker" / "stage2-hook.sh"
def _block() -> str:
text = STAGE2_HOOK.read_text()
- start = text.index("_remote_config_mode=0")
+ start = text.index("# --- Migrate persisted config schema ---")
end = text.index("\nfi\n", text.index("docker_config_migrate.py failed", start)) + len("\nfi\n")
return text[start:end]
+def _env(root: Path, backend_env: str | None) -> dict:
+ env = {k: v for k, v in os.environ.items() if k != "HERMES_CONFIG_BACKEND"}
+ env.update(PYTHONPATH=str(REPO), HERMES_MANAGED_DIR=str(root / "managed"))
+ if backend_env is not None:
+ env["HERMES_CONFIG_BACKEND"] = backend_env
+ return env
+
+
def _run(home: Path, backend_env: str | None) -> str:
+ """The checked-in stage2 block, with only the migration replaced by a marker."""
if shutil.which("sh") is None:
pytest.skip("sh not available")
- env_line = "unset HERMES_CONFIG_BACKEND\n" if backend_env is None else f"HERMES_CONFIG_BACKEND='{backend_env}'\n"
- bin_dir = home.parent / "bin" # s6-setuidgid is not a valid sh function name: stub it on PATH
+ root = home.parent
+ bin_dir = root / "bin" # s6-setuidgid is not a valid sh function name: stub it on PATH
bin_dir.mkdir(exist_ok=True)
stub = bin_dir / "s6-setuidgid"
- stub.write_text("#!/bin/sh\necho MIGRATE_RAN\n")
+ stub.write_text('#!/bin/sh\nshift\ncase "$*" in *docker_config_migrate.py*) echo MIGRATE_RAN ;; *) exec "$@" ;; esac\n')
stub.chmod(0o755)
+ venv_bin = root / "install" / ".venv" / "bin"
+ venv_bin.mkdir(parents=True, exist_ok=True)
+ python = venv_bin / "python" # a wrapper, not a symlink: a moved venv symlink loses its site-packages
+ python.write_text(f'#!/bin/sh\nexec "{sys.executable}" "$@"\n')
+ python.chmod(0o755)
script = (
- "set -eu\n" + env_line
- + f'PATH="{bin_dir}:$PATH"\nHERMES_HOME="{home}"\nINSTALL_DIR=/nonexistent\n'
+ "set -eu\n"
+ + f'PATH="{bin_dir}:$PATH"\nHERMES_HOME="{home}"\nINSTALL_DIR="{root / "install"}"\n'
+ _block()
)
- proc = subprocess.run(["sh", "-c", script], capture_output=True, text=True, timeout=30,
- stdin=subprocess.DEVNULL)
+ proc = subprocess.run(["sh", "-c", script], capture_output=True, text=True, timeout=60,
+ stdin=subprocess.DEVNULL, env=_env(root, backend_env))
assert proc.returncode == 0, proc.stderr
+ assert "could not determine the config backend" not in proc.stdout, proc.stdout
return proc.stdout
+def _runtime_selects(home: Path, backend_env: str | None) -> str:
+ """The backend a real Hermes start selects: the first config read (``hermes_cli.config`` reads
+ at import time) or ``load_hermes_dotenv``'s boot, whichever selects first, reports the backend
+ and ends the process right there (there is no plane in this test)."""
+ code = ("import os, sys\n"
+ "from hermes_cli.config_backend import FileBackend\n"
+ "from plugins.config_backends.remote.backend import RemoteBackend\n"
+ "def report(name):\n"
+ " def stop(self, *args, **kwargs):\n"
+ " print(name, flush=True)\n"
+ " os._exit(0)\n"
+ " return stop\n"
+ "RemoteBackend._state = report('remote')\n"
+ "FileBackend.boot = report('file')\n"
+ "from hermes_cli import env_loader\n"
+ "env_loader.load_hermes_dotenv(hermes_home=sys.argv[1], load_external_secrets=False)\n"
+ "sys.exit('no backend was selected')\n")
+ env = _env(home.parent, backend_env)
+ env["HERMES_HOME"] = str(home)
+ proc = subprocess.run([sys.executable, "-c", code, str(home)], capture_output=True, text=True,
+ timeout=60, env=env, cwd=str(home.parent))
+ assert proc.returncode == 0, proc.stderr
+ return proc.stdout.strip().splitlines()[-1]
+
+
@pytest.fixture
def home(tmp_path):
home = tmp_path / "home"
home.mkdir()
(home / "config.yaml").write_text("model: {}\n")
+ (tmp_path / "managed").mkdir()
return home
@@ -56,13 +102,24 @@ def test_remote_from_container_env_skips(home):
assert "MIGRATE_RAN" not in out and "skipping docker_config_migrate.py" in out
[email protected]("line", ["HERMES_CONFIG_BACKEND=remote", 'HERMES_CONFIG_BACKEND="remote"',
- "export HERMES_CONFIG_BACKEND='remote'"])
-def test_remote_from_dotenv_skips(home, line):
- (home / ".env").write_text(f"OTHER=1\n{line}\n")
[email protected]("dotenv, container", [
+ ("HERMES_CONFIG_BACKEND=remote", None),
+ ('HERMES_CONFIG_BACKEND="remote"', None),
+ ("export HERMES_CONFIG_BACKEND='remote'", None),
+ ("HERMES_CONFIG_BACKEND=remote # fleet-managed", None),
+ ("HERMES_CONFIG_BACKEND = remote", None),
+ ("HERMES_CONFIG_BACKEND=file\n# HERMES_CONFIG_BACKEND=remote", None),
+ ("HERMES_CONFIG_BACKEND=file", "remote"),
+ ("HERMES_CONFIG_BACKEND=remote", "file"),
+])
+def test_stage2_decides_like_the_runtime(home, dotenv, container):
+ (home / ".env").write_text(f"OTHER=1\n{dotenv}\n")
+ runtime = _runtime_selects(home, container)
+ migrated = "MIGRATE_RAN" in _run(home, container)
+ assert migrated == (runtime == "file"), (dotenv, container, runtime)
+
+
+def test_remote_from_managed_dotenv_skips(home):
+ (home.parent / "managed" / ".env").write_text("HERMES_CONFIG_BACKEND=remote\n")
+ assert _runtime_selects(home, None) == "remote"
assert "MIGRATE_RAN" not in _run(home, None)
-
-
-def test_other_dotenv_values_still_migrate(home):
- (home / ".env").write_text("HERMES_CONFIG_BACKEND=file\n# HERMES_CONFIG_BACKEND=remote\n")
- assert "MIGRATE_RAN" in _run(home, None)
diff --git a/tools/environments/local.py b/tools/environments/local.py
index 7b8edef590c..8e2e93df5ed 100644
--- a/tools/environments/local.py
+++ b/tools/environments/local.py
@@ -511,9 +511,12 @@ def strip_launch_profile_env(env: dict, target_home: str | Path | None = None) -
# LAUNCH profile's source supplied is not the target profile's to see; the caller's scope
# overlay puts back exactly the ones the target's own sources supply. The administrator-managed
# .env is NOT residue: its values are policy for every profile (``_apply_managed_env`` applies
- # it last, with override, so it beats the user's own .env) — leave them in place.
+ # it last, with override, so it beats the user's own .env) — leave them in place. Neither is
+ # the config backend's deployment (which backend, which plane, which instance): it is the
+ # instance's, and a child without it reads the target profile's config from another backend.
+ from hermes_cli.config_backend import deployment_env_names
from hermes_cli.env_loader import launch_dotenv_keys, managed_dotenv_keys, source_supplied_names
- managed_names = {key.upper() for key in managed_dotenv_keys()}
+ managed_names = {key.upper() for key in managed_dotenv_keys()} | set(deployment_env_names())
residue_names = {
key.upper() for key in
set(load_env_file(launch_home / ".env")) | set(launch_dotenv_keys())
diff --git a/tui_gateway/methods_config_set.py b/tui_gateway/methods_config_set.py
index 5ca0a5175a1..ad8979c983a 100644
--- a/tui_gateway/methods_config_set.py
+++ b/tui_gateway/methods_config_set.py
@@ -519,7 +519,11 @@ def _(rid, params: dict) -> dict:
handler = _set_display_toggle
if handler is None:
return _err(rid, 4002, f"unknown config key: {key}")
- return handler(rid, params, key, value, session)
+ from hermes_cli.config_backend import ConfigWriteError
+ try:
+ return handler(rid, params, key, value, session)
+ except ConfigWriteError as exc: # a locked key or a value the config backend refused: nothing saved
+ return _err(rid, 4002, str(exc))
def register(server) -> None:
diff --git a/tui_gateway/rpc_dispatch.py b/tui_gateway/rpc_dispatch.py
index b2a2587c42d..a31b553b07e 100644
--- a/tui_gateway/rpc_dispatch.py
+++ b/tui_gateway/rpc_dispatch.py
@@ -29,11 +29,17 @@ def _handle_admitted_request(req: dict) -> dict | None:
params, problem = _contracts.validate_params(contract, params)
if problem is not None:
return _err(rid, 4000, problem)
+ from hermes_cli.config_backend import ConfigBackendUnavailable
+
token = _current_rpc_method.set(method)
try:
response = fn(rid, params)
except ProfileUnavailableError as exc:
return _err(rid, 4064, str(exc))
+ except ConfigBackendUnavailable as exc:
+ # A SystemExit on purpose (readers must not fall back to defaults), so no ``except
+ # Exception`` below catches it: this request still owes its caller one answer.
+ return _err(rid, 4064, str(exc.code))
finally:
_current_rpc_method.reset(token)
if contract is not None and isinstance(response, dict) and isinstance(response.get("result"), dict):
diff --git a/tui_gateway/server.py b/tui_gateway/server.py
index 00849ea49e5..b86e15dfd2f 100644
--- a/tui_gateway/server.py
+++ b/tui_gateway/server.py
@@ -1032,9 +1032,14 @@ def _load_cfg() -> dict:
def _save_cfg(cfg: dict):
global _cfg_cache, _cfg_sig, _cfg_path
from hermes_cli.config import atomic_config_replace
- from hermes_cli.config_backend import config_cache_version
+ from hermes_cli.config_backend import config_cache_version, supports_file_tooling
path = _active_config_path()
atomic_config_replace(path, cfg)
+ if not supports_file_tooling():
+ # The backend saves what it accepts (a bulk save omits locked keys), not what was proposed:
+ # drop the cache so the next raw read returns the accepted document.
+ _drop_cfg_cache()
+ return
with _cfg_lock:
_cfg_cache, _cfg_path = copy.deepcopy(cfg), path
try:
@@ -1043,6 +1048,12 @@ def _save_cfg(cfg: dict):
_cfg_sig = None
+def _drop_cfg_cache() -> None:
+ global _cfg_cache, _cfg_sig, _cfg_path
+ with _cfg_lock:
+ _cfg_cache, _cfg_sig, _cfg_path = None, None, None
+
+
def _session_for_key(session_key: str) -> dict | None:
"""First live record with this session_key (snapshot under the lock: pool handlers mutate ``_sessions``)."""
with _sessions_lock:
@@ -1337,6 +1348,15 @@ def _append_model_switch_marker(session: dict | None, *, model: str, provider: s
def _write_config_key(key_path: str, value):
+ from hermes_cli.config_backend import Changes, get_config_backend, supports_file_tooling
+ if not supports_file_tooling():
+ # One explicit key: a keyed write, which refuses a locked key (ConfigLockedError) instead
+ # of the bulk save's silent omission, so the caller never reports a refused value as saved.
+ try:
+ get_config_backend().write_changes(_active_config_path().parent, Changes(set={key_path: value}))
+ finally:
+ _drop_cfg_cache()
+ return
# Write-back round-trip: raw read is mandatory — saving the overlaid/expanded view would persist it.
cfg = current = _load_cfg_raw()
*parents, leaf = key_path.split(".")
diff --git a/website/docs/user-guide/remote-config.md b/website/docs/user-guide/remote-config.md
index 28b59cbd467..20e416bd272 100644
--- a/website/docs/user-guide/remote-config.md
+++ b/website/docs/user-guide/remote-config.md
@@ -17,9 +17,9 @@ machine with one user should keep `config.yaml`.
## Turning it on
-Set the following in the environment or in `~/.hermes/.env`. A value in
-`config.yaml` cannot select the backend, because the backend is what serves
-config.
+Set the following in the environment, in `~/.hermes/.env`, or in the managed
+`/etc/hermes/.env` (which wins over the other two). A value in `config.yaml`
+cannot select the backend, because the backend is what serves config.
| Variable | Meaning |
|---|---|
@@ -41,6 +41,14 @@ These values, and the `HERMES_CONFIG_*` variables above, must come from
(1Password, Bitwarden, a command) supplies any of them, Hermes refuses to
start.
+A Hermes process started for another profile (a `hermes -p <name>` worker, a
+profile's cron or bot child) keeps the `HERMES_CONFIG_*` deployment of the
+process that started it, so it reads that profile's config from the plane too.
+It does not inherit the plane credential: on-prem, put the
+`GATEWAY_RELAY_IDP_*` values where every profile reads them (the managed
+`/etc/hermes/.env`) or in each profile's `.env`. Without one, the process
+exits with an error rather than read a local file.
+
## What changes
- **Startup fetches config and fails closed.** Every Hermes process (gateway,
@@ -57,12 +65,15 @@ start.
running agent within one interval.
- **Writes go to the profile's own level.** `hermes config set`, `/model`,
the dashboard, and every other config writer send only the keys that changed,
- guarded against concurrent edits. Removing a key removes it from the profile
- level only: if an upper level also sets it, that value still applies, and
- Hermes warns you.
-- **Locked keys are refused.** `hermes config set` on a locked key prints which
- level locks it and changes nothing. When a whole document is saved, locked
- keys are left out, and Hermes prints a note listing them.
+ guarded against concurrent edits. If another process changed the profile
+ meanwhile, Hermes re-reads it and applies only its own change on top, never
+ undoing the other one. Removing a key removes it from the profile level only:
+ if an upper level also sets it, that value still applies, and Hermes warns
+ you.
+- **Locked keys are refused.** `hermes config set` (and a single setting
+ changed in the TUI or desktop app) on a locked key reports which level locks
+ it and changes nothing. When a whole document is saved, locked keys are left
+ out, and Hermes prints a note listing them.
- **Secrets stay out of the plane.** A secret-shaped key such as `api_key`
accepts only a `${VAR}` reference, never the secret itself. Put the value in
`.env` or a secret source and set the reference:
@@ -76,7 +87,11 @@ start.
- **Schema migrations run in memory.** A document written by an older Hermes is
migrated when it is read, and the result is not written back. A later write
from a newer Hermes keeps the profile's stored schema version unless that
- data is already current, so every later reader still migrates it.
+ data is already current, so every later reader still migrates it. For the
+ same reason, setting a value that one of those migrations rewrites (for
+ example `compression.threshold_tokens: 256000`, an old default, on a profile
+ stored at an older schema) is refused with a message naming the key: it would
+ be saved and then never read back.
- **Unknown keys** in the fetched document are ignored with a warning, for
example a key added by a newer Hermes.
@@ -92,7 +107,8 @@ warns at startup, and `hermes doctor` flags the file. The managed
These commands copy or edit `config.yaml` and are refused in remote mode:
- `hermes config edit` (use `hermes config set` instead)
-- profile clone and profile distribution install
+- profile clone, profile rename (the plane keeps a profile's settings under its
+ name), and profile distribution install
- `hermes backup`, `hermes import`, and snapshot restore
- `hermes gateway --config <file>`