HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.
Python agent-apiagent-frameworkagent-harnessai-agentsai-infrastructurecoding-agentconformance-testingdeepseek-harnessharness-engineeringhermesllmopen-protocolopen-sourceopen-standardpi-coding-agentself-hosteduhpunified-apiunified-harness-protocolunified-interface
runner: hermes checks its work once, before it answers, and keeps the check out of the reply (#526)
github-actions[bot]dade07dfab6e
diff --git a/runner/server.py b/runner/server.pyindex 20e1ea67..99e8b932 100644--- a/runner/server.py+++ b/runner/server.py@@ -1883,6 +1883,21 @@ def _plugin_launchers(cwd: str, servers: list[dict] | None) -> list[dict]:_AGENTS_BEGIN = "<!-- harness-skills:begin -->"_AGENTS_END = "<!-- harness-skills:end -->"+# What replaces hermes' own stop-time check, which _hermes_prepare_env switches off. The check+# stays (hermes' coding prompt asks for it too); what changes is when it runs and who hears about+# it. Before the final reply, once, so no second answer follows the first; and out of the reply,+# which is about the person's work: the person never asked how the agent checked itself, and a+# reply about test commands, validators and a script under some path is runtime noise to them.+_HERMES_CHECK_SECTION = [+ "## Checking your work", "",+ "If you changed code or data files, check them before your final reply: run the project's "+ "own test or build command when it has one, otherwise a short script you keep in `tmp/`. "+ "Fix what the check finds. Check before you answer, never after: your reply ends the turn.", "",+ "The check is for you, not the user. Your reply is about their work: what you made or "+ "changed, and anything they need to do next. Leave out tests, linters, validators, check "+ "scripts and their paths, and whether the workspace has a test command, unless the user "+ "asked about them. If something still does not work, say what, in terms of their work.", ""]+def _agent_doc_path(cwd: str, backend: str) -> pathlib.Path:"""The agent's instruction file: AGENTS.md for Codex, Hermes, Pi, dsh and opencode (all read@@ -1966,7 +1981,14 @@ def _write_agent_doc(cwd: str, backend: str, agent_doc: str | None, skills_meta:relative paths by habit, and the first dsh deck task (deepseek-v4-pro, 2026-08-21) saved to theworkspace PARENT, the user saw an empty turn, and three turns went to copying files into view.An instruction is the right mechanism here: writes cannot be walled in a sandbox whose point is- real bash, and widening collection would ship every scratch file as a deliverable."""+ real bash, and widening collection would ship every scratch file as a deliverable.++ Scratch files have a place INSIDE the workspace, `tmp/`, which is already what the platform+ treats as scratch: git-ignored (_git_ensure), so never a produced file; left out of the+ checkpoint (CHECKPOINT_EXCLUDE); hidden from the gateway's file listing. Without it the rule+ read as "writes are restricted to the workspace" while nothing enforces that, and hermes, told+ by its own stop-time check to write under /tmp, spent tool calls and its reply on the conflict+ (2026-10-19, see _hermes_prepare_env). One rule, and the reality agrees with it."""if backend in ("systemone", "qoder"):return # reads no workspace; its instructions travel in the job (_build_systemone, _build_qoder)p = _agent_doc_path(cwd, backend)@@ -1974,9 +1996,13 @@ def _write_agent_doc(cwd: str, backend: str, agent_doc: str | None, skills_meta:lines = [_AGENTS_BEGIN, "## Workspace", "","Your working directory is this task's workspace and the ONLY place the user can see ""files. Save every deliverable (documents, decks, code, exports) to a relative path "- "under it. Never write output to an absolute path outside it (/tmp, /app, the "- "workspace parent directory, or your home directory): those files are not collected, "- "and the user will never see them.", ""]+ "under it. Files you need only while working (a quick check script, intermediate "+ "data) go in its `tmp/` folder, which is never shown to the user or saved with the "+ "work. Keep every file you write inside the workspace: nothing written outside it "+ "(/tmp, /app, the workspace parent directory, or your home directory) is collected, "+ "and the user will never see it.", ""]+ if backend == "hermes":+ lines += _HERMES_CHECK_SECTIONlines += environments.doc_section(environment)if skills_meta:lines += ["## Available skills", "",@@ -6172,9 +6198,23 @@ def _hermes_prepare_env(provider: str | None, auth: Auth, cwd: str, env: dict,cfg["model"]["api_mode"] = api_modeif p == "bedrock":cfg["bedrock"] = {"region": auth.aws_region or "us-east-1"}+ # hermes' verify-on-stop, OFF. It is a second turn after the final answer: when the turn edited+ # code and hermes saw no check it recognises, it keeps the answer, appends a nudge and runs the+ # model again, up to twice a turn (agent/verification_stop.py, max_attempts=2). The nudge tells+ # the model to write a `hermes-verify-` script under the OS temp dir and to "summarize it+ # explicitly as ad-hoc verification rather than suite green", and its ledger counts that script+ # only OUTSIDE the project, so one the model moved into the workspace (as the agent doc's+ # Workspace rule asks) left the turn unverified and drew the second nudge. Every nudged answer+ # is already in state.db, which this runner reads as the reply: two answers, an extra model+ # call, and the check narrated to the person (staging kits, and a QA company's design reviews+ # posted as card comments, 2026-10-19). Upstream turned it off by default in its config v31,+ # but only by migrating an installed config; this file is written fresh every turn, so the+ # schema default "auto" (on for the CLI) applied. The check itself moves into the agent doc,+ # before the answer and out of it (_HERMES_CHECK_SECTION).+ cfg["agent"] = {"verify_on_stop": False}if max_turns:# hermes 0.19.0 has no --max-turns CLI flag; agent.max_turns in config.yaml is the knob.- cfg["agent"] = {"max_turns": int(max_turns)}+ cfg["agent"]["max_turns"] = int(max_turns)if vision_auth and vision_auth.get("model"):# hermes asks its image questions separately from the conversation (vision_analyze, the# browser tools) and, left alone, asks the model the harness writes with. That made adiff --git a/runner/tests/test_hermes_quiet_check.py b/runner/tests/test_hermes_quiet_check.pynew file mode 100644index 00000000..67cca9ee--- /dev/null+++ b/runner/tests/test_hermes_quiet_check.py@@ -0,0 +1,87 @@+"""hermes checks its work once, before it answers, and keeps the check out of the reply.++What this pins (2026-10-19): hermes' own verify-on-stop ran a second turn after the final answer,+told the model to write a `hermes-verify-` script under the OS temp dir while the agent doc said+files belong in the workspace, and asked for the check to be summarised to the person. The+person read about test commands and /tmp, and on one card got two answers."""+import subprocess+import sys+from pathlib import Path++import yaml++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))+import server # noqa: E402+++def _hermes_cfg(tmp_path, **kw) -> dict:+ env = {"HOME": str(tmp_path)}+ server._hermes_prepare_env("bedrock", server.Auth(), str(tmp_path), env, model="m", **kw)+ return yaml.safe_load((tmp_path / ".hermes" / "config.yaml").read_text())+++def test_no_verify_turn_after_the_final_answer(tmp_path):+ """verify-on-stop is the one thing that runs the model again after it has answered. It is off+ in the file hermes reads, explicitly: left out, hermes' schema default ("auto") turns it on."""+ assert _hermes_cfg(tmp_path)["agent"]["verify_on_stop"] is False+++def test_max_turns_rides_beside_it(tmp_path):+ agent = _hermes_cfg(tmp_path, max_turns=7)["agent"]+ assert agent == {"verify_on_stop": False, "max_turns": 7}+++def _doc(tmp_path, backend: str) -> str:+ server._write_agent_doc(str(tmp_path), backend, "", [])+ return server._agent_doc_path(str(tmp_path), backend).read_text()+++def test_one_file_rule_scratch_goes_inside_the_workspace(tmp_path):+ """Scratch has one place, `tmp/` in the workspace, and nothing points anywhere else: no+ instruction to write under /tmp beside one that says files outside are lost."""+ doc = _doc(tmp_path, "hermes")+ assert "`tmp/` folder" in doc and "Keep every file you write inside the workspace" in doc+ # /tmp is named once, as a place outside the workspace, never as a place to write to+ assert doc.count("/tmp") == 1 and "(/tmp, /app" in doc+ assert "hermes-verify" not in doc and "tempfile" not in doc+ # the config carries no path rule of its own+ _hermes_cfg(tmp_path)+ assert "tmp" not in (tmp_path / ".hermes" / "config.yaml").read_text()+++def test_the_scratch_folder_is_scratch_in_fact(tmp_path):+ """The instruction and the runtime agree: a file in `tmp/` is never a produced file and never+ travels in the checkpoint, while a deliverable beside it is collected."""+ ws = tmp_path / "ws"+ ws.mkdir()+ (ws / "tmp").mkdir()+ (ws / "tmp" / "check.py").write_text("print('ok')\n")+ (ws / "report.md").write_text("# report\n")+ paths = [f["path"] for f in server._produced_list(str(ws))]+ assert paths == ["report.md"]+ assert "./tmp" in server.CHECKPOINT_EXCLUDE+ ignored = subprocess.run(["git", "-C", str(ws), "check-ignore", "-q", "tmp/check.py"])+ assert ignored.returncode == 0+++def test_the_check_stays_out_of_the_reply(tmp_path):+ doc = _doc(tmp_path, "hermes")+ assert "## Checking your work" in doc+ assert "before your final reply" in doc and "never after" in doc+ assert "The check is for you, not the user" in doc+ for kept_out in ("tests", "linters", "validators", "check scripts and their paths",+ "whether the workspace has a test command"):+ assert kept_out in doc+ # inside the managed block, so a harness's own doc cannot be overwritten by it or shadow it+ begin, end = doc.index(server._AGENTS_BEGIN), doc.index(server._AGENTS_END)+ assert begin < doc.index("## Checking your work") < end+++def test_the_check_section_is_hermes_only(tmp_path):+ """It replaces hermes' own stop-time check; no other base had one."""+ for backend in ("codex", "claude", "pi"):+ d = tmp_path / backend+ d.mkdir()+ doc = _doc(d, backend)+ assert "## Checking your work" not in doc+ assert "`tmp/` folder" in doc # the file rule itself is every base's