Skip to content

HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.

conformance: the guard follows every way a check holds an answer, and the suite has one version (2026.10.4.post9) (#414)

github-actions[bot]d40da2092358
diff --git a/protocol/CHANGELOG.md b/protocol/CHANGELOG.md
index 983c0eb3..ad3ff4a5 100644
--- a/protocol/CHANGELOG.md
+++ b/protocol/CHANGELOG.md
@@ -2,6 +2,28 @@
All notable changes to the Unified Harness Protocol.
+## Conformance suite 2026.10.4.post9 (2026-10-06)
+
+No change to the protocol, the schema or any verdict a conformant server gets.
+
+A check that quotes the server's answer in its failure message could not read it. Eighteen messages,
+reached by eight of the nine Environments checks and all ten Memories checks (in the check itself or
+in a helper they share), were built from `r.text`, and a client result had `body`, `json` and
+`header()` but no `text`. A message is
+evaluated only when its assertion fails, so nothing showed against a conformant server. Against a
+server that really refused the call, the check raised `AttributeError`, the suite filed it as ERROR,
+"the check itself broke", and the server's own sentence about why it refused never reached the
+report: a real non-conformance read as a defect in the suite. A result now has `text`, the body
+decoded and never raising, and those checks FAIL with the status and the server's words. Reported
+and fixed by @Jiaqiiii523 (#411), with a test that reads the checks and refuses any attribute a
+result does not have; it now also follows an answer that comes back from a helper and one read
+straight off the call, which between them are a third of the answers the checks read.
+
+A report names the suite that produced it. `suite_version` was a second copy of the version and
+had not moved since `2026.9.12.post4`, so every report written by `2026.10.4` through
+`2026.10.4.post8` carried that older name. The version is now written once, in the package, and
+the build reads it from there. No report published under `reports/` was written in that window.
+
## 2026-10-04 (patched the same day)
Additive to `2026-09-28`: every request and object valid under the previous version is valid here.
diff --git a/protocol/conformance/pyproject.toml b/protocol/conformance/pyproject.toml
index 38233143..b791c843 100644
--- a/protocol/conformance/pyproject.toml
+++ b/protocol/conformance/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "uhp-conformance"
-version = "2026.10.4.post8"
+dynamic = ["version"]
description = "Conformance suite for the Unified Harness Protocol (UHP)"
readme = "README.md"
requires-python = ">=3.10"
@@ -21,5 +21,9 @@ build-backend = "setuptools.build_meta"
[tool.setuptools]
packages = ["uhp_conformance"]
+# The version is the package's own `__version__`, the string a report prints. One place to change.
+[tool.setuptools.dynamic]
+version = { attr = "uhp_conformance.__version__" }
+
[tool.setuptools.package-data]
uhp_conformance = ["uhp-*.schema.json"]
diff --git a/protocol/conformance/tests/test_failure_evidence.py b/protocol/conformance/tests/test_failure_evidence.py
index bb36f3f2..4f9febe5 100644
--- a/protocol/conformance/tests/test_failure_evidence.py
+++ b/protocol/conformance/tests/test_failure_evidence.py
@@ -45,37 +45,121 @@ def _dotted(node: ast.AST) -> str:
return ".".join(reversed(parts))
-def _reads_in_scope(func: ast.AST) -> tuple[set[str], set[str]]:
- """One function's own answer variables and the attributes read straight off them.
-
+def _is_answer(node: ast.AST, helpers: set[str]) -> bool:
+ """A call that hands back a Result: a client verb, or a helper of the module that returns one."""
+ if not isinstance(node, ast.Call):
+ return False
+ parts = _dotted(node.func).split(".")
+ if len(parts) >= 2 and parts[-1] in RESULT_VERBS and parts[-2] == "client":
+ return True
+ return isinstance(node.func, ast.Name) and node.func.id in helpers
+
+
+def _bound(func: ast.AST, helpers: set[str]) -> set[str]:
+ """The names one function gives an answer to. Gathered over the whole function before any read
+ is looked at: an assignment three blocks deep comes later in a walk than a read one block deep,
+ and a scan that binds as it goes walks past that read."""
+ names = set()
+ for node in ast.walk(func):
+ targets = (node.targets if isinstance(node, ast.Assign)
+ else [node.target] if isinstance(node, (ast.AnnAssign, ast.NamedExpr)) else [])
+ if targets and _is_answer(getattr(node, "value", None), helpers):
+ names |= {t.id for t in targets if isinstance(t, ast.Name)}
+ return names
+
+
+def _functions(tree: ast.AST) -> list:
+ return [n for n in ast.walk(tree) if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))]
+
+
+def _answer_helpers(tree: ast.AST) -> set[str]:
+ """The module's functions that return a Result: one that returns a client call, a name bound
+ to one, or another such helper's answer. Repeated until nothing is added, so a helper that
+ wraps a helper counts."""
+ helpers: set[str] = set()
+ while True:
+ found = set()
+ for func in _functions(tree):
+ if func.name in helpers:
+ continue
+ bound = _bound(func, helpers)
+ for node in ast.walk(func):
+ if isinstance(node, ast.Return) and node.value is not None and (
+ _is_answer(node.value, helpers)
+ or (isinstance(node.value, ast.Name) and node.value.id in bound)):
+ found.add(func.name)
+ if not found:
+ return helpers
+ helpers |= found
+
+
+def _scan(source: str) -> tuple[int, set[str]]:
+ """How many answers the source reads, and every attribute it reads off one.
+
+ Three ways the checks hold an answer, all of them looked at: a name assigned from a client
+ call, a name assigned from a helper that returns one, and no name at all (`client.get(p).json`).
Scoped per function on purpose: `e` is a Result in one and a dict in the next, and a
module-wide name set would blame whichever pairing happens to be read later.
"""
- bound, read = set(), set()
- for node in ast.walk(func):
- if isinstance(node, ast.Assign) and isinstance(node.value, ast.Call):
- parts = _dotted(node.value.func).split(".")
- if len(parts) >= 2 and parts[-1] in RESULT_VERBS and parts[-2] == "client":
- for target in node.targets:
- if isinstance(target, ast.Name):
- bound.add(target.id)
- elif isinstance(node, ast.Attribute) and isinstance(node.value, ast.Name):
- if node.value.id in bound:
+ tree = ast.parse(source)
+ helpers = _answer_helpers(tree)
+ answers, read = 0, set()
+ for func in _functions(tree):
+ bound = _bound(func, helpers)
+ answers += len(bound)
+ for node in ast.walk(func):
+ if not isinstance(node, ast.Attribute):
+ continue
+ if isinstance(node.value, ast.Name) and node.value.id in bound:
read.add(node.attr)
- return bound, read
+ elif _is_answer(node.value, helpers):
+ answers += 1
+ read.add(node.attr)
+ return answers, read
+
+
+def _members() -> set[str]:
+ return set(Result.__dataclass_fields__) | {n for n in dir(Result) if not n.startswith("__")}
def test_everything_the_checks_read_off_a_result_is_something_a_result_has():
"""A typo in a failure message waits until a server misbehaves to surface, so catch it here."""
- tree = ast.parse(inspect.getsource(checks))
- members = set(Result.__dataclass_fields__) | {n for n in dir(Result) if not n.startswith("__")}
- total, missing = 0, set()
- for func in [n for n in ast.walk(tree) if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))]:
- bound, read = _reads_in_scope(func)
- total += len(bound)
- missing |= read - members
- assert total, "the scan bound no client response anywhere, which means it stopped working"
- assert not missing, f"checks read {sorted(missing)} off a Result, which has only {sorted(members)}"
+ answers, read = _scan(inspect.getsource(checks))
+ assert answers, "the scan found no client response anywhere, which means it stopped working"
+ missing = read - _members()
+ assert not missing, f"checks read {sorted(missing)} off a Result, which has only {sorted(_members())}"
+
+
[email protected]("source", [
+ # a name assigned from a client call, the read inside a message that never runs
+ "def c(ctx):\n r = ctx.client.get('/x')\n assert r.status == 200, r.txt\n",
+ # the assignment deeper in the function than the read: a scan that binds as it walks misses it
+ "def c(ctx, ps):\n if ps:\n for p in ps:\n r = ctx.client.get(p)\n return r.txt\n",
+ # the answer comes back from a helper of the module
+ "def _h(ctx):\n return ctx.client.post('/x')\n\ndef c(ctx):\n r = _h(ctx)\n assert r.status == 200, r.txt\n",
+ # ...or from a helper that wraps that helper, or one that returns a name it bound
+ "def _h(ctx):\n r = ctx.client.post('/x')\n return r\n\ndef _g(ctx):\n return _h(ctx)\n\n"
+ "def c(ctx):\n done = _g(ctx)\n assert done.status == 200, done.txt\n",
+ # no name at all: the attribute is read straight off the call
+ "def c(ctx):\n return ctx.client.get('/x').txt\n",
+ "def _h(ctx):\n return ctx.client.post('/x')\n\ndef c(ctx):\n return _h(ctx).txt\n",
+ # an annotated assignment and an assignment expression bind like a plain one
+ "def c(ctx):\n r: object = ctx.client.get('/x')\n return r.txt\n",
+ "def c(ctx):\n if (r := ctx.client.get('/x')).status:\n return r.txt\n",
+], ids=["assigned", "bound-deeper-than-read", "from-a-helper", "helper-of-a-helper", "straight-off-the-call",
+ "straight-off-a-helper", "annotated", "assignment-expression"])
+def test_the_scan_sees_every_way_a_check_holds_an_answer(source):
+ """The scan is only worth its place if a wrong name cannot hide behind how the answer is held."""
+ answers, read = _scan(source)
+ assert answers and "txt" in read, (answers, read)
+ assert "txt" not in _members()
+
+
+def test_the_scan_blames_nothing_that_is_not_an_answer():
+ """A dict's own `get`, and a name that is an answer in one function and a dict in the next."""
+ source = ("def a(ctx):\n e = ctx.client.get('/x')\n return e.status\n\n"
+ "def b(d):\n e = d.get('k')\n e.items()\n return d.get('x').keys()\n")
+ assert _scan(source) == (1, {"status"})
def _run(check_id: str, base: str):
diff --git a/protocol/conformance/tests/test_schema_distribution.py b/protocol/conformance/tests/test_schema_distribution.py
index 1ddc9d4c..55ad682c 100644
--- a/protocol/conformance/tests/test_schema_distribution.py
+++ b/protocol/conformance/tests/test_schema_distribution.py
@@ -54,6 +54,22 @@ def distributions(tmp_path_factory):
return {"wheel": next(output.glob("*.whl")), "sdist": next(output.glob("*.tar.gz"))}
+def test_the_built_distribution_carries_the_version_a_report_prints(distributions):
+ """One version, in the package, read by the build. It was written twice, and for nine revisions
+ only the packaging copy moved: every report of 2026.10.4 to 2026.10.4.post8 named itself
+ 2026.9.12.post4. A report is evidence, and evidence names the suite that produced it."""
+ from uhp_conformance import __version__
+ assert distributions["wheel"].name == f"uhp_conformance-{__version__}-py3-none-any.whl"
+ assert distributions["sdist"].name == f"uhp_conformance-{__version__}.tar.gz"
+
+
+def test_the_version_is_written_in_one_place():
+ packaging = (PROJECT / "pyproject.toml").read_text()
+ project = packaging.split("[project]", 1)[1].split("\n[", 1)[0]
+ assert 'dynamic = ["version"]' in project and "\nversion" not in project, project
+ assert 'version = { attr = "uhp_conformance.__version__" }' in packaging
+
+
@pytest.mark.parametrize("kind", ["wheel", "sdist"])
def test_installed_distribution_validates_instead_of_skipping(distributions, tmp_path, kind):
target = tmp_path / "installed"
diff --git a/protocol/conformance/uhp_conformance/__init__.py b/protocol/conformance/uhp_conformance/__init__.py
index 0dcc969f..b1d7277c 100644
--- a/protocol/conformance/uhp_conformance/__init__.py
+++ b/protocol/conformance/uhp_conformance/__init__.py
@@ -1,5 +1,9 @@
"""UHP conformance suite — the definition of "conformant" for the Unified Harness Protocol."""
-__version__ = "2026.9.12.post4"
+# The suite's one version: the build reads it from here (pyproject.toml, [tool.setuptools.dynamic])
+# and a report prints it as `suite_version`, so a report always names the code that produced it.
+# It used to be written twice, and from 2026.10.4 to 2026.10.4.post8 only the packaging copy moved:
+# every report of those nine revisions called itself 2026.9.12.post4.
+__version__ = "2026.10.4.post9"
UHP_VERSION = "2026-10-04"
# The MCP server the plugin checks point a plugin at. It must resolve and answer: a host is
# allowed to refuse a server it cannot reach at configuration time and record that in `skipped`,
diff --git a/protocol/conformance/uv.lock b/protocol/conformance/uv.lock
index 46f96b82..9039a928 100644
--- a/protocol/conformance/uv.lock
+++ b/protocol/conformance/uv.lock
@@ -320,7 +320,6 @@ wheels = [
[[package]]
name = "uhp-conformance"
-version = "2026.9.12"
source = { editable = "." }
dependencies = [
{ name = "jsonschema" },