HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.
Python agent-apiagent-frameworkagent-harnessai-agentsai-infrastructurecoding-agentconformance-testingdeepseek-harnessharness-engineeringhermesllmopen-protocolopen-sourceopen-standardpi-coding-agentself-hosteduhpunified-apiunified-harness-protocolunified-interface
chore(conformance): a workflow that keeps the recorded run honest
richard-epsillaaebccba7d842
diff --git a/.github/workflows/conformance-remeasure.yml b/.github/workflows/conformance-remeasure.ymlnew file mode 100644index 00000000..f174166d--- /dev/null+++ b/.github/workflows/conformance-remeasure.yml@@ -0,0 +1,102 @@+# Re-measure the reference implementation against the FULL conformance suite, and open a PR that+# replaces the recorded-run block in protocol/conformance/README.md with what was actually measured.+#+# This exists because that block goes stale silently: every new check makes the recorded number one+# the repository can no longer reproduce, and a conformance README whose own headline figure is+# unreproduced is lying about the thing it exists to establish. Regenerating it by hand invites the+# same drift back; regenerating it from a live run, in a PR anyone can inspect before it lands,+# keeps the number honest.+#+# The PR is authored by github-actions[bot], so the record of "who measured this" is the workflow+# itself rather than whoever happened to trigger it. A red run opens no PR: a suite that failed+# must never retire the number it failed to reproduce.+#+# Dispatch inputs name the target; UHP_TARGET_KEY (repository secret) is the API key. The default+# target is the public self-hosted reference instance.+name: conformance-remeasure++on:+ workflow_dispatch:+ inputs:+ base_url:+ description: 'Server root to measure (a live reference deployment)'+ default: 'https://20-98-237-6.sslip.io/api/harness'+ server_label:+ description: 'How the README names what was measured'+ default: 'HarnessRouter Community Edition'++permissions:+ contents: write+ pull-requests: write++jobs:+ remeasure:+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with:+ python-version: "3.12"+ - run: pip install -e protocol/conformance++ # --plain: the transcript goes into a README, and ANSI codes in a code block are noise.+ # The CLI exits 1 on any FAIL/ERROR, which fails this step and stops the PR from opening.+ - name: Run the full suite against the live reference+ env:+ UHP_API_KEY: ${{ secrets.UHP_TARGET_KEY }}+ run: |+ uhp-conformance --base-url '${{ inputs.base_url }}' --class full --plain \+ --json report.json | tee run.txt++ - name: Splice the measured result into the README+ run: |+ python3 - <<'PY'+ import json, re, datetime++ report = json.load(open("report.json"))+ out = open("run.txt").read()+ # The Summary block, verbatim from the tool: the README shows what the tool said,+ # not a paraphrase of it.+ m = re.search(r"( {2}Summary\n(?: {4}.*\n?)+)", out)+ assert m, "no Summary block in the suite output"+ summary = m.group(1).rstrip()++ day = datetime.date.today().isoformat()+ version = report.get("suite_version", "")+ block = f"""<!-- conformance-recorded-run:start — regenerated by .github/workflows/conformance-remeasure.yml; do not edit by hand -->+ ${{ inputs.server_label }}, the reference implementation in this repository, run against a+ live instance on {day} (suite {version}):++ ```+ {summary}+ ```+ <!-- conformance-recorded-run:end -->"""+ # The heredoc indents the template; the README wants it flush left.+ block = "\n".join(ln[10:] if ln.startswith(" " * 10) else ln for ln in block.split("\n"))++ p = "protocol/conformance/README.md"+ s = open(p).read()+ new = re.sub(r"<!-- conformance-recorded-run:start.*?conformance-recorded-run:end -->",+ block, s, count=1, flags=re.S)+ assert new != s, "the recorded-run markers were not found"+ open(p, "w").write(new)+ PY++ - name: Open the PR+ env:+ GH_TOKEN: ${{ github.token }}+ run: |+ set -euo pipefail+ git config user.name "github-actions[bot]"+ git config user.email "41898282+github-actions[bot]@users.noreply.github.com"+ BRANCH="conformance/remeasure-$(date -u +%Y%m%d-%H%M)"+ git checkout -b "$BRANCH"+ git add protocol/conformance/README.md+ git commit -m "conformance: retire the stale recorded run for one actually measured++ Measured now, against ${{ inputs.base_url }}, by the conformance-remeasure+ workflow. The transcript in the README is the tool's own output; a run with+ any failure opens no PR at all."+ git push origin "$BRANCH"+ gh pr create --title "conformance: retire the stale recorded run for one actually measured" \+ --body "$(printf 'Automated remeasure of the reference implementation, full class, against `%s`.\n\nThe recorded-run block in `protocol/conformance/README.md` is replaced with the transcript of a run that just happened, so the headline number is one this repository has actually reproduced. Opened by the `conformance-remeasure` workflow; a run with any FAIL or ERROR fails the job and opens nothing.\n\n```\n%s\n```' '${{ inputs.base_url }}' "$(sed -n '/^ Summary/,$p' run.txt)")"diff --git a/protocol/conformance/README.md b/protocol/conformance/README.mdindex 33a7d7c7..c796d5f0 100644--- a/protocol/conformance/README.md+++ b/protocol/conformance/README.md@@ -108,6 +108,7 @@ for a conformance claim and its reader is frequently not the person who ran it:## Reference implementation results+<!-- conformance-recorded-run:start — regenerated by .github/workflows/conformance-remeasure.yml; do not edit by hand -->HarnessRouter Community Edition 0.3.0, the reference implementation in this repository, run againsta live instance on 2026-08-11:@@ -117,9 +118,12 @@ a live instance on 2026-08-11:CONFORMANT — UHP 2026-08-11 (full)```-That run predates the `T-` additions and the `R-` series, so it is a 52-check result. Both have-since been measured against the reference implementation directly, live on a self-hosted-instance, and both measured a real defect before passing:+That run predates the `T-` additions and the `R-` series, so it is a 52-check result of the+62-check suite. A remeasured run replaces this block via the `conformance-remeasure` workflow.+<!-- conformance-recorded-run:end -->++Both new series were measured against the reference implementation directly, live on a+self-hosted instance, and both measured a real defect before passing:- `R-01`…`R-07` all pass. Against the reference as it stood, all seven skipped (its sharedialect wanted `{"enabled": true}` and published no view URL), and behind those skips sat a