Skip to content

HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.

conformance: the plugin checks point at a server that answers, and P-11 proves the agent reached it (#300)

github-actions[bot]cf21fc00ee37
diff --git a/protocol/CHANGELOG.md b/protocol/CHANGELOG.md
index 8258007c..f1638792 100644
--- a/protocol/CHANGELOG.md
+++ b/protocol/CHANGELOG.md
@@ -2,6 +2,23 @@
All notable changes to the Unified Harness Protocol.
+## Conformance suite 2026.9.12.post3 (2026-09-27)
+
+The protocol is unchanged; the suite that measures it is.
+
+- The plugin checks point a plugin at a server that resolves and answers. A host may refuse an
+ MCP server it cannot reach at configuration time and record that in `skipped`; with the
+ unresolvable placeholder the checks used before, such a host failed P-02, P-04, P-06 and P-07
+ for refusing the placeholder rather than for anything about plugins (the hosted HarnessRouter,
+ 2026-09-27). The suite ships the server (`uhp-conformance-fixture`, `pip install
+ "uhp-conformance[fixture]"`), defaults to a public copy of it, and takes `--plugin-mcp-url`
+ for a copy the host under test can reach.
+- **P-11**, new: a plugin's MCP server answers the agent's tool call. The agent is asked to call
+ the fixture's tool with a nonce and the fixture's answer, a function of the nonce, must come
+ back. It is the only plugin check that runs a task, and the only one that shows an agent
+ reaching a plugin's server rather than a server deriving and refusing correctly.
+- Measured implementations are re-measured under this suite before their badges are shown again.
+
## 2026-09-12
Additive to `2026-08-11`: every request and object valid under the previous version is valid
diff --git a/protocol/conformance/README.md b/protocol/conformance/README.md
index dfe9e3d6..6f79cecb 100644
--- a/protocol/conformance/README.md
+++ b/protocol/conformance/README.md
@@ -34,26 +34,27 @@ check.
| `--harness-id` | Run tasks against a specific harness instead of the first one listed. |
| `--model` | Run tasks with a specific model. Required when the harness named by `--harness-id` has no default model: the suite says so and stops before the first task rather than waiting out the task timeout on each. |
| `--task-timeout` | Seconds to allow one agent task. Default 300. |
+| `--plugin-mcp-url` | A reachable streamable-HTTP MCP server the plugin checks point a plugin at, or set `UHP_PLUGIN_MCP_URL`. Defaults to the public copy of the suite's own fixture (below). |
| `--only` | Comma-separated check ids, for iterating on one failure. |
| `--json` | Write a machine-readable report. |
| `--plain` | No ANSI colour, for CI logs. |
Exit code is `0` when nothing failed and `1` otherwise, so it drops into CI unchanged.
-**The suite runs real agent tasks.** It sends about six of them, which costs real model tokens and a
+**The suite runs real agent tasks.** It sends about seven of them, which costs real model tokens and a
few minutes. That is deliberate: the defects worth catching — a stream that never flushes, a
cancellation that never terminates, an artifact that cannot be downloaded — are invisible to
anything that only inspects a schema.
## What it checks
-75 checks across three classes.
+76 checks across three classes.
| Class | Checks | Covers |
|---|---|---|
| **Core** | 40 | Discovery, version negotiation, authentication, the error envelope, harnesses, models, task execution (streaming and not), the event stream, sessions, cancellation, reserved request fields |
| **Extended** | +8 | Session listing and inspection, file input, artifacts, download headers, path-traversal probes |
-| **Full** | +25 | Harness create / update / delete, refusal of an unsupported base, skill-folder round trip, MCP and disabled-tool persistence, session sharing, plugins (package round trip, derivation, refusals, export) |
+| **Full** | +26 | Harness create / update / delete, refusal of an unsupported base, skill-folder round trip, MCP and disabled-tool persistence, session sharing, plugins (package round trip, derivation, refusals, export, and a tool call through a plugin's server) |
Every check names the section of the specification it enforces, so a failure points at the sentence
it violates rather than at a test name.
@@ -83,6 +84,11 @@ A few of them are worth calling out, because they catch things a schema check ne
mistake that makes every read-then-write install them twice; P-06 fails an export that carries
the operator's credentials, and one that omits them without saying so. Both series skip, never
fail, on a server that reports the `plugins` capability false, because the chapter is optional.
+- **P-11** — the one plugin check that runs a task. It installs a plugin whose only server is the
+ fixture the run was pointed at, asks the agent to call the fixture's tool with a nonce, and
+ expects the fixture's answer back (a function of the nonce the agent cannot produce without
+ calling it). P-02 to P-10 read what the server derived and refused; only this one shows an
+ agent reaching a plugin's server at run time.
- **T-08/T-09/T-10** — the reserved fields `tools` and `include` are accepted, and reported as
ignored. Both halves matter: before these, the suite sent neither field, so a server that
rejected them outright and a server that silently acted on them scored the same as a correct
@@ -107,6 +113,27 @@ A few of them are worth calling out, because they catch things a schema check ne
`SKILL.md`, or that empties a bundle when the harness is renamed, passes every other check: the
config still looks right, and the loss only shows up later as an agent behaving oddly.
+## The plugin fixture
+
+Every plugin check points a plugin at an MCP server, and a host is allowed to refuse a server it
+cannot reach at configuration time and record that in `skipped`. Until 2026.9.12.post3 that server
+was an unresolvable placeholder, so on such a host the checks measured the refusal instead of the
+plugin. The suite now ships the server it needs:
+
+```bash
+pip install "uhp-conformance[fixture]"
+uhp-conformance-fixture --host 0.0.0.0 --port 8080 # serves http://<host>:8080/mcp
+uhp-conformance --base-url https://your-uhp-server --api-key "$UHP_API_KEY" --class full \
+ --plugin-mcp-url https://fixture.example.com/mcp
+```
+
+It is a streamable-HTTP MCP server with two tools: `uhp_echo`, whose answer is a function of its
+input (the prefix `UHP-FIXTURE:` and the text reversed), which is how P-11 proves the agent called
+it; and `uhp_time`. A run defaults to a public copy of the same code (`DEFAULT_PLUGIN_MCP_URL` in
+the package; `fixture/Dockerfile` is how it is built), reachable from any host with outbound HTTPS.
+A host under test whose agents cannot reach the public internet needs a copy it can reach, and the
+address goes on `--plugin-mcp-url`.
+
## Outcomes
| Outcome | Meaning |
diff --git a/protocol/conformance/fixture/Dockerfile b/protocol/conformance/fixture/Dockerfile
new file mode 100644
index 00000000..f4ee8882
--- /dev/null
+++ b/protocol/conformance/fixture/Dockerfile
@@ -0,0 +1,8 @@
+# The public copy of uhp-conformance-fixture: the suite's own package with its `fixture` extra.
+# Build from protocol/conformance as the context: docker build -f fixture/Dockerfile .
+FROM python:3.12-slim
+WORKDIR /src
+COPY . /src
+RUN pip install --no-cache-dir "/src[fixture]"
+EXPOSE 8080
+CMD ["uhp-conformance-fixture", "--host", "0.0.0.0", "--port", "8080"]
diff --git a/protocol/conformance/pyproject.toml b/protocol/conformance/pyproject.toml
index ea8f0460..fa16db3f 100644
--- a/protocol/conformance/pyproject.toml
+++ b/protocol/conformance/pyproject.toml
@@ -1,14 +1,18 @@
[project]
name = "uhp-conformance"
-version = "2026.9.12.post2"
+version = "2026.9.12.post3"
description = "Conformance suite for the Unified Harness Protocol (UHP)"
readme = "README.md"
requires-python = ">=3.10"
license = { text = "Apache-2.0" }
dependencies = ["jsonschema>=4.0"]
+[project.optional-dependencies]
+fixture = ["mcp>=1.10,<2"]
+
[project.scripts]
uhp-conformance = "uhp_conformance.cli:main"
+uhp-conformance-fixture = "uhp_conformance.fixture:main"
[build-system]
requires = ["setuptools>=68"]
diff --git a/protocol/conformance/reports/harnessrouter-ce/2026-09-27T2059Z-full.json b/protocol/conformance/reports/harnessrouter-ce/2026-09-27T2059Z-full.json
new file mode 100644
index 00000000..a6c8ec75
--- /dev/null
+++ b/protocol/conformance/reports/harnessrouter-ce/2026-09-27T2059Z-full.json
@@ -0,0 +1,635 @@
+{
+ "protocol": "uhp",
+ "protocol_version": "2026-09-12",
+ "suite_protocol_version": "2026-09-12",
+ "suite_version": "2026.9.12.post3",
+ "generated_at": "2026-09-27T20:59:52Z",
+ "target": "https://20-98-237-6.sslip.io/api/harness",
+ "target_label": "HarnessRouter CE 0.25.6 (hr-test)",
+ "implementation": {
+ "name": "HarnessRouter Community Edition",
+ "version": "0.25.6"
+ },
+ "requested_class": "full",
+ "conformant": true,
+ "conformant_with_skips": true,
+ "skipped_not_verified": [],
+ "highest_class_passed": "full",
+ "summary": {
+ "pass": 76,
+ "fail": 0,
+ "skip": 0,
+ "error": 0,
+ "total": 76
+ },
+ "checks": [
+ {
+ "id": "D-01",
+ "title": "Discovery document is served",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#2-capability-discovery",
+ "outcome": "pass",
+ "detail": "conformance_class=full versions=['2026-09-12', '2026-08-11']"
+ },
+ {
+ "id": "D-02",
+ "title": "Discovery is served without authentication",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#2-capability-discovery",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "D-03",
+ "title": "Discovery document matches the schema",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/schema.md",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "D-04",
+ "title": "default_version is one of versions",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#2-capability-discovery",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "D-05",
+ "title": "conformance_class agrees with capabilities",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#2-capability-discovery",
+ "outcome": "pass",
+ "detail": "class=full"
+ },
+ {
+ "id": "V-01",
+ "title": "UHP-Version header on every response",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#1-version-negotiation",
+ "outcome": "pass",
+ "detail": "UHP-Version: 2026-09-12"
+ },
+ {
+ "id": "V-02",
+ "title": "A supported version is honoured",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#1-version-negotiation",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "V-03",
+ "title": "An unsupported version is refused, not silently substituted",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#1-version-negotiation",
+ "outcome": "pass",
+ "detail": "supported=['2026-09-12', '2026-08-11']"
+ },
+ {
+ "id": "A-01",
+ "title": "Authenticated endpoints reject an absent credential",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/architecture.md#5-authentication",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "A-02",
+ "title": "Authenticated endpoints reject an invalid credential",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/architecture.md#5-authentication",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "E-01",
+ "title": "Errors use the structured envelope",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/errors.md#1-the-error-envelope",
+ "outcome": "pass",
+ "detail": "code=harness_not_found"
+ },
+ {
+ "id": "E-02",
+ "title": "Unknown harness reports harness_not_found",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/errors.md#31-request-and-routing",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "E-03",
+ "title": "Unknown response reports response_not_found",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/errors.md#31-request-and-routing",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "E-04",
+ "title": "Error messages carry no stack traces",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/errors.md#1-the-error-envelope",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "H-01",
+ "title": "Harness list is served and well formed",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#1-discovering-harnesses",
+ "outcome": "pass",
+ "detail": "53 harness(es)"
+ },
+ {
+ "id": "H-02",
+ "title": "A harness can be fetched by id",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#1-discovering-harnesses",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "H-03",
+ "title": "Model catalogue is served and availability is a boolean",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#3-models",
+ "outcome": "pass",
+ "detail": "688/701 models available"
+ },
+ {
+ "id": "H-04",
+ "title": "Per-harness model list is served",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#3-models",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "T-01",
+ "title": "A non-streaming task returns a valid Response",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/tasks.md#3-the-response-object",
+ "outcome": "pass",
+ "detail": "status=completed in 10.8s"
+ },
+ {
+ "id": "T-02",
+ "title": "A finished task is in a terminal state",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#3-task-lifecycle",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "T-03",
+ "title": "The response names the model that actually ran",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/tasks.md#13-model-selection-and-substitution",
+ "outcome": "pass",
+ "detail": "model=gpt-5.4"
+ },
+ {
+ "id": "T-04",
+ "title": "The response reports its session",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/lifecycle.md#4-session-lifecycle",
+ "outcome": "pass",
+ "detail": "session_id=hsess16f4b34984f9410fb9c836161caa24d5"
+ },
+ {
+ "id": "T-05",
+ "title": "usage is an object or explicitly null, never fabricated",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/tasks.md#3-the-response-object",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "T-06",
+ "title": "A stored response can be read back",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/tasks.md#4-reading-a-task-back",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "T-07",
+ "title": "Response ids are prefixed",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/architecture.md#3-object-model",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "S-01",
+ "title": "A streaming task emits events",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/streaming.md#1-opening-a-stream",
+ "outcome": "pass",
+ "detail": "9 events"
+ },
+ {
+ "id": "S-02",
+ "title": "Stream content type is text/event-stream",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/streaming.md#1-opening-a-stream",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "S-03",
+ "title": "Every event validates against the schema",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/schema.md",
+ "outcome": "pass",
+ "detail": "9 events valid"
+ },
+ {
+ "id": "S-04",
+ "title": "sequence_number starts at 0 and has no gaps",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/streaming.md#3-ordering-guarantees",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "S-05",
+ "title": "The first event is response.created",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/streaming.md#3-ordering-guarantees",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "S-06",
+ "title": "The stream ends with exactly one terminal event",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/streaming.md#4-terminal-events",
+ "outcome": "pass",
+ "detail": "terminal=response.completed"
+ },
+ {
+ "id": "S-07",
+ "title": "The terminal event carries the complete response",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/streaming.md#4-terminal-events",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "S-08",
+ "title": "Streaming and non-streaming agree on the result shape",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/streaming.md#6-non-streaming",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "S-09",
+ "title": "The stream is progressive, not buffered to the end",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/streaming.md#1-opening-a-stream",
+ "outcome": "pass",
+ "detail": "events spread over 8.0s"
+ },
+ {
+ "id": "C-01",
+ "title": "A task can continue a session",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/sessions.md#1-continuing-a-session",
+ "outcome": "pass",
+ "detail": "session hsess16f4b34984f9410fb9c836161caa24d5 extended"
+ },
+ {
+ "id": "C-02",
+ "title": "Cancelling a terminal task succeeds and changes nothing",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/sessions.md#4-cancelling",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "C-03",
+ "title": "A running task can be cancelled and ends non-running",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/sessions.md#4-cancelling",
+ "outcome": "pass",
+ "detail": "final status=cancelled"
+ },
+ {
+ "id": "X-01",
+ "title": "Sessions can be listed",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/sessions.md#2-listing-sessions",
+ "outcome": "pass",
+ "detail": "5 session(s)"
+ },
+ {
+ "id": "X-02",
+ "title": "Session listing reports the end of pagination explicitly",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/sessions.md#2-listing-sessions",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "X-03",
+ "title": "A session can be inspected",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/sessions.md#3-inspecting-a-session",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "X-04",
+ "title": "A session's turn history is available, in the specified shape",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/sessions.md#3-inspecting-a-session",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "X-05",
+ "title": "A file can be sent as task input",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/files.md#1-sending-files-in",
+ "outcome": "pass",
+ "detail": "the harness read the file and echoed the token"
+ },
+ {
+ "id": "X-06",
+ "title": "Artifacts of a session can be listed",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/files.md#22-by-listing-the-session",
+ "outcome": "pass",
+ "detail": "1 artifact(s)"
+ },
+ {
+ "id": "X-07",
+ "title": "An artifact downloads as raw bytes with nosniff",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/files.md#3-downloading",
+ "outcome": "pass",
+ "detail": "12 bytes, text/plain; charset=utf-8"
+ },
+ {
+ "id": "X-08",
+ "title": "Artifact ids do not traverse outside their container",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/files.md#5-retention-and-scope",
+ "outcome": "pass",
+ "detail": "traversal probes refused"
+ },
+ {
+ "id": "X-09",
+ "title": "A file uploaded through POST /v1/files is accepted and can be sent as task input",
+ "class": "extended",
+ "spec": "protocol/versions/2026-09-12/files.md#12-by-upload",
+ "outcome": "pass",
+ "detail": "uploaded, referenced by id, and the harness echoed the token"
+ },
+ {
+ "id": "F-01",
+ "title": "A harness can be created, updated and deleted",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#4-managing-harnesses",
+ "outcome": "pass",
+ "detail": "created, updated and deleted chrn_417f1db7e6a046ca86e09785a9091e43"
+ },
+ {
+ "id": "F-03",
+ "title": "A skill folder round-trips through create and read",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#42-skills",
+ "outcome": "pass",
+ "detail": "3 files incl. nested + binary"
+ },
+ {
+ "id": "F-04",
+ "title": "An unrelated harness edit does not destroy skill contents",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#42-skills",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "F-05",
+ "title": "A skill bundle without SKILL.md is refused at config time",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#42-skills",
+ "outcome": "pass",
+ "detail": "refused with HTTP 400"
+ },
+ {
+ "id": "F-06",
+ "title": "MCP servers and disabled tools round-trip",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#41-mcp-servers",
+ "outcome": "pass",
+ "detail": "mcp + disabledTools preserved"
+ },
+ {
+ "id": "P-01",
+ "title": "The plugins capability names the manifest schemas it installs",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#7-discovery",
+ "outcome": "pass",
+ "detail": "plugin_schemas=['https://agent-plugins.org/schemas/1.0.0/plugin.schema.json']"
+ },
+ {
+ "id": "P-02",
+ "title": "A plugin package round-trips, and is derived rather than copied",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#2-the-plugin-object",
+ "outcome": "pass",
+ "detail": "5 files, 2 servers, 1 skill derived"
+ },
+ {
+ "id": "P-03",
+ "title": "A package without plugin.json is refused at configuration time",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#21-files",
+ "outcome": "pass",
+ "detail": "refused with HTTP 422 plugin_invalid"
+ },
+ {
+ "id": "P-04",
+ "title": "A component name collision across the harness and a plugin is refused",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#42-one-namespace-checked-at-configuration-time",
+ "outcome": "pass",
+ "detail": "refused with 409 plugin_conflict"
+ },
+ {
+ "id": "P-05",
+ "title": "An unrelated harness edit does not destroy plugin contents",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#3-installing-a-plugin",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "P-06",
+ "title": "A harness exports as a package that installs, without its credentials",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#5-exporting-a-harness-as-a-plugin",
+ "outcome": "pass",
+ "detail": "exported 5 files as 'uhp-conformance-35c3e3', reinstalled"
+ },
+ {
+ "id": "P-07",
+ "title": "A component that fails to load is recorded, not silently dropped",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#22-what-the-server-derives",
+ "outcome": "pass",
+ "detail": "skipped: mcp.json#/mcpServers/conformance-broken"
+ },
+ {
+ "id": "P-08",
+ "title": "A disabled plugin stays installed, and stays disabled",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#3-installing-a-plugin",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "P-09",
+ "title": "An unsupported manifest schema is refused, naming the supported ones",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#22-what-the-server-derives",
+ "outcome": "pass",
+ "detail": "refused; supported=['https://agent-plugins.org/schemas/1.0.0/plugin.schema.json']"
+ },
+ {
+ "id": "P-11",
+ "title": "A plugin's MCP server answers the agent's tool call",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#4-binding-at-run-time",
+ "outcome": "pass",
+ "detail": "the fixture answered through the plugin in 16.2s"
+ },
+ {
+ "id": "P-10",
+ "title": "Two plugins with the same name are refused",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/plugins.md#3-installing-a-plugin",
+ "outcome": "pass",
+ "detail": "refused with 409 plugin_conflict"
+ },
+ {
+ "id": "F-07",
+ "title": "Configured harnesses are cleaned up",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#53-delete",
+ "outcome": "pass",
+ "detail": "10 removed"
+ },
+ {
+ "id": "F-08",
+ "title": "A session can be deleted at the protocol path",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#6-deleting",
+ "outcome": "pass",
+ "detail": "deleted hsess3ba743959eb14566adb3f6e2ed8f38e3, now 404"
+ },
+ {
+ "id": "F-02",
+ "title": "Creating a harness with an unsupported base is refused",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/harnesses.md#41-create",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "T-08",
+ "title": "A request carrying reserved fields is accepted, not rejected",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/tasks.md#11-request-fields",
+ "outcome": "pass",
+ "detail": "accepted and ran"
+ },
+ {
+ "id": "T-09",
+ "title": "Reserved fields are reported in metadata.ignored_fields",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/tasks.md#14-reserved-fields-tools-and-include",
+ "outcome": "pass",
+ "detail": "ignored_fields=['tools', 'include']"
+ },
+ {
+ "id": "T-10",
+ "title": "A task that sends no reserved field is not told one was ignored",
+ "class": "core",
+ "spec": "protocol/versions/2026-09-12/tasks.md#11-request-fields",
+ "outcome": "pass",
+ "detail": ""
+ },
+ {
+ "id": "R-01",
+ "title": "A shared session is published, and the link alone opens it",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#5-session-sharing",
+ "outcome": "pass",
+ "detail": "readable unauthenticated at /share/shrd3e43f506c184a0596951297f62b6519"
+ },
+ {
+ "id": "R-02",
+ "title": "The share can be read back from the endpoint that minted it",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#5-session-sharing",
+ "outcome": "pass",
+ "detail": "GET agrees with POST"
+ },
+ {
+ "id": "R-03",
+ "title": "A share id is not a credential for the API",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#5-session-sharing",
+ "outcome": "pass",
+ "detail": "refused on 2 endpoint(s)"
+ },
+ {
+ "id": "R-04",
+ "title": "The shared view is read-only",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#5-session-sharing",
+ "outcome": "pass",
+ "detail": "7 write probes refused"
+ },
+ {
+ "id": "R-05",
+ "title": "The shared view exposes no credentials",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#5-session-sharing",
+ "outcome": "pass",
+ "detail": "no credential-shaped fields"
+ },
+ {
+ "id": "R-06",
+ "title": "Revocation kills every link minted for the session",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#5-session-sharing",
+ "outcome": "pass",
+ "detail": "revoked via DELETE /v1/sessions/hsess16f4b34984f9410fb9c836161caa24d5/share; 1 link(s) minted, all now 404"
+ },
+ {
+ "id": "R-07",
+ "title": "Deleting a session takes its shared view with it",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#6-deleting",
+ "outcome": "pass",
+ "detail": "session deleted, then HTTP 404"
+ },
+ {
+ "id": "R-08",
+ "title": "A bodyless POST publishes, as \u00a75 says it does",
+ "class": "full",
+ "spec": "protocol/versions/2026-09-12/sessions.md#5-session-sharing",
+ "outcome": "pass",
+ "detail": "published with no body"
+ }
+ ]
+}
diff --git a/protocol/conformance/tests/test_plugin_checks.py b/protocol/conformance/tests/test_plugin_checks.py
index ee5c4150..60a446aa 100644
--- a/protocol/conformance/tests/test_plugin_checks.py
+++ b/protocol/conformance/tests/test_plugin_checks.py
@@ -4,7 +4,9 @@ The Plugins chapter is a set of derivations, refusals and omissions, and a check
passes cannot tell a server that refuses from one that does not. So every P- check runs against
a deliberately wrong server (plugin_stub.py) carrying one defect at a time. Each defect must be
caught by the check that claims to cover it, by no other, and a clean server must pass all ten.
-No task is ever run, so this needs no credentials, no network and no agent tokens.
+No task is ever run, so this needs no credentials, no network and no agent tokens. P-11 is the
+one plugin check that runs a task (through the fixture server); it is tested in
+test_plugin_fixture.py with a fake client and left out of this series.
"""
from __future__ import annotations
@@ -66,16 +68,16 @@ def _run_series(defect: str) -> dict[str, Outcome]:
else:
pytest.fail(f"the stub did not start for defect {defect!r}")
ctx = Context(client=Client(f"http://127.0.0.1:{port}", "stub-key"), task_timeout=30.0)
- return {c.id: c.run(ctx).outcome for c in REGISTRY if c.id.startswith("P-")}
+ return {c.id: c.run(ctx).outcome for c in REGISTRY if c.id.startswith("P-") and c.id != "P-11"}
finally:
srv.terminate()
srv.wait(timeout=5)
def test_the_series_is_registered_at_full():
- ids = [c.id for c in REGISTRY if c.id.startswith("P-")]
+ ids = [c.id for c in REGISTRY if c.id.startswith("P-") and c.id != "P-11"]
assert ids == [f"P-{i:02d}" for i in range(1, 11)]
- assert {c.cls for c in REGISTRY if c.id.startswith("P-")} == {"full"}
+ assert {c.cls for c in REGISTRY if c.id.startswith("P-")} == {"full"} # P-11 included: it is full too
def test_a_conformant_server_passes_every_check():
diff --git a/protocol/conformance/tests/test_plugin_fixture.py b/protocol/conformance/tests/test_plugin_fixture.py
new file mode 100644
index 00000000..94012c14
--- /dev/null
+++ b/protocol/conformance/tests/test_plugin_fixture.py
@@ -0,0 +1,86 @@
+"""The plugin fixture and the address that reaches it.
+
+The fixture's answer is a function of its input, which is what lets P-11 tell a tool call from a
+guess; the plugin checks take the server's address from the context, so a run can be pointed at a
+copy the host under test can reach; and P-11 must fail a reply that does not carry the answer.
+"""
+from __future__ import annotations
+
+from uhp_conformance import DEFAULT_PLUGIN_MCP_URL
+from uhp_conformance import checks # noqa: F401 — importing populates the registry
+from uhp_conformance.context import Context
+from uhp_conformance.fixture import PREFIX, TOOL_NAME, fixture_answer
+from uhp_conformance.registry import REGISTRY
+
+
+def test_the_answer_is_a_function_of_the_input_and_names_the_fixture():
+ assert fixture_answer("abc123") == PREFIX + "321cba"
+ assert fixture_answer("") == PREFIX
+ assert TOOL_NAME == "uhp_echo"
+
+
+def test_the_context_carries_the_address_and_defaults_to_the_public_copy():
+ ctx = Context(client=None)
+ assert ctx.plugin_mcp_url == DEFAULT_PLUGIN_MCP_URL
+ assert DEFAULT_PLUGIN_MCP_URL.startswith("https://") and DEFAULT_PLUGIN_MCP_URL.endswith("/mcp")
+ own = Context(client=None, plugin_mcp_url="http://10.0.0.5:8080/mcp")
+ mcp = checks._plugin_mcp(own)
+ assert mcp["mcpServers"]["conformance-http"]["url"] == "http://10.0.0.5:8080/mcp"
+ assert "example.invalid" not in str(checks._plugin_files_of(own))
+
+
+def test_p11_is_a_full_class_check_that_runs_a_task():
+ p11 = next(c for c in REGISTRY if c.id == "P-11")
+ assert p11.cls == "full"
+ assert "tool call" in p11.title
+
+
+class _Resp:
+ def __init__(self, status, json, body=b""):
+ self.status, self.json, self.body, self.elapsed_s = status, json, body, 0.1
+
+
+class _Client:
+ """A server that derives the plugin and then answers the task without calling the tool."""
+ def __init__(self, reply: str):
+ self.reply = reply
+ def get(self, path, **kw):
+ if path == "/v1/harnesses":
+ return _Resp(200, {"harnesses": [{"id": "h1", "base": "claude-code", "defaultModel": "m"}]})
+ if path == "/v1/discovery":
+ return _Resp(200, {"capabilities": {"plugins": True}})
+ return _Resp(200, {"id": "h2", "plugins": [{"name": "uhp-conformance-plugin",
+ "mcpServers": [{"name": "conformance-http"}], "skipped": []}]})
+ def post(self, path, **kw):
+ if path == "/v1/harnesses":
+ return _Resp(200, {"id": "h2"})
+ return _Resp(200, {"status": "completed", "output": [{"type": "message", "content": [{"type": "output_text", "text": self.reply}]}]})
+ def delete(self, path, **kw):
+ return _Resp(200, {})
+
+
+def _run_p11(reply: str):
+ ctx = Context(client=_Client(reply)); ctx.state["discovery"] = {"capabilities": {"plugins": True}}
+ p11 = next(c for c in REGISTRY if c.id == "P-11")
+ return p11.run(ctx)
+
+
+def test_p11_fails_a_reply_without_the_fixture_answer_and_passes_one_with_it():
+ from uhp_conformance.registry import Outcome
+ bad = _run_p11("I called the tool and it said hello")
+ assert bad.outcome == Outcome.FAIL and "never reached the plugin's server" in (bad.detail or "")
+ # the passing reply must carry the answer for the nonce the check chose, which the fake
+ # server cannot know: so pass through whatever text the check sent, reversed, as the fixture would
+ import uhp_conformance.checks as ck
+ real_post = _Client.post
+ def echo_post(self, path, **kw):
+ if path == "/v1/responses":
+ text = kw["body"]["input"]; nonce = text.split(" with the text ")[1].split(" ")[0]
+ return _Resp(200, {"status": "completed", "output": [{"type": "message", "content": [{"type": "output_text", "text": fixture_answer(nonce)}]}]})
+ return real_post(self, path, **kw)
+ _Client.post = echo_post
+ try:
+ good = _run_p11("")
+ finally:
+ _Client.post = real_post
+ assert good.outcome == Outcome.PASS, good.detail
diff --git a/protocol/conformance/uhp_conformance/__init__.py b/protocol/conformance/uhp_conformance/__init__.py
index 995debc1..7507a3c6 100644
--- a/protocol/conformance/uhp_conformance/__init__.py
+++ b/protocol/conformance/uhp_conformance/__init__.py
@@ -1,3 +1,8 @@
"""UHP conformance suite — the definition of "conformant" for the Unified Harness Protocol."""
-__version__ = "2026.9.12.post2"
+__version__ = "2026.9.12.post3"
UHP_VERSION = "2026-09-12"
+# The MCP server the plugin checks point a plugin at. It must resolve and answer: a host is
+# allowed to refuse a server it cannot reach at configuration time and record that in `skipped`,
+# so an unresolvable placeholder made those checks measure the refusal instead of the plugin. The
+# suite ships the server (`uhp-conformance-fixture`, see fixture.py); this is a public copy of it.
+DEFAULT_PLUGIN_MCP_URL = "https://uhp-fixture.calmrock-23b12d3f.eastus2.azurecontainerapps.io/mcp"
diff --git a/protocol/conformance/uhp_conformance/checks.py b/protocol/conformance/uhp_conformance/checks.py
index 067d0ebb..4d723c1f 100644
--- a/protocol/conformance/uhp_conformance/checks.py
+++ b/protocol/conformance/uhp_conformance/checks.py
@@ -846,7 +846,7 @@ def f05(ctx):
@check("F-06", "MCP servers and disabled tools round-trip", "full",
f"{SPEC}/harnesses.md#41-mcp-servers")
def f06(ctx):
- mcp = [{"name": "conformance-mcp", "url": "https://mcp.example.invalid/mcp",
+ mcp = [{"name": "conformance-mcp", "url": ctx.plugin_mcp_url,
"transport": "http", "enabled": False}]
h = _managed_harness(ctx, mcp_servers=mcp, disabled_tools=["WebSearch"])
got = ctx.client.get(f"/v1/harnesses/{h['id']}").json or {}
@@ -901,13 +901,25 @@ def _package(name: str = "uhp-conformance-plugin", *, manifest: dict | None = No
_PLUGIN_MANIFEST = {"$schema": _AP_MANIFEST_SCHEMA, "name": "uhp-conformance-plugin",
"version": "1.0.0", "description": "A conformance fixture."}
-_PLUGIN_MCP = {"$schema": _AP_MCP_SCHEMA, "mcpServers": {
- "conformance-http": {"type": "streamable-http", "url": "https://mcp.example.invalid/mcp"},
- "conformance-stdio": {"type": "stdio", "command": "./bin/tool.bin",
- "args": ["--data", "${PLUGIN_DATA}"]},
-}}
-_PLUGIN_FILES = _package(manifest=_PLUGIN_MANIFEST, mcp=_PLUGIN_MCP)
-_PLUGIN_PATHS = sorted(f["path"] for f in _PLUGIN_FILES)
+
+
+def _plugin_mcp(ctx) -> dict:
+ """The fixture plugin's mcp.json. Its HTTP server is the one the run was pointed at
+ (`--plugin-mcp-url`), which resolves and answers: a host may refuse a server it cannot reach at
+ configuration time and record that in `skipped`, and with an unresolvable placeholder these
+ checks measured that refusal instead of the plugin (the hosted HarnessRouter, 2026-09-27)."""
+ return {"$schema": _AP_MCP_SCHEMA, "mcpServers": {
+ "conformance-http": {"type": "streamable-http", "url": ctx.plugin_mcp_url},
+ "conformance-stdio": {"type": "stdio", "command": "./bin/tool.bin",
+ "args": ["--data", "${PLUGIN_DATA}"]},
+ }}
+
+
+def _plugin_files_of(ctx) -> list[dict]:
+ return _package(manifest=_PLUGIN_MANIFEST, mcp=_plugin_mcp(ctx))
+
+
+_PLUGIN_PATHS = sorted(f["path"] for f in _package(manifest=_PLUGIN_MANIFEST, mcp={"mcpServers": {}}))
def _error_code(r) -> str:
@@ -938,7 +950,7 @@ def p01(ctx):
f"{SPEC}/plugins.md#2-the-plugin-object")
def p02(ctx):
_plugins_supported(ctx)
- h = _managed_harness(ctx, plugins=[{"files": _PLUGIN_FILES}])
+ h = _managed_harness(ctx, plugins=[{"files": _plugin_files_of(ctx)}])
got = ctx.client.get(f"/v1/harnesses/{h['id']}").json or {}
plugins = got.get("plugins") or []
assert len(plugins) == 1, f"installed one plugin, read back {len(plugins)}"
@@ -983,7 +995,7 @@ def p03(ctx):
base = _supported_base(ctx)
r = ctx.client.post("/v1/harnesses", body={
"name": "uhp-conformance-nomanifest", "base": base,
- "plugins": [{"files": _package(manifest=None, mcp=_PLUGIN_MCP)}]})
+ "plugins": [{"files": _package(manifest=None, mcp=_plugin_mcp(ctx))}]})
if r.status == 200:
ctx.client.delete(f"/v1/harnesses/{(r.json or {}).get('id')}")
raise AssertionError(
@@ -1000,11 +1012,10 @@ def p03(ctx):
def p04(ctx):
_plugins_supported(ctx)
base = _supported_base(ctx)
- direct = [{"name": "conformance-http", "url": "https://direct.example.invalid/mcp",
- "transport": "http"}]
+ direct = [{"name": "conformance-http", "url": ctx.plugin_mcp_url, "transport": "http"}]
r = ctx.client.post("/v1/harnesses", body={
"name": "uhp-conformance-collision", "base": base,
- "mcp_servers": direct, "plugins": [{"files": _PLUGIN_FILES}]})
+ "mcp_servers": direct, "plugins": [{"files": _plugin_files_of(ctx)}]})
if r.status == 200:
ctx.client.delete(f"/v1/harnesses/{(r.json or {}).get('id')}")
raise AssertionError(
@@ -1023,7 +1034,7 @@ def p04(ctx):
f"{SPEC}/plugins.md#3-installing-a-plugin")
def p05(ctx):
_plugins_supported(ctx)
- h = _managed_harness(ctx, plugins=[{"files": _PLUGIN_FILES}])
+ h = _managed_harness(ctx, plugins=[{"files": _plugin_files_of(ctx)}])
hid = h["id"]
got = ctx.client.get(f"/v1/harnesses/{hid}").json or {}
if got.get("mcpServers") or got.get("skills"):
@@ -1050,7 +1061,7 @@ def p05(ctx):
f"{SPEC}/plugins.md#5-exporting-a-harness-as-a-plugin")
def p06(ctx):
_plugins_supported(ctx)
- direct = [{"name": "conformance-direct", "url": "https://mcp.example.invalid/mcp",
+ direct = [{"name": "conformance-direct", "url": ctx.plugin_mcp_url,
"transport": "http", "headers": {"X-Team": "conformance"}, "auth": "secret-token"}]
h = _managed_harness(ctx, mcp_servers=direct, skills=[_SKILL_BUNDLE])
r = ctx.client.get(f"/v1/harnesses/{h['id']}/plugin")
@@ -1094,8 +1105,8 @@ def p06(ctx):
def p07(ctx):
_plugins_supported(ctx)
mcp = {"$schema": _AP_MCP_SCHEMA, "mcpServers": {
- "conformance-http": {"type": "streamable-http", "url": "https://mcp.example.invalid/mcp"},
- "conformance-broken": {"type": "carrier-pigeon", "url": "https://mcp.example.invalid/x"}}}
+ "conformance-http": {"type": "streamable-http", "url": ctx.plugin_mcp_url},
+ "conformance-broken": {"type": "carrier-pigeon", "url": ctx.plugin_mcp_url}}}
h = _managed_harness(ctx, plugins=[{"files": _package(manifest=_PLUGIN_MANIFEST, mcp=mcp)}])
pl = ((ctx.client.get(f"/v1/harnesses/{h['id']}").json or {}).get("plugins") or [{}])[0]
names = [s.get("name") for s in pl.get("mcpServers") or []]
@@ -1114,7 +1125,7 @@ def p07(ctx):
f"{SPEC}/plugins.md#3-installing-a-plugin")
def p08(ctx):
_plugins_supported(ctx)
- h = _managed_harness(ctx, plugins=[{"files": _PLUGIN_FILES, "enabled": False}])
+ h = _managed_harness(ctx, plugins=[{"files": _plugin_files_of(ctx), "enabled": False}])
pl = ((ctx.client.get(f"/v1/harnesses/{h['id']}").json or {}).get("plugins") or [{}])[0]
assert pl.get("name") == "uhp-conformance-plugin", f"the disabled plugin was not kept: {pl}"
assert pl.get("enabled") is False, (
@@ -1146,6 +1157,51 @@ def p09(ctx):
return f"refused; supported={supported}"
+@check("P-11", "A plugin's MCP server answers the agent's tool call", "full",
+ f"{SPEC}/plugins.md#4-binding-at-run-time")
+def p11(ctx):
+ """The one plugin check that runs a task. P-02 to P-10 read what the server derived and
+ refused; none of them shows that an agent can reach the plugin's server. This one installs a
+ plugin whose only server is the fixture the run was pointed at, asks the agent to call the
+ fixture's tool with a nonce, and expects the fixture's answer back: the tool's output is a
+ function of the nonce that the agent cannot produce without calling it."""
+ _plugins_supported(ctx)
+ from .fixture import fixture_answer, TOOL_NAME
+ base_h = _harness(ctx)
+ nonce = uuid.uuid4().hex[:10]
+ mcp = {"$schema": _AP_MCP_SCHEMA, "mcpServers": {
+ "conformance-http": {"type": "streamable-http", "url": ctx.plugin_mcp_url}}}
+ body = {"name": f"uhp-conformance-p11-{nonce[:4]}", "base": base_h.get("base") or _supported_base(ctx),
+ "plugins": [{"files": _package(manifest=_PLUGIN_MANIFEST, mcp=mcp, skill=False)}]}
+ model = ctx.model or str(base_h.get("defaultModel") or "").strip()
+ if model:
+ body["default_model"] = model
+ r = ctx.client.post("/v1/harnesses", body=body)
+ if r.status != 200:
+ raise Skip(f"could not create a harness carrying the plugin (HTTP {r.status}: {r.body[:160]!r})")
+ h = r.json or {}
+ ctx.state.setdefault("_cleanup_harnesses", []).append(h.get("id"))
+ pl = ((ctx.client.get(f"/v1/harnesses/{h['id']}").json or {}).get("plugins") or [{}])[0]
+ names = [s.get("name") for s in pl.get("mcpServers") or []]
+ assert names == ["conformance-http"], (
+ f"the plugin's server was not derived (servers={names}, skipped={pl.get('skipped')}); the "
+ f"fixture at {ctx.plugin_mcp_url} must be reachable from this server")
+ task = {"input": (f"Call the tool {TOOL_NAME} with the text {nonce} and reply with exactly what it "
+ "returns, nothing else."),
+ "metadata": {"harness_id": h["id"]}, "stream": False}
+ if model:
+ task["model"] = model
+ resp = ctx.client.post("/v1/responses", body=task)
+ assert resp.status == 200, f"POST /v1/responses returned HTTP {resp.status}: {resp.body[:200]!r}"
+ text = json.dumps((resp.json or {}).get("output") or [])
+ want = fixture_answer(nonce)
+ assert want in text, (
+ f"the reply does not carry the fixture's answer {want!r} (status={(resp.json or {}).get('status')}, "
+ f"output={text[:240]!r}). The agent never reached the plugin's server: either the host did "
+ f"not bind it at run time or the sandbox cannot reach {ctx.plugin_mcp_url}.")
+ return f"the fixture answered through the plugin in {resp.elapsed_s:.1f}s"
+
+
@check("P-10", "Two plugins with the same name are refused", "full",
f"{SPEC}/plugins.md#3-installing-a-plugin")
def p10(ctx):
@@ -1154,7 +1210,7 @@ def p10(ctx):
twin = _package(manifest=_PLUGIN_MANIFEST, mcp=None, skill=False)
r = ctx.client.post("/v1/harnesses", body={
"name": "uhp-conformance-twins", "base": base,
- "plugins": [{"files": _PLUGIN_FILES}, {"files": twin}]})
+ "plugins": [{"files": _plugin_files_of(ctx)}, {"files": twin}]})
if r.status == 200:
ctx.client.delete(f"/v1/harnesses/{(r.json or {}).get('id')}")
raise AssertionError("two plugins named uhp-conformance-plugin were installed side by side; "
@@ -1241,7 +1297,7 @@ def _run_with_reserved(ctx) -> dict:
"tools": [{"type": "function", "name": "uhp_conformance_probe",
"description": "Must never be offered to the agent.",
"parameters": {"type": "object", "properties": {}}},
- {"name": "uhp-conformance-mcp", "url": "https://mcp.example.invalid/mcp",
+ {"name": "uhp-conformance-mcp", "url": ctx.plugin_mcp_url,
"transport": "http"}],
"include": ["uhp.conformance.not_a_real_value"]}
if ctx.model:
diff --git a/protocol/conformance/uhp_conformance/cli.py b/protocol/conformance/uhp_conformance/cli.py
index ded89b67..f856ed46 100644
--- a/protocol/conformance/uhp_conformance/cli.py
+++ b/protocol/conformance/uhp_conformance/cli.py
@@ -7,6 +7,7 @@ import sys
from . import UHP_VERSION, checks # noqa: F401 — importing checks populates the registry
from .client import Client
+from . import DEFAULT_PLUGIN_MCP_URL
from .context import Context
from .registry import CLASSES, Outcome, checks_for
from .report import render, to_json
@@ -34,6 +35,10 @@ def main(argv=None) -> int:
help="Extra header sent on every request, repeatable. For a target that "
"needs something a stock client never sends, e.g. a bring-your-own-key "
"host wanting the caller's model key. Never echoed into the report.")
+ p.add_argument("--plugin-mcp-url", default=os.environ.get("UHP_PLUGIN_MCP_URL", DEFAULT_PLUGIN_MCP_URL),
+ help="A reachable streamable-HTTP MCP server the plugin checks point a plugin at "
+ "(default: the public copy of uhp-conformance-fixture, or UHP_PLUGIN_MCP_URL). "
+ "Run `uhp-conformance-fixture` to host your own.")
p.add_argument("--label", default="",
help="How the report names the implementation measured, e.g. 'SuperQode 2.3.1'")
a = p.parse_args(argv)
@@ -47,7 +52,7 @@ def main(argv=None) -> int:
extra[name.strip()] = value.strip()
ctx = Context(client=Client(a.base_url, a.api_key, headers=extra), harness_id=a.harness_id,
- model=a.model, task_timeout=a.task_timeout)
+ model=a.model, task_timeout=a.task_timeout, plugin_mcp_url=a.plugin_mcp_url)
# The harness the tasks run on is resolved BEFORE the first task, and named. A --harness-id
# that matches nothing (a harness name given for an id, a typo) once fell back silently to
diff --git a/protocol/conformance/uhp_conformance/context.py b/protocol/conformance/uhp_conformance/context.py
index 437c4556..ca0a0d81 100644
--- a/protocol/conformance/uhp_conformance/context.py
+++ b/protocol/conformance/uhp_conformance/context.py
@@ -10,7 +10,7 @@ try:
except ImportError: # pragma: no cover
jsonschema = None
-from . import UHP_VERSION
+from . import DEFAULT_PLUGIN_MCP_URL, UHP_VERSION
from .client import Client
SCHEMA_RESOURCE = resources.files("uhp_conformance").joinpath(f"uhp-{UHP_VERSION}.schema.json")
@@ -22,6 +22,8 @@ class Context:
harness_id: str = ""
model: str = ""
task_timeout: float = 300.0
+ # A reachable MCP server for the plugin checks (see DEFAULT_PLUGIN_MCP_URL).
+ plugin_mcp_url: str = DEFAULT_PLUGIN_MCP_URL
state: dict = field(default_factory=dict)
_schema: dict | None = None
diff --git a/protocol/conformance/uhp_conformance/fixture.py b/protocol/conformance/uhp_conformance/fixture.py
new file mode 100644
index 00000000..5548f28d
--- /dev/null
+++ b/protocol/conformance/uhp_conformance/fixture.py
@@ -0,0 +1,56 @@
+"""The MCP server the plugin checks point a plugin at, shipped with the suite.
+
+A streamable-HTTP MCP server with two tools. `uhp_echo` answers with a function of its input
+(`fixture_answer`), so a conformance run can prove an agent called it: the answer cannot be
+produced without the call. `uhp_time` says what time the fixture thinks it is, for a human at the
+other end of a debugging session.
+
+Run it anywhere a host under test can reach: `uhp-conformance-fixture --host 0.0.0.0 --port 8080`
+serves at http://<host>:8080/mcp, and `uhp-conformance --plugin-mcp-url <that>` points a run at
+it. The public copy the suite defaults to is the same code (protocol/conformance/fixture/).
+"""
+from __future__ import annotations
+
+import argparse
+import datetime as _dt
+
+TOOL_NAME = "uhp_echo"
+PREFIX = "UHP-FIXTURE:"
+
+
+def fixture_answer(text: str) -> str:
+ """What `uhp_echo` returns for `text`: the prefix and the text reversed."""
+ return PREFIX + str(text)[::-1]
+
+
+def build_server(host: str, port: int):
+ try:
+ from mcp.server.fastmcp import FastMCP
+ except ImportError as e: # pragma: no cover - the extra is named in the error
+ raise SystemExit("the fixture needs the MCP SDK: pip install 'uhp-conformance[fixture]'") from e
+ server = FastMCP("uhp-conformance-fixture", host=host, port=port, stateless_http=True)
+
+ @server.tool(name=TOOL_NAME, description="Echo back the text, transformed the way only this "
+ "fixture does, so a caller can prove the call happened.")
+ def uhp_echo(text: str) -> str:
+ return fixture_answer(text)
+
+ @server.tool(name="uhp_time", description="The fixture's current UTC time.")
+ def uhp_time() -> str:
+ return _dt.datetime.now(_dt.timezone.utc).isoformat(timespec="seconds")
+
+ return server
+
+
+def main(argv=None) -> int:
+ p = argparse.ArgumentParser(prog="uhp-conformance-fixture",
+ description="The MCP server the UHP conformance plugin checks point a plugin at.")
+ p.add_argument("--host", default="127.0.0.1")
+ p.add_argument("--port", type=int, default=8080)
+ a = p.parse_args(argv)
+ build_server(a.host, a.port).run(transport="streamable-http")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())