Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(reporting): complete private runtime reports and saved-run acceptance (#345)
Narendran Raghavan55d8b8d427e9
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bf3b330a..2a4311de 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -24,6 +24,11 @@ All notable changes to SkillEvaluator are documented in this file.### Added+- Private runtime runs retain sanitized loading, coverage, MCP, behavior and+ comparison views before cleanup. The saved snapshot supports offline reports+ in all six existing formats, with explicit missing evidence and omissions.+ Public command activation remains deferred.+- `skillevaluator tier3 feedback RUN_DIR [--skill PATH]` opens a terminal chatabout one completed Tier 3 run. Answers are grounded in that run's scores,with-skill and baseline comparisons, every case's attempts with per-metricdiff --git a/docs/superpowers/plans/2026-11-05-private-runtime-reports.md b/docs/superpowers/plans/2026-11-05-private-runtime-reports.mdnew file mode 100644index 00000000..0a910995--- /dev/null+++ b/docs/superpowers/plans/2026-11-05-private-runtime-reports.md@@ -0,0 +1,62 @@+# Private Runtime Reports Implementation Plan++> **For agentic workers:** Use the existing execution workflow, focused helpers and an independent final review. Track completed checks in the task ledger.++**Goal:** Complete item 24 with consistent private runtime views and durable offline replay.++**Architecture:** Reduce existing results/provenance once into an immutable sanitized display projection. Retain it before cleanup and render it through the existing six private report adapters.++**Tech Stack:** Python 3.11+, dataclasses, shared secure filesystem/JSON readers, Jinja and existing reporters; no added dependency.++**Spec:** `docs/superpowers/specs/2026-11-05-private-runtime-reports-design.md`++## Global Constraints++- Public CLI/configuration/detection/exports remain disabled until item 25.+- Keep unknown values distinct from zero; staging never proves loading or execution.+- Reports are passive and advisory and do not alter scores, gates or source results.+- Redact before truncating, escape each output format and bound input/output work.+- Use existing statistics without recomputing lifts, eligibility, confidence intervals or costs.++## Review Focus++- Missing trajectories/startup evidence must remain incomplete despite successful execution.+- Missing or unsupported arms must retain their measurement/exclusion reasons.+- Hostile strings, malformed values and report budget exhaustion must remain safe and explicit.+- Saved snapshots must replay after source/jobs removal and refuse unsafe or malformed files.+- Publication failure must preserve callbacks, evaluation status and cleanup.++### Task 1: Projection and retained snapshot++Files: new `reporting/plugin_runtime.py`, `reporting/plugin_runtime_io.py`;+modify `tier3/plugin_execution.py`; new focused reporting/lifecycle tests.++Interfaces: `project_plugin_runtime(result, *, provenance=None) -> RuntimePluginReport | None`;+`RuntimePluginReport.payload() -> dict`; `save_plugin_runtime_report(report, run_dir)`;+`load_plugin_runtime_report(run_dir) -> RuntimePluginReport`.++- [x] Write failing tests for run identity, load states, coverage/signals/MCP, comparisons, measured costs, unknowns and explicit omissions.+- [x] Implement fixed-schema bounded projection using existing sanitization and section budgets.+- [x] Test strict no-follow, bounded and version/schema-checked snapshot round trips, including malformed/oversized/missing/link inputs.+- [x] Retain the snapshot in the private publication callback, keeping failures advisory and caller publication independent.+- [x] Run focused projection, replay and lifecycle regressions.++### Task 2: Six report formats++Files: new `reporting/plugin_runtime_render.py`, runtime HTML template and tests;+minimal reuse/refactor of `reporting/plugin_static_render.py`.++Interface: `render_plugin_runtime_reports(results, *, report, reporters) -> dict[str, str]`.++- [x] Write failing cross-format tests for identical payloads, escaping, missing evidence, combined static/runtime output and ordinary-output compatibility.+- [x] Render the same projection through all six existing reporters, with no new collection or public exports.+- [x] Run runtime/static reporting tests and check a generated HTML report visually.++### Task 3: Assembled acceptance and delivery++Files: new `tests/tier3/test_private_runtime_report_acceptance.py`, private usage notes and release note.++- [x] Exercise real discovery, deterministic validation, preparation, service/runner and collection with local execution fixtures: five cross-component cases, two attempts, four arms.+- [x] Verify saved replay after source/tasks/jobs removal, shared comparison estimates, missing evidence and ordinary-run isolation.+- [ ] Run `make lint`, `make test`, `make build`; verify installed base-only replay and private CLI boundaries.+- [ ] Independently review, fix actionable findings, sign off commits, push and open/attach a focused PR with exact validation limits.diff --git a/docs/superpowers/specs/2026-11-05-private-runtime-reports-design.md b/docs/superpowers/specs/2026-11-05-private-runtime-reports-design.mdnew file mode 100644index 00000000..44f3dc74--- /dev/null+++ b/docs/superpowers/specs/2026-11-05-private-runtime-reports-design.md@@ -0,0 +1,66 @@+# Private runtime reports (split-plan item 24)++Item 24 completes loading, coverage, behavior, comparison and saved-run views+over the private workflow introduced in items 15–23. Reviewers need the same+facts after temporary tasks and jobs have been removed. Public commands,+configuration, detection and exports remain item 25.++Use one immutable, versioned, bounded display projection of existing engine+results and private provenance. Preserve declared/staged/listed/loaded states,+missing evidence, unsupported modes, comparison exclusions, uncertainty and+unknown costs. Execution status and evidence completeness are separate facts.+Advisory observations never change scores, validation findings or exit status.++Persist that sanitized projection as `plugin_runtime_report.json` during the+private publication callback, before cleanup and caller publication. A fixed-file+secure reader validates the saved schema and reuses the projection for offline+reporting. Missing older snapshots are explicitly unavailable. Replay does not+reconstruct traces, load source files, launch agents or contact providers.++Reuse the six existing private static adapters and table renderers for CLI,+JSON, Markdown, HTML, SARIF and benchmark output. Keep ordinary reporter+entrypoints byte-compatible. Display omissions and missing values explicitly;+escape untrusted text for each output language and redact before truncation.++Alternatives considered: rebuilding views from raw sidecars loses private load+decisions and preparation provenance; embedding ad-hoc views in six reporters+duplicates policy and risks public activation. The retained shared projection+fits the existing private adapter architecture and needs no new dependency.++Acceptance exercises discovery, deterministic validation, preparation, the+existing service/runner, all four arms, evidence, publication, cleanup and+offline replay. Substitute external provider/Harbor execution with deterministic+local fixtures; state this boundary accurately. Verify five cross-component+cases and two attempts, missing evidence, publication failure and ordinary+workflow isolation. Also build/install the distributions and check replay+without optional runtime dependencies.++## Private API usage++`evaluate_prepared_plugin` retains the snapshot automatically when the existing+engine produces a run directory. Caller publication sees it before the temporary+wrapper is removed. A storage failure is advisory: `publication_error` and+`provenance["runtime_report_error"]` explain it, and caller publication and cleanup+still run. A successfully scored run may have incomplete runtime evidence.++Offline consumers select the concrete retained run directory, load the snapshot+and supply their existing validation results to the private adapter:++```python+from skillevaluator.reporting import HTMLReporter, JSONReporter+from skillevaluator.reporting.plugin_runtime_io import load_plugin_runtime_report+from skillevaluator.reporting.plugin_runtime_render import render_plugin_runtime_reports++report = load_plugin_runtime_report(run_dir)+documents = render_plugin_runtime_reports(+ validation_results,+ report=report,+ reporters=[HTMLReporter(), JSONReporter()],+)+```++Replay uses the saved display schema; it does not re-read source files, re-grade+trials or recover omitted details. A missing snapshot from an older run raises+an explicit unavailable error. Select a concrete directory rather than the+`latest` symlink. Treat the snapshot as retained observations, not authenticated+proof of a run. Keep the original artifacts for deeper investigation.diff --git a/src/skillevaluator/reporting/plugin_runtime.py b/src/skillevaluator/reporting/plugin_runtime.pynew file mode 100644index 00000000..d0b566bf--- /dev/null+++ b/src/skillevaluator/reporting/plugin_runtime.py@@ -0,0 +1,455 @@+# SPDX-License-Identifier: Apache-2.0+"""Passive, bounded views of retained private runtime observations."""++from __future__ import annotations++import json+from collections.abc import Mapping+from dataclasses import dataclass+from itertools import islice+from pathlib import PurePosixPath+from typing import Any++from skillevaluator.reporting.plugin_static import ( # noqa: F401 - re-exported bound+ MAX_OUTPUT_BYTES,+ _clean_text,+ _count,+ _mapping,+ _Projection,+)++SCHEMA_VERSION = 1+_PAIR_LABELS = {+ "effectiveness": "Effectiveness lift (full minus baseline)",+ "integration": "Bundle lift (full minus member skills)",+}+SECTIONS = {+ "run": ("Run and source", ("Field", "Value")),+ "diagnostics": ("Evidence limitations", ("Source", "Detail")),+ "conditions": (+ "Arm execution coverage",+ ("Agent", "Arm", "Execution", "Expected cases", "Scored cases", "Expected attempts", "Scored attempts"),+ ),+ "loading": ("Component loading", ("Agent", "Mode", "Component type", "Component", "State", "Evidence")),+ "coverage": ("Activation coverage", ("Agent", "Arm", "Component", "State", "Exercise rate")),+ "signals": ("Behavior signals", ("Agent", "Arm", "Signal", "Status", "Count", "Skipped", "Value")),+ "mcp": ("MCP proof", ("Server", "Proof", "Tools", "Detail")),+ "mcp_calls": ("Agent-observed MCP calls", ("Agent", "Arm", "Server", "Total", "Succeeded", "Failed", "Unknown")),+ "comparisons": (+ "Paired comparisons",+ ("Agent", "Measure", "Estimate", "CI low", "CI high", "Cases", "Precision", "Partial"),+ ),+ "eligibility": ("Measurement eligibility", ("Agent", "Measure", "Eligible", "Verdict", "Reason")),+ "exclusions": ("Excluded cases", ("Agent", "Case", "Reason")),+ "cost": ("Recorded cost", ("Agent", "Arm", "Tokens", "USD", "Successes", "Tokens/success", "USD/success")),+ "context": (+ "Measured context",+ ("Agent", "Status", "Delta tokens", "Pairs", "Partial", "Excluded hosted search", "Missing tokens", "Reason"),+ ),+}+++@dataclass(frozen=True)+class RuntimePluginReport:+ """An immutable display snapshot; no trace, score or execution authority."""++ _serialized: str++ def payload(self) -> dict[str, Any]:+ return json.loads(self._serialized)+++class _RuntimeProjection:+ def __init__(self) -> None:+ self.bound = _Projection()+ self.rows: dict[str, list[list]] = {key: [] for key in SECTIONS}+ self.totals = dict.fromkeys(SECTIONS, 0)+ self.complete = True++ def add(self, key: str, *cells: object) -> None:+ self.totals[key] += 1+ # Count every supplied row but retain only the section's display budget.+ if len(self.rows[key]) < 64:+ self.rows[key].append(list(cells))++ def items(self, raw: object, *, limit: int = 32):+ data = _mapping(raw)+ if len(data) > limit:+ self.bound.truncated = True+ self.issue("report bounds", f"{len(data) - limit} input groups omitted")+ return islice(data.items(), limit)++ def issue(self, source: str, detail: object) -> None:+ self.complete = False+ self.add("diagnostics", source, detail)++ def records(self, key: str, raw: object) -> list[dict]:+ records = self.bound.records(raw)+ if isinstance(raw, (list, tuple)):+ self.totals[key] += len(raw) - len(records)+ return records++ def labels(self, key: str, raw: object) -> list[str]:+ if not isinstance(raw, (list, tuple)):+ return []+ length = min(len(raw), self.bound.input_remaining, 4096)+ self.bound.input_remaining -= length+ labels = [item for item in islice(raw, length) if isinstance(item, str)]+ self.totals[key] += len(raw) - len(labels)+ self.bound.truncated |= length < len(raw)+ return labels++ def errors(self, source: str, raw: object) -> None:+ if isinstance(raw, list):+ for error in islice(raw, 64):+ if isinstance(error, str):+ self.issue(source, error)+ if len(raw) > 64:+ self.bound.truncated = True+ self.issue(source, f"{len(raw) - 64} more diagnostics omitted")++ def finish(self) -> list[dict[str, Any]]:+ # Limitations and eligibility stay visible before large member inventories.+ order = [+ "run",+ "diagnostics",+ "eligibility",+ "conditions",+ "comparisons",+ "mcp",+ "context",+ "cost",+ "loading",+ "coverage",+ "signals",+ "mcp_calls",+ "exclusions",+ ]+ for key in order:+ title, columns = SECTIONS[key]+ self.bound.section(+ key,+ title,+ columns,+ self.rows[key],+ total=self.totals[key],+ notes=(+ ["Skipped expectations concern components unsupported in that arm."]+ if key == "signals"+ else []+ if self.totals[key]+ else ["No retained observations for this section."]+ ),+ )+ return self.bound.finish()+++def _load_views(projection: _RuntimeProjection, native: dict, decisions: dict) -> None:+ for agent, raw in projection.items(native):+ row = _mapping(raw)+ mode = row.get("mode") or _mapping(decisions.get(agent)).get("mode") or "unknown"+ for state in ("loaded", "listed", "staged", "not_loaded"):+ for item in projection.records("loading", row.get(state)):+ projection.add(+ "loading",+ agent,+ mode,+ item.get("type"),+ item.get("name"),+ state,+ item.get("reason") or item.get("evidence"),+ )+ if row.get("fallback_trials") or row.get("listed") or row.get("staged"):+ projection.issue(agent, "Some trials have only staging evidence; harness loading is unverified")+ for agent, raw in projection.items(decisions):+ decision = _mapping(raw)+ projection.add("run", f"{agent} loading", decision.get("mode"))+ if decision.get("reason"):+ projection.add("run", f"{agent} loading reason", decision["reason"])+ if decision.get("mode") == "native" and not _mapping(native.get(agent)):+ projection.issue(agent, "Native loading evidence is unavailable")+++def _signal_views(projection: _RuntimeProjection, agents: object) -> None:+ if not _mapping(agents):+ projection.issue("agents", "No retained agent observations")+ for agent, raw in projection.items(agents):+ info = _mapping(raw)+ for arm, raw_condition in projection.items(info.get("conditions"), limit=4):+ condition = _mapping(raw_condition)+ projection.add(+ "conditions",+ agent,+ arm,+ condition.get("execution_status"),+ *(+ _count(condition.get(key))+ for key in ("expected_cases", "scored_cases", "expected_attempts", "scored_attempts")+ ),+ )+ summaries = _mapping(info.get("plugin_signals_summary"))+ if not summaries:+ projection.issue(agent, "Runtime behavior evidence is unavailable")+ for arm, raw_summary in projection.items(summaries, limit=4):+ summary = _mapping(raw_summary)+ if summary.get("evidence_complete") is not True:+ projection.issue(f"{agent}/{arm}", "Runtime behavior evidence is incomplete")+ for key in ("n_trials", "n_missing_trials", "n_missing_trajectory", "n_incomplete_trajectory"):+ projection.add(+ "signals",+ agent,+ arm,+ key,+ "recorded" if _count(summary.get(key)) is not None else "unknown",+ _count(summary.get(key)),+ None,+ None,+ )+ for key, metric in (+ ("routing", "f1"),+ ("tool_selection", "f1"),+ ("arguments", "pass_rate"),+ ("order", "satisfaction_rate"),+ ("handoff", "pass_rate"),+ ("conflict", "pass_rate"),+ ):+ value = _mapping(summary.get(key))+ projection.add(+ "signals",+ agent,+ arm,+ key,+ value.get("status") or "unavailable",+ _count(value.get("n_scored")),+ _count(value.get("skipped")),+ value.get(metric),+ )+ coverage = _mapping(summary.get("activation_coverage"))+ declared = coverage.get("declared")+ if isinstance(declared, list):+ # Membership work is bounded even for an in-memory hostile input.+ states = {}+ for key in ("exercised", "unavailable"):+ values = coverage.get(key, [])+ if not isinstance(values, list):+ projection.issue(f"{agent}/{arm}", f"Malformed {key} coverage evidence")+ values = []+ states[key] = {value for value in islice(values, 4096) if isinstance(value, str)}+ for label in islice(declared, 4096):+ if isinstance(label, str):+ state = (+ "exercised"+ if label in states["exercised"]+ else "unavailable"+ if label in states["unavailable"]+ else "unverified"+ )+ projection.add(+ "coverage", agent, arm, label, state, _mapping(coverage.get("exercise_rate")).get(label)+ )+ projection.totals["coverage"] += max(0, len(declared) - 4096)+ for server, counts in projection.items(_mapping(summary.get("mcp_calls")).get("by_server"), limit=64):+ counts = _mapping(counts)+ projection.add(+ "mcp_calls",+ agent,+ arm,+ server,+ *(_count(counts.get(field)) for field in ("total", "succeeded", "failed", "unknown")),+ )+++def _comparison_views(projection: _RuntimeProjection, comparisons: dict) -> None:+ if comparisons and comparisons.get("status") != "recorded":+ projection.issue("comparisons", "Comparison evidence is incomplete")+ projection.errors("comparisons", comparisons.get("errors"))+ for agent, raw in projection.items(comparisons.get("agents")):+ stats = _mapping(raw)+ ladder = _mapping(stats.get("lift_ladder"))+ for key in ("basis", "expected_cases", "n_cases", "partial"):+ if key in ladder:+ projection.add("run", f"{agent} comparison {key}", ladder[key])+ for key in ("confidence", "resamples", "method", "interval"):+ if key in _mapping(ladder.get("bootstrap")):+ projection.add("run", f"{agent} bootstrap {key}", ladder["bootstrap"][key])+ estimates = [*projection.items(ladder.get("steps"), limit=4)]+ estimates.extend((name, ladder[name]) for name in ("bundle", "glue") if isinstance(ladder.get(name), dict))+ estimates.extend(+ (+ _PAIR_LABELS.get(name, _PAIR_LABELS["effectiveness"]),+ step,+ )+ for name, step in projection.items(stats.get("lift_uncertainty"), limit=2)+ )+ for name, raw_step in estimates:+ step = _mapping(raw_step)+ projection.add(+ "comparisons",+ agent,+ name,+ step.get("estimate", step.get("lift")),+ step.get("ci_low"),+ step.get("ci_high"),+ step.get("n_cases", step.get("n")),+ step.get("precision"),+ step.get("partial", ladder.get("partial")),+ )+ claim = _mapping(stats.get("integration_claim"))+ projection.add(+ "eligibility",+ agent,+ claim.get("label") or "Integration lift",+ claim.get("eligible"),+ claim.get("verdict"),+ claim.get("reason") or ("claim unavailable" if not claim else None),+ )+ loose = _mapping(stats.get("loose_parts"))+ if loose.get("measured") is not True:+ projection.add(+ "eligibility",+ agent,+ "Loose-parts control",+ False,+ "not measured",+ loose.get("not_measured_reason") or "Measurement evidence unavailable",+ )+ else:+ projection.add(+ "eligibility",+ agent,+ "Content parity",+ loose.get("equal_across_arms"),+ "recorded",+ loose.get("content_basis"),+ )+ for arm, pieces in projection.items(_mapping(loose.get("load_evidence")).get("pieces"), limit=4):+ for piece, raw_counts in projection.items(pieces, limit=256):+ counts = _mapping(raw_counts)+ state = (+ "not_loaded"+ if counts.get("not_loaded")+ else "unverified"+ if counts.get("unverifiable")+ else "loaded"+ if counts.get("loaded")+ else "unknown"+ )+ projection.add(+ "loading",+ agent,+ arm,+ "comparison piece",+ piece,+ state,+ f"loaded={_count(counts.get('loaded'))}; not_loaded={_count(counts.get('not_loaded'))}; unverified={_count(counts.get('unverifiable'))}",+ )+ seen_exclusions = set()+ for entry in projection.records("exclusions", ladder.get("excluded_cases")):+ case, reason = entry.get("case_id", entry.get("case")), entry.get("reason")+ projection.add("exclusions", agent, case, reason)+ if isinstance(case, str) and isinstance(reason, str):+ seen_exclusions.add((case, reason))+ eligibility = _mapping(stats.get("comparison_eligibility"))+ for measure, label in _PAIR_LABELS.items():+ mismatches = _mapping(eligibility.get(measure)).get("score_policy_mismatches")+ for case in projection.labels("exclusions", mismatches):+ reason = f"{label}: score_policy_mismatch"+ if (case, reason) not in seen_exclusions:+ projection.add("exclusions", agent, case, reason)+ seen_exclusions.add((case, reason))+ for arm, raw_cost in projection.items(stats.get("cost"), limit=4):+ cost = _mapping(raw_cost)+ projection.add(+ "cost",+ agent,+ arm,+ *(+ cost.get(key)+ for key in ("total_tokens", "total_usd", "successes", "tokens_per_success", "usd_per_success")+ ),+ )+ context = _mapping(stats.get("context_cost_measured"))+ excluded = _mapping(context.get("excluded"))+ projection.add(+ "context",+ agent,+ context.get("status") or "unavailable",+ context.get("delta_tokens_mean"),+ context.get("n_pairs"),+ context.get("partial"),+ excluded.get("hosted_search"),+ excluded.get("missing_tokens"),+ context.get("reason"),+ )+++def project_plugin_runtime(+ result: Mapping[str, Any], *, provenance: Mapping[str, Any] | None = None+) -> RuntimePluginReport | None:+ """Copy recorded observations into bounded views without rescoring or I/O."""+ provenance = dict(provenance or {})+ if not any(+ key in result for key in ("runtime_evidence", "native_evidence", "private_comparisons")+ ) and not provenance.get("plugin"):+ return None+ projection = _RuntimeProjection()+ observed = _mapping(provenance.get("runtime_evidence") or result.get("runtime_evidence"))+ comparisons = _mapping(provenance.get("private_comparisons") or result.get("private_comparisons"))+ native = _mapping(provenance.get("native_evidence") or result.get("native_evidence"))+ if observed.get("evidence_complete") is not True:+ projection.issue("runtime", "Runtime evidence is unavailable or incomplete")+ if provenance.get("complete") is False:+ projection.issue("preparation", "The prepared package has incomplete evidence")+ projection.errors("runtime", observed.get("errors"))+ # ``error`` is the engine's compatibility copy of ``execution_errors``.+ execution_errors = result.get("execution_errors")+ projection.errors("execution", execution_errors if isinstance(execution_errors, list) else result.get("error"))+ for key in ("native_evidence_error", "runtime_report_error"):+ if error := provenance.get(key) or result.get(key):+ projection.issue(key, error)+ for reason in islice(provenance.get("incomplete_reasons") or (), 64):+ projection.issue("preparation", reason)+ run_id = PurePosixPath(str(result.get("run_dir") or "unknown").replace("\\", "/")).name+ projection.add("run", "Run", run_id)+ projection.add("run", "Execution status", result.get("execution_status") or "unknown")+ if "attempts" in provenance:+ projection.add("run", "Setup attempts", _count(provenance["attempts"]))+ for reason in islice(provenance.get("retry_reasons") or (), 2):+ projection.add("run", "Setup retry reason", reason)+ source = _mapping(provenance.get("source") or observed.get("source"))+ source = {+ key: _clean_text(source[key])[0]+ for key in ("manifest", "manifest_sha256", "source_tree_sha256")+ if isinstance(source.get(key), str)+ }+ for key, value in source.items():+ projection.add("run", key, value)+ _load_views(projection, native, _mapping(provenance.get("load_decisions")))+ _signal_views(projection, result.get("agents"))+ for name, raw in projection.items(observed.get("mcp_proof"), limit=64):+ proof = _mapping(raw)+ projection.add(+ "mcp",+ name,+ proof.get("status") or "unknown",+ projection.bound.names(proof.get("tools")),+ proof.get("detail"),+ )+ _comparison_views(projection, comparisons)+ sections = projection.finish()+ payload = {+ "schema_version": SCHEMA_VERSION,+ "run_id": _clean_text(run_id)[0],+ "name": _clean_text(str(provenance.get("plugin") or "Plugin runtime"))[0],+ "execution_status": _clean_text(str(result.get("execution_status") or "unknown"))[0],+ "status": "complete" if projection.complete else "incomplete",+ "evidence_complete": projection.complete,+ "truncated": projection.bound.truncated,+ "source": source,+ "sections": sections,+ }+ # Settle sanitization through the validator that replay uses, so the live+ # report, its provenance copy and the saved snapshot are the same bytes.+ from skillevaluator.reporting.plugin_runtime_io import _validate++ return _validate(payload)diff --git a/src/skillevaluator/reporting/plugin_runtime_io.py b/src/skillevaluator/reporting/plugin_runtime_io.pynew file mode 100644index 00000000..ef301538--- /dev/null+++ b/src/skillevaluator/reporting/plugin_runtime_io.py@@ -0,0 +1,151 @@+# SPDX-License-Identifier: Apache-2.0+"""Versioned private display snapshots, independent of optional runtime packages."""++from __future__ import annotations++import json+from pathlib import Path++from skillevaluator.reporting.plugin_runtime import SCHEMA_VERSION, SECTIONS, RuntimePluginReport+from skillevaluator.reporting.plugin_static import MAX_OUTPUT_BYTES, _clean_text, _number+from skillevaluator.utils.secure_fs import SecureRoot+from skillevaluator.utils.structured_data import load_bounded_json++RUNTIME_REPORT_FILENAME = "plugin_runtime_report.json"+++def _settle(value: str) -> tuple[str, bool]:+ # _clean_text redacts before it joins lines, so one pass over multi-line+ # text is not always stable. Repeat until a pass changes nothing: a saved+ # snapshot then replays to exactly the report that was saved. Also report+ # whether any pass cut the text at the display bound.+ cut = False+ for _ in range(8):+ cleaned, truncated = _clean_text(value)+ cut |= truncated+ if cleaned == value:+ break+ value = cleaned+ return value, cut+++def _validate(payload: object) -> RuntimePluginReport:+ if (+ not isinstance(payload, dict)+ or type(payload.get("schema_version")) is not int+ or payload["schema_version"] != SCHEMA_VERSION+ ):+ raise ValueError("Unsupported private runtime report schema")+ if not isinstance(payload.get("status"), str) or payload["status"] not in {"complete", "incomplete"}:+ raise ValueError("Invalid runtime evidence status")+ for key in ("evidence_complete", "truncated"):+ if type(payload.get(key)) is not bool:+ raise ValueError(f"Invalid runtime report {key}")+ if payload["evidence_complete"] is not (payload["status"] == "complete"):+ raise ValueError("Inconsistent runtime report completeness")+ cut = False++ def settle(value: str) -> str:+ nonlocal cut+ value, truncated = _settle(value)+ cut |= truncated+ return value++ for key in ("name", "run_id", "execution_status"):+ if not isinstance(payload.get(key), str):+ raise ValueError(f"Invalid runtime report {key}")+ payload[key] = settle(payload[key])+ source = payload.get("source")+ if (+ not isinstance(source, dict)+ or not source.keys() <= {"manifest", "manifest_sha256", "source_tree_sha256"}+ or not all(isinstance(value, str) for value in source.values())+ ):+ raise ValueError("Invalid runtime report source")+ payload["source"] = {key: settle(value) for key, value in source.items()}+ sections = payload.get("sections")+ if not isinstance(sections, list) or len(sections) != len(SECTIONS):+ raise ValueError("Invalid runtime report sections")+ seen, row_count = set(), 0+ for section in sections:+ if (+ not isinstance(section, dict)+ or not isinstance(section.get("key"), str)+ or section["key"] not in SECTIONS+ or section["key"] in seen+ ):+ raise ValueError("Invalid runtime report section")+ key = section["key"]+ if section.keys() != {"key", "title", "columns", "rows", "total", "omitted", "notes"}:+ raise ValueError("Unexpected runtime report section fields")+ seen.add(key)+ title, columns = SECTIONS[key]+ if section.get("title") != title or section.get("columns") != list(columns):+ raise ValueError("Incompatible runtime report columns")+ rows, notes = section.get("rows"), section.get("notes")+ if not isinstance(rows, list) or len(rows) > 64 or not isinstance(notes, list) or len(notes) > 8:+ raise ValueError("Invalid runtime report rows")+ for row in rows:+ if not isinstance(row, list) or len(row) != len(columns):+ raise ValueError("Invalid runtime report row width")+ for index, value in enumerate(row):+ if value is not None and type(value) not in (str, int, float):+ raise ValueError("Invalid runtime report cell")+ if type(value) in (int, float) and _number(value) is None:+ raise ValueError("Invalid runtime report number")+ if isinstance(value, str):+ row[index] = settle(value)+ if not all(isinstance(note, str) for note in notes):+ raise ValueError("Invalid runtime report notes")+ section["notes"] = [settle(note) for note in notes]+ total, omitted = section.get("total"), section.get("omitted")+ if (+ type(total) is not int+ or type(omitted) is not int+ or not 0 <= omitted <= total+ or total != len(rows) + omitted+ ):+ raise ValueError("Invalid runtime report omission counts")+ row_count += len(rows)+ if row_count > 256:+ raise ValueError("Runtime report row bound exceeded")+ allowed = {+ "schema_version",+ "run_id",+ "name",+ "execution_status",+ "status",+ "evidence_complete",+ "truncated",+ "source",+ "sections",+ }+ if payload.keys() != allowed:+ raise ValueError("Unexpected runtime report fields")+ # Text cut while settling is as incomplete as text cut during projection.+ payload["truncated"] = payload["truncated"] or cut+ serialized = json.dumps(payload, ensure_ascii=True, allow_nan=False)+ if len(serialized.encode("utf-8")) > MAX_OUTPUT_BYTES:+ raise ValueError("Runtime report exceeds its publication bound")+ return RuntimePluginReport(serialized)+++def save_plugin_runtime_report(report: RuntimePluginReport, run_dir: Path) -> None:+ """Atomically retain one bounded sanitized snapshot in the selected run directory."""+ # Writing needs the tier3 output helpers; replay stays free of tier3 imports.+ from skillevaluator.tier3.output_provenance import write_output_file_atomically++ checked = _validate(report.payload())+ encoded = checked._serialized.encode("utf-8")+ write_output_file_atomically(run_dir / RUNTIME_REPORT_FILENAME, encoded)+++def load_plugin_runtime_report(run_dir: Path) -> RuntimePluginReport:+ """Replay a selected snapshot; missing older snapshots are explicitly unavailable."""+ try:+ with SecureRoot(run_dir) as root:+ raw, _ = root.read_bytes(Path(RUNTIME_REPORT_FILENAME), MAX_OUTPUT_BYTES)+ payload = load_bounded_json(raw.decode("utf-8"), max_nodes=50_000, max_tokens=50_000, max_collection_items=4096)+ return _validate(payload)+ except (OSError, UnicodeError, ValueError) as exc:+ raise ValueError(f"Saved private runtime report unavailable: {exc}") from excdiff --git a/src/skillevaluator/reporting/plugin_runtime_render.py b/src/skillevaluator/reporting/plugin_runtime_render.pynew file mode 100644index 00000000..30d4574f--- /dev/null+++ b/src/skillevaluator/reporting/plugin_runtime_render.py@@ -0,0 +1,68 @@+# SPDX-License-Identifier: Apache-2.0+"""Private runtime adapters over one retained, passive evidence projection."""++from __future__ import annotations++import json+from typing import TYPE_CHECKING++from skillevaluator.reporting.plugin_static_render import _cli, _markdown, render_plugin_static_reports++if TYPE_CHECKING:+ from collections.abc import Iterable++ from skillevaluator.models.result import ValidationResult+ from skillevaluator.reporting.base import ReporterBase+ from skillevaluator.reporting.plugin_runtime import RuntimePluginReport++_TITLE = "Plugin runtime evidence"+_NOTICE = "Advisory observations do not change scores or validation gates."+_TRUNCATION_NOTICE = "Some runtime evidence was omitted by report bounds; see section counts."+++def _augment(reporter: ReporterBase, document: str, payload: dict) -> str:+ if reporter.name == "json":+ data = json.loads(document)+ data["plugin_runtime"] = payload+ return json.dumps(data, indent=reporter.indent, ensure_ascii=True, allow_nan=False)+ if reporter.name == "sarif":+ data = json.loads(document)+ data["runs"][0].setdefault("properties", {})["pluginRuntime"] = payload+ return json.dumps(data, indent=reporter.indent, ensure_ascii=True, allow_nan=False)+ if reporter.name == "html":+ fragment = reporter._env.get_template("plugin_runtime_sections.html.j2").render(plugin_runtime=payload)+ if "</main>" not in document:+ raise ValueError("The HTML reporter did not provide its main content boundary")+ return document.replace("</main>", fragment + "\n</main>", 1)+ options = {+ "title": _TITLE,+ "notice": _NOTICE,+ "truncation_notice": _TRUNCATION_NOTICE,+ "status_label": "Evidence status",+ "header_fields": (("Run", payload["run_id"]), ("Execution status", payload["execution_status"])),+ }+ if reporter.name in {"markdown", "benchmark"}:+ fragment = _markdown(payload, publication=reporter.name == "benchmark", **options)+ return document.rstrip() + "\n\n" + fragment.rstrip() + "\n"+ if reporter.name == "cli":+ return document.rstrip() + "\n\n" + _cli(payload, **options)+ raise ValueError(f"Unsupported private runtime report format: {reporter.name}")+++def render_plugin_runtime_reports(+ results: list[ValidationResult],+ *,+ report: RuntimePluginReport | None,+ reporters: Iterable[ReporterBase],+) -> dict[str, str]:+ """Compose static and runtime evidence without collecting or changing results.++ A saved report follows the same path as a live report. The private adapter+ does not add public dispatch or require an optional execution dependency.+ """+ selected = tuple(reporters)+ rendered = render_plugin_static_reports(results, reporters=selected)+ if report is None:+ return rendered+ payload = report.payload()+ return {reporter.name: _augment(reporter, rendered[reporter.name], payload) for reporter in selected}diff --git a/src/skillevaluator/reporting/plugin_static_render.py b/src/skillevaluator/reporting/plugin_static_render.pyindex 4818a76f..5a731f75 100644--- a/src/skillevaluator/reporting/plugin_static_render.py+++ b/src/skillevaluator/reporting/plugin_static_render.py@@ -20,7 +20,16 @@ def _display(value: object) -> str:return "unknown" if value is None else str(value)-def _markdown(payload: dict, *, publication: bool = False) -> str:+def _markdown(+ payload: dict,+ *,+ publication: bool = False,+ title: str = "Plugin static evidence",+ notice: str = "Static evidence does not demonstrate runtime loading or behavior.",+ truncation_notice: str = "Some static evidence was omitted by report bounds; see section counts.",+ status_label: str = "",+ header_fields: tuple[tuple[str, object], ...] = (),+) -> str:if publication:from skillevaluator.reporting.benchmark import _md_cell as escapeelse:@@ -35,22 +44,23 @@ def _markdown(payload: dict, *, publication: bool = False) -> str:return encodedlines = [- "## Plugin static evidence",+ f"## {title}","",- f"**{cell(payload['name'])}** — {payload['status']}",+ f"**{cell(payload['name'])}** — {status_label + ': ' if status_label else ''}{cell(payload['status'])}","",- "Static evidence does not demonstrate runtime loading or behavior.",+ *[f"{label}: {cell(value)}\n" for label, value in header_fields],+ notice,"",]if payload["truncated"]:- lines.extend(["Some static evidence was omitted by report bounds; see section counts.", ""])+ lines.extend([truncation_notice, ""])for section in payload["sections"]:- lines.extend([f"### {section['title']}", ""])+ lines.extend([f"### {cell(section['title'])}", ""])lines.extend(f"{cell(note)}\n" for note in section["notes"] if note is not None)if section["rows"]:lines.extend([- "| " + " | ".join(section["columns"]) + " |",+ "| " + " | ".join(cell(column) for column in section["columns"]) + " |","| " + " | ".join("---" for _ in section["columns"]) + " |",])@@ -61,19 +71,28 @@ def _markdown(payload: dict, *, publication: bool = False) -> str:return "\n".join(lines)-def _cli(payload: dict) -> str:+def _cli(+ payload: dict,+ *,+ title: str = "Plugin static evidence",+ notice: str = "Static evidence does not demonstrate runtime loading or behavior.",+ truncation_notice: str = "Static evidence was omitted by report bounds; see section counts.",+ status_label: str = "",+ header_fields: tuple[tuple[str, object], ...] = (),+) -> str:from rich.console import Consolefrom rich.text import Textoutput = io.StringIO()console = Console(file=output, color_system=None, emoji=False)lines = [- "Plugin static evidence",- f"{payload['name']} — {payload['status']}",- "Static evidence does not demonstrate runtime loading or behavior.",+ title,+ f"{payload['name']} — {status_label + ': ' if status_label else ''}{payload['status']}",+ *[f"{label}: {_display(value)}" for label, value in header_fields],+ notice,]if payload["truncated"]:- lines.append("Static evidence was omitted by report bounds; see section counts.")+ lines.append(truncation_notice)for section in payload["sections"]:lines.extend(["", section["title"], " | ".join(section["columns"])])lines.extend(_display(note) for note in section["notes"] if note is not None)diff --git a/src/skillevaluator/reporting/templates/plugin_runtime_sections.html.j2 b/src/skillevaluator/reporting/templates/plugin_runtime_sections.html.j2new file mode 100644index 00000000..8a90176f--- /dev/null+++ b/src/skillevaluator/reporting/templates/plugin_runtime_sections.html.j2@@ -0,0 +1,20 @@+<section class="table-section" id="plugin-runtime-evidence" aria-label="Plugin runtime evidence">+ <h2>Plugin runtime evidence: {{ plugin_runtime.name }}</h2>+ <p>Evidence status: {{ plugin_runtime.status }}. Execution status: {{ plugin_runtime.execution_status }}.</p>+ <p>Run: {{ 'unknown' if plugin_runtime.run_id is none else plugin_runtime.run_id }}</p>+ <p>Advisory observations do not change scores or validation gates.</p>+ {% if plugin_runtime.truncated %}<p>Some runtime evidence was omitted by report bounds; see section counts.</p>{% endif %}+ {% for section in plugin_runtime.sections %}+ <section aria-label="{{ section.title }}">+ <h3>{{ section.title }}</h3>+ {% for note in section.notes %}{% if note is not none %}<p>{{ note }}</p>{% endif %}{% endfor %}+ {% if section.rows %}+ <div class="table-scroll" tabindex="0" role="region" aria-label="{{ section.title }}"><table class="contributor-table plugin-runtime-table">+ <thead><tr>{% for column in section.columns %}<th scope="col">{{ column }}</th>{% endfor %}</tr></thead>+ <tbody>{% for row in section.rows %}<tr>{% for cell in row %}<td>{{ 'unknown' if cell is none else cell }}</td>{% endfor %}</tr>{% endfor %}</tbody>+ </table></div>+ {% endif %}+ {% if section.omitted %}<p>{{ section.omitted }} more rows omitted ({{ section.total }} total).</p>{% endif %}+ </section>+ {% endfor %}+</section>diff --git a/src/skillevaluator/tier3/harbor/collector.py b/src/skillevaluator/tier3/harbor/collector.pyindex e9bc495c..39cd53c7 100644--- a/src/skillevaluator/tier3/harbor/collector.py+++ b/src/skillevaluator/tier3/harbor/collector.py@@ -6649,6 +6649,8 @@ def _condition_execution_summary("execution_error_details_total": 0,"execution_error_details_shown": 0,"execution_error_details_truncated": False,+ "expected_cases": 0,+ "scored_cases": 0,"expected_attempts": 0,"scored_attempts": 0,**_failure_list_metadata("runtime_failure_details", runtime_failures),@@ -6807,6 +6809,8 @@ def _condition_execution_summary("execution_error_details_total": len(all_errors),"execution_error_details_shown": len(errors),"execution_error_details_truncated": len(errors) < len(all_errors),+ "expected_cases": expected_count,+ "scored_cases": len(expected_set.intersection(by_case)) if expected_ids else len(by_case),"expected_attempts": expected_attempts,"scored_attempts": scored_attempts,**(diff --git a/src/skillevaluator/tier3/plugin_comparisons.py b/src/skillevaluator/tier3/plugin_comparisons.pyindex f94e12a7..17f16fb4 100644--- a/src/skillevaluator/tier3/plugin_comparisons.py+++ b/src/skillevaluator/tier3/plugin_comparisons.py@@ -419,9 +419,9 @@ class ComparisonRuntime:)by_case.setdefault(record.case_id, []).append(row)failures = load_failures_by_case(evidence)- cross_cases = {- case for case in self.expected_case_ids if observer.context.case_spec(case).get("cross_component")- }+ # Signal specs deliberately omit dataset classification fields.+ # Reuse the context's canonical two-member classification instead.+ cross_cases = getattr(observer.context, "cross_component_cases", None)stats = build_agent_statistics(arms,expected_case_ids=self.expected_case_ids,diff --git a/src/skillevaluator/tier3/plugin_execution.py b/src/skillevaluator/tier3/plugin_execution.pyindex d49520fc..ab90c5c3 100644--- a/src/skillevaluator/tier3/plugin_execution.py+++ b/src/skillevaluator/tier3/plugin_execution.py@@ -489,6 +489,27 @@ def evaluate_prepared_plugin(outcome.provenance.update(runtime_evidence=observed, mcp_proof=observed["mcp_proof"])if comparison_runtime is not None:outcome.provenance["private_comparisons"] = comparison_runtime.provenance()+ # Preserve a passive view while preparation provenance and all arm+ # summaries are still available. Its failure cannot suppress caller+ # publication or change the engine's status and cleanup semantics.+ try:+ from skillevaluator.reporting.plugin_runtime import project_plugin_runtime+ from skillevaluator.reporting.plugin_runtime_io import save_plugin_runtime_report++ report = project_plugin_runtime(+ outcome.result,+ provenance={**outcome.provenance, "attempts": outcome.attempts, "retry_reasons": outcome.retry_reasons},+ )+ if report is not None:+ outcome.provenance["runtime_report"] = report.payload()+ if run_dir := outcome.result.get("run_dir"):+ save_plugin_runtime_report(report, Path(run_dir))+ except Exception as exc:+ from skillevaluator.utils.redaction import redact_printable_text++ error = redact_printable_text(f"Runtime report: {type(exc).__name__}: {exc}", max_len=600)+ outcome.provenance["runtime_report_error"] = error+ outcome.publication_error = errorif publish is not None:publish(outcome)diff --git a/tests/reporting/test_private_runtime_projection.py b/tests/reporting/test_private_runtime_projection.pynew file mode 100644index 00000000..592a1b64--- /dev/null+++ b/tests/reporting/test_private_runtime_projection.py@@ -0,0 +1,395 @@+# SPDX-License-Identifier: Apache-2.0+"""Private views preserve evidence strength and the existing numerical basis."""++from __future__ import annotations++import importlib+import json+from copy import deepcopy++import pytest+++def api():+ return importlib.import_module("skillevaluator.reporting.plugin_runtime")+++def sample():+ return {+ "run_dir": "/retained/run-42",+ "execution_status": "succeeded",+ "runtime_evidence": {+ "evidence_complete": True,+ "source": {"manifest_sha256": "a" * 64, "source_tree_sha256": "b" * 64},+ "mcp_proof": {"lookup": {"status": "declared", "tools": [], "detail": "probe not requested"}},+ "errors": [],+ },+ "native_evidence": {+ "claude-code": {+ "mode": "native",+ "trials": 5,+ "loaded": [{"type": "skill", "name": "member", "evidence": "startup event"}],+ "listed": [{"type": "rule", "name": "policy", "evidence": "setup listing"}],+ "staged": [{"type": "mcp", "name": "lookup", "evidence": "staged only"}],+ "not_loaded": [{"type": "hook", "name": "startup", "reason": "permission refused"}],+ }+ },+ "agents": {+ "claude-code": {+ "plugin_signals_summary": {+ "with_skill": {+ "n_trials": 5,+ "n_missing_trials": 0,+ "n_missing_trajectory": 0,+ "evidence_complete": True,+ "routing": {"status": "scored", "n_scored": 5, "f1": 0.8},+ "arguments": {"status": "not_applicable", "n_scored": 0, "pass_rate": None},+ "activation_coverage": {+ "declared": ["Skill:member", "MCP:lookup"],+ "exercised": ["Skill:member"],+ "unverified": ["MCP:lookup"],+ "exercise_rate": {"Skill:member": 1.0, "MCP:lookup": 0.0},+ },+ "mcp_calls": {"by_server": {"lookup": {"total": 1, "succeeded": 0, "failed": 1, "unknown": 0}}},+ }+ }+ }+ },+ "private_comparisons": {+ "status": "recorded",+ "errors": [],+ "agents": {+ "claude-code": {+ "lift_ladder": {+ "partial": False,+ "n_cases": 5,+ "steps": {+ "skill": {+ "estimate": 0.2,+ "ci_low": 0.1,+ "ci_high": 0.3,+ "n_cases": 5,+ "precision": "adequate",+ }+ },+ "excluded_cases": [{"case_id": "excluded", "reason": "mixed judges"}],+ },+ "integration_claim": {"eligible": False, "verdict": "inconclusive", "reason": "load unverified"},+ "loose_parts": {"measured": False, "not_measured_reason": "unsupported adapter"},+ "cost": {+ "with_skill": {+ "total_tokens": 0,+ "total_usd": None,+ "successes": 0,+ "tokens_per_success": None,+ "usd_per_success": None,+ }+ },+ "context_cost_measured": {+ "status": "unavailable",+ "delta_tokens_mean": None,+ "n_pairs": 0,+ "partial": False,+ "excluded": {"hosted_search": 0, "missing_tokens": 5},+ "reason": "missing first-turn tokens",+ },+ }+ },+ },+ }+++def sections(payload):+ return {section["key"]: section for section in payload["sections"]}+++def test_projection_keeps_staging_loading_unknowns_and_recorded_statistics_separate():+ result = sample()+ before = deepcopy(result)+ report = api().project_plugin_runtime(result, provenance={"plugin": "fixture", "complete": True})+ payload = report.payload()+ assert payload["run_id"] == "run-42" and payload["execution_status"] == "succeeded"+ assert payload["name"] == "fixture" and payload["schema_version"] == 1+ assert payload["source"]["manifest_sha256"] == "a" * 64+ rows = sections(payload)+ assert {row[4] for row in rows["loading"]["rows"]} == {"loaded", "listed", "staged", "not_loaded"}+ assert any("permission refused" in row for row in rows["loading"]["rows"])+ assert ["claude-code", "with_skill", "MCP:lookup", "unverified", 0.0] in rows["coverage"]["rows"]+ assert ["claude-code", "skill", 0.2, 0.1, 0.3, 5, "adequate", "no"] in rows["comparisons"]["rows"]+ assert ["claude-code", "with_skill", 0, None, 0, None, None] in rows["cost"]["rows"]+ assert any("load unverified" in row for row in rows["eligibility"]["rows"])+ assert any("unsupported adapter" in row for row in rows["eligibility"]["rows"])+ assert any("mixed judges" in row for row in rows["exclusions"]["rows"])+ assert result == before+ payload["source"].clear()+ assert report.payload()["source"]+++def complete_sample():+ """A fixture with no limitation of its own, so each check is observable."""+ result = sample()+ native = result["native_evidence"]["claude-code"]+ native.pop("listed")+ native.pop("staged")+ return result+++def diagnostics(payload):+ return sections(payload)["diagnostics"]["rows"]+++def test_complete_baseline_has_no_evidence_limitations():+ payload = api().project_plugin_runtime(complete_sample()).payload()+ assert payload["status"] == "complete" and payload["evidence_complete"] is True+ assert diagnostics(payload) == []+++ ("scope", "expected"),+ [+ ("runtime", ["runtime", "Runtime evidence is unavailable or incomplete"]),+ ("arm", ["claude-code/with_skill", "Runtime behavior evidence is incomplete"]),+ ],+)+def test_successful_execution_with_missing_evidence_is_not_a_complete_report(scope, expected):+ result = complete_sample()+ summary = result["agents"]["claude-code"]["plugin_signals_summary"]["with_skill"]+ summary.update(n_missing_trials=2, n_missing_trajectory=3)+ if scope == "runtime":+ result["runtime_evidence"]["evidence_complete"] = False+ else:+ summary["evidence_complete"] = False+ payload = api().project_plugin_runtime(result).payload()+ assert payload["execution_status"] == "succeeded"+ assert payload["status"] == "incomplete" and payload["evidence_complete"] is False+ assert diagnostics(payload) == [expected]+ signals = sections(payload)["signals"]["rows"]+ assert ["claude-code", "with_skill", "n_missing_trials", "recorded", 2, None, None] in signals+ assert ["claude-code", "with_skill", "n_missing_trajectory", "recorded", 3, None, None] in signals+++def test_execution_error_alias_is_not_listed_twice():+ result = complete_sample()+ result["execution_errors"] = ["claude-code: launch failed", "codex: launch failed"]+ # The engine keeps ``error`` as a compatibility copy of execution_errors.+ result["error"] = list(result["execution_errors"])+ payload = api().project_plugin_runtime(result).payload()+ assert diagnostics(payload) == [+ ["execution", "claude-code: launch failed"],+ ["execution", "codex: launch failed"],+ ]+ legacy = complete_sample()+ legacy["error"] = ["legacy launch failure"]+ assert diagnostics(api().project_plugin_runtime(legacy).payload()) == [["execution", "legacy launch failure"]]+++def test_ordinary_result_has_no_private_projection():+ assert api().project_plugin_runtime({"execution_status": "succeeded", "agents": {}}) is None+++def test_projection_sanitizes_all_identifiers_and_does_not_accept_invalid_numbers():+ result = sample()+ bad = "\x1b[31m<script>secret</script> token=fixture-not-a-real-secret\x00"+ result["runtime_evidence"]["errors"] = [bad]+ result["private_comparisons"]["agents"]["claude-code"]["cost"]["with_skill"]["total_tokens"] = float("nan")+ payload = api().project_plugin_runtime(result, provenance={"plugin": bad}).payload()+ text = json.dumps(payload, allow_nan=False)+ assert "fixture-not-a-real-secret" not in text+ assert "\\u001b" not in text and "\\u0000" not in text+ assert sections(payload)["cost"]["rows"][0][2] is None+++def test_large_coverage_has_explicit_omissions_with_bounded_output():+ result = sample()+ coverage = result["agents"]["claude-code"]["plugin_signals_summary"]["with_skill"]["activation_coverage"]+ coverage["declared"] = [f"Skill:member-{i}" for i in range(10000)]+ payload = api().project_plugin_runtime(result).payload()+ section = sections(payload)["coverage"]+ assert payload["truncated"] is True+ assert section["total"] == 10000 and section["omitted"] == 10000 - len(section["rows"])+ assert len(json.dumps(payload).encode()) <= api().MAX_OUTPUT_BYTES+ assert sum(len(section["rows"]) for section in payload["sections"]) <= 256+++def test_recorded_comparison_basis_and_attempt_coverage_survive_the_view():+ result = sample()+ result["agents"]["claude-code"]["conditions"] = {+ "with_skill": {+ "execution_status": "succeeded",+ "expected_cases": 5,+ "scored_cases": 5,+ "expected_attempts": 10,+ "scored_attempts": 10,+ }+ }+ stats = result["private_comparisons"]["agents"]["claude-code"]+ stats["lift_ladder"].update(basis="common scored dimensions", bootstrap={"confidence": 0.95, "resamples": 2000})+ stats["lift_uncertainty"] = {"integration": {"estimate": 0.4, "n_cases": 5}}+ payload = api().project_plugin_runtime(result).payload()+ views = sections(payload)+ assert ["claude-code", "with_skill", "succeeded", 5, 5, 10, 10] in views["conditions"]["rows"]+ assert any("common scored dimensions" in row for row in views["run"]["rows"])+ assert any(0.95 in row for row in views["run"]["rows"])+ assert any(row[1] == "Bundle lift (full minus member skills)" for row in views["comparisons"]["rows"])+++def test_staging_only_native_observations_stay_incomplete_despite_readable_trajectories():+ result = sample()+ payload = api().project_plugin_runtime(result).payload()+ assert payload["evidence_complete"] is False++[email protected]("expected,skipped", [({}, 0), ({"expected_order": [["command:review", "skill:member"]]}, 1)])+def test_unsupported_signal_expectations_retain_their_skipped_count(expected, skipped):+ from skillevaluator.tier3.eval_core.plugin_signals import compute_plugin_signals, summarize_plugin_signals++ signal = compute_plugin_signals(+ {"steps": [], "agent": {"name": "codex"}},+ expected,+ declared={"command": ["review"], "skill": ["member"]},+ )+ summary = summarize_plugin_signals([signal])+ summary["evidence_complete"] = True+ result = {+ "execution_status": "succeeded",+ "runtime_evidence": {"evidence_complete": True},+ "agents": {"codex": {"plugin_signals_summary": {"with_skill": summary}}},+ }+ payload = api().project_plugin_runtime(result).payload()+ section = sections(payload)["signals"]+ assert "Skipped" in section["columns"]+ row = next(row for row in section["rows"] if row[2] == "order")+ assert row[section["columns"].index("Skipped")] == skipped+ assert any("unsupported" in note for note in section["notes"])+++def test_load_inventory_bounds_report_all_withheld_rows():+ result = sample()+ native = result["native_evidence"]["claude-code"]+ native.clear()+ native["loaded"] = [{"type": "skill", "name": f"member-{i}"} for i in range(10000)]+ payload = api().project_plugin_runtime(result).payload()+ section = sections(payload)["loading"]+ assert section["total"] == 10000+ assert section["omitted"] == 10000 - len(section["rows"])++[email protected]("value", [None, [], True, "unavailable"])+def test_malformed_optional_evidence_is_explicit_and_does_not_crash(value):+ result = sample()+ result.update(native_evidence=value, private_comparisons=value)+ result["agents"]["claude-code"]["plugin_signals_summary"] = value+ payload = api().project_plugin_runtime(result).payload()+ assert payload["status"] == "incomplete"++[email protected]("value", [42, {}, True, "unexpected"])+def test_malformed_coverage_lists_cannot_invent_activation(value):+ result = complete_sample()+ coverage = result["agents"]["claude-code"]["plugin_signals_summary"]["with_skill"]["activation_coverage"]+ coverage.update(exercised=value, unavailable=value)+ payload = api().project_plugin_runtime(result).payload()+ assert all(row[3] == "unverified" for row in sections(payload)["coverage"]["rows"])+ assert payload["status"] == "incomplete"+ assert diagnostics(payload) == [+ ["claude-code/with_skill", "Malformed exercised coverage evidence"],+ ["claude-code/with_skill", "Malformed unavailable coverage evidence"],+ ]+++def test_case_coverage_uses_real_condition_summaries_for_every_arm():+ from skillevaluator.tier3.harbor.collector import _condition_execution_summary+ from skillevaluator.tier3.harbor.metrics import DEFAULT_METRIC_SET, DEFAULT_METRICS++ rewards = [+ {+ "entry_id": "recorded",+ "_trial_root_name": f"recorded__attempt{attempt}",+ "metric_set": DEFAULT_METRIC_SET,+ **dict.fromkeys(DEFAULT_METRICS, 0.8),+ }+ for attempt in (1, 2)+ ]+ condition = _condition_execution_summary(+ rewards,+ expected_case_ids=["recorded", "missing"],+ expected_cases=2,+ n_attempts=2,+ job_failure="",+ )+ result = sample()+ result["agents"]["claude-code"]["conditions"] = dict.fromkeys(+ ("with_skill", "without_skill", "sum_of_parts", "loose_parts"), condition+ )+ rows = sections(api().project_plugin_runtime(result).payload())["conditions"]["rows"]+ assert rows == [+ ["claude-code", "with_skill", "failed", 2, 1, 4, 2],+ ["claude-code", "without_skill", "failed", 2, 1, 4, 2],+ ["claude-code", "sum_of_parts", "failed", 2, 1, 4, 2],+ ["claude-code", "loose_parts", "failed", 2, 1, 4, 2],+ ]++[email protected]("member_skills", [False, True])+def test_pairwise_policy_exclusions_survive_without_a_loose_parts_ladder(member_skills):+ from skillevaluator.tier3.harbor.metrics import (+ DEFAULT_METRIC_SET,+ DEFAULT_METRICS,+ LEGACY_METRIC_SET,+ LEGACY_METRICS,+ )+ from skillevaluator.tier3.harbor.stats import ArmObservations, TrialObservation, build_agent_statistics++ def trial(case, value, *, legacy=False):+ return TrialObservation(+ case,+ {+ "entry_id": case,+ "metric_set": LEGACY_METRIC_SET if legacy else DEFAULT_METRIC_SET,+ **dict.fromkeys(LEGACY_METRICS if legacy else DEFAULT_METRICS, value),+ },+ )++ arms = {+ "with_skill": ArmObservations("succeeded", (trial("kept", 0.8), trial("excluded-policy", 0.8))),+ "without_skill": ArmObservations("succeeded", (trial("kept", 0.4), trial("excluded-policy", 0.4, legacy=True))),+ }+ if member_skills:+ arms["sum_of_parts"] = ArmObservations(+ "succeeded", (trial("kept", 0.6), trial("excluded-policy", 0.6, legacy=True))+ )+ stats = build_agent_statistics(+ arms,+ expected_case_ids=["kept", "excluded-policy"],+ n_attempts=1,+ stop_on_pass=False,+ pass_threshold=0.5,+ sum_of_parts_requested=member_skills,+ )+ assert stats["lift_ladder"] is None+ # Repeated retained entries must not create duplicate displayed exclusions.+ stats["comparison_eligibility"]["effectiveness"]["score_policy_mismatches"] *= 2+ result = sample()+ result["private_comparisons"]["agents"]["claude-code"] = stats+ excluded = sections(api().project_plugin_runtime(result).payload())["exclusions"]+ expected = [["claude-code", "excluded-policy", "Effectiveness lift (full minus baseline): score_policy_mismatch"]]+ if member_skills:+ expected.append(+ ["claude-code", "excluded-policy", "Bundle lift (full minus member skills): score_policy_mismatch"]+ )+ assert excluded["rows"] == expected+ assert excluded["total"] == len(expected) and excluded["omitted"] == 0+++def test_pairwise_policy_exclusion_bounds_count_every_withheld_entry():+ result = sample()+ stats = result["private_comparisons"]["agents"]["claude-code"]+ stats["lift_ladder"] = None+ stats["comparison_eligibility"] = {+ "effectiveness": {"score_policy_mismatches": [f"case-{i}" for i in range(10000)]}+ }+ payload = api().project_plugin_runtime(result).payload()+ excluded = sections(payload)["exclusions"]+ assert excluded["total"] == 10000+ assert excluded["rows"] and excluded["omitted"] == 10000 - len(excluded["rows"])+ assert payload["truncated"] is Truediff --git a/tests/reporting/test_private_runtime_render.py b/tests/reporting/test_private_runtime_render.pynew file mode 100644index 00000000..8d5d4283--- /dev/null+++ b/tests/reporting/test_private_runtime_render.py@@ -0,0 +1,261 @@+# SPDX-License-Identifier: Apache-2.0+"""Private runtime adapters preserve shared evidence and inert display text."""++from __future__ import annotations++import html+import importlib+import json+import subprocess+import sys+from copy import deepcopy++import pytest++from skillevaluator.models.result import ValidationResult+from skillevaluator.reporting import (+ BenchmarkReporter,+ CLIReporter,+ HTMLReporter,+ JSONReporter,+ MarkdownReporter,+ SARIFReporter,+)+from skillevaluator.reporting.plugin_static_render import render_plugin_static_reports+++class RetainedReport:+ """Exercise rendering before the independent projection is available."""++ def __init__(self, payload: dict) -> None:+ self.serialized = json.dumps(payload)+ self.reads = 0++ def payload(self) -> dict:+ self.reads += 1+ return json.loads(self.serialized)+++def renderers():+ return importlib.import_module("skillevaluator.reporting.plugin_runtime_render")+++def reporters():+ return [+ CLIReporter(),+ JSONReporter(include_timestamp=False),+ MarkdownReporter(include_timestamp=False),+ HTMLReporter(include_timestamp=False),+ SARIFReporter(include_timestamp=False),+ BenchmarkReporter(include_timestamp=False),+ ]+++def payload():+ return {+ "schema_version": 1,+ "name": "runtime-demo",+ "status": "incomplete",+ "run_id": "retained-run-7",+ "execution_status": "succeeded",+ "evidence_complete": False,+ "source": {"manifest_sha256": "a" * 64, "source_tree_sha256": "b" * 64},+ "truncated": True,+ "sections": [+ {+ "key": "coverage",+ "title": "Activation coverage",+ "columns": ["Agent", "Arm", "Component", "State", "Calls", "Cost"],+ "rows": [["claude-code", "with_skill", "docs-server", "unverified", 0, None]],+ "total": 3,+ "omitted": 2,+ "notes": ["One trajectory is missing; costs were not recorded."],+ },+ {+ "key": "comparison",+ "title": "Comparison uncertainty",+ "columns": ["Comparison", "Estimate", "Precision", "Reason"],+ "rows": [["Wrapper effect", 0.2, "insufficient", "Only one paired case"]],+ "total": 1,+ "omitted": 0,+ "notes": [],+ },+ ],+ }+++def bundle(*, static=False):+ result = ValidationResult(validator_name="Private checks")+ if static:+ result.metadata["plugin"] = {+ "name": "static-demo",+ "component_inventory": {+ "complete": True,+ "components": [{"type": "skill", "name": "member", "path": "skills/member"}],+ },+ }+ return [result]+++def test_all_six_formats_compose_static_and_runtime_without_losing_shared_facts():+ report = RetainedReport(payload())+ output = renderers().render_plugin_runtime_reports(bundle(static=True), report=report, reporters=iter(reporters()))++ assert set(output) == {"cli", "json", "markdown", "html", "sarif", "benchmark"}+ expected = payload()+ assert json.loads(output["json"])["plugin_runtime"] == expected+ assert json.loads(output["sarif"])["runs"][0]["properties"]["pluginRuntime"] == expected+ assert json.loads(output["json"])["plugin_static"]["name"] == "static-demo"+ for name in ("cli", "markdown", "html", "benchmark"):+ document = html.unescape(output[name]).replace("\\_", "_")+ for marker in (+ "Plugin runtime evidence",+ "runtime-demo",+ "retained-run-7",+ "Execution status: succeeded",+ "Evidence status: incomplete",+ "Advisory observations do not change scores or validation gates.",+ "docs-server",+ "unverified",+ "Wrapper effect",+ "Only one paired case",+ "2 more rows omitted (3 total).",+ "static-demo",+ ):+ assert marker in document, (name, marker)+ assert 'id="plugin-runtime-evidence"' in output["html"]+ assert 'class="contributor-table plugin-runtime-table"' in output["html"]+ assert output["html"].count('id="plugin-static-evidence"') == 1+ assert output["html"].count('id="plugin-runtime-evidence"') == 1+ assert report.reads == 1++[email protected]("static", [False, True])+def test_absent_runtime_report_keeps_existing_outputs_exactly(static):+ results = bundle(static=static)+ report_list = reporters()+ expected = render_plugin_static_reports(results, reporters=report_list)++ assert renderers().render_plugin_runtime_reports(results, report=None, reporters=iter(report_list)) == expected++[email protected]("force_color", [False, True])+def test_runtime_markup_stays_inert_in_html_markdown_and_rich(monkeypatch, force_color):+ monkeypatch.setenv("FORCE_COLOR", "1" if force_color else "0")+ raw = payload()+ raw["name"] = '<script>alert("name")</script>'+ raw["run_id"] = '<img src=x onerror="alert(1)">'+ raw["sections"][0]["title"] = "[heading](javascript:alert(1))"+ raw["sections"][0]["columns"][0] = "[label](javascript:alert(1))"+ raw["sections"][0]["rows"][0][2] = "[click](javascript:alert(1)) | `code` [bold red]text[/]"+ raw["sections"][0]["notes"] = ['<iframe src="javascript:alert(1)"></iframe>']+ output = renderers().render_plugin_runtime_reports(bundle(), report=RetainedReport(raw), reporters=reporters())++ fragment = output["html"].split('id="plugin-runtime-evidence"', 1)[1].split("</main>", 1)[0]+ assert "<script>" not in fragment and "<img " not in fragment and "<iframe " not in fragment+ assert "<script>" in fragment and "<img " in fragment and "<iframe " in fragment+ for name in ("markdown", "benchmark"):+ runtime = output[name].split("## Plugin runtime evidence", 1)[1]+ assert "<script>" not in runtime and "<img " not in runtime and "<iframe " not in runtime+ assert "[click](javascript:alert(1))" not in runtime+ assert "[heading](javascript:alert(1))" not in runtime+ assert "[label](javascript:alert(1))" not in runtime+ assert "| `code`" not in runtime+ runtime_cli = output["cli"].split("Plugin runtime evidence", 1)[1]+ assert "[bold red]text[/]" in runtime_cli+ assert "\x1b[" not in runtime_cli+++def test_unknown_and_zero_remain_distinct_and_rendering_does_not_mutate_inputs():+ results = bundle(static=True)+ before = deepcopy(results)+ raw = payload()+ report = RetainedReport(raw)+ encoded = report.serialized+ output = renderers().render_plugin_runtime_reports(results, report=report, reporters=reporters())++ assert results == before and report.serialized == encoded+ assert json.loads(output["json"])["plugin_runtime"]["sections"][0]["rows"][0][-2:] == [0, None]+ for name in ("cli", "markdown", "html", "benchmark"):+ assert "unknown" in output[name]+ assert "| 0 | unknown |" in output["markdown"]+ assert "<td>0</td><td>unknown</td>" in output["html"]+++def test_runtime_rendering_does_not_collect_source_or_network_evidence(monkeypatch):+ import socket++ from skillevaluator.plugin_paths import PluginRootReader++ def forbidden(*args, **kwargs):+ raise AssertionError("Runtime report rendering cannot collect new evidence")++ monkeypatch.setattr(socket, "create_connection", forbidden)+ monkeypatch.setattr(subprocess, "Popen", forbidden)+ monkeypatch.setattr(PluginRootReader, "read_text", forbidden)++ output = renderers().render_plugin_runtime_reports(+ bundle(), report=RetainedReport(payload()), reporters=reporters()+ )+ assert json.loads(output["json"])["plugin_runtime"]["execution_status"] == "succeeded"+++def test_runtime_replay_and_render_need_no_optional_runtime_dependencies(tmp_path):+ script = """+import json, sys+from pathlib import Path++# Allow only what a base install provides: the standard library, this package,+# the import names of the base dependencies in pyproject.toml and their own+# requirements (colorama is click's on Windows), and whatever the interpreter+# had already loaded at startup (coverage, for example). Optional extras and+# everything they pull in are refused.+BASE = {'skillevaluator', 'click', 'colorama', 'idna', 'jinja2', 'markupsafe', 'markdown_it', 'mdurl',+ 'packaging', 'pydantic', 'pydantic_core', 'annotated_types', 'typing_extensions',+ 'typing_inspection', 'yaml', '_yaml', 'regex', 'rich', 'pygments'}+ALLOWED = set(sys.stdlib_module_names) | BASE | {name.split('.')[0] for name in sys.modules}++def allowed(name):+ # _sysconfigdata_<abi>_<platform> is stdlib but named per platform; Cython+ # extensions in base dependencies register their runtime modules directly.+ top = name.split('.')[0]+ return top in ALLOWED or top.startswith(('_sysconfigdata_', '_cython_')) or top == 'cython_runtime'++class Blocker:+ def find_spec(self, name, path=None, target=None):+ if not allowed(name):+ raise ModuleNotFoundError(f'not provided by a base install: {name}')+ return None++sys.meta_path.insert(0, Blocker())+import skillevaluator.reporting+from skillevaluator.models.result import ValidationResult+before = set(sys.modules)+from skillevaluator.reporting.plugin_runtime import SECTIONS+from skillevaluator.reporting.plugin_runtime_io import RUNTIME_REPORT_FILENAME, load_plugin_runtime_report+from skillevaluator.reporting.plugin_runtime_render import render_plugin_runtime_reports+run_dir = Path(sys.argv[1])+sections = [{'key': key, 'title': title, 'columns': list(columns), 'rows': [], 'total': 0, 'omitted': 0,+ 'notes': []} for key, (title, columns) in SECTIONS.items()]+snapshot = {'schema_version': 1, 'name': 'offline', 'status': 'incomplete', 'run_id': 'saved',+ 'execution_status': 'succeeded', 'evidence_complete': False, 'source': {}, 'truncated': False,+ 'sections': sections}+(run_dir / RUNTIME_REPORT_FILENAME).write_text(json.dumps(snapshot), encoding='utf-8')+reporting = skillevaluator.reporting+reporters = [reporting.CLIReporter(), reporting.JSONReporter(include_timestamp=False),+ reporting.MarkdownReporter(include_timestamp=False), reporting.HTMLReporter(include_timestamp=False),+ reporting.SARIFReporter(include_timestamp=False), reporting.BenchmarkReporter(include_timestamp=False)]+output = render_plugin_runtime_reports([ValidationResult(validator_name='retained')],+ report=load_plugin_runtime_report(run_dir), reporters=reporters)+assert json.loads(output['json'])['plugin_runtime']['name'] == 'offline'+assert len(output) == 6+new = set(sys.modules) - before+assert not sorted(name for name in new if not allowed(name)), sorted(new)+forbidden = ('skillevaluator.inference', 'skillevaluator.tier3', 'skillevaluator.validators')+assert not sorted(name for name in new if name.startswith(forbidden)), sorted(new)+"""+ # Generous: a cold interpreter under coverage can take most of 20 seconds.+ completed = subprocess.run(+ [sys.executable, "-c", script, str(tmp_path)], capture_output=True, text=True, timeout=180+ )+ assert completed.returncode == 0, completed.stderrdiff --git a/tests/reporting/test_private_runtime_replay.py b/tests/reporting/test_private_runtime_replay.pynew file mode 100644index 00000000..04b7e46a--- /dev/null+++ b/tests/reporting/test_private_runtime_replay.py@@ -0,0 +1,215 @@+# SPDX-License-Identifier: Apache-2.0+"""Retained reports replay without a source tree or runtime dependencies."""++from __future__ import annotations++import importlib+import json+import os++import pytest+from test_private_runtime_projection import api, sample+++def io_api():+ return importlib.import_module("skillevaluator.reporting.plugin_runtime_io")+++def test_snapshot_round_trip_is_self_contained(tmp_path):+ report = api().project_plugin_runtime(sample(), provenance={"plugin": "fixture"})+ io_api().save_plugin_runtime_report(report, tmp_path)+ assert io_api().load_plugin_runtime_report(tmp_path).payload() == report.payload()+ assert sorted(path.name for path in tmp_path.iterdir()) == ["plugin_runtime_report.json"]+++def test_missing_older_snapshot_has_explicit_unavailable_error(tmp_path):+ with pytest.raises(ValueError, match="unavailable"):+ io_api().load_plugin_runtime_report(tmp_path)++[email protected]("kind", ["root_link", "file_link", "hardlink", "directory"])+def test_replay_refuses_unsafe_selected_paths(tmp_path, kind):+ real = tmp_path / "real"+ real.mkdir()+ report = api().project_plugin_runtime(sample())+ io_api().save_plugin_runtime_report(report, real)+ target = real / "plugin_runtime_report.json"+ root = real+ if kind == "root_link":+ root = tmp_path / "linked"+ root.symlink_to(real, target_is_directory=True)+ elif kind == "file_link":+ target.rename(real / "original.json")+ target.symlink_to(real / "original.json")+ elif kind == "hardlink":+ os.link(target, real / "alias.json")+ else:+ target.unlink()+ target.mkdir()+ with pytest.raises(ValueError):+ io_api().load_plugin_runtime_report(root)+++ "change", ["version", "columns", "row_width", "status", "status_shape", "status_object", "unknown_section"]+)+def test_replay_refuses_incompatible_snapshot_schema(tmp_path, change):+ payload = api().project_plugin_runtime(sample()).payload()+ if change == "version":+ payload["schema_version"] = 999+ elif change == "columns":+ payload["sections"][0]["columns"] = ["unexpected"]+ elif change == "row_width":+ payload["sections"][0]["rows"][0].append("extra")+ elif change == "status":+ payload["status"] = "passed"+ elif change == "status_shape":+ payload["status"] = []+ elif change == "status_object":+ payload["status"] = {}+ else:+ payload["sections"][0]["key"] = "unknown"+ (tmp_path / "plugin_runtime_report.json").write_text(json.dumps(payload))+ with pytest.raises(ValueError, match="unavailable"):+ io_api().load_plugin_runtime_report(tmp_path)+++def _strings(value):+ if isinstance(value, str):+ yield value+ elif isinstance(value, dict):+ for item in value.values():+ yield from _strings(item)+ elif isinstance(value, list):+ for item in value:+ yield from _strings(item)+++ ("error", "kept", "dropped"),+ [+ (+ "claude-code/with_skill: RuntimeError: census read failed\napi_key: placeholder\n"+ "rerun with --keep-jobs to inspect trials",+ "census read failed",+ "placeholder",+ ),+ # Three sanitizer passes: joining lines exposes the assignments, their+ # redaction pushes the key footer past the cell bound, and the next pass+ # redacts the now-unterminated key header.+ (+ "token:\n," * 8 + " -----BEGIN PRIVATE KEY----- " + "y" * 150 + " -----END x",+ "token:<redacted>",+ "BEGIN PRIVATE KEY",+ ),+ ],+ ids=["two_passes", "three_passes"],+)+def test_saved_snapshot_is_the_live_report_for_multiline_diagnostics(tmp_path, error, kept, dropped):+ # Runtime observer errors keep line breaks; redaction must not differ between+ # the live report and the file that offline replay reads.+ result = sample()+ result["runtime_evidence"]["errors"] = [error]+ report = api().project_plugin_runtime(result, provenance={"plugin": "fixture"})+ io_api().save_plugin_runtime_report(report, tmp_path)+ saved = (tmp_path / "plugin_runtime_report.json").read_text(encoding="utf-8")+ assert json.loads(saved) == report.payload()+ assert io_api().load_plugin_runtime_report(tmp_path).payload() == report.payload()+ assert kept in saved and dropped not in saved+ assert all(api()._clean_text(text)[0] == text for text in _strings(report.payload()))+++def test_text_cut_while_settling_marks_the_report_truncated(tmp_path):+ # The first pass leaves this under the cell bound; redaction on a later pass+ # grows it past the bound, so the cut must still be reported.+ result = sample()+ result["runtime_evidence"]["errors"] = ["token:\n," * 30 + " rerun with --keep-jobs to inspect trials"]+ report = api().project_plugin_runtime(result)+ assert api().project_plugin_runtime(sample()).payload()["truncated"] is False+ assert report.payload()["truncated"] is True+ io_api().save_plugin_runtime_report(report, tmp_path)+ assert io_api().load_plugin_runtime_report(tmp_path).payload() == report.payload()+++def _bounded_rows_payload():+ payload = api().project_plugin_runtime(sample()).payload()+ remaining = 257+ for section in payload["sections"]:+ count = min(64, remaining)+ remaining -= count+ section["rows"] = [["x"] * len(section["columns"]) for _row in range(count)]+ section["total"], section["omitted"] = count, 0+ return payload+++ ("change", "reason"),+ [+ ("completeness_mismatch", "Inconsistent runtime report completeness"),+ ("omitted_negative", "Invalid runtime report omission counts"),+ ("total_mismatch", "Invalid runtime report omission counts"),+ ("extra_field", "Unexpected runtime report fields"),+ ("truncated_shape", "Invalid runtime report truncated"),+ ("row_bound", "Runtime report row bound exceeded"),+ ],+)+def test_replay_refuses_snapshots_that_misstate_completeness_or_omissions(tmp_path, change, reason):+ payload = _bounded_rows_payload() if change == "row_bound" else api().project_plugin_runtime(sample()).payload()+ run = payload["sections"][0]+ if change == "completeness_mismatch":+ payload["status"] = "complete"+ elif change == "omitted_negative":+ run["omitted"], run["total"] = -1, len(run["rows"]) - 1+ elif change == "total_mismatch":+ run["total"] += 1+ elif change == "extra_field":+ payload["unexpected"] = "x"+ elif change == "truncated_shape":+ payload["truncated"] = "yes"+ (tmp_path / "plugin_runtime_report.json").write_text(json.dumps(payload), encoding="utf-8")+ with pytest.raises(ValueError, match=f"unavailable: {reason}"):+ io_api().load_plugin_runtime_report(tmp_path)+++ "content",+ ['{"schema_version": NaN}', "[" * 101, '"unterminated', '"' + "x" * (256 * 1024) + '"'],+ ids=["non-finite", "too-deep", "malformed", "oversized"],+)+def test_replay_rejects_invalid_or_unbounded_json(tmp_path, content):+ (tmp_path / "plugin_runtime_report.json").write_text(content)+ with pytest.raises(ValueError):+ io_api().load_plugin_runtime_report(tmp_path)+++def test_replay_rejects_snapshot_that_exceeds_byte_bound_after_sanitization(tmp_path):+ payload = api().project_plugin_runtime(sample()).payload()+ for section in payload["sections"]:+ section["rows"] = []+ section["total"] = section["omitted"] = 0+ if section["key"] in {"comparisons", "context", "cost", "loading"}:+ section["rows"] = [["\u202e" * 40 for _column in section["columns"]] for _row in range(64)]+ section["total"] = 64+ encoded = json.dumps(payload, ensure_ascii=False).encode("utf-8")+ assert len(encoded) < 256 * 1024+ (tmp_path / "plugin_runtime_report.json").write_bytes(encoded)++ with pytest.raises(ValueError, match=r"unavailable.*publication bound"):+ io_api().load_plugin_runtime_report(tmp_path)++[email protected]("value", [True, 2**100, {"unexpected": "nested"}])+def test_replay_refuses_cells_outside_the_display_schema(tmp_path, value):+ payload = api().project_plugin_runtime(sample()).payload()+ payload["sections"][0]["rows"][0][1] = value+ (tmp_path / "plugin_runtime_report.json").write_text(json.dumps(payload))+ with pytest.raises(ValueError):+ io_api().load_plugin_runtime_report(tmp_path)+++def test_replay_refuses_unrecognized_section_fields(tmp_path):+ payload = api().project_plugin_runtime(sample()).payload()+ payload["sections"][0]["unexpected"] = "unchecked data"+ (tmp_path / "plugin_runtime_report.json").write_text(json.dumps(payload))+ with pytest.raises(ValueError):+ io_api().load_plugin_runtime_report(tmp_path)diff --git a/tests/test_harbor_collector_runtime_failures.py b/tests/test_harbor_collector_runtime_failures.pyindex 8d366b74..3ce97f90 100644--- a/tests/test_harbor_collector_runtime_failures.py+++ b/tests/test_harbor_collector_runtime_failures.py@@ -951,6 +951,8 @@ def test_complete_low_score_is_execution_success(tmp_path: Path) -> None:"execution_error_details_total": 0,"execution_error_details_shown": 0,"execution_error_details_truncated": False,+ "expected_cases": 1,+ "scored_cases": 1,"expected_attempts": 1,"scored_attempts": 1,"runtime_failure_details_total": 0,diff --git a/tests/tier3/test_private_case_coverage_review.py b/tests/tier3/test_private_case_coverage_review.pynew file mode 100644index 00000000..c9551fed--- /dev/null+++ b/tests/tier3/test_private_case_coverage_review.py@@ -0,0 +1,76 @@+# SPDX-License-Identifier: Apache-2.0+"""Case coverage retains its own denominator alongside logical attempts."""++from __future__ import annotations++import pytest++from skillevaluator.tier3.harbor.collector import _condition_execution_summary+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRIC_SET, DEFAULT_METRICS+++def _reward(case: str, attempt: int = 1, value: float = 0.8) -> dict:+ return {+ "entry_id": case,+ "_trial_root_name": f"{case}__attempt{attempt}",+ "_attempt_ordinal": attempt,+ "metric_set": DEFAULT_METRIC_SET,+ **dict.fromkeys(DEFAULT_METRICS, value),+ }+++ ("rewards", "options", "coverage", "status"),+ [+ (+ [_reward("a", 1), _reward("a", 2), _reward("b", 1), _reward("b", 2)],+ {},+ (2, 2, 4, 4),+ "succeeded",+ ),+ ([_reward("a", 1)], {}, (2, 1, 4, 1), "failed"),+ (+ [_reward("a", 1), _reward("b", 1, 0.2), _reward("b", 2)],+ {"n_attempts": 3, "stop_on_pass": True},+ (2, 2, 3, 3),+ "succeeded",+ ),+ ([], {"skipped": True}, (0, 0, 0, 0), "skipped"),+ (+ [_reward("a"), _reward("extra"), _reward("b", value=float("nan"))],+ {"n_attempts": 1},+ (2, 1, 2, 2),+ "failed",+ ),+ (+ [_reward("a"), _reward("b")],+ {"expected_case_ids": None, "expected_cases": 3, "n_attempts": 1},+ (3, 2, 3, 2),+ "failed",+ ),+ (+ [_reward("a", 2)],+ {+ "expected_case_ids": ["a"],+ "expected_cases": 1,+ "n_attempts": 3,+ "stop_on_pass": True,+ "recovered_attempts": [{"case_id": "a", "trial": "a__attempt1", "attempt": 1}],+ },+ (1, 1, 1, 1),+ "succeeded",+ ),+ ],+ ids=["multi-attempt", "missing-attempts", "stop-on-pass", "skipped", "extra-invalid", "count-only", "recovered"],+)+def test_condition_counts_cases_without_inferring_them_from_attempts(rewards, options, coverage, status):+ condition = _condition_execution_summary(+ rewards,+ job_failure="",+ **{"expected_case_ids": ["a", "b"], "expected_cases": 2, "n_attempts": 2, **options},+ )+ assert condition["execution_status"] == status+ assert (+ tuple(condition.get(key) for key in ("expected_cases", "scored_cases", "expected_attempts", "scored_attempts"))+ == coverage+ )diff --git a/tests/tier3/test_private_comparison_artifacts.py b/tests/tier3/test_private_comparison_artifacts.pyindex 955662ba..d8d23916 100644--- a/tests/tier3/test_private_comparison_artifacts.py+++ b/tests/tier3/test_private_comparison_artifacts.py@@ -90,7 +90,7 @@ def _runtime(tmp_path, *, case_count=1, n_attempts=1, n_pieces=0):logical_trials={agent: dict.fromkeys(arms if agent == "claude-code" else arms[:2], records) for agent in agents},- context=SimpleNamespace(case_spec=lambda _case: {"cross_component": True}),+ context=SimpleNamespace(cross_component_cases=frozenset(cases)),)native = SimpleNamespace(observer=observer, decisions={}, collect_evidence=lambda *_args: None)package = SimpleNamespace(diff --git a/tests/tier3/test_private_runtime_report_acceptance.py b/tests/tier3/test_private_runtime_report_acceptance.pynew file mode 100644index 00000000..6732d2e3--- /dev/null+++ b/tests/tier3/test_private_runtime_report_acceptance.py@@ -0,0 +1,378 @@+# SPDX-License-Identifier: Apache-2.0+"""Private reports traverse the service and replay without live execution."""++from __future__ import annotations++import json+import os+import shutil+from functools import partial+from pathlib import Path+from types import SimpleNamespace++import pytest++from skillevaluator.evaluation.options import EvaluationOptions+from skillevaluator.evaluation.service import EvaluationService+from skillevaluator.evaluation.tier3_report import render_agent_eval_html_report+from skillevaluator.plugin_inventory import plugin_inventory_for_root+from skillevaluator.provider_config import ProviderConfig+from skillevaluator.tier3 import commands, plugin_evidence, plugin_execution+from skillevaluator.tier3.harbor import runner, runtime_preflight+from skillevaluator.tier3.plugin_comparisons import ComparisonOptions+from skillevaluator.tier3.plugin_prepare import SourceIdentity, prepare_runtime_package+from skillevaluator.validators.private_plugin_schema import validate_inventory_schema+from tier3.test_private_comparison_engine_acceptance import _harbor_result++SNAPSHOT = "plugin_runtime_report.json"+++def _prepared(root: Path):+ files = {+ ".claude-plugin/plugin.json": {+ "name": "acceptance",+ "description": "Offline acceptance fixture.",+ "mcpServers": {"fixture": {"url": "https://fixture.example.invalid/mcp"}},+ },+ "skills/member/SKILL.md": "---\nname: member\ndescription: Fixture member.\n---\nRead fixture inputs.\n",+ "skills/support/SKILL.md": "---\nname: support\ndescription: Fixture support.\n---\nSupport the member task.\n",+ "skills/member/evals/evals.json": [+ {+ "id": f"case-{index}",+ "prompt": f"Perform fixture task {index}.",+ "cross_component": True,+ "expected_skills": ["member", "support"],+ "expected_tools": ["SKILL:member", "SKILL:support", "MCP:fixture/search"],+ }+ for index in range(1, 6)+ ],+ }+ for relative, value in files.items():+ path = root / relative+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(value if isinstance(value, str) else json.dumps(value), encoding="utf-8")+ inventory = plugin_inventory_for_root(root)+ assert inventory is not None and inventory.complete+ validation = validate_inventory_schema(inventory)+ assert validation.passed, validation.errors+ identity = SourceIdentity.of(inventory)+ package = prepare_runtime_package(inventory, expected=identity)+ assert package.identity == identity+ return package, validation+++def _trajectory(variant: str) -> dict:+ calls, observations = [], []+ if variant != "without":+ for member in ("member", "support"):+ calls.append(+ {+ "tool_call_id": f"read-{member}",+ "function_name": "Read",+ "arguments": {"file_path": f"/skills/{member}/SKILL.md"},+ }+ )+ observations.append({"source_call_id": f"read-{member}", "content": f"Fixture {member} instructions."})+ if variant in {"with", "looseparts"}:+ server = "plugin_acceptance_fixture" if variant == "with" else "fixture"+ calls.append(+ {+ "tool_call_id": "search",+ "function_name": f"mcp__{server}__search",+ "arguments": {"query": "private-argument-value-is-not-display-evidence"},+ }+ )+ observations.append({"source_call_id": "search", "content": "found", "extra": {"tool_result_is_error": False}})+ return {+ "schema_version": "ATIF-v1.7",+ "trajectory_id": f"fixture-{variant}",+ "agent": {"name": "claude-code", "version": "fixture"},+ "final_metrics": {"total_prompt_tokens": 23, "total_completion_tokens": 5, "total_cost_usd": None},+ "steps": [+ {+ "step_id": 1,+ "source": "agent",+ "message": "Fixture response.",+ "metrics": {"prompt_tokens": 23, "completion_tokens": 5},+ "tool_calls": calls,+ "observation": {"results": observations},+ }+ ],+ }+++def _offline_engine(monkeypatch, *, missing: str | None = None):+ # Configuration, routing, task emission, collection and report rendering stay+ # real. These substitutions stop provider and container execution only.+ for name in tuple(os.environ):+ if name.startswith(("SKILL_EVAL_", "OPENAI_", "ANTHROPIC_", "CLAUDE_", "NVIDIA_", "AWS_", "LLM_JUDGE_")):+ monkeypatch.delenv(name)+ provider = ProviderConfig(+ provider="anthropic",+ model="claude-sonnet-4-6",+ api_key="offline-fixture-credential",+ base_url="https://provider.example.invalid",+ litellm_model="anthropic/claude-sonnet-4-6",+ )+ monkeypatch.setattr(commands, "resolve_llm_provider", lambda: provider)+ monkeypatch.setattr(runner, "resolve_llm_provider", lambda: provider)+ monkeypatch.setattr(runner, "_check_prerequisites", lambda **_kwargs: [])+ monkeypatch.setattr(+ runtime_preflight,+ "probe_model",+ lambda selected: SimpleNamespace(+ ok=True,+ provider=selected.provider,+ model=selected.model,+ detail="offline fixture",+ failure_kind=None,+ http_status=None,+ ),+ )+ monkeypatch.setattr(+ runner, "render_agent_eval_html_report", partial(render_agent_eval_html_report, use_llm_judge=False)+ )++ def no_host_probe(*_args, **_kwargs):+ pytest.fail("default private evidence must not probe the declared fixture endpoint")++ monkeypatch.setattr(plugin_evidence, "probe_mcp_servers", no_host_probe)+ launches = []++ def launch(**kwargs):+ tasks = sorted(path for path in kwargs["dataset"].iterdir() if path.is_dir())+ launches.append((kwargs["job_name"].rsplit("-", 1)[-1], [path.name for path in tasks]))+ assert len(tasks) == 5 and kwargs["n_attempts"] == 2+ job = kwargs["jobs_dir"] / kwargs["job_name"]+ # This helper runs emitted setup/census scripts with writable paths in+ # tmp_path and writes real Harbor model shapes, but starts no agent.+ _harbor_result(job, tasks, kwargs["n_attempts"])+ variant = kwargs["job_name"].rsplit("-", 1)[-1]+ for trial in job.iterdir():+ if not trial.is_dir():+ continue+ if variant != "without":+ startup_path = trial / "agent/claude-code.txt"+ startup = json.loads(startup_path.read_text(encoding="utf-8"))+ startup["skills"] = (+ ["acceptance:member", "acceptance:support"] if variant == "with" else ["member", "support"]+ )+ startup_path.write_text(json.dumps(startup) + "\n", encoding="utf-8")+ if not (missing == "trajectory" and variant == "with"):+ (trial / "agent/trajectory.json").write_text(json.dumps(_trajectory(variant)), encoding="utf-8")+ if missing == "load" and variant == "with":+ (trial / "agent/claude-code.txt").unlink()+ return True, ""++ monkeypatch.setattr(runner, "_run_harbor", launch)+ return launches+++def _run_private(tmp_path, monkeypatch, *, missing=None, publish=None, comparisons=True):+ package, validation = _prepared(tmp_path / "plugin")+ launches = _offline_engine(monkeypatch, missing=missing)+ work = tmp_path / "work"+ work.mkdir()+ seen = {}++ def observer_factory(current_package, wrapper, hooks, options):+ # Configuration belongs to this temporary wrapper; the source and its+ # analyzed digest are unchanged. Use the real configuration reader.+ (wrapper.path / "evals/config.yml").write_text(+ "schema_version: 1\nharbor:\n base_image_mode: disabled\n", encoding="utf-8"+ )+ seen["wrapper"] = wrapper.path+ observer = plugin_evidence.RuntimeEvidenceObserver(current_package, wrapper, hooks, options)+ seen["observer"] = observer+ return observer++ def published(outcome):+ run_dir = Path(outcome.result["run_dir"])+ assert (run_dir / SNAPSHOT).is_file(), "private publication must retain the report before caller publication"+ assert seen["wrapper"].exists()+ seen["payload"] = json.loads((run_dir / SNAPSHOT).read_text(encoding="utf-8"))+ if publish is not None:+ publish(outcome)++ outcome = plugin_execution.evaluate_prepared_plugin(+ package,+ EvaluationOptions(+ skill_path=Path(package.identity.root),+ agents="claude-code",+ results_dir=tmp_path / "results",+ n_attempts=2,+ n_concurrent=2,+ max_cases=5,+ agent_runtime_preflight=False,+ ),+ work_dir=work,+ load="native",+ comparison_options=ComparisonOptions(loose_parts=True) if comparisons else None,+ observer_factory=observer_factory,+ publish=published,+ )+ assert plugin_execution.active_runtime() is None+ assert not seen["wrapper"].exists() and list(work.iterdir()) == []+ return outcome, package, validation, launches, seen+++def test_service_run_retains_one_snapshot_before_cleanup_and_replays_six_formats(tmp_path, monkeypatch):+ outcome, package, validation, launches, seen = _run_private(tmp_path, monkeypatch)+ assert outcome.status == "succeeded", outcome.result+ assert outcome.cleanup_error is None and outcome.publication_error is None+ assert {variant for variant, _tasks in launches} == {"with", "without", "sumofparts", "looseparts"}+ assert len({tuple(tasks) for _variant, tasks in launches}) == 1+ run_dir = Path(outcome.result["run_dir"])+ assert not (run_dir / "_harbor-jobs").exists() and not (run_dir / "_harbor-tasks").exists()+ ladder = outcome.result["agents"]["claude-code"]["lift_ladder"]+ for condition in outcome.result["agents"]["claude-code"]["conditions"].values():+ assert condition["expected_cases"] == 5 and condition["scored_cases"] == 5+ assert ladder["n_cases"] == 5 and ladder["excluded_cases"] == []+ assert len(seen["observer"].context.cross_component_cases) == 5+ assert ladder["by_case_type"]["cross_component"]["n_cases"] == 5+ assert ladder["glue"]["n"] == 5+ assert {name: step["estimate"] for name, step in ladder["steps"].items()} == pytest.approx(+ {"skill": 0.2, "capability": 0.2, "integration": 0.2, "plugin": 0.6}+ )+ assert seen["payload"]["execution_status"] == "succeeded"+ assert seen["payload"]["evidence_complete"] is True and seen["payload"]["status"] == "complete"+ assert seen["payload"]["source"]["manifest_sha256"] == package.identity.manifest_sha256+ assert outcome.provenance["mcp_proof"]["fixture"]["status"] == "used-successfully"+ sections = {section["key"]: section for section in seen["payload"]["sections"]}+ assert ["claude-code", "with_skill", "skill:support", "exercised", 1.0] in sections["coverage"]["rows"]+ assert ["claude-code", "with_skill", 280, None, 5, 56.0, None] in sections["cost"]["rows"]+ shutil.rmtree(Path(package.identity.root))++ def forbidden(*_args, **_kwargs):+ pytest.fail("saved private reporting must not collect source or runtime evidence")++ monkeypatch.setattr(runner, "_run_harbor", forbidden)+ monkeypatch.setattr(runtime_preflight, "probe_model", forbidden)+ monkeypatch.setattr(plugin_evidence, "probe_mcp_servers", forbidden)+ from skillevaluator.plugin_paths import PluginRootReader+ from skillevaluator.reporting import (+ BenchmarkReporter,+ CLIReporter,+ HTMLReporter,+ JSONReporter,+ MarkdownReporter,+ SARIFReporter,+ )+ from skillevaluator.reporting.plugin_runtime_io import load_plugin_runtime_report+ from skillevaluator.reporting.plugin_runtime_render import render_plugin_runtime_reports++ monkeypatch.setattr(PluginRootReader, "read_text", forbidden)+ loaded = load_plugin_runtime_report(run_dir)+ assert loaded.payload() == seen["payload"]+ output = render_plugin_runtime_reports(+ [validation],+ report=loaded,+ reporters=[+ CLIReporter(),+ JSONReporter(include_timestamp=False),+ HTMLReporter(include_timestamp=False),+ MarkdownReporter(include_timestamp=False),+ SARIFReporter(include_timestamp=False),+ BenchmarkReporter(include_timestamp=False),+ ],+ )+ assert set(output) == {"cli", "json", "html", "markdown", "sarif", "benchmark"}+ assert json.loads(output["json"])["plugin_runtime"] == loaded.payload()+ assert json.loads(output["sarif"])["runs"][0]["properties"]["pluginRuntime"] == loaded.payload()+ for document in output.values():+ assert "acceptance" in document+ assert "used-successfully" in document+ assert "280" in document+ assert "offline-fixture-credential" not in document+ assert "private-argument-value-is-not-display-evidence" not in document++[email protected]("missing", ["load", "trajectory"])+def test_successful_scoring_does_not_turn_missing_runtime_evidence_into_complete_report(tmp_path, monkeypatch, missing):+ outcome, _package, _validation, _launches, seen = _run_private(+ tmp_path, monkeypatch, missing=missing, comparisons=missing == "load"+ )+ assert outcome.status == "succeeded", outcome.result+ assert outcome.publication_error is None+ if missing == "load":+ assert outcome.provenance["native_evidence"]["claude-code"]["loaded"] == []+ else:+ summary = outcome.result["agents"]["claude-code"]["plugin_signals_summary"]["with_skill"]+ assert summary["n_missing_trajectory"] == 10+ assert seen["payload"]["execution_status"] == "succeeded"+ assert seen["payload"]["evidence_complete"] is False+ assert seen["payload"]["status"] == "incomplete"+++def test_caller_publication_failure_preserves_saved_snapshot_status_and_cleanup(tmp_path, monkeypatch):+ def failed_publication(_outcome):+ raise OSError("fixture publication failed")++ outcome, _package, _validation, _launches, seen = _run_private(+ tmp_path, monkeypatch, publish=failed_publication, comparisons=False+ )+ assert outcome.status == "succeeded"+ assert outcome.publication_error == "OSError: fixture publication failed"+ assert outcome.cleanup_error is None+ assert json.loads((Path(outcome.result["run_dir"]) / SNAPSHOT).read_text(encoding="utf-8")) == seen["payload"]+++def test_snapshot_storage_failure_keeps_caller_publication_and_cleanup(tmp_path, monkeypatch):+ """Isolate the publication lifecycle; engine execution is covered above."""+ from skillevaluator.reporting import plugin_runtime_io++ package, _validation = _prepared(tmp_path / "plugin")+ work, run_dir = tmp_path / "work", tmp_path / "run"+ work.mkdir()+ run_dir.mkdir()+ observed = {}++ def no_storage(_report, _run_dir):+ raise OSError("fixture snapshot storage failure")++ def completed_engine(options):+ observed["wrapper"] = options.skill_path+ return {"execution_status": "succeeded", "run_dir": str(run_dir)}++ def published(outcome):+ observed["called"] = True+ assert observed["wrapper"].exists()+ assert "fixture snapshot storage failure" in outcome.provenance["runtime_report_error"]++ monkeypatch.setattr(plugin_runtime_io, "save_plugin_runtime_report", no_storage)+ outcome = plugin_execution.evaluate_prepared_plugin(+ package,+ EvaluationOptions(skill_path=Path(package.identity.root), agents="claude-code"),+ work_dir=work,+ evaluate=completed_engine,+ publish=published,+ )+ assert outcome.status == "succeeded" and observed["called"]+ assert "fixture snapshot storage failure" in outcome.publication_error+ assert outcome.cleanup_error is None and list(work.iterdir()) == []+ assert not observed["wrapper"].exists() and not (run_dir / SNAPSHOT).exists()+ assert plugin_execution.active_runtime() is None+++def test_ordinary_service_run_does_not_retain_a_private_snapshot(tmp_path, monkeypatch):+ _package, _validation = _prepared(tmp_path / "plugin")+ launches = _offline_engine(monkeypatch)+ skill = tmp_path / "plugin/skills/member"+ (skill / "evals/config.yml").write_text(+ "schema_version: 1\nharbor:\n base_image_mode: disabled\n", encoding="utf-8"+ )+ result = EvaluationService().evaluate(+ EvaluationOptions(+ skill_path=skill,+ agents="claude-code",+ results_dir=tmp_path / "results",+ n_attempts=2,+ n_concurrent=2,+ agent_runtime_preflight=False,+ )+ )+ assert result["execution_status"] == "succeeded", result+ assert {variant for variant, _tasks in launches} == {"with", "without"}+ assert plugin_execution.active_runtime() is None+ assert "runtime_evidence" not in result and "private_comparisons" not in result+ assert not (Path(result["run_dir"]) / SNAPSHOT).exists()