Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(reporting): add quantitative skill performance dashboard (#179)
Narendran Raghavan526dd30ed035
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ac42c517..da09de8f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,8 +7,15 @@ All notable changes to SkillEvaluator are documented in this file.### Added- Optional `skillevaluator dashboard` tables for comparing saved Tier 3 reports- and retained runs across skills, models, harnesses, and skill conditions,- with paired token, agent runtime, and recorded USD cost differences.+ and retained runs across skills, models, harnesses, recorded harness+ features, and skill conditions. Paired with/without differences cover score,+ pass@k, tokens, and agent execution time, plus USD cost when retained runs+ record it; the performance table also shows per-trial means alongside+ totals. Comparisons require matching complete trial coverage (repeated+ attempts pair by case, as in the HTML report) and name the reason when a+ pair is not comparable. Inputs are read once and reused between+ interactions (**Reload reports** re-reads them); browser uploads are limited+ to 16 MB. Install it with the `dashboard` extra.- Tier 3 support for Harbor GKE execution mode (`--env-mode gke`) and `--ek`argument forwarding. API keys, bearer tokens, and passwords passed in `--ek`keys or values are rejected to keep credentials out of process listings, while@@ -305,10 +312,6 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed-- Preserve missing Tier 3 token counters as `null` in canonical JSON exports- instead of reporting unmeasured usage as zero.-- Keep dashboard comparisons unavailable for contradictory trial identities or- invalid coverage counts, and enforce the documented 16 MB browser upload limit.- Report non-string YAML keys as validation errors in skills, rules, workflows,and plugin manifests instead of raising a `TypeError`, including frontmatter`metadata` keys. Show boolean, null, and date keys in a readable YAML form.diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex e4d3e25b..fda94771 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -504,9 +504,10 @@ skillevaluator compare ./my-skill## dashboard-Open the optional local tables dashboard. Install `skillevaluator[dashboard]`-first, then pass JSON reports, retained run directories, or directories containing-reports. With no paths, upload JSON files in the browser.+Open the optional local tables dashboard. Install the+[`dashboard` extra](installation.mdx#choosing-extras) first, then pass JSON+reports, retained run directories, or directories containing reports. With no+paths, upload JSON files in the browser.```bashskillevaluator dashboard reports/run-a.json reports/run-b.json@@ -515,7 +516,7 @@ skillevaluator dashboard --no-browser --port 8502| Option | Default | Description || --- | --- | --- |-| `--port` | `8501` | Local server port, from 1 through 65535. |+| `--port` | `8501`, or the next free port | Local port on 127.0.0.1, from 1 through 65535. Pass it to require a specific port; the command fails if that port is busy. || `--no-browser` | off | Start the server without opening a browser. |The server listens on `127.0.0.1`; Ctrl+C stops it. See thediff --git a/docs/installation.mdx b/docs/installation.mdxindex 14729f6a..b2ce0929 100644--- a/docs/installation.mdx+++ b/docs/installation.mdx@@ -140,10 +140,11 @@ and environment backends.| `tier3` | Docker, local, and cloud live-agent evaluation through Harbor (pins `harbor==0.24.0` and requires `python-dotenv>=1.2.0`); includes `llm` || `security` | Bandit and pip-audit || `all` | `tier2`, `tier3`, and `security` in one bundle |+| `dashboard` | Local Streamlit tables for comparing saved reports and retained runs (`skillevaluator dashboard`); not included in `all` || `dev` | Build, test, coverage, and formatting tools for contributors |<Tip>-`[all]` installs every SkillEvaluator extra. External scanner tools, Docker, agent CLIs, and the Harbor extras for managed sandbox backends (such as `harbor[modal]==0.24.0`) stay separate no matter which extra you pick — see [Agents & Sandboxes](agents-and-sandboxes.mdx#where-trials-run).+`[all]` installs the evaluator extras (`tier2`, `tier3`, `security`); add `dashboard` with `[all,dashboard]`. External scanner tools, Docker, agent CLIs, and the Harbor extras for managed sandbox backends (such as `harbor[modal]==0.24.0`) stay separate no matter which extra you pick — see [Agents & Sandboxes](agents-and-sandboxes.mdx#where-trials-run).</Tip>## System toolsdiff --git a/docs/reports.mdx b/docs/reports.mdxindex c2663ee9..3760236b 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -542,7 +542,7 @@ Install the optional Python dashboard and open saved JSON reports or individualretained Tier 3 run directories:```bash-pip install "skillevaluator[dashboard]"+uv tool install --python 3.13 --reinstall "skillevaluator[all,dashboard] @ git+https://github.com/NVIDIA/SkillEvaluator.git"skillevaluator dashboard reports/run-a.json reports/run-b.jsonskillevaluator dashboard ./my-skill/evals/results/<run-id># Start with an empty dashboard and upload several JSON reports in the browser:@@ -550,40 +550,82 @@ skillevaluator dashboard```From a checkout, use `uv run --extra dashboard skillevaluator dashboard ...`.-The server runs at `http://127.0.0.1:8501`; use `--port 8502` for another port,-`--no-browser` to suppress opening a browser, and Ctrl+C to stop it.+For live reload while developing the dashboard, run+`PYTHONPATH=src uv run --extra dashboard streamlit run src/skillevaluator/reporting/dashboard_ui/skillevaluator_dashboard.py -- PATH`;+`PYTHONPATH=src` lets Streamlit watch the package modules for changes.+The server runs at `http://127.0.0.1:8501`, or the next free port when 8501 is+busy; pass `--port 8502` to require a specific port. Use `--no-browser` to+suppress opening a browser, and Ctrl+C to stop it.+The dashboard runs from its own package directory, so Streamlit settings or+secrets in your project's `.streamlit/` folder are not applied. It accepts+browser connections only for 127.0.0.1 and localhost.Browser uploads accept up to 16 MB (16,000,000 bytes) per file. Local JSON filesaccept up to 16 MiB (16,777,216 bytes); larger files produce a notice while othervalid reports remain available.Filter the sortable tables by skill, model, harness, recorded harness features,-condition, and execution status. Download the visible performance or comparison-rows as CSV. Each performance row represents one run, harness, and condition.+condition, and execution status. The status and condition filters narrow the+performance table only, so comparisons keep both arms; the other filters also+remove arms from comparisons. Download the visible performance or comparison+rows as CSV. Each performance row is one run, agent (harness and model),+recorded feature configuration, and condition.+Per-trial columns are means over the row's measured trials and stay blank when+any trial is unmeasured. They help compare rows with different trial counts,+but runs on different datasets are still not like-for-like.The comparison table calculates **with skill minus without skill**, and percentchange relative to without skill. A negative usage difference means less usage;a positive score difference means higher performance. Comparisons stay within-the same run and require successful conditions with matching trial coverage.--Missing measurements stay unavailable: older JSON reports may contain token-counts without time or cost, and cost is shown only when recorded in USD.-Some older exports encoded missing token counters as zero. Open their retained-run directories to recover recorded counters, or regenerate the canonical JSON-with this version to preserve missing measurements as `null`.-For retained runs, duration measures agent execution rather than the entire-evaluation workflow. Cached tokens are shown separately and are not added again+the same report and agent and require successful conditions with matching trial+coverage, the same dataset digest, and the same attempt policy.+When a pair's differences stay blank, the **Comparison reason** column says why.+Repeated attempts pair by case, as in the HTML report: each arm must record the+same number of attempts for every case, and each attempt must have its own+trial ID. A trajectory above the report loader's size bound leaves that trial's+token counts unavailable in canonical JSON but does not block score comparisons.+Truncation limited to diagnostic details, such as evidence, raw reward fields,+or non-best agent details, also leaves comparisons available, with a notice.+Truncation that drops trials or other trial artifacts, or that is not+explained, keeps comparisons unavailable.++The dashboard reads each input once and reuses it while you filter or switch+views. Rewriting a JSON file passed on the command line, or adding an entry+directly inside a directory input, is picked up on the next interaction. Edits+deeper inside a directory input (for example a run that is still writing trials)+or in reports a catalog references appear after **Reload reports** in the+sidebar; a browser refresh does not re-read them. Overlapping inputs, such as a+results directory and a run inside it, are loaded once. Byte-identical uploads+are skipped with a notice.++Missing measurements stay unavailable, and cost is shown only when recorded in+USD. Canonical Tier 3 JSON records each single-step trial's agent execution time+(`agent_seconds`) when Harbor's timestamps carry a timezone. Timestamps without a+timezone are treated as unavailable, as in the HTML report's+[measured agent resources](#measured-agent-resources). Agent time excludes+environment setup and verification. Canonical JSON never records USD cost; open+the retained run directory to show Harbor's recorded `agent_result.cost_usd` and+multi-step totals. For retained runs whose trajectory lacks token totals or+cannot be read, the dashboard falls back to Harbor's `agent_result` token+counters and keeps the trial's agent time and cost.+Exports written before v0.5.0 could encode missing token counters as zero, so+when a canonical JSON report's `evaluator_version` is earlier than 0.5.0 the+dashboard shows its zero token counters as unavailable and adds a notice; open+the retained run directory to recover recorded counters.+Cached tokens are shown separately and are not added againto prompt plus completion tokens. No pricing estimates or new agent evaluationsare performed. Adverse conditions and harness features are displayed whenexplicitly recorded; the dashboard does not run these experiments.-Canonical JSON can omit resource ownership for multi-step trials. When several-reward rows share one physical trial, the dashboard keeps their usage totals-unavailable and shows a notice. Open the retained run directory to recover-totals from authoritative step results.+Since v0.5.0, canonical JSON collapses multi-step trials without per-trial token+or time totals, so they show as unmeasured. Older exports that repeat a trial ID+across reward rows show an ambiguity notice. Open the retained run directory to+recover totals from authoritative step results.### Recorded harness features-Features are experiment metadata, not inferred from the harness name. In a-canonical Tier 3 JSON payload, an agent or condition can include:+Features are experiment metadata, not inferred from the harness name. Current+SkillEvaluator commands do not record this field; add it to the canonical JSON+or to `run_config.json` by hand. In a canonical Tier 3 JSON payload, an agent or+condition can include:```json"dashboard_metadata": {@@ -591,17 +633,21 @@ canonical Tier 3 JSON payload, an agent or condition can include:}```-For retained runs, the dashboard also reads this metadata from `run_config.json`-(at the root or within the agent configuration) and each condition's-`summary.json`. An empty feature list records that no features were enabled;-an absent field is displayed as **unknown**. Differing recorded models or feature-configurations are kept separate when pairing conditions.+`harness_features` can be a list of strings, a non-empty string, or a JSON+object. For canonical JSON the condition's value wins over the agent's, which+wins over the report's. For retained runs the order is the condition's+`summary.json`, then the agent entry in `run_config.json`, then the root of+`run_config.json`. Edit the run directory's `run_config.json`; the copy embedded+in `result.json` is used only when that file cannot be read. An empty feature+list records that no features were enabled and is shown as **none (recorded)**;+an absent field is displayed as **unknown**. Differing recorded models or+feature configurations are kept separate when pairing conditions.An explicitly imported adverse condition uses `conditions.adverse`,`trials_adverse`, `num_trials_adverse`, and `adverse` for its score in the sameagent payload. It appears in the performance table; usage differences are-calculated for with/without pairs only. Existing evaluator runs do not create-adverse trials through the dashboard.+calculated for with/without pairs only. Adverse conditions are import-only+metadata: no evaluator command produces them, and the dashboard does not run them.## Machine-readable contractdiff --git a/pyproject.toml b/pyproject.tomlindex 0b6e2e05..4bb9b0dd 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -79,7 +79,8 @@ security = []# Optional local UI for comparing saved reports; never imported by base commands.dashboard = [- "streamlit>=1.49,<2",+ # server.allowedHosts, which limits the dashboard to loopback host names, needs 1.61.+ "streamlit>=1.61,<2",]# Full public install convenience bundle.all = [diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex ecae8262..1b3f7cae 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -3056,9 +3056,14 @@ def models_command(limit: int, as_json: bool) -> None:@cli.command("dashboard")@click.argument("paths", nargs=-1, type=click.Path(exists=True, path_type=Path))[email protected]("--port", type=click.IntRange(1, 65535), default=8501, show_default=True)+ "--port",+ type=click.IntRange(1, 65535),+ default=None,+ help="Local port on 127.0.0.1 (default 8501, or the next free port).",+)@click.option("--no-browser", is_flag=True, help="Start the server without opening a browser.")-def dashboard_command(paths: tuple[Path, ...], port: int, no_browser: bool) -> None:+def dashboard_command(paths: tuple[Path, ...], port: int | None, no_browser: bool) -> None:"""Compare saved JSON reports or retained Tier 3 run directories in tables.Install skillevaluator[dashboard] first. With no PATHS, upload JSON reportsdiff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.pyindex 4121a597..f1267fcd 100644--- a/src/skillevaluator/evaluation/tier3_report.py+++ b/src/skillevaluator/evaluation/tier3_report.py@@ -461,8 +461,13 @@ def agent_eval_result_from_directory(evaluator_version: str | None = None,evaluated_source: dict[str, Any] | None = None,use_llm_judge: bool = True,+ enforce_embedded_size: bool = True,) -> ValidationResult | None:- """Build the canonical ``AGENT_EVAL`` result for one explicit Harbor run."""+ """Build the canonical ``AGENT_EVAL`` result for one explicit Harbor run.++ Read-only consumers that never embed the payload pass+ ``enforce_embedded_size=False`` to keep every agent's trial details.+ """# Imported lazily so base-only Tier 1 workflows do not load Tier 3 helpers.from skillevaluator.tier3.harbor.report_data import (load_agent_data,@@ -511,6 +516,7 @@ def agent_eval_result_from_directory(publication_target_conflict=run_truth.get("publication_target_conflict"),evaluated_source=evaluated_source,use_llm_judge=use_llm_judge,+ enforce_embedded_size=enforce_embedded_size,)return _validation_result_from_payload(payload)@@ -621,6 +627,7 @@ def build_agent_eval_payload(publication_target_conflict: str | None = None,evaluated_source: dict[str, Any] | None = None,use_llm_judge: bool = True,+ enforce_embedded_size: bool = True,) -> dict[str, Any] | None:"""Assemble the canonical Tier 3 ``agent_eval`` payload from loaded agent data.@@ -633,7 +640,8 @@ def build_agent_eval_payload(``trials`` + ``pass_at_k``) feeds the Trials tab, deterministic + LLM``conclusions`` / ``recommendations`` / ``suggestions`` feed the Insights tab,and ``provenance`` (raw evaluators, raw lift, raw trial rewards) feeds the- Diagnostics tab.+ Diagnostics tab. ``enforce_embedded_size=False`` skips the embedded-report+ size pruning; loader truncation is still recorded in ``report_truncation``."""from skillevaluator.tier3.harbor.report_data import (build_dataset_snapshot,@@ -920,7 +928,7 @@ def build_agent_eval_payload(evidence_links,)payload = _sanitize_json_numbers(payload)- _enforce_report_payload_budget(payload, report_budget)+ _enforce_report_payload_budget(payload, report_budget, enforce_size=enforce_embedded_size)return payload@@ -1188,7 +1196,9 @@ def _prune_pass_at_k_pairing_diagnostics(payload: dict[str, Any], report_budget:report_budget.omit("unpaired_case_ids", omitted)-def _enforce_report_payload_budget(payload: dict[str, Any], report_budget: _ReportBudget) -> None:+def _enforce_report_payload_budget(+ payload: dict[str, Any], report_budget: _ReportBudget, *, enforce_size: bool = True+) -> None:"""Keep the complete self-contained payload within a hard serialized budget.Cardinality limits normally keep the payload comfortably below the cap. The@@ -1207,7 +1217,7 @@ def _enforce_report_payload_budget(payload: dict[str, Any], report_budget: _Repo# best-agent and top-level projections._prune_pass_at_k_pairing_diagnostics(payload, report_budget)refresh_signal()- if _serialized_payload_size(payload) <= _MAX_EMBEDDED_REPORT_BYTES:+ if not enforce_size or _serialized_payload_size(payload) <= _MAX_EMBEDDED_REPORT_BYTES:returnprovenance = payload.get("provenance")diff --git a/src/skillevaluator/reporting/dashboard.py b/src/skillevaluator/reporting/dashboard.pyindex 244dd46b..1c37d805 100644--- a/src/skillevaluator/reporting/dashboard.py+++ b/src/skillevaluator/reporting/dashboard.py@@ -6,34 +6,84 @@from __future__ import annotationsimport importlib.util+import os+import signalimport subprocessimport sys+import threadingfrom pathlib import Path+from typing import Anyimport click+# Streamlit runs from this directory, so the caller's .streamlit/ settings and+# modules cannot replace the dashboard's own.+_APP = Path(__file__).with_name("dashboard_ui") / "skillevaluator_dashboard.py"-def launch_dashboard(paths: tuple[Path, ...], *, port: int = 8501, no_browser: bool = False) -> int:++def launch_dashboard(paths: tuple[Path, ...], *, port: int | None = None, no_browser: bool = False) -> int:"""Run Streamlit on loopback with explicit paths and usage telemetry disabled."""if importlib.util.find_spec("streamlit") is None:- raise click.ClickException('Install the dashboard extra: pip install "skillevaluator[dashboard]"')- app = Path(__file__).with_name("dashboard_app.py")+ raise click.ClickException(+ "Install the dashboard extra: uv tool install --python 3.13 --reinstall "+ '"skillevaluator[all,dashboard] @ git+https://github.com/NVIDIA/SkillEvaluator.git" '+ "(from a checkout: uv run --extra dashboard skillevaluator dashboard, "+ 'or python -m pip install -e ".[dashboard]")'+ )+ # -P keeps the working directory off sys.path. An explicit headless=false+ # would trigger the first-run email prompt and override Streamlit's+ # display detection, so the flag is only ever set to true.command = [sys.executable,+ "-P","-m","streamlit","run",- str(app),+ str(_APP),"--server.address=127.0.0.1",- f"--server.port={port}",+ # A port on the command line counts as manually set, and Streamlit then+ # exits instead of moving to the next free port, so pass only a chosen one.+ *([f"--server.port={port}"] if port is not None else []),"--server.maxUploadSize=16",- f"--server.headless={str(no_browser).lower()}",+ "--server.showEmailPrompt=false",+ "--server.fileWatcherType=none",+ "--server.enableCORS=true",+ "--server.enableXsrfProtection=true",+ "--client.toolbarMode=viewer",+ *(["--server.headless=true"] if no_browser else []),"--browser.gatherUsageStats=false","--theme.primaryColor=#76b900","--",*(str(path.resolve()) for path in paths),]+ env = {**os.environ, "STREAMLIT_SERVER_ALLOWED_HOSTS": "127.0.0.1 localhost"}+ proc = subprocess.Popen(command, cwd=str(_APP.parent), env=env)+ # A SIGTERM, SIGHUP, or SIGINT sent only to this process would otherwise+ # leave Streamlit running on the port.+ previous: dict[int, Any] = {}+ if threading.current_thread() is threading.main_thread():+ for sig in (signal.SIGTERM, getattr(signal, "SIGHUP", None)):+ if sig is not None:+ previous[sig] = signal.signal(sig, lambda signum, _frame: proc.send_signal(signum))try:- return subprocess.call(command)+ code = proc.wait()except KeyboardInterrupt:- return 0+ # Ctrl+C reaches the whole process group, but kill -INT on this pid does+ # not; a repeated SIGINT only repeats Streamlit's stop request.+ if sys.platform != "win32":+ proc.send_signal(signal.SIGINT)+ try:+ proc.wait(timeout=10)+ except subprocess.TimeoutExpired:+ proc.terminate()+ proc.wait()+ code = 0+ finally:+ for sig, handler in previous.items():+ signal.signal(sig, handler)+ if port is not None and code > 0:+ click.echo(+ f"Streamlit exited with status {code}. If port {port} is in use, choose another with --port.", err=True+ )+ # A child killed by signal N reports -N; exit with 128 + N as a shell would.+ return 128 - code if code < 0 else codediff --git a/src/skillevaluator/reporting/dashboard_app.py b/src/skillevaluator/reporting/dashboard_app.pyindex d1096bfb..1cd827d0 100644--- a/src/skillevaluator/reporting/dashboard_app.py+++ b/src/skillevaluator/reporting/dashboard_app.py@@ -10,18 +10,27 @@ starting Streamlit. Only normalized quantitative data is displayed or exported.from __future__ import annotationsimport csv+import hashlibimport ioimport jsonimport sys+import unicodedatafrom pathlib import Pathfrom typing import Anyimport streamlit as st-from skillevaluator.reporting.dashboard_data import comparison_rows, load_dashboard_path, parse_dashboard_report+from skillevaluator.reporting.dashboard_data import (+ DashboardData,+ comparison_rows,+ dedupe_rows,+ load_dashboard_path,+ parse_dashboard_report,+)_DIMENSIONS = ("skill", "model", "harness", "features")_MAX_UPLOAD_BYTES = 16_000_000+_CACHE_ENTRIES = 64_CONDITION_NAMES = {"with_skill": "With skill", "without_skill": "Without skill", "adverse": "Adverse skill"}_USAGE_METRICS = ("total_tokens", "prompt_tokens", "completion_tokens", "cached_tokens", "duration_seconds", "cost_usd")_PERFORMANCE_COLUMNS = (@@ -30,6 +39,9 @@ _PERFORMANCE_COLUMNS = ("harness","condition","score",+ "total_tokens_per_trial",+ "duration_seconds_per_trial",+ "cost_usd_per_trial","total_tokens","duration_seconds","cost_usd",@@ -56,6 +68,7 @@ _COMPARISON_COLUMNS = ("cost_usd_delta","cost_usd_percent","comparison_status",+ "comparison_reason","features","coverage","with_score",@@ -82,6 +95,32 @@ _COMPARISON_COLUMNS = ()+def _input_fingerprint(path: Path) -> tuple[int, int, int] | None:+ """Cheap identity for one CLI input; nested directory edits need Reload reports."""+ try:+ info = path.stat()+ except OSError:+ return None+ return (info.st_mtime_ns, info.st_size, info.st_ino)++[email protected]_data(max_entries=_CACHE_ENTRIES, show_spinner="Loading reports…")+def _load_path_cached(path: str, fingerprint: tuple[int, int, int] | None) -> DashboardData:+ return load_dashboard_path(Path(path)) # fingerprint is a cache key only+++# Keyed on file_id, which is stable while the file stays in the widget;+# the leading underscore stops Streamlit hashing the uploaded bytes.[email protected]_data(max_entries=_CACHE_ENTRIES, show_spinner=False)+def _parse_upload_cached(file_id: str, source: str, name: str, _contents: bytes) -> tuple[str, DashboardData]:+ digest = hashlib.sha256(_contents).hexdigest()+ try:+ payload = json.loads(_contents)+ except (ValueError, RecursionError): # UnicodeError is a ValueError subclass+ return digest, DashboardData(warnings=[f"{name}: invalid JSON. Upload a UTF-8 JSON evaluation report."])+ return digest, parse_dashboard_report(payload, source=source)++def rows_csv(rows: list[dict[str, Any]], columns: tuple[str, ...]) -> str:"""Export visible quantitative columns with spreadsheet-safe text cells."""buffer = io.StringIO(newline="")@@ -91,10 +130,11 @@ def rows_csv(rows: list[dict[str, Any]], columns: tuple[str, ...]) -> str:cells: dict[str, Any] = {}for column in columns:value = row.get(column)- if isinstance(value, str) and (- value.startswith(("\t", "\r", "\n")) or value.lstrip().startswith(("=", "+", "-", "@"))- ):- value = "'" + value+ if isinstance(value, str):+ # NFKC folds full-width prefixes such as U+FF1D to their ASCII forms.+ probe = unicodedata.normalize("NFKC", value).lstrip()+ if value.startswith(("\t", "\r", "\n")) or probe.startswith(("=", "+", "-", "@")):+ value = "'" + valuecells[column] = valuewriter.writerow(cells)return buffer.getvalue()@@ -120,7 +160,12 @@ def _performance_column_config() -> dict[str, Any]:"condition": st.column_config.TextColumn("Condition"),"score": st.column_config.NumberColumn("Score", format="%.3f", width="small"),"pass_rate": st.column_config.NumberColumn("Pass rate", format="percent", width="small"),- "duration_seconds": st.column_config.NumberColumn("Duration (s)", format="%.2f", width="small"),+ "total_tokens_per_trial": st.column_config.NumberColumn("Tokens / trial", format="%.1f", width="small"),+ "duration_seconds_per_trial": st.column_config.NumberColumn(+ "Agent time / trial (s)", format="%.2f", width="small"+ ),+ "cost_usd_per_trial": st.column_config.NumberColumn("Cost / trial (USD)", format="$%.4f", width="small"),+ "duration_seconds": st.column_config.NumberColumn("Agent time (s)", format="%.2f", width="small"),"cost_usd": st.column_config.NumberColumn("Cost (USD)", format="$%.4f", width="small"),"status": st.column_config.TextColumn("Status"),"trials": st.column_config.NumberColumn("Trials", format="%d"),@@ -139,8 +184,10 @@ def _comparison_column_config(columns: tuple[str, ...]) -> dict[str, Any]:"score_delta": "Score Δ","total_tokens_delta": "Tokens Δ","total_tokens_percent": "Tokens Δ (%)",- "duration_seconds_delta": "Time Δ (s)",- "duration_seconds_percent": "Time Δ (%)",+ "duration_seconds_delta": "Agent time Δ (s)",+ "duration_seconds_percent": "Agent time Δ (%)",+ "with_duration_seconds": "With skill: agent time (s)",+ "without_duration_seconds": "Without skill: agent time (s)","cost_usd_delta": "Cost Δ (USD)","cost_usd_percent": "Cost Δ (%)",}@@ -182,10 +229,14 @@ def _show_performance(rows: list[dict[str, Any]]) -> None:file_name="skill-performance.csv",mime="text/csv",key="download_performance",+ on_click="ignore",)def _show_comparisons(rows: list[dict[str, Any]]) -> None:+ if not rows:+ st.info("No rows match these filters. Clear a filter to include more results.")+ returncomparisons = comparison_rows(rows)st.caption("Conditions are paired within each run, skill, model, harness, and feature configuration.")if not comparisons:@@ -197,7 +248,7 @@ def _show_comparisons(rows: list[dict[str, Any]]) -> None:key="comparison_metrics",)identity = ("skill", "model", "harness")- details = ("comparison_status", "features", "coverage", "run_id", "source")+ details = ("comparison_status", "comparison_reason", "features", "coverage", "run_id", "source")metric_sets = {"Overview": ("score_delta",@@ -232,6 +283,8 @@ def _show_comparisons(rows: list[dict[str, Any]]) -> None:}columns = (*identity, *metric_sets[view], *details) if view in metric_sets else _COMPARISON_COLUMNSvisible = [{column: row.get(column) for column in columns} for row in comparisons]+ if not any(row["comparison_status"] == "comparable" for row in comparisons):+ st.info("No pair is comparable. The Comparison reason column explains why.")st.dataframe(visible,column_order=columns,@@ -250,6 +303,7 @@ def _show_comparisons(rows: list[dict[str, Any]]) -> None:file_name="skill-comparison.csv",mime="text/csv",key="download_comparisons",+ on_click="ignore",)@@ -257,7 +311,7 @@ def main(paths: list[str] | None = None) -> None:"""Render the app, accepting CLI paths or a supplied path list for embedding."""st.set_page_config(page_title="Skill performance · SkillEvaluator", page_icon="▦", layout="wide")st.title("Skill performance")- st.caption("SKILLEVALUATOR · Compare agent quality, tokens, duration, and cost across evaluation runs.")+ st.caption("SKILLEVALUATOR · Compare agent quality, tokens, agent time, and cost across evaluation runs.")st.sidebar.header("Reports")uploads = st.sidebar.file_uploader("Upload JSON reports",@@ -266,9 +320,14 @@ def main(paths: list[str] | None = None) -> None:help="Use Tier 3 JSON reports or consolidated SkillEvaluator JSON reports, up to 16 MB (16,000,000 bytes) per file.",key="report_uploads",)+ if paths and st.sidebar.button(+ "Reload reports", key="reload_reports", help="Re-read report files and run directories from disk."+ ):+ _load_path_cached.clear()rows: list[dict[str, Any]] = []warnings: list[str] = []seen_paths: set[Path] = set()+ inputs: list[Path] = []for supplied_path in paths or []:try:path = Path(supplied_path).expanduser().resolve()@@ -278,26 +337,44 @@ def main(paths: list[str] | None = None) -> None:if path in seen_paths:continueseen_paths.add(path)- data = load_dashboard_path(path)+ inputs.append(path)+ # A directory's fingerprint misses in-place rewrites of files inside it, so load+ # deeper inputs first and let dedupe_rows keep their fresher copy of shared rows.+ for path in sorted(inputs, key=lambda item: len(item.parts), reverse=True):+ try:+ data = _load_path_cached(str(path), _input_fingerprint(path))+ except Exception as exc: # one unreadable input must not hide the others+ warnings.append(f"{path}: cannot load dashboard input: {exc}")+ continuerows.extend(data.rows)warnings.extend(data.warnings)+ seen_uploads: set[str] = set()for index, uploaded in enumerate(uploads):- contents = uploaded.getvalue()- if len(contents) > _MAX_UPLOAD_BYTES:- warnings.append(f"{uploaded.name}: report exceeds 16 MB (16,000,000 bytes). Upload a smaller evaluation report.")+ if uploaded.size > _MAX_UPLOAD_BYTES:+ warnings.append(+ f"{uploaded.name}: report exceeds 16 MB (16,000,000 bytes). Upload a smaller evaluation report."+ )continue- try:- payload = json.loads(contents)- except (ValueError, UnicodeError, RecursionError):- warnings.append(f"{uploaded.name}: invalid JSON. Upload a UTF-8 JSON evaluation report.")+ digest, data = _parse_upload_cached(+ uploaded.file_id, f"upload:{index + 1}:{uploaded.name}", uploaded.name, uploaded.getvalue()+ )+ if digest in seen_uploads:+ warnings.append(f"{uploaded.name}: identical to an earlier upload; skipped.")continue- data = parse_dashboard_report(payload, source=f"upload:{index + 1}:{uploaded.name}")+ seen_uploads.add(digest)rows.extend(data.rows)warnings.extend(data.warnings)+ # Overlapping CLI paths (a directory and a run inside it) reload rows under the same source.+ rows, duplicate_count = dedupe_rows(rows)+ if duplicate_count:+ plural = "s" if duplicate_count != 1 else ""+ warnings.append(f"Overlapping inputs: {duplicate_count} duplicate condition row{plural} loaded once.")if warnings:- with st.expander(f"Report notices ({len(warnings)})", expanded=True):- for warning in dict.fromkeys(warnings):- st.warning(warning)+ notices = list(dict.fromkeys(warnings))+ with st.expander(f"Report notices ({len(notices)})", expanded=True):+ st.warning("Some inputs were skipped or have incomplete measurements.")+ for notice in notices:+ st.text(notice) # st.text parses neither Markdown nor HTMLif not rows:st.info("Upload JSON evaluation reports to start comparing results.")st.caption("You can also open saved run directories with skillevaluator dashboard PATH [PATH …].")@@ -308,9 +385,9 @@ def main(paths: list[str] | None = None) -> None:dimension_rows = rowsfor field in _DIMENSIONS:dimension_rows = _filter_rows(dimension_rows, field, rows)- dimension_rows = _filter_rows(dimension_rows, "status", rows)- performance_rows = _filter_rows(dimension_rows, "condition", rows)- st.sidebar.caption("The condition filter applies to performance. Comparisons retain both conditions.")+ performance_rows = _filter_rows(dimension_rows, "status", rows)+ performance_rows = _filter_rows(performance_rows, "condition", rows)+ st.sidebar.caption("Status and condition filters apply to performance. Comparisons retain both conditions.")run_count = len({(row["source"], row["run_id"]) for row in performance_rows})skill_count = len({row["skill"] for row in performance_rows})@@ -320,7 +397,15 @@ def main(paths: list[str] | None = None) -> None:summary[1].metric("Skills", skill_count)summary[2].metric("Condition rows", len(performance_rows))summary[3].metric("Scored rows", scored_count)- st.caption("Blank cells mean a measurement is missing, incomplete, or ambiguous. Unknown settings remain labeled unknown.")+ st.caption(+ "Blank cells mean a measurement is missing, incomplete, or ambiguous. Unknown settings remain labeled unknown."+ )+ # Evaluator commands do not write harness features; "none (recorded)" counts as recorded.+ if all(row.get("features") == "unknown" for row in rows):+ st.caption(+ "No harness features are recorded in these reports. Add dashboard_metadata.harness_features to compare "+ "feature configurations; see Recorded harness features in the reports guide."+ )performance_tab, comparison_tab = st.tabs(["Performance", "With vs without skill"])with performance_tab:_show_performance(performance_rows)diff --git a/src/skillevaluator/reporting/dashboard_data.py b/src/skillevaluator/reporting/dashboard_data.pyindex 0ab37c36..98e430bf 100644--- a/src/skillevaluator/reporting/dashboard_data.py+++ b/src/skillevaluator/reporting/dashboard_data.py@@ -17,17 +17,32 @@ import reimport statfrom collections import Counterfrom dataclasses import dataclass, field-from datetime import datetimefrom itertools import islicefrom pathlib import Pathfrom typing import Any+from packaging.version import Version++from skillevaluator.tier3.harbor.resource_usage import agent_execution_seconds+_USAGE = ("prompt_tokens", "completion_tokens", "total_tokens", "cached_tokens", "duration_seconds", "cost_usd")_IDENTITY = ("source", "run_id", "skill", "model", "harness", "features")_PAIRING_IDENTITY = (*_IDENTITY, "_report_index", "_agent_key", "_feature_key")_MAX_FILE_BYTES = 16 * 1024 * 1024_MAX_FILES = 1000_MAX_PATHS = 20_000+_MAX_COUNT = 1_000_000 # far above report_data's 512-trial bound; larger values are malformed+# Mirror report_data._MAX_TRIALS_PER_CONDITION / _MAX_TRIAL_PATHS_SCANNED so the+# dashboard totals the same bounded trial population as the canonical loader.+_MAX_NATIVE_TRIALS = 512+_MAX_NATIVE_TRIAL_ENTRIES = 4096+_CATALOG_LIMIT_NOTICE = "Dashboard catalog limit reached; some inputs were omitted."+_DISCOVERY_LIMIT_NOTICE = "Dashboard discovery limit reached; some inputs were omitted."+# Loader bounds on these artifacts drop only per-trial telemetry (tokens, steps).+# Canonical trials then omit those counters, so coverage counts keep totals blank.+_TELEMETRY_ARTIFACTS = frozenset({"trajectory"})+# Canonical exports from earlier evaluators wrote missing token counters as zero.+_ZERO_TOKEN_COUNTERS_FIXED_IN = Version("0.5.0")@dataclass@@ -63,7 +78,7 @@ def _measurement(value: Any) -> float | None:def _count(value: Any) -> int | None:- return value if isinstance(value, int) and not isinstance(value, bool) and value >= 0 else None+ return value if isinstance(value, int) and not isinstance(value, bool) and 0 <= value <= _MAX_COUNT else Nonedef _feature_config(*owners: dict[str, Any]) -> tuple[str, str]:@@ -98,7 +113,16 @@ def _is_native_result(value: dict[str, Any]) -> bool:def _trial_coverage_truncated(marker: dict[str, Any]) -> bool:if not marker.get("truncated"):return False- if marker.get("artifact_loading"):+ loading = marker.get("artifact_loading")+ if loading and (+ not isinstance(loading, list)+ or any(+ not isinstance(reason, dict)+ or not isinstance(reason.get("artifact"), str)+ or reason["artifact"] not in _TELEMETRY_ARTIFACTS+ for reason in loading+ )+ ):return Trueomitted = _dict(marker.get("omitted"))# These budgets affect diagnostics, not the normalized trial population.@@ -116,7 +140,9 @@ def _trial_coverage_truncated(marker: dict[str, Any]) -> bool:"custom_metric_discovery_trials","non_best_agent_details",}- return not omitted or any(section not in diagnostics for section in omitted)+ if not omitted:+ return not loading # an unexplained truncation still fails closed+ return any(section not in diagnostics for section in omitted)def _payloads(value: Any, depth: int = 0):@@ -170,11 +196,6 @@ def _trial_keys(trials: list[dict[str, Any]]) -> tuple[tuple[str, str], ...] | Nif len(set(normalized_attempts)) > 1:return Noneattempt = normalized_attempts[0] if normalized_attempts else None- # Native canonical reports omit attempt numbers, but Harbor trial IDs- # carry __attemptN when attempts are repeated.- matches = re.findall(r"(?:__|_|-)attempt[_-]?(\d+)(?=__|$)", trial_id)- if attempt is None and matches:- attempt = matches[-1].lstrip("0") or "0"if trial_id and trial_id in seen_trials:if seen_trials[trial_id] != (entry, attempt):return None@@ -186,27 +207,57 @@ def _trial_keys(trials: list[dict[str, Any]]) -> tuple[tuple[str, str], ...] | Nif attempt is None:implicit.add(entry)keys.append((entry, str(attempt) if attempt is not None else f"ordinal:{counts[entry]}"))- if any(counts[entry] > 1 for entry in implicit):+ # Harbor names repeated attempts <task>__<random> without an ordinal, so+ # unnumbered attempts pair by case count, as in the HTML report, only when+ # each one has its own trial ID.+ implicit_ids: dict[str, set[str]] = {}+ for trial_id, (entry, attempt) in seen_trials.items():+ if attempt is None:+ implicit_ids.setdefault(entry, set()).add(trial_id)+ if any(counts[entry] > 1 and len(implicit_ids.get(entry, ())) != counts[entry] for entry in implicit):return Nonereturn tuple(sorted(keys))-def _usage(trial: dict[str, Any]) -> dict[str, float | None]:+def _zero_encodes_missing_tokens(report: dict[str, Any]) -> bool:+ # The version names the run's evaluator, so re-exports of older runs also fail closed.+ version = report.get("evaluator_version") or _dict(report.get("summary")).get("evaluator_version")+ if not isinstance(version, str):+ return False+ try:+ return Version(version) < _ZERO_TOKEN_COUNTERS_FIXED_IN+ except ValueError: # InvalidVersion, or more digits than int() converts+ return False+++def _has_zero_token_counter(trials: list[dict[str, Any]]) -> bool:+ return any(+ _measurement(_dict(trial.get("tokens")).get(key)) == 0+ for trial in trials+ for key in ("prompt", "completion", "cached")+ )+++def _usage(trial: dict[str, Any], *, zero_tokens_missing: bool = False) -> dict[str, float | None]:tokens = _dict(trial.get("tokens"))- prompt = _measurement(tokens.get("prompt"))- completion = _measurement(tokens.get("completion"))+ prompt, completion, cached = (_measurement(tokens.get(key)) for key in ("prompt", "completion", "cached"))+ if zero_tokens_missing:+ prompt, completion, cached = (value or None for value in (prompt, completion, cached))return {"prompt_tokens": prompt,"completion_tokens": completion,"total_tokens": prompt + completion if prompt is not None and completion is not None else None,- "cached_tokens": _measurement(tokens.get("cached")),- "duration_seconds": _measurement(trial.get("duration_seconds")),+ "cached_tokens": cached,+ # Canonical Tier 3 JSON records agent_seconds; duration_seconds is only+ # read from hand-built payloads and never replaces an invalid agent_seconds.+ "duration_seconds": _measurement(+ trial["agent_seconds"] if "agent_seconds" in trial else trial.get("duration_seconds")+ ),"cost_usd": _measurement(trial.get("cost_usd")),}-def _set_usage(row: dict[str, Any], usages: list[dict[str, Any]], expected: int | None = None) -> None:- expected = row["trials"] if expected is None else expected+def _set_usage(row: dict[str, Any], usages: list[dict[str, Any]], *, expected: int) -> None:row["_usage_coverage"] = {}for metric in _USAGE:values = [_measurement(usage.get(metric)) for usage in usages]@@ -217,13 +268,32 @@ def _set_usage(row: dict[str, Any], usages: list[dict[str, Any]], expected: intif values and available == len(values) == expectedelse None)+ # A total exists only when every expected trial is measured, so this is an exact mean.+ for metric in ("total_tokens", "duration_seconds", "cost_usd"):+ total = row[metric]+ row[f"{metric}_per_trial"] = _number(total / expected) if total is not None and expected else Noneeligibility = "complete" if row["_complete"] else "incomplete or unknown trial coverage"row["coverage"] = (- f"{row['trials']} trials ({eligibility}); tokens {row['_usage_coverage']['total_tokens']}, "+ f"{row['trials']} trial{'' if row['trials'] == 1 else 's'} ({eligibility}); "+ f"tokens {row['_usage_coverage']['total_tokens']}, "f"time {row['_usage_coverage']['duration_seconds']}, cost {row['_usage_coverage']['cost_usd']}")+def _ambiguous_resource_notice(source: str) -> str:+ return (+ f"{source}: canonical multi-step resource ownership is ambiguous; "+ "usage totals are unavailable. Load the retained run directory for authoritative step measurements."+ )+++def _zero_token_notice(source: str) -> str:+ return (+ f"{source}: exports before v0.5.0 may record missing token counters as zero; "+ "zero token counters are shown as unavailable. Load the retained run directory for recorded counters."+ )++def parse_dashboard_report(payload: Any, source: str) -> DashboardData:"""Normalize canonical Tier 3, validate-report, and embedded catalog JSON."""output = DashboardData()@@ -243,6 +313,10 @@ def parse_dashboard_report(payload: Any, source: str) -> DashboardData:run_id = str(report.get("evaluated_at") or summary.get("evaluated_at") or "unknown")truncation = _dict(report.get("report_truncation")) or _dict(report.get("truncation"))coverage_truncated = _trial_coverage_truncated(truncation)+ zero_tokens_missing = _zero_encodes_missing_tokens(report)+ non_best_pruned = bool(truncation.get("truncated")) and "non_best_agent_details" in _dict(+ truncation.get("omitted")+ )if truncation.get("truncated"):message = ("trial data may be truncated; comparisons are unavailable."@@ -255,6 +329,15 @@ def parse_dashboard_report(payload: Any, source: str) -> DashboardData:if not agent:continueconditions = _dict(agent.get("conditions"))+ # The embedded-size budget empties a non-best agent's trial details+ # but keeps its declared counts. Older reports also dropped its+ # conditions; current ones keep a compacted copy.+ details_pruned = (+ non_best_pruned+ and str(name) != str(report.get("best_agent") or "")+ and not agent.get("trials")+ and not agent.get("trials_baseline")+ )for condition, trial_field, score_field, count_field in (("with_skill", "trials", "with_skill", "num_trials"),("without_skill", "trials_baseline", "baseline", "num_trials_baseline"),@@ -288,6 +371,7 @@ def parse_dashboard_report(payload: Any, source: str) -> DashboardData:declared = _count(agent.get(count_field))if declared is None:declared = len(trials)+ pruned = details_pruned and not trialsexpected = _count(details.get("expected_attempts"))scored = _count(details.get("scored_attempts"))keys = _trial_keys(trials)@@ -295,10 +379,11 @@ def parse_dashboard_report(payload: Any, source: str) -> DashboardData:physical_counts = Counter(str(trial["trial_id"]) for trial in trials if trial.get("trial_id"))ambiguous_ids = {trial_id for trial_id, count in physical_counts.items() if count > 1}if ambiguous_ids:- notice = (- f"{source}: canonical multi-step resource ownership is ambiguous; "- "usage totals are unavailable. Load the retained run directory for authoritative step measurements."- )+ notice = _ambiguous_resource_notice(source)+ if notice not in output.warnings:+ output.warnings.append(notice)+ if zero_tokens_missing and _has_zero_token_counter(trials):+ notice = _zero_token_notice(source)if notice not in output.warnings:output.warnings.append(notice)complete = (@@ -320,13 +405,15 @@ def parse_dashboard_report(payload: Any, source: str) -> DashboardData:"harness": str(agent.get("harness") or agent.get("name") or name),"condition": condition,"status": status,- "trials": logical_count,+ "trials": declared if pruned else logical_count,"score": _number(agent.get(score_field)) if complete else None,"pass_rate": _number(_dict(_dict(agent.get("pass_at_k")).get(condition)).get("rate"))if completeelse None,"_complete": complete,"_invalid_counts": invalid_counts,+ "_coverage_truncated": coverage_truncated,+ "_details_pruned": pruned,"_report_index": report_index,"_agent_key": str(name),"_trial_keys": keys,@@ -343,9 +430,13 @@ def parse_dashboard_report(payload: Any, source: str) -> DashboardData:[]if invalid_countselse [- {} if str(trial.get("trial_id") or "") in ambiguous_ids else _usage(trial) for trial in trials+ {}+ if str(trial.get("trial_id") or "") in ambiguous_ids+ else _usage(trial, zero_tokens_missing=zero_tokens_missing)+ for trial in trials],- expected=declared + max(0, row["_expected_usage_trials"] - logical_count),+ # Trials missing from the declared count are already part of it.+ expected=declared + max(0, row["_expected_usage_trials"] - max(logical_count, declared)),)output.rows.append(row)if not output.rows and not output.warnings:@@ -353,6 +444,42 @@ def parse_dashboard_report(payload: Any, source: str) -> DashboardData:return output+def dedupe_rows(rows: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], int]:+ """Drop rows reloaded from the same resolved source; keys are unique within one parse."""+ seen: set[tuple[Any, ...]] = set()+ kept = []+ for row in rows:+ key = (row.get("source"), row.get("_report_index"), row.get("_agent_key"), row.get("condition"))+ if key not in seen:+ seen.add(key)+ kept.append(row)+ return kept, len(rows) - len(kept)+++def _ineligible_reason(with_row: dict[str, Any], without_row: dict[str, Any]) -> str:+ """Name the first check that keeps a with/without pair from being comparable."""+ arms = (("with-skill", with_row), ("without-skill", without_row))+ for label, row in arms:+ if row.get("status") != "succeeded":+ return f"{label} arm status {row.get('status') or 'unknown'}"+ if with_row.get("_invalid_counts") or without_row.get("_invalid_counts"):+ return "invalid trial counts"+ if with_row.get("_coverage_truncated") or without_row.get("_coverage_truncated"):+ return "trial data truncated"+ if with_row.get("_details_pruned") or without_row.get("_details_pruned"):+ return "trial details pruned from embedded report; load the retained run directory"+ for label, row in arms:+ if not row.get("_complete"):+ return f"{label} trial coverage incomplete"+ if with_row.get("_trial_keys") != without_row.get("_trial_keys"):+ return "different cases or attempts"+ if with_row.get("_dataset_digest") != without_row.get("_dataset_digest"):+ return "different dataset"+ if with_row.get("_attempt_policy") != without_row.get("_attempt_policy"):+ return "different attempt policy"+ return ""++def comparison_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:"""Compute with-minus-without differences only within an identical run."""groups: dict[tuple[Any, ...], list[dict[str, Any]]] = {}@@ -365,9 +492,11 @@ def comparison_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:without_rows = [row for row in arms if row["condition"] == "without_skill"]result = dict(zip(_PAIRING_IDENTITY, identity, strict=True))status = "missing condition"+ reason = f"no {'without' if with_rows else 'with'}-skill arm for this run and configuration"paired = Falseif len(with_rows) > 1 or len(without_rows) > 1:status = "ambiguous duplicate conditions"+ reason = "same arm loaded more than once"elif with_rows and without_rows:with_row, without_row = with_rows[0], without_rows[0]paired = (@@ -378,7 +507,9 @@ def comparison_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:and with_row.get("_attempt_policy") == without_row.get("_attempt_policy"))status = "comparable" if paired else "incomplete or mismatched coverage"+ reason = "" if paired else _ineligible_reason(with_row, without_row)result["comparison_status"] = status+ result["comparison_reason"] = reasonresult["coverage"] = (f"{with_rows[0]['trials']} with / {without_rows[0]['trials']} without"if with_rows and without_rows@@ -402,7 +533,8 @@ def comparison_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:def _read_json(path: Path) -> Any:- if not stat.S_ISREG(path.stat().st_mode) or path.stat().st_size > _MAX_FILE_BYTES:+ info = path.stat()+ if not stat.S_ISREG(info.st_mode) or info.st_size > _MAX_FILE_BYTES:raise ValueError("JSON must be a regular file no larger than 16 MiB")with path.open("rb") as stream:data = stream.read(_MAX_FILE_BYTES + 1)@@ -431,22 +563,17 @@ def _read_native_json(path: Path, run_root: Path) -> Any:return _read_json(path)-def _duration(result: dict[str, Any]) -> float | None:- execution = _dict(result.get("agent_execution"))- try:- started = datetime.fromisoformat(str(execution["started_at"]))- finished = datetime.fromisoformat(str(execution["finished_at"]))- return _measurement((finished - started).total_seconds())- except (KeyError, TypeError, ValueError, OverflowError):- return None---def _native_usage(trial_dir: Path, run_root: Path, *, require_steps: bool = False) -> dict[str, Any]:- trajectory = (- _dict(_read_native_json(trial_dir / "trajectory.json", run_root))- if (trial_dir / "trajectory.json").is_file()- else {}- )+def _native_usage(+ trial_dir: Path, run_root: Path, warnings: list[str], *, require_steps: bool = False+) -> dict[str, Any]:+ trajectory: dict[str, Any] = {}+ if (trial_dir / "trajectory.json").is_file():+ try:+ trajectory = _dict(_read_native_json(trial_dir / "trajectory.json", run_root))+ except (OSError, ValueError, RecursionError) as exc:+ # As in canonical JSON, this loses only the trajectory's token totals;+ # result.json still records time, cost, and agent_result counters.+ warnings.append(f"{trial_dir}: cannot read trajectory: {exc}")final = _dict(trajectory.get("final_metrics"))result = (_dict(_read_native_json(trial_dir / "result.json", run_root)) if (trial_dir / "result.json").is_file() else {}@@ -476,7 +603,7 @@ def _native_usage(trial_dir: Path, run_root: Path, *, require_steps: bool = Falsprompt = step_total("n_input_tokens")completion = step_total("n_output_tokens")cached = step_total("n_cache_tokens")- durations = [_duration(_dict(step)) for step in constituents]+ durations = [agent_execution_seconds(_dict(step)) for step in constituents]costs = [_measurement(_dict(_dict(step).get("agent_result")).get("cost_usd")) for step in constituents]return {"_resource_scope_valid": True,@@ -496,16 +623,29 @@ def _load_native(path: Path) -> DashboardData:engine_result = _dict(_read_native_json(path / "result.json", path))skill = str(engine_result.get("skill_name") or "unknown")- result = agent_eval_result_from_directory(Path(skill), path, engine_result=engine_result, use_llm_judge=False)+ # The payload is parsed in memory, never embedded, so keep every agent's trials.+ result = agent_eval_result_from_directory(+ Path(skill), path, engine_result=engine_result, use_llm_judge=False, enforce_embedded_size=False+ )if result is None:return DashboardData(warnings=[f"{path}: no supported retained agent summaries found."])canonical = result.metadata["agent_eval"]canonical["run_id"] = engine_result.get("run_id") or path.name+ # The runner writes the same config into result.json, but run_config.json is+ # the copy users edit to add dashboard_metadata, so it wins when readable.run_config = _dict(engine_result.get("run_config"))- if not run_config and (path / "run_config.json").is_file():- run_config = _dict(_read_native_json(path / "run_config.json", path))+ config_warning = None+ if (path / "run_config.json").is_file():+ try:+ run_config = _dict(_read_native_json(path / "run_config.json", path)) or run_config+ except (OSError, ValueError, RecursionError) as exc:+ config_warning = f"{path / 'run_config.json'}: cannot read run config: {exc}"canonical["dashboard_metadata"] = run_config.get("dashboard_metadata")output = parse_dashboard_report(canonical, str(path))+ # Retained usage is read from disk below, so the export notice does not apply.+ output.warnings = [warning for warning in output.warnings if warning != _zero_token_notice(str(path))]+ if config_warning:+ output.warnings.append(config_warning)unresolved_scope = Falsefor row in output.rows:# The report's agent keys identify repeated harness occurrences on disk.@@ -536,20 +676,43 @@ def _load_native(path: Path) -> DashboardData:_set_usage(row, [], expected=row["_expected_usage_trials"])continueif trials_dir.is_dir():- trial_paths, truncated = _directory_entries(trials_dir, 4096)- trial_dirs = [candidate for candidate in trial_paths if candidate.is_dir() and not candidate.is_symlink()]- truncated |= len(trial_dirs) > 512- for trial_dir in trial_dirs[:512]:- if not trial_dir.is_dir() or trial_dir.is_symlink():+ try:+ trial_paths, truncated = _directory_entries(trials_dir, _MAX_NATIVE_TRIAL_ENTRIES)+ trial_dirs = [+ candidate for candidate in trial_paths if candidate.is_dir() and not candidate.is_symlink()+ ]+ except (OSError, ValueError) as exc:+ output.warnings.append(f"{trials_dir}: cannot read retained trials: {exc}")+ _set_usage(row, [], expected=row["_expected_usage_trials"])+ continue+ truncated |= len(trial_dirs) > _MAX_NATIVE_TRIALS+ loaded: list[tuple[Path, dict[str, Any] | None]] = []+ for trial_dir in trial_dirs[:_MAX_NATIVE_TRIALS]:+ try:+ loaded.append((trial_dir, _dict(_read_native_json(trial_dir / "reward.json", path))))+ except (OSError, ValueError, RecursionError) as exc:+ output.warnings.append(f"{trial_dir}: cannot read usage: {exc}")+ loaded.append((trial_dir, None))+ # Canonical reports collapse reward rows that share one physical+ # trial, so count shared identities in the retained rows themselves.+ physical_counts = Counter(+ str(reward.get("trial_id") or trial_dir.name) for trial_dir, reward in loaded if reward is not None+ )+ for trial_dir, reward in loaded:+ if reward is None:+ usages_by_trial[trial_dir.name] = {}+ continue+ physical_id = str(reward.get("trial_id") or trial_dir.name)+ if physical_id in usages_by_trial:continuetry:- reward = _dict(_read_native_json(trial_dir / "reward.json", path))- physical_id = str(reward.get("trial_id") or trial_dir.name)- if physical_id not in usages_by_trial:- usages_by_trial[physical_id] = _native_usage(- trial_dir, path, require_steps=physical_id in row["_ambiguous_trial_ids"]- )- unresolved_scope |= usages_by_trial[physical_id].get("_resource_scope_valid") is False+ usages_by_trial[physical_id] = _native_usage(+ trial_dir,+ path,+ output.warnings,+ require_steps=physical_counts[physical_id] > 1 or physical_id in row["_ambiguous_trial_ids"],+ )+ unresolved_scope |= usages_by_trial[physical_id].get("_resource_scope_valid") is Falseexcept (OSError, ValueError, RecursionError) as exc:output.warnings.append(f"{trial_dir}: cannot read usage: {exc}")usages_by_trial[trial_dir.name] = {}@@ -558,10 +721,11 @@ def _load_native(path: Path) -> DashboardData:usages_by_trial["__omitted_trials__"] = {}row["_expected_usage_trials"] = max(row["_expected_usage_trials"], len(usages_by_trial))_set_usage(row, list(usages_by_trial.values()), expected=row["_expected_usage_trials"])+ notice = _ambiguous_resource_notice(str(path))if not unresolved_scope:- output.warnings = [- warning for warning in output.warnings if "canonical multi-step resource ownership" not in warning- ]+ output.warnings = [warning for warning in output.warnings if warning != notice]+ elif notice not in output.warnings:+ output.warnings.append(notice)return output@@ -575,7 +739,7 @@ def _load_report_file(if path in budget.seen:return DashboardData()if budget.files_remaining <= 0 or budget.references_remaining <= 0:- return DashboardData(warnings=["Dashboard catalog limit reached; some inputs were omitted."])+ return DashboardData(warnings=[_CATALOG_LIMIT_NOTICE])budget.seen.add(path)budget.files_remaining -= 1if path.name == "result.json" and (path.parent / "run_config.json").is_file():@@ -591,7 +755,7 @@ def _load_report_file(if not isinstance(report_dir, str) or not isinstance(report_name, str):continueif budget.files_remaining <= 0 or budget.references_remaining <= 0:- output.warnings.append("Dashboard catalog limit reached; some inputs were omitted.")+ output.warnings.append(_CATALOG_LIMIT_NOTICE)breakbudget.references_remaining -= 1try:@@ -604,7 +768,7 @@ def _load_report_file(output.rows.extend(report.rows)output.warnings.extend(report.warnings)if len(payload["skills"]) > _MAX_FILES:- output.warnings.append("Dashboard catalog limit reached; some inputs were omitted.")+ output.warnings.append(_CATALOG_LIMIT_NOTICE)if not output.rows:output.warnings.extend(data.warnings)return output@@ -642,9 +806,15 @@ def load_dashboard_path(path: Path) -> DashboardData:while pending:root = pending.pop()if visited >= _MAX_PATHS or files >= _MAX_FILES:- output.warnings.append("Dashboard discovery limit reached; some inputs were omitted.")+ output.warnings.append(_DISCOVERY_LIMIT_NOTICE)break- if (root / "result.json").is_file() and (root / "run_config.json").is_file():+ try:+ # is_file() re-raises EACCES for a subdirectory without search permission.+ native_run = (root / "result.json").is_file() and (root / "run_config.json").is_file()+ except OSError as exc:+ output.warnings.append(f"{root}: cannot discover reports: {exc}")+ continue+ if native_run:try:data = _load_native(root)except (OSError, ValueError, RecursionError, RuntimeError, TypeError, AttributeError) as exc:@@ -660,7 +830,7 @@ def load_dashboard_path(path: Path) -> DashboardData:continuevisited += len(entries)if truncated:- output.warnings.append("Dashboard discovery limit reached; some inputs were omitted.")+ output.warnings.append(_DISCOVERY_LIMIT_NOTICE)subdirectories = []for entry in entries:if files >= _MAX_FILES:diff --git a/src/skillevaluator/reporting/dashboard_ui/__init__.py b/src/skillevaluator/reporting/dashboard_ui/__init__.pynew file mode 100644index 00000000..52a7a9da--- /dev/null+++ b/src/skillevaluator/reporting/dashboard_ui/__init__.py@@ -0,0 +1,2 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0diff --git a/src/skillevaluator/reporting/dashboard_ui/skillevaluator_dashboard.py b/src/skillevaluator/reporting/dashboard_ui/skillevaluator_dashboard.pynew file mode 100644index 00000000..2307c8f1--- /dev/null+++ b/src/skillevaluator/reporting/dashboard_ui/skillevaluator_dashboard.py@@ -0,0 +1,14 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Streamlit entry script for the dashboard.++Streamlit puts this script's directory first on sys.path, so the directory holds+only this uniquely named file and cannot shadow the standard library or a dependency.+"""++import sys++from skillevaluator.reporting.dashboard_app import main++main(sys.argv[1:])diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex 3d855170..36f93bf9 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -184,7 +184,6 @@"type": "path"},{- "default": "8501","name": "port","opts": ["--port"diff --git a/tests/reporting/test_dashboard_app.py b/tests/reporting/test_dashboard_app.pyindex f8e79e2e..cf71c901 100644--- a/tests/reporting/test_dashboard_app.py+++ b/tests/reporting/test_dashboard_app.py@@ -8,15 +8,24 @@ from __future__ import annotationsimport csvimport ioimport json+import osfrom pathlib import Pathimport pytestpytest.importorskip("streamlit")+# st.dataframe otherwise imports these inside AppTest's timed script thread.+pytest.importorskip("pandas")+pytest.importorskip("pyarrow")+import streamlit as stfrom streamlit.testing.v1 import AppTest+from skillevaluator.reporting import dashboard_appfrom skillevaluator.reporting.dashboard_app import rows_csv+# A ceiling for cold imports under xdist and coverage; passing runs finish far sooner.+_APP_TIMEOUT_SECONDS = 60+def _write_report(path: Path,@@ -27,6 +36,7 @@ def _write_report(features: list[str] | None = None,baseline_status: str = "succeeded",measured: bool = True,+ record_features: bool = True,) -> Path:def trial(score: float, tokens: int) -> dict:value = {"entry_id": "case-1", "trial_id": "case-1__attempt1", "overall": score}@@ -63,13 +73,37 @@ def _write_report(},},}+ if not record_features:+ del report["agents"][harness]["dashboard_metadata"]path.write_text(json.dumps(report), encoding="utf-8")return pathdef _app(paths: list[Path]) -> AppTest:script = f"from skillevaluator.reporting.dashboard_app import main\nmain({[str(path) for path in paths]!r})\n"- return AppTest.from_string(script, default_timeout=15).run()+ return AppTest.from_string(script, default_timeout=_APP_TIMEOUT_SECONDS).run()+++def _upload_app() -> AppTest:+ app = _app([])+ if not hasattr(app, "file_uploader"):+ pytest.skip("This Streamlit version's AppTest cannot drive st.file_uploader")+ return app+++def _rewrite_score(path: Path, score: float) -> None:+ report = json.loads(path.read_text(encoding="utf-8"))+ report["agents"]["opencode"]["with_skill"] = score+ report["agents"]["opencode"]["padding"] = "x" * 10 # change the size too, for coarse mtimes+ path.write_text(json.dumps(report), encoding="utf-8")++[email protected](autouse=True)+def _clear_dashboard_cache():+ # Cached loads are process-wide, so keep each test's inputs isolated.+ st.cache_data.clear()+ yield+ st.cache_data.clear()def test_empty_dashboard_has_upload_direction() -> None:@@ -97,7 +131,26 @@ def test_dashboard_renders_numeric_performance_and_paired_comparison(tmp_path: Passert comparison["comparison_status"].iloc[0] == "comparable"assert "raw_transcript" not in performance.columnsassert all(not column.startswith("_") for column in performance.columns)+ assert json.loads(app.dataframe[0].proto.columns)["duration_seconds"]["label"] == "Agent time (s)"assert [metric.value for metric in app.metric] == ["1", "1", "2", "2"]+ downloads = app.get("download_button")+ assert len(downloads) == 2+ assert all(button.proto.ignore_rerun for button in downloads)+++def test_performance_table_shows_per_trial_usage(tmp_path: Path) -> None:+ path = _write_report(tmp_path / "run.json")+ app = _app([path])+ assert not app.exception+ performance = app.dataframe[0]+ assert performance.value["total_tokens_per_trial"].tolist() == [140.0, 120.0]+ config = json.loads(performance.proto.columns)+ assert config["total_tokens_per_trial"]["label"] == "Tokens / trial"+ for field in ("total_tokens_per_trial", "duration_seconds_per_trial", "cost_usd_per_trial"):+ assert config[field]["type_config"]["type"] == "number", field+ rows = dashboard_app.load_dashboard_path(path).rows+ header = next(csv.reader(io.StringIO(rows_csv(rows, dashboard_app._PERFORMANCE_COLUMNS))))+ assert header.index("total_tokens_per_trial") == header.index("score") + 1def test_dimension_filters_and_condition_retain_baseline_for_comparison(tmp_path: Path) -> None:@@ -123,7 +176,7 @@ def test_dimension_filters_and_condition_retain_baseline_for_comparison(tmp_pathassert not app.exceptionassert len(app.dataframe) == 0assert "No rows match these filters" in app.info[0].value- assert "No comparisons are available" in app.info[1].value+ assert "No rows match these filters" in app.info[1].valuefor widget in app.multiselect:widget.set_value([])@@ -147,7 +200,35 @@ def test_unmeasured_and_failed_baseline_are_not_zero_or_comparable(tmp_path: Patapp.multiselect(key="filter_status").set_value(["succeeded"]).run()assert not app.exceptionassert len(app.dataframe[0].value) == 1- assert app.dataframe[1].value["comparison_status"].iloc[0] == "missing condition"+ # Status narrows performance only, so the comparison still names the failed arm.+ comparison = app.dataframe[1].value+ assert comparison["comparison_status"].iloc[0] == "incomplete or mismatched coverage"+ assert comparison["comparison_reason"].iloc[0] == "without-skill arm status failed"+ assert comparison["coverage"].iloc[0] == "1 with / 1 without"+ assert comparison["score_delta"].isna().all()+ app.multiselect(key="filter_status").set_value(["failed"]).run()+ assert not app.exception+ assert app.dataframe[0].value["condition"].tolist() == ["without_skill"]+ assert app.dataframe[1].value["comparison_reason"].tolist() == ["without-skill arm status failed"]+++def test_no_comparable_pair_hint(tmp_path: Path) -> None:+ app = _app([_write_report(tmp_path / "failed.json", baseline_status="failed")])+ assert not app.exception+ assert any("No pair is comparable" in message.value for message in app.info)+ app = _app([_write_report(tmp_path / "paired.json")])+ assert not app.exception+ assert not any("No pair is comparable" in message.value for message in app.info)+++def test_unrecorded_features_caption(tmp_path: Path) -> None:+ app = _app([_write_report(tmp_path / "unrecorded.json", record_features=False)])+ assert not app.exception+ assert app.dataframe[0].value["features"].tolist() == ["unknown", "unknown"]+ assert any("No harness features are recorded" in caption.value for caption in app.caption)+ app = _app([_write_report(tmp_path / "recorded.json")])+ assert not app.exception+ assert not any("No harness features are recorded" in caption.value for caption in app.caption)def test_bad_report_does_not_hide_other_results(tmp_path: Path) -> None:@@ -156,23 +237,144 @@ def test_bad_report_does_not_hide_other_results(tmp_path: Path) -> None:invalid.write_text("{broken JSON", encoding="utf-8")app = _app([invalid, valid, tmp_path / "missing.json"])assert not app.exception- assert len(app.warning) == 2+ assert len(app.warning) == 1+ assert len(app.text) == 2assert len(app.dataframe[0].value) == 2[email protected](os.name == "nt" or os.geteuid() == 0, reason="Needs POSIX permissions enforced for this user")+def test_unreadable_subdirectory_does_not_hide_other_results(tmp_path: Path) -> None:+ runs = tmp_path / "runs"+ runs.mkdir()+ _write_report(runs / "run.json")+ locked = runs / "locked"+ locked.mkdir()+ other = _write_report(tmp_path / "other.json", model="model-b")+ locked.chmod(0)+ try:+ app = _app([runs, other])+ finally:+ locked.chmod(0o755)+ assert not app.exception+ assert len(app.dataframe[0].value) == 4+ assert any(text.value.startswith(f"{locked.resolve()}: cannot discover reports") for text in app.text)+++def test_input_that_fails_to_load_does_not_hide_other_results(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ real = dashboard_app.load_dashboard_path+ unreadable = _write_report(tmp_path / "unreadable.json")+ valid = _write_report(tmp_path / "valid.json", model="model-b")++ def failing(path: Path):+ # Stands in for any error the loader does not anticipate.+ if path == unreadable.resolve():+ raise PermissionError(13, "Permission denied", str(path))+ return real(path)++ monkeypatch.setattr(dashboard_app, "load_dashboard_path", failing)+ app = _app([unreadable, valid])+ assert not app.exception+ assert app.dataframe[0].value["model"].tolist() == ["model-b", "model-b"]+ assert any(text.value.startswith(f"{unreadable.resolve()}: cannot load dashboard input:") for text in app.text)+++def test_oversized_trial_count_keeps_other_reports(tmp_path: Path) -> None:+ valid = _write_report(tmp_path / "valid.json")+ oversized = _write_report(tmp_path / "oversized.json")+ report = json.loads(oversized.read_text(encoding="utf-8"))+ agent = report["agents"]["opencode"]+ agent["num_trials"] = 10**20+ for trial in agent["trials"]:+ trial.pop("entry_id")+ oversized.write_text(json.dumps(report), encoding="utf-8")+ app = _app([valid, oversized])+ assert not app.exception+ assert len(app.dataframe[0].value) == 4+ assert any("invalid trial coverage counts" in text.value for text in app.text)+++def test_report_notices_are_plain_text(tmp_path: Path) -> None:+ path = _write_report(tmp_path / "beacon.json")+ report = json.loads(path.read_text(encoding="utf-8"))+ agent = report["agents"].pop("opencode")+ agent["num_trials"] = -1+ report["agents"][""] = agent+ path.write_text(json.dumps(report), encoding="utf-8")+ app = _app([path])+ assert not app.exception+ assert not any("](" in warning.value for warning in app.warning)+ assert any("beacon.example.test" in text.value for text in app.text)+++def test_notice_count_matches_unique_notices(tmp_path: Path) -> None:+ report = _write_report(tmp_path / "run.json")+ payload = json.loads(report.read_text(encoding="utf-8"))+ payload["agents"]["opencode"]["num_trials"] = -1+ report.write_text(json.dumps(payload), encoding="utf-8")+ # The directory walk and the direct file path report the same notice for one file.+ app = _app([tmp_path, report])+ assert not app.exception+ assert app.expander[0].label == "Report notices (2)"+ assert len(app.text) == 2+ assert app.text[1].value == "Overlapping inputs: 2 duplicate condition rows loaded once."+++def test_directory_and_file_inside_it_load_once(tmp_path: Path) -> None:+ report = _write_report(tmp_path / "run.json")+ app = _app([tmp_path, report])+ assert not app.exception+ assert len(app.dataframe[0].value) == 2+ comparison = app.dataframe[1].value+ assert comparison["comparison_status"].tolist() == ["comparable"]+ assert comparison["score_delta"].iloc[0] == pytest.approx(0.3)+ assert any("duplicate" in text.value for text in app.text)+++def test_rewritten_file_inside_directory_input_is_reloaded(tmp_path: Path) -> None:+ report = _write_report(tmp_path / "run.json")+ # The directory comes first, and its own fingerprint misses an in-place rewrite of the file.+ app = _app([tmp_path, report])+ _rewrite_score(report, 0.5)+ app.run()+ assert not app.exception+ assert app.dataframe[0].value["score"].tolist() == [0.5, 0.6]+++def test_results_dir_and_latest_symlink_load_once(tmp_path: Path) -> None:+ results = tmp_path / "results"+ run = results / "run-1"+ run.mkdir(parents=True)+ _write_report(run / "report.json")+ try:+ (results / "latest").symlink_to(run, target_is_directory=True)+ except OSError:+ pytest.skip("Symlinks are unavailable on this host")+ app = _app([results, results / "latest"])+ assert not app.exception+ assert len(app.dataframe[0].value) == 2+ comparison = app.dataframe[1].value+ assert comparison["comparison_status"].tolist() == ["comparable"]+ assert comparison["score_delta"].iloc[0] == pytest.approx(0.3)+ assert any("duplicate" in text.value for text in app.text)++def test_status_filter_does_not_pair_arms_from_separate_embedded_reports(tmp_path: Path) -> None:path = _write_report(tmp_path / "combined.json", baseline_status="failed")- first = json.loads(path.read_text())- second = json.loads(path.read_text())+ first = json.loads(path.read_text(encoding="utf-8"))+ second = json.loads(path.read_text(encoding="utf-8"))second["agents"]["opencode"]["conditions"]["with_skill"]["execution_status"] = "failed"second["agents"]["opencode"]["conditions"]["without_skill"]["execution_status"] = "succeeded"- path.write_text(json.dumps([first, second]))+ path.write_text(json.dumps([first, second]), encoding="utf-8")app = _app([path])app.multiselect(key="filter_status").set_value(["succeeded"]).run()assert not app.exceptioncomparisons = app.dataframe[1].valueassert len(comparisons) == 2- assert comparisons["comparison_status"].tolist() == ["missing condition", "missing condition"]+ assert comparisons["comparison_status"].tolist() == ["incomplete or mismatched coverage"] * 2+ assert comparisons["comparison_reason"].tolist() == [+ "without-skill arm status failed",+ "with-skill arm status failed",+ ]assert comparisons["score_delta"].isna().all()@@ -185,8 +387,122 @@ def test_cyclic_input_path_does_not_hide_valid_report(tmp_path: Path) -> None:pytest.skip("Symlinks are unavailable on this host")app = _app([loop, valid])assert not app.exception- assert len(app.warning) == 1- assert str(loop) in app.warning[0].value+ assert len(app.text) == 1+ assert str(loop) in app.text[0].value+ assert len(app.dataframe[0].value) == 2+++def test_filter_reruns_do_not_reload_inputs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[Path] = []+ real = dashboard_app.load_dashboard_path++ def counting(path: Path):+ calls.append(path)+ return real(path)++ monkeypatch.setattr(dashboard_app, "load_dashboard_path", counting)+ path = _write_report(tmp_path / "run.json")+ app = _app([path])+ for value in (["model-a"], [], ["model-a"]):+ app.multiselect(key="filter_model").set_value(value).run()+ assert not app.exception+ assert calls == [path.resolve()]+++def test_rewritten_report_file_is_reloaded_on_next_rerun(tmp_path: Path) -> None:+ path = _write_report(tmp_path / "run.json")+ app = _app([path])+ assert app.dataframe[0].value["score"].tolist() == [0.9, 0.6]+ _rewrite_score(path, 0.5)+ app.run()+ assert not app.exception+ assert app.dataframe[0].value["score"].tolist() == [0.5, 0.6]+++def test_reload_button_rereads_nested_directory_edits(tmp_path: Path) -> None:+ nested = tmp_path / "runs" / "a"+ nested.mkdir(parents=True)+ path = _write_report(nested / "run.json")+ app = _app([tmp_path / "runs"])+ assert app.dataframe[0].value["score"].tolist() == [0.9, 0.6]+ _rewrite_score(path, 0.5)+ app.button(key="reload_reports").click().run()+ assert not app.exception+ assert app.dataframe[0].value["score"].tolist() == [0.5, 0.6]+++def test_uploads_are_parsed_once_per_uploaded_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ real = dashboard_app.parse_dashboard_report++ def counting(payload, source):+ calls.append(source)+ return real(payload, source)++ monkeypatch.setattr(dashboard_app, "parse_dashboard_report", counting)+ content = _write_report(tmp_path / "upload.json").read_bytes()+ app = _upload_app()+ app.file_uploader(key="report_uploads").set_value([("upload.json", content, "application/json")]).run()+ assert len(app.dataframe[0].value) == 2+ app.multiselect(key="filter_condition").set_value(["with_skill"]).run()+ assert not app.exception+ assert calls == ["upload:1:upload.json"]+++def test_uploaded_report_renders_with_upload_source(tmp_path: Path) -> None:+ content = _write_report(tmp_path / "run.json").read_bytes()+ app = _upload_app()+ app.file_uploader(key="report_uploads").set_value(("run.json", content, "application/json")).run()+ assert not app.exception+ performance = app.dataframe[0].value+ assert performance["source"].tolist() == ["upload:1:run.json", "upload:1:run.json"]+ assert performance["total_tokens"].tolist() == [140.0, 120.0]+++def test_upload_batch_reports_oversize_and_invalid_and_keeps_valid(tmp_path: Path) -> None:+ content = _write_report(tmp_path / "run.json").read_bytes()+ oversized = content + b" " * (16_000_001 - len(content))+ app = _upload_app()+ app.file_uploader(key="report_uploads").set_value(+ [+ ("big.json", oversized, "application/json"),+ ("broken.json", b"{", "application/json"),+ ("deep.json", b"[" * 100_000, "application/json"),+ ("run.json", content, "application/json"),+ ]+ ).run()+ assert not app.exception+ notices = [text.value for text in app.text]+ assert any(notice.startswith("big.json: report exceeds 16 MB") for notice in notices), notices+ assert sum("invalid JSON" in notice for notice in notices) == 2, notices+ assert len(app.dataframe[0].value) == 2+++def test_identical_uploads_are_loaded_once(tmp_path: Path) -> None:+ blob = _write_report(tmp_path / "run.json").read_bytes()+ app = _upload_app()+ app.file_uploader(key="report_uploads").set_value([("run.json", blob, "application/json")] * 2).run()+ assert not app.exception+ assert len(app.dataframe[0].value) == 2+ assert [metric.value for metric in app.metric] == ["1", "1", "2", "2"]+ assert any("identical to an earlier upload; skipped." in text.value for text in app.text)++ other = _write_report(tmp_path / "other.json", model="model-b").read_bytes()+ app.file_uploader(key="report_uploads").set_value(+ [("run.json", blob, "application/json"), ("run.json", other, "application/json")]+ ).run()+ assert not app.exception+ assert len(app.dataframe[0].value) == 4+ assert not any("identical to an earlier upload" in text.value for text in app.text)+++def test_upload_at_exact_limit_is_accepted(tmp_path: Path) -> None:+ content = _write_report(tmp_path / "run.json").read_bytes()+ exact = content + b" " * (16_000_000 - len(content))+ assert len(exact) == 16_000_000+ app = _upload_app()+ app.file_uploader(key="report_uploads").set_value(("exact.json", exact, "application/json")).run()+ assert not app.exceptionassert len(app.dataframe[0].value) == 2@@ -210,6 +526,7 @@ def test_comparison_views_keep_measurements_numeric(tmp_path: Path, view: str, mconfig = json.loads(comparison.proto.columns)assert config[measured_column]["type_config"]["type"] == "number"assert config["comparison_status"]["type_config"]["type"] == "text"+ assert config["comparison_reason"]["type_config"]["type"] == "text"assert comparison.value["comparison_status"].iloc[0] == "comparable"assert len(comparison.value.columns) == len(set(comparison.value.columns))if view == "All metrics":@@ -232,3 +549,13 @@ def test_csv_preserves_numeric_deltas_and_escapes_formula_strings() -> None:assert cells[1] == {"skill": "'\tprivate-name", "score_delta": "0.0", "cost_usd": "0.0"}assert cells[2]["skill"] == "'@SUM(A1)"assert "must not be exported" not in exported+++ "value",+ ["\r=1", "\n=1", "\xa0=1", "\u3000=1", "\uff1d1", "\uff0b1", "\uff0d1", "\uff20A1"],+)+def test_csv_escapes_formula_prefixes_after_whitespace_and_width_folding(value: str) -> None:+ exported = rows_csv([{"skill": value, "score_delta": -0.25, "cost_usd": 7}], ("skill", "score_delta", "cost_usd"))+ cells = list(csv.DictReader(io.StringIO(exported)))+ assert cells == [{"skill": "'" + value, "score_delta": "-0.25", "cost_usd": "7"}]diff --git a/tests/reporting/test_dashboard_cli.py b/tests/reporting/test_dashboard_cli.pyindex 45c21b45..174d57b9 100644--- a/tests/reporting/test_dashboard_cli.py+++ b/tests/reporting/test_dashboard_cli.py@@ -5,49 +5,240 @@from __future__ import annotations+import importlib.machinery+import importlib.util+import os+import signal+import subprocessimport sysfrom pathlib import Path+import pytestfrom click.testing import CliRunnerfrom skillevaluator.cli import clifrom skillevaluator.reporting import dashboard+class FakePopen:+ """Stand in for the Streamlit child: record the launch, waits, and forwarded signals."""++ def __init__(self, command: list[str], **kwargs) -> None:+ self.command = command+ self.kwargs = kwargs+ self.sent: list[int] = []+ self.wait_timeouts: list[float | None] = []++ def wait(self, timeout: float | None = None) -> int:+ self.wait_timeouts.append(timeout)+ return self.exit_code()++ def exit_code(self) -> int:+ return 7++ def send_signal(self, signum: int) -> None:+ self.sent.append(signum)++ def terminate(self) -> None:+ self.send_signal(signal.SIGTERM)+++def _fake_streamlit_spec(monkeypatch, spec: importlib.machinery.ModuleSpec | None) -> None:+ """Fake only Streamlit's lookup; coverage and pytest call find_spec for other modules."""+ find_spec = importlib.util.find_spec+ monkeypatch.setattr(+ dashboard.importlib.util,+ "find_spec",+ lambda name, *args, **kwargs: spec if name == "streamlit" else find_spec(name, *args, **kwargs),+ )+++def _capture_launch(monkeypatch, *args: str, popen: type[FakePopen] = FakePopen):+ """Invoke the dashboard command with Streamlit present and the child process faked."""+ launched: list[FakePopen] = []+ _fake_streamlit_spec(monkeypatch, importlib.machinery.ModuleSpec("streamlit", None))++ def start(command, **kwargs):+ launched.append(popen(command, **kwargs))+ return launched[-1]++ monkeypatch.setattr(dashboard.subprocess, "Popen", start)+ result = CliRunner().invoke(cli, ["dashboard", *args])+ return result, launched[0]++def test_dashboard_help_and_input_validation() -> None:runner = CliRunner()help_result = runner.invoke(cli, ["dashboard", "--help"])assert help_result.exit_code == 0assert "retained Tier 3" in help_result.output+ assert "next free port" in " ".join(help_result.output.split())assert runner.invoke(cli, ["dashboard", "--port", "0"]).exit_code == 2assert runner.invoke(cli, ["dashboard", "missing-report-170.json"]).exit_code == 2def test_missing_extra_has_actionable_error(monkeypatch) -> None:- monkeypatch.setattr(dashboard.importlib.util, "find_spec", lambda _name: None)+ _fake_streamlit_spec(monkeypatch, None)result = CliRunner().invoke(cli, ["dashboard"])assert result.exit_code == 1- assert 'pip install "skillevaluator[dashboard]"' in result.output+ assert "skillevaluator[all,dashboard]" in result.output+ assert "uv run --extra dashboard" in result.output+ assert 'pip install "skillevaluator[dashboard]"' not in result.outputdef test_launch_uses_loopback_and_literal_absolute_paths(tmp_path: Path, monkeypatch) -> None:report = tmp_path / "report with spaces; literal.json"report.write_text("{}", encoding="utf-8")- commands = []- monkeypatch.setattr(dashboard.importlib.util, "find_spec", lambda _name: object())-- def capture(command):- commands.append(command)- return 7-- monkeypatch.setattr(dashboard.subprocess, "call", capture)- result = CliRunner().invoke(cli, ["dashboard", str(report), "--port", "8517", "--no-browser"])+ result, fake = _capture_launch(monkeypatch, str(report), "--port", "8517", "--no-browser")assert result.exit_code == 7- command = commands[0]- assert command[:3] == [sys.executable, "-m", "streamlit"]+ command, kwargs = fake.command, fake.kwargs+ assert command[:4] == [sys.executable, "-P", "-m", "streamlit"]+ app = Path(command[command.index("run") + 1])+ assert app.name == "skillevaluator_dashboard.py"+ assert app.parent.name == "dashboard_ui"+ assert Path(kwargs["cwd"]) == app.parent+ assert kwargs["env"]["STREAMLIT_SERVER_ALLOWED_HOSTS"] == "127.0.0.1 localhost"assert "--server.address=127.0.0.1" in commandassert "--server.port=8517" in commandassert "--server.maxUploadSize=16" in command- assert "--server.headless=true" in command+ for flag in (+ "--server.showEmailPrompt=false",+ "--server.fileWatcherType=none",+ "--server.enableCORS=true",+ "--server.enableXsrfProtection=true",+ "--client.toolbarMode=viewer",+ "--server.headless=true",+ ):+ assert flag in commandassert "--browser.gatherUsageStats=false" in commandassert command[-2:] == ["--", str(report.resolve())]+++def test_launch_settings_exist_in_installed_streamlit(monkeypatch) -> None:+ # Streamlit silently ignores settings it does not define; server.allowedHosts needs 1.61.+ config = pytest.importorskip("streamlit.config")+ # Filled on import; config._config_options stays None until Streamlit loads its config.+ options = config._config_options_template+ monkeypatch.delenv("STREAMLIT_SERVER_ALLOWED_HOSTS", raising=False)+ _result, fake = _capture_launch(monkeypatch, "--port", "8517", "--no-browser")+ command, env = fake.command, fake.kwargs["env"]+ flags = command[command.index("run") + 2 : command.index("--")]+ assert flags+ for flag in flags:+ name, separator, _value = flag.removeprefix("--").partition("=")+ assert flag.startswith("--") and separator, flag+ assert name in options and not options[name].deprecated, flag+ env_options = {option.env_var: name for name, option in options.items()}+ added = {name for name, value in env.items() if os.environ.get(name) != value}+ assert {env_options.get(name) for name in added} == {"server.allowedHosts"}+++def test_default_launch_lets_streamlit_choose_port(monkeypatch) -> None:+ # Streamlit moves to the next free port only when the port was not set explicitly.+ _result, fake = _capture_launch(monkeypatch)+ assert not any(arg.startswith("--server.port") for arg in fake.command)+++def test_explicit_port_failure_has_hint(monkeypatch) -> None:+ class BusyPortPopen(FakePopen):+ def exit_code(self) -> int:+ return 1++ result, _fake = _capture_launch(monkeypatch, "--port", "8517", popen=BusyPortPopen)+ assert result.exit_code == 1+ assert "choose another with --port" in result.output+++def test_default_launch_does_not_force_headless(monkeypatch) -> None:+ # An explicit headless=false makes Streamlit prompt for an email and exit without a TTY.+ _result, fake = _capture_launch(monkeypatch)+ assert not any(arg.startswith("--server.headless") for arg in fake.command)+++def test_sigterm_is_forwarded_to_streamlit(monkeypatch) -> None:+ original = signal.getsignal(signal.SIGTERM)++ class TerminatedPopen(FakePopen):+ def exit_code(self) -> int:+ # A process manager sends SIGTERM to the launcher alone.+ signal.getsignal(signal.SIGTERM)(signal.SIGTERM, None)+ return -15++ result, fake = _capture_launch(monkeypatch, popen=TerminatedPopen)+ assert fake.sent == [signal.SIGTERM]+ assert signal.getsignal(signal.SIGTERM) is original+ assert result.exit_code == 128 + signal.SIGTERM++[email protected](not hasattr(signal, "SIGHUP"), reason="SIGHUP is POSIX-only")+def test_streamlit_killed_by_signal_exits_like_a_shell(monkeypatch) -> None:+ class HungUpPopen(FakePopen):+ def exit_code(self) -> int:+ # Streamlit has no SIGHUP handler, so the forwarded signal kills it.+ signal.getsignal(signal.SIGHUP)(signal.SIGHUP, None)+ return -signal.SIGHUP++ result, fake = _capture_launch(monkeypatch, "--port", "8517", popen=HungUpPopen)+ assert fake.sent == [signal.SIGHUP]+ assert result.exit_code == 128 + signal.SIGHUP+ # The busy-port hint is only for Streamlit's own failures.+ assert "choose another with --port" not in result.output+++def test_ctrl_c_returns_zero(monkeypatch) -> None:+ class InterruptedPopen(FakePopen):+ def exit_code(self) -> int:+ if len(self.wait_timeouts) == 1:+ raise KeyboardInterrupt+ return 0++ result, fake = _capture_launch(monkeypatch, popen=InterruptedPopen)+ assert result.exit_code == 0+ # The launcher waits for Streamlit to finish its own Ctrl+C shutdown, and+ # passes SIGINT on in case it was sent to the launcher alone.+ assert fake.wait_timeouts == [None, 10]+ assert fake.sent == ([] if sys.platform == "win32" else [signal.SIGINT])+++def test_ctrl_c_terminates_streamlit_that_does_not_stop(monkeypatch) -> None:+ class StuckPopen(FakePopen):+ def exit_code(self) -> int:+ if len(self.wait_timeouts) == 1:+ raise KeyboardInterrupt+ if self.wait_timeouts[-1] is not None:+ raise subprocess.TimeoutExpired(self.command, self.wait_timeouts[-1])+ return -signal.SIGTERM++ result, fake = _capture_launch(monkeypatch, popen=StuckPopen)+ assert result.exit_code == 0+ assert fake.wait_timeouts == [None, 10, None]+ assert fake.sent == ([] if sys.platform == "win32" else [signal.SIGINT]) + [signal.SIGTERM]+++def test_streamlit_script_directory_cannot_shadow_modules() -> None:+ # Streamlit puts the script directory first on sys.path in the server.+ app = dashboard._APP+ assert app.is_file()+ stems = {path.stem for path in app.parent.glob("*.py")}+ assert stems <= {"__init__", app.stem}+ assert not stems & set(sys.stdlib_module_names)+ assert importlib.util.find_spec(app.stem) is None+++def test_streamlit_is_not_shadowed_by_cwd(tmp_path: Path, monkeypatch) -> None:+ pytest.importorskip("streamlit")+ _result, fake = _capture_launch(monkeypatch)+ command = fake.command+ # Restore the real subprocess.Popen before starting Python below.+ monkeypatch.undo()+ (tmp_path / "streamlit.py").write_text("raise SystemExit(3)\n", encoding="utf-8")+ completed = subprocess.run(+ [*command[:4], "version"],+ cwd=tmp_path,+ capture_output=True,+ text=True,+ encoding="utf-8",+ timeout=60,+ )+ assert completed.returncode == 0, completed.stderr+ assert "Streamlit" in completed.stdoutdiff --git a/tests/reporting/test_dashboard_contract.py b/tests/reporting/test_dashboard_contract.pynew file mode 100644index 00000000..58dd9abf--- /dev/null+++ b/tests/reporting/test_dashboard_contract.py@@ -0,0 +1,86 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Contract test: dashboard measurements must agree with the canonical Tier 3 ``resources`` block.++Builds the canonical payload with the producer's own loader/builder (agent_eval_result_from_directory),+so the test fails as soon as the producer and the dashboard disagree about field names or pairing rules.+"""++from __future__ import annotations++import json+from pathlib import Path++import pytest++from skillevaluator.evaluation.tier3_report import agent_eval_result_from_directory+from skillevaluator.reporting.dashboard_data import comparison_rows, load_dashboard_path, parse_dashboard_report+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS+++def _write(path: Path, value: object) -> None:+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(json.dumps(value), encoding="utf-8")+++def _retained_run(run: Path, *, attempts: int) -> Path:+ scores = dict.fromkeys(DEFAULT_METRICS, 1.0)+ _write(run / "run_config.json", {"agents": {"opencode": {"agent": "opencode", "model": "model-a"}}})+ _write(run / "result.json", {"skill_name": "demo", "run_id": run.name})+ for arm, prompt, seconds in (("with-skill", 20, 3), ("without-skill", 10, 5)):+ condition = run / "opencode" / arm+ count = 2 * attempts+ _write(+ condition / "summary.json",+ {+ "model": "model-a",+ "scores": scores,+ "metrics": list(DEFAULT_METRICS),+ "num_trials": count,+ "num_reward_rows": count,+ "execution_status": "succeeded",+ "expected_attempts": count,+ "scored_attempts": count,+ "pass_at_k": {"k": attempts, "rate": 1.0},+ },+ )+ for case in ("case-a", "case-b"):+ for attempt in range(attempts):+ # Harbor names repeated attempts <task>__<random>; no attempt ordinal is recorded.+ trial = condition / "trials" / f"{case}__trial{attempt}"+ _write(trial / "reward.json", {**scores, "entry_id": case, "trial_id": trial.name})+ _write(+ trial / "trajectory.json",+ {"steps": [{}], "final_metrics": {"total_prompt_tokens": prompt, "total_completion_tokens": 2}},+ )+ _write(+ trial / "result.json",+ {+ "agent_execution": {+ "started_at": "2026-10-11T00:00:00Z",+ "finished_at": f"2026-10-11T00:00:0{seconds}Z",+ }+ },+ )+ return run++[email protected]("attempts", [1, 3])[email protected]("source", ["canonical", "native"])+def test_dashboard_matches_main_resources(tmp_path: Path, attempts: int, source: str) -> None:+ run = _retained_run(tmp_path / "run", attempts=attempts)+ canonical = agent_eval_result_from_directory(tmp_path, run, use_llm_judge=False).metadata["agent_eval"]+ resources = canonical["agents"]["opencode"]["resources"]+ assert resources["pairing"] == "matching case counts"++ data = load_dashboard_path(run) if source == "native" else parse_dashboard_report(canonical, "report.json")+ by_condition = {row["condition"]: row for row in data.rows}+ for condition, arm in (("with_skill", "with_skill"), ("without_skill", "baseline")):+ assert by_condition[condition]["score"] is not None+ assert by_condition[condition]["total_tokens"] == resources["tokens"][arm]["total"]+ assert by_condition[condition]["duration_seconds"] == resources["agent_seconds"][arm]["total"]+ comparison = comparison_rows(data.rows)[0]+ assert comparison["comparison_status"] == "comparable"+ assert comparison["total_tokens_delta"] == resources["tokens"]["delta"]+ assert comparison["duration_seconds_delta"] == resources["agent_seconds"]["delta"]diff --git a/tests/reporting/test_dashboard_data.py b/tests/reporting/test_dashboard_data.pyindex 7ba2b809..acab15e1 100644--- a/tests/reporting/test_dashboard_data.py+++ b/tests/reporting/test_dashboard_data.py@@ -5,12 +5,19 @@ from __future__ import annotationsimport copyimport json+import osimport shutilfrom pathlib import Pathimport pytest-from skillevaluator.reporting.dashboard_data import comparison_rows, load_dashboard_path, parse_dashboard_report+from skillevaluator.evaluation.tier3_report import agent_eval_result_from_directory+from skillevaluator.reporting.dashboard_data import (+ comparison_rows,+ dedupe_rows,+ load_dashboard_path,+ parse_dashboard_report,+)def _report() -> dict:@@ -20,7 +27,9 @@ def _report() -> dict:"trial_id": "case-a__random","overall": 0.8,"tokens": {"prompt": prompt, "completion": completion, "cached": 0},- "duration_seconds": seconds,+ "agent_seconds": seconds,+ # Tier 3 canonical JSON does not produce cost_usd; it is kept to+ # exercise the reader for hand-imported payloads."cost_usd": cost,}@@ -59,6 +68,7 @@ def test_with_without_differences_and_explicit_features() -> None:assert data.rows[0]["features"] == "memory, rules"comparison = comparison_rows(data.rows)[0]assert comparison["comparison_status"] == "comparable"+ assert comparison["comparison_reason"] == ""assert comparison["score_delta"] == pytest.approx(0.2)assert comparison["total_tokens_delta"] == 30assert comparison["total_tokens_percent"] == 25@@ -66,6 +76,10 @@ def test_with_without_differences_and_explicit_features() -> None:assert comparison["cost_usd_percent"] == 100+def test_single_trial_coverage_label_is_singular() -> None:+ assert parse_dashboard_report(_report(), "r").rows[0]["coverage"].startswith("1 trial (complete)")++def test_authoritative_validator_results_exclude_top_level_mirror() -> None:first, second = _report(), _report()second["skill_name"] = "second-skill"@@ -93,6 +107,32 @@ def test_partial_missing_metrics_never_become_zero_or_partial_total() -> None:assert "cost 0/1" in data.rows[0]["coverage"]+def test_pre_v050_zero_token_counters_are_unavailable() -> None:+ report = _report()+ report["evaluator_version"] = "0.4.0"+ report["agents"]["claude-code"]["trials_baseline"][0]["tokens"] = {"prompt": 0, "completion": 0, "cached": 0}+ data = parse_dashboard_report(report, "legacy.json")+ with_row, without_row = data.rows+ assert without_row["prompt_tokens"] is None+ assert without_row["total_tokens"] is None+ assert with_row["total_tokens"] == 150+ assert with_row["cached_tokens"] is None+ comparison = comparison_rows(data.rows)[0]+ assert comparison["comparison_status"] == "comparable"+ assert comparison["score_delta"] == pytest.approx(0.2)+ assert comparison["total_tokens_delta"] is None+ assert any("before v0.5.0" in warning for warning in data.warnings)+ report.pop("evaluator_version")+ report["summary"] = {"evaluator_version": "0.4.1"}+ assert parse_dashboard_report(report, "legacy.json").rows[1]["total_tokens"] is None+ # Current, missing, or unparsable versions keep recorded zeros.+ for version in ("0.5.0", None, "not-a-version", "9" * 5000):+ report["summary"]["evaluator_version"] = version+ data = parse_dashboard_report(report, "current.json")+ assert data.rows[1]["total_tokens"] == 0+ assert not data.warnings++def test_canonical_missing_attempt_keeps_usage_totals_unknown() -> None:report = _report()agent = report["agents"]["claude-code"]@@ -103,7 +143,59 @@ def test_canonical_missing_attempt_keeps_usage_totals_unknown() -> None:assert "tokens 1/2" in row["coverage"][email protected]("value", [True, -1, 1.5, "1"])+def test_canonical_missing_trial_counts_each_declared_trial_once() -> None:+ report = _report()+ agent = report["agents"]["claude-code"]+ agent["num_trials"] = 10+ agent["conditions"]["with_skill"].update(expected_attempts=10, scored_attempts=10)+ agent["trials"] = [+ {**copy.deepcopy(agent["trials"][0]), "entry_id": f"case-{index}", "trial_id": f"case-{index}__trial0"}+ for index in range(9)+ ]+ row = parse_dashboard_report(report, "report.json").rows[0]+ assert row["total_tokens"] is None+ assert "tokens 9/10" in row["coverage"]+ # Without case identities the shortfall still keeps totals unknown.+ for trial in agent["trials"]:+ trial.pop("entry_id")+ row = parse_dashboard_report(report, "report.json").rows[0]+ assert row["total_tokens"] is None+ assert "tokens 9/10" in row["coverage"]+++def test_per_trial_usage_normalizes_trial_counts() -> None:+ per_trial = []+ for count in (2, 4):+ report = _report()+ agent = report["agents"]["claude-code"]+ for field, count_field, arm in (+ ("trials", "num_trials", "with_skill"),+ ("trials_baseline", "num_trials_baseline", "without_skill"),+ ):+ agent[field] = [+ {**copy.deepcopy(agent[field][0]), "entry_id": f"case-{index}", "trial_id": f"case-{index}__trial0"}+ for index in range(count)+ ]+ agent[count_field] = count+ agent["conditions"][arm].update(expected_attempts=count, scored_attempts=count)+ rows = parse_dashboard_report(report, f"trials-{count}.json").rows+ # Totals grow with the trial count; per-trial means do not.+ assert [row["total_tokens"] for row in rows] == [150 * count, 120 * count]+ per_trial.append([(row["total_tokens_per_trial"], row["duration_seconds_per_trial"]) for row in rows])+ assert per_trial == [[(150.0, 8.0), (120.0, 10.0)]] * 2+++def test_per_trial_usage_is_blank_when_any_trial_is_unmeasured() -> None:+ report = _repeated_attempts_report(("case-a__trial0", "case-a__trial1"))+ report["agents"]["claude-code"]["trials"][1]["tokens"].pop("completion")+ with_row, without_row = parse_dashboard_report(report, "report.json").rows+ assert with_row["total_tokens"] is None+ assert with_row["total_tokens_per_trial"] is None+ assert with_row["duration_seconds_per_trial"] == 8.0+ assert without_row["total_tokens_per_trial"] == 120.0++[email protected]("value", [True, -1, 1.5, "1", 10**30])@pytest.mark.parametrize("field", ["declared", "expected", "both_attempt_counts"])def test_present_invalid_coverage_counts_never_become_inferred_complete_counts(value: object, field: str) -> None:report = _report()@@ -135,8 +227,18 @@ def test_unrecorded_legacy_coverage_counts_preserve_observed_trial_pairing() ->assert comparison_rows(parse_dashboard_report(report, "legacy.json").rows)[0]["comparison_status"] == "comparable"[email protected]("mutation", ["failed", "missing_trial", "truncated", "different_case", "missing_attempt"])-def test_incomplete_or_unmatched_arms_block_all_deltas(mutation: str) -> None:+ ("mutation", "reason"),+ [+ ("failed", "without-skill arm status failed"),+ ("missing_trial", "without-skill trial coverage incomplete"),+ ("truncated", "trial data truncated"),+ ("different_case", "different cases or attempts"),+ ("missing_attempt", "without-skill trial coverage incomplete"),+ ("invalid_count", "invalid trial counts"),+ ],+)+def test_incomplete_or_unmatched_arms_block_all_deltas(mutation: str, reason: str) -> None:report = _report()agent = report["agents"]["claude-code"]if mutation == "failed":@@ -149,25 +251,58 @@ def test_incomplete_or_unmatched_arms_block_all_deltas(mutation: str) -> None:agent["trials_baseline"][0]["entry_id"] = "case-b"elif mutation == "missing_attempt":agent["conditions"]["without_skill"]["expected_attempts"] = 2+ elif mutation == "invalid_count":+ agent["num_trials_baseline"] = -1comparison = comparison_rows(parse_dashboard_report(report, "same.json").rows)[0]assert comparison["comparison_status"] == "incomplete or mismatched coverage"+ assert comparison["comparison_reason"] == reasonassert comparison["score_delta"] is Noneassert comparison["total_tokens_delta"] is None+ ("field", "reason"),+ [("_dataset_digest", "different dataset"), ("_attempt_policy", "different attempt policy")],+)+def test_tautological_pair_checks_have_reasons(field: str, reason: str) -> None:+ rows = parse_dashboard_report(_report(), "same.json").rows+ rows[1][field] = "other"+ comparison = comparison_rows(rows)[0]+ assert comparison["comparison_status"] == "incomplete or mismatched coverage"+ assert comparison["comparison_reason"] == reason+ assert comparison["score_delta"] is None++def test_disjoint_runs_features_and_duplicate_conditions_cannot_pair() -> None:data = parse_dashboard_report(_report(), "first.json")rows = copy.deepcopy(data.rows)rows[1]["source"] = "second.json"- assert all(row["comparison_status"] == "missing condition" for row in comparison_rows(rows))+ comparisons = comparison_rows(rows)+ assert all(row["comparison_status"] == "missing condition" for row in comparisons)+ assert [row["comparison_reason"] for row in comparisons] == [+ "no without-skill arm for this run and configuration",+ "no with-skill arm for this run and configuration",+ ]rows = copy.deepcopy(data.rows)rows[1]["features"] = "rules"- assert all(row["comparison_status"] == "missing condition" for row in comparison_rows(rows))+ comparisons = comparison_rows(rows)+ assert all(row["comparison_status"] == "missing condition" for row in comparisons)+ assert all(row["comparison_reason"].startswith("no ") for row in comparisons)duplicate = comparison_rows([*data.rows, data.rows[0]])[0]assert duplicate["comparison_status"] == "ambiguous duplicate conditions"+ assert duplicate["comparison_reason"] == "same arm loaded more than once"assert duplicate["score_delta"] is None+def test_dedupe_rows_drops_only_reloaded_rows() -> None:+ rows = parse_dashboard_report(_report(), "x").rows+ deduped, dropped = dedupe_rows(rows + copy.deepcopy(rows))+ assert (deduped, dropped) == (rows, 2)+ # Identical embedded reports in one file are separate occurrences, not reloads.+ assert dedupe_rows(parse_dashboard_report([_report(), _report()], "x").rows)[1] == 0+ assert comparison_rows(deduped)[0]["comparison_status"] == "comparable"++@pytest.mark.parametrize("scope", ["reports", "agents"])def test_status_filter_cannot_pair_different_report_or_agent_occurrences(scope: str) -> None:first, second = _report(), _report()@@ -210,20 +345,25 @@ def test_zero_attempt_identity_and_conflicting_attempt_fields() -> None:assert comparison["score_delta"] is None-def test_long_numeric_attempt_suffix_is_normalized_without_integer_conversion() -> None:+def test_long_numeric_attempt_field_is_normalized_without_integer_conversion() -> None:report = _report()agent = report["agents"]["claude-code"]- for field in ("trials", "trials_baseline"):- agent[field][0]["trial_id"] = "case-a__attempt" + "9" * 5000+ # Longer than int()'s default 4300-digit limit for decimal strings.+ agent["trials"][0]["attempt"] = "0" * 10 + "9" * 5000+ agent["trials_baseline"][0]["attempt"] = "9" * 5000comparison = comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]assert comparison["comparison_status"] == "comparable"+ agent["trials_baseline"][0]["attempt"] = "9" * 4999 + "8"+ comparison = comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]+ assert comparison["comparison_status"] == "incomplete or mismatched coverage"+ assert comparison["comparison_reason"] == "different cases or attempts"def test_zero_baseline_and_nonfinite_values() -> None:report = _report()agent = report["agents"]["claude-code"]agent["trials_baseline"][0]["cost_usd"] = 0- agent["trials"][0]["duration_seconds"] = float("inf")+ agent["trials"][0]["agent_seconds"] = float("inf")agent["trials"][0]["tokens"]["cached"] = Truereport["dashboard_metadata"] = {"harness_features": {"invalid": float("nan")}}data = parse_dashboard_report(report, "report.json")@@ -235,6 +375,24 @@ def test_zero_baseline_and_nonfinite_values() -> None:assert comparison["duration_seconds_delta"] is None+def test_invalid_agent_seconds_does_not_fall_back_to_legacy_duration() -> None:+ report = _report()+ trial = report["agents"]["claude-code"]["trials"][0]+ trial.update(agent_seconds=-1, duration_seconds=5)+ row = parse_dashboard_report(report, "report.json").rows[0]+ assert row["duration_seconds"] is None+ assert "time 0/1" in row["coverage"]+++def test_legacy_trial_duration_seconds_is_still_read() -> None:+ report = _report()+ agent = report["agents"]["claude-code"]+ for field in ("trials", "trials_baseline"):+ agent[field][0]["duration_seconds"] = agent[field][0].pop("agent_seconds")+ data = parse_dashboard_report(report, "report.json")+ assert [row["duration_seconds"] for row in data.rows] == [8.0, 10.0]++def test_malformed_status_and_aggregate_overflow_fail_closed() -> None:report = _report()agent = report["agents"]["claude-code"]@@ -248,27 +406,72 @@ def test_malformed_status_and_aggregate_overflow_fail_closed() -> None:assert row["total_tokens"] is None-def test_repeated_attempts_require_recorded_identity_and_match_exact_set() -> None:+def _repeated_attempts_report(trial_ids: tuple[str, ...]) -> dict:report = _report()agent = report["agents"]["claude-code"]for field, count_field, arm in (("trials", "num_trials", "with_skill"),("trials_baseline", "num_trials_baseline", "without_skill"),):- agent[field].append(copy.deepcopy(agent[field][0]))- agent[field][0]["trial_id"] = "case-a__attempt1__abc"- agent[field][1]["trial_id"] = "case-a__attempt2__def"- agent[count_field] = 2- agent["conditions"][arm].update(expected_attempts=2, scored_attempts=2)+ agent[field] = [{**copy.deepcopy(agent[field][0]), "trial_id": trial_id} for trial_id in trial_ids]+ agent[count_field] = len(trial_ids)+ agent["conditions"][arm].update(expected_attempts=len(trial_ids), scored_attempts=len(trial_ids))+ return report+++def test_repeated_attempts_pair_by_case_with_distinct_trial_ids() -> None:+ report = _repeated_attempts_report(("case-a__trial0", "case-a__trial1"))+ agent = report["agents"]["claude-code"]+ comparison = comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]+ assert comparison["comparison_status"] == "comparable"+ assert comparison["score_delta"] == pytest.approx(0.2)+ # Harbor's attempt names are random, so arms pair by case count, not by name.+ agent["trials_baseline"][1]["trial_id"] = "case-a__trial2"assert comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["comparison_status"] == "comparable"- agent["trials_baseline"][1]["trial_id"] = "case-a__attempt3__def"- assert comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["score_delta"] is Nonefor field in ("trials", "trials_baseline"):for index, trial in enumerate(agent[field]):trial["trial_id"] = f"case-a__unrelated-{index}"+ assert comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["comparison_status"] == "comparable"++ report = _repeated_attempts_report(("case-a__trial0", "case-a__trial1"))+ report["agents"]["claude-code"]["trials"][1]["trial_id"] = ""+ assert comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["score_delta"] is None++ report = _repeated_attempts_report(("case-a__trial0", "case-a__trial1"))+ report["agents"]["claude-code"]["trials"][1].update(entry_id="case-b", trial_id="case-a__trial0")+ assert comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["score_delta"] is None++ report = _repeated_attempts_report(("case-a__trial0", "case-a__trial1"))+ agent = report["agents"]["claude-code"]+ agent["trials_baseline"].pop()+ agent["num_trials_baseline"] = 1+ agent["conditions"]["without_skill"].update(expected_attempts=1, scored_attempts=1)+ comparison = comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]+ assert comparison["comparison_status"] == "incomplete or mismatched coverage"+ assert comparison["score_delta"] is None++ report = _repeated_attempts_report(("case-a__trial0", "case-a__trial1"))+ agent = report["agents"]["claude-code"]+ for field, attempts in (("trials", (1, 2)), ("trials_baseline", (1, 3))):+ for trial, attempt in zip(agent[field], attempts, strict=True):+ trial["attempt"] = attempt+ comparison = comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]+ assert comparison["comparison_status"] == "incomplete or mismatched coverage"+ assert comparison["score_delta"] is None++ report = _repeated_attempts_report(("case-a__trial0", "case-a__trial1"))+ report["agents"]["claude-code"]["trials"][0]["attempt"] = 1assert comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["score_delta"] is None+def test_authored_case_id_containing_attempt_text_is_not_parsed() -> None:+ report = _repeated_attempts_report(("retry-attempt1__aaa", "retry-attempt1__bbb"))+ for field in ("trials", "trials_baseline"):+ for trial in report["agents"]["claude-code"][field]:+ trial["entry_id"] = "retry-attempt1"+ assert comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["comparison_status"] == "comparable"++def test_multistep_rewards_keep_logical_scores_without_ambiguous_resource_sums() -> None:report = _report()agent = report["agents"]["claude-code"]@@ -333,6 +536,26 @@ def test_diagnostic_truncation_preserves_complete_trial_comparisons(omitted: strassert comparison_rows(data.rows)[0]["comparison_status"] == "comparable"+def test_pruned_non_best_agent_reports_declared_trials_and_reason() -> None:+ report = _report()+ report["best_agent"] = "best"+ report["agents"]["best"] = copy.deepcopy(report["agents"]["claude-code"])+ agent = report["agents"]["claude-code"]+ agent.update(num_trials=5, num_trials_baseline=5, conditions={}, trials=[], trials_baseline=[])+ report["report_truncation"] = {"truncated": True, "omitted": {"non_best_agent_details": 12}}+ data = parse_dashboard_report(report, "report.json")+ rows = [row for row in data.rows if row["_agent_key"] == "claude-code"]+ assert [row["trials"] for row in rows] == [5, 5]+ assert all(row["score"] is None and "tokens 0/5" in row["coverage"] for row in rows)+ comparisons = {row["_agent_key"]: row for row in comparison_rows(data.rows)}+ assert comparisons["best"]["comparison_status"] == "comparable"+ assert comparisons["claude-code"]["comparison_status"] == "incomplete or mismatched coverage"+ assert comparisons["claude-code"]["comparison_reason"] == (+ "trial details pruned from embedded report; load the retained run directory"+ )+ assert comparisons["claude-code"]["score_delta"] is None++def test_artifact_loading_truncation_blocks_comparisons() -> None:report = _report()report["report_truncation"] = {@@ -343,14 +566,71 @@ def test_artifact_loading_truncation_blocks_comparisons() -> None:assert comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["score_delta"] is None-def test_bad_input_is_diagnostic_and_directory_skips_symlinks(tmp_path: Path) -> None:+_TRAJECTORY_BYTES = {"code": "json_bytes", "artifact": "trajectory", "limit": 2 * 1024 * 1024}+++ ("marker", "status"),+ [+ ({"truncated": True, "omitted": {}, "artifact_loading": [_TRAJECTORY_BYTES]}, "comparable"),+ (+ {+ "truncated": True,+ "omitted": {},+ "artifact_loading": [+ _TRAJECTORY_BYTES,+ {"code": "json_bytes", "artifact": "reward", "limit": 2 * 1024 * 1024},+ ],+ },+ "incomplete or mismatched coverage",+ ),+ (+ {"truncated": True, "artifact_loading": [{"artifact": "with-skill", "code": "trial_limit"}]},+ "incomplete or mismatched coverage",+ ),+ (+ {"truncated": True, "artifact_loading": [{"artifact": "mystery", "code": "json_bytes"}]},+ "incomplete or mismatched coverage",+ ),+ (+ {"truncated": True, "artifact_loading": [{"artifact": ["trajectory"], "code": "json_bytes"}]},+ "incomplete or mismatched coverage",+ ),+ ({"truncated": True}, "incomplete or mismatched coverage"),+ ({"truncated": True, "artifact_loading": _TRAJECTORY_BYTES}, "incomplete or mismatched coverage"),+ ],+)+def test_only_trajectory_loader_bounds_keep_trial_coverage(marker: dict, status: str) -> None:+ report = _report()+ report["report_truncation"] = marker+ data = parse_dashboard_report(report, "report.json")+ assert data.warnings+ assert comparison_rows(data.rows)[0]["comparison_status"] == status+++def test_trajectory_reason_does_not_hide_missing_trial() -> None:+ report = _report()+ report["agents"]["claude-code"]["trials_baseline"] = []+ report["report_truncation"] = {"truncated": True, "omitted": {}, "artifact_loading": [_TRAJECTORY_BYTES]}+ status = comparison_rows(parse_dashboard_report(report, "report.json").rows)[0]["comparison_status"]+ assert status == "incomplete or mismatched coverage"+++def test_bad_input_is_diagnostic(tmp_path: Path) -> None:malformed = tmp_path / "broken.json"malformed.write_text("{", encoding="utf-8")assert load_dashboard_path(malformed).warningsassert load_dashboard_path(tmp_path / "missing").warnings+++def test_directory_skips_symlinks(tmp_path: Path) -> None:+ (tmp_path / "broken.json").write_text("{", encoding="utf-8")good = tmp_path / "report.json"good.write_text(json.dumps(_report()), encoding="utf-8")- (tmp_path / "loop").symlink_to(tmp_path, target_is_directory=True)+ try:+ (tmp_path / "loop").symlink_to(tmp_path, target_is_directory=True)+ except OSError:+ pytest.skip("Symlinks are unavailable on this host")data = load_dashboard_path(tmp_path)assert len(data.rows) == 2assert any("cannot read" in warning for warning in data.warnings)@@ -359,7 +639,7 @@ def test_bad_input_is_diagnostic_and_directory_skips_symlinks(tmp_path: Path) ->def test_catalog_index_loads_authoritative_local_reports_with_containment(tmp_path: Path) -> None:reports = tmp_path / "calculator"reports.mkdir()- (reports / "current.json").write_text(json.dumps(_report()))+ (reports / "current.json").write_text(json.dumps(_report()), encoding="utf-8")index = tmp_path / "catalog-summary.json"index.write_text(json.dumps(@@ -369,7 +649,8 @@ def test_catalog_index_loads_authoritative_local_reports_with_containment(tmp_pa{"report_dir": "..", "json_report": "outside.json"},]}- )+ ),+ encoding="utf-8",)data = load_dashboard_path(index)assert len(data.rows) == 2@@ -379,8 +660,12 @@ def test_catalog_index_loads_authoritative_local_reports_with_containment(tmp_padef _native_run(path: Path) -> None:metrics = ["security", "skill_execution", "skill_efficiency", "accuracy", "goal_accuracy", "behavior_check"]path.mkdir()- (path / "result.json").write_text(json.dumps({"skill_name": "native-skill", "run_id": "native-run"}))- (path / "run_config.json").write_text(json.dumps({"agents": {"opencode": {"agent": "opencode", "model": "m"}}}))+ run_config = {"agents": {"opencode": {"agent": "opencode", "model": "m"}}}+ # Like the runner, result.json embeds the config that run_config.json holds.+ (path / "result.json").write_text(+ json.dumps({"skill_name": "native-skill", "run_id": "native-run", "run_config": run_config}), encoding="utf-8"+ )+ (path / "run_config.json").write_text(json.dumps(run_config), encoding="utf-8")for variant, input_tokens in (("with-skill", 200), ("without-skill", 100)):condition = path / "opencode" / varianttrial = condition / "trials" / "case-a__random"@@ -397,7 +682,8 @@ def _native_run(path: Path) -> None:"expected_attempts": 1,"scored_attempts": 1,}- )+ ),+ encoding="utf-8",)(trial / "reward.json").write_text(json.dumps(@@ -407,7 +693,8 @@ def _native_run(path: Path) -> None:"trial_id": "case-a__random","overall": 0.8,}- )+ ),+ encoding="utf-8",)(trial / "trajectory.json").write_text(json.dumps(@@ -415,7 +702,8 @@ def _native_run(path: Path) -> None:"steps": [],"final_metrics": {"total_prompt_tokens": input_tokens, "total_completion_tokens": 10},}- )+ ),+ encoding="utf-8",)(trial / "result.json").write_text(json.dumps(@@ -423,14 +711,15 @@ def _native_run(path: Path) -> None:"agent_result": {"n_input_tokens": input_tokens, "n_output_tokens": 10, "cost_usd": None},"agent_execution": {"started_at": "2026-10-01T00:00:00Z", "finished_at": "2026-10-01T00:00:10Z"},}- )+ ),+ encoding="utf-8",)def test_native_results_use_real_usage_and_prune_nested_mirrors(tmp_path: Path) -> None:run = tmp_path / "run"_native_run(run)- (run / "report.json").write_text(json.dumps(_report()))+ (run / "report.json").write_text(json.dumps(_report()), encoding="utf-8")data = load_dashboard_path(tmp_path)assert len(data.rows) == 2assert data.rows[0]["skill"] == "native-skill"@@ -444,28 +733,156 @@ def test_native_results_use_real_usage_and_prune_nested_mirrors(tmp_path: Path)assert load_dashboard_path(run / "result.json").rows == load_dashboard_path(run).rows+def test_native_per_trial_usage_uses_retained_measurements(tmp_path: Path) -> None:+ run = tmp_path / "run"+ _native_run(run)+ rows = load_dashboard_path(run).rows+ assert [row["total_tokens_per_trial"] for row in rows] == [210.0, 110.0]+ assert [row["duration_seconds_per_trial"] for row in rows] == [10.0, 10.0]+ assert [row["cost_usd_per_trial"] for row in rows] == [None, None]+++def _canonical_export(run: Path) -> dict:+ result = agent_eval_result_from_directory(Path("native-skill"), run, use_llm_judge=False)+ assert result is not None+ return json.loads(json.dumps(result.metadata["agent_eval"]))+++def test_canonical_export_agent_seconds_matches_retained_run(tmp_path: Path) -> None:+ run = tmp_path / "run"+ _native_run(run)+ canonical = parse_dashboard_report(_canonical_export(run), "r.json")+ retained = load_dashboard_path(run)+ assert (+ [row["duration_seconds"] for row in canonical.rows]+ == [row["duration_seconds"] for row in retained.rows]+ == [10.0, 10.0]+ )+ assert all("time 1/1" in row["coverage"] for row in canonical.rows)+ comparison = comparison_rows(canonical.rows)[0]+ assert comparison["comparison_status"] == "comparable"+ assert comparison["duration_seconds_delta"] == 0.0+++def test_unreadable_trajectory_keeps_retained_time_and_token_fallback(tmp_path: Path) -> None:+ run = tmp_path / "run"+ _native_run(run)+ trajectory = run / "opencode" / "with-skill" / "trials" / "case-a__random" / "trajectory.json"+ trajectory.write_text("{not json", encoding="utf-8")+ retained = load_dashboard_path(run)+ canonical = parse_dashboard_report(_canonical_export(run), "r.json")+ assert retained.rows[0]["duration_seconds"] == canonical.rows[0]["duration_seconds"] == 10.0+ assert comparison_rows(retained.rows)[0]["duration_seconds_delta"] == 0.0+ # Harbor's agent_result counters stand in for the unreadable trajectory totals.+ assert retained.rows[0]["total_tokens"] == 210+ assert any("cannot read trajectory" in warning for warning in retained.warnings)+++def test_retained_run_ignores_export_zero_token_notice(tmp_path: Path) -> None:+ run = tmp_path / "run"+ _native_run(run)+ engine_result = json.loads((run / "result.json").read_text(encoding="utf-8"))+ engine_result["evaluator_version"] = "0.4.0"+ (run / "result.json").write_text(json.dumps(engine_result), encoding="utf-8")+ for trajectory_path in run.glob("opencode/*/trials/*/trajectory.json"):+ trajectory = json.loads(trajectory_path.read_text(encoding="utf-8"))+ trajectory["final_metrics"]["total_cached_tokens"] = 0+ trajectory_path.write_text(json.dumps(trajectory), encoding="utf-8")+ assert any("before v0.5.0" in warning for warning in parse_dashboard_report(_canonical_export(run), "r").warnings)+ data = load_dashboard_path(run)+ # Retained trajectories hold recorded counters, so their zeros stay measured.+ assert [row["cached_tokens"] for row in data.rows] == [0, 0]+ assert not any("before v0.5.0" in warning for warning in data.warnings)+++def test_retained_naive_timestamps_follow_canonical_timezone_policy(tmp_path: Path) -> None:+ run = tmp_path / "run"+ _native_run(run)+ for result_path in run.glob("*/*/trials/*/result.json"):+ result = json.loads(result_path.read_text(encoding="utf-8"))+ result["agent_execution"] = {"started_at": "2026-10-01T00:00:00", "finished_at": "2026-10-01T00:00:10"}+ result_path.write_text(json.dumps(result), encoding="utf-8")+ rows = load_dashboard_path(run).rows+ assert len(rows) == 2+ assert all(row["duration_seconds"] is None for row in rows)+ assert all("time 0/1" in row["coverage"] for row in rows)+ agent = _canonical_export(run)["agents"]["opencode"]+ assert all("agent_seconds" not in trial for trial in agent["trials"] + agent["trials_baseline"])+++def test_oversized_trajectory_keeps_scores_comparable(tmp_path: Path) -> None:+ run = tmp_path / "run"+ _native_run(run)+ trajectory = run / "opencode" / "with-skill" / "trials" / "case-a__random" / "trajectory.json"+ # Larger than the report loader's 2 MiB JSON bound, below the dashboard's own.+ trajectory.write_text(json.dumps({"steps": [], "padding": "x" * (3 * 1024 * 1024)}), encoding="utf-8")+ comparison = comparison_rows(load_dashboard_path(run).rows)[0]+ assert comparison["comparison_status"] == "comparable"+ assert comparison["score_delta"] == 0+ canonical = _canonical_export(run)+ assert canonical["report_truncation"]["artifact_loading"][0]["artifact"] == "trajectory"+ rows = parse_dashboard_report(canonical, "r.json").rows+ assert comparison_rows(rows)[0]["comparison_status"] == "comparable"+ assert rows[0]["condition"] == "with_skill"+ assert rows[0]["total_tokens"] is None+++def test_retained_run_keeps_non_best_agent_trials_beyond_embedded_budget(tmp_path: Path, monkeypatch) -> None:+ from skillevaluator.evaluation import tier3_report++ run = tmp_path / "run"+ _native_run(run)+ shutil.copytree(run / "opencode", run / "codex")+ config = json.loads((run / "run_config.json").read_text(encoding="utf-8"))+ config["agents"]["codex"] = {"agent": "codex", "model": "m"}+ (run / "run_config.json").write_text(json.dumps(config), encoding="utf-8")+ for summary_path in run.glob("codex/*/summary.json"):+ summary = json.loads(summary_path.read_text(encoding="utf-8"))+ summary["agent"] = "codex"+ summary_path.write_text(json.dumps(summary), encoding="utf-8")+ # A budget just below the full payload prunes only the non-best agent's details.+ budget = tier3_report._serialized_payload_size(_canonical_export(run)) - 200+ monkeypatch.setattr(tier3_report, "_MAX_EMBEDDED_REPORT_BYTES", budget)+ canonical = _canonical_export(run)+ assert canonical["report_truncation"]["omitted"]["non_best_agent_details"]+ pruned = next(name for name in canonical["agents"] if name != canonical["best_agent"])++ rows = load_dashboard_path(run).rows+ assert len(rows) == 4+ assert all(row["trials"] == 1 and row["score"] == 0.8 and row["total_tokens"] is not None for row in rows)+ assert all(row["comparison_status"] == "comparable" for row in comparison_rows(rows))+ # The bounded canonical export still fails closed for the pruned agent.+ comparison = next(+ row for row in comparison_rows(parse_dashboard_report(canonical, "r.json").rows) if row["_agent_key"] == pruned+ )+ assert comparison["coverage"] == "1 with / 1 without"+ assert comparison["comparison_reason"].startswith("trial details pruned from embedded report")++def test_catalog_native_and_mixed_references_are_loaded_without_cycles(tmp_path: Path) -> None:run = tmp_path / "run"_native_run(run)index = tmp_path / "catalog.json"reference = {"report_dir": "run", "json_report": "result.json"}- index.write_text(json.dumps({"skills": [reference]}))+ index.write_text(json.dumps({"skills": [reference]}), encoding="utf-8")assert load_dashboard_path(index).rows == load_dashboard_path(run).rows- index.write_text(json.dumps({"skills": [{"tier3": _report()}, reference]}))+ index.write_text(json.dumps({"skills": [{"tier3": _report()}, reference]}), encoding="utf-8")assert len(load_dashboard_path(index).rows) == 4- index.write_text(json.dumps({"skills": [reference, {"report_dir": ".", "json_report": "catalog.json"}]}))+ index.write_text(+ json.dumps({"skills": [reference, {"report_dir": ".", "json_report": "catalog.json"}]}), encoding="utf-8"+ )data = load_dashboard_path(index)assert len(data.rows) == 2assert any("cyclic" in warning for warning in data.warnings)def test_nested_catalog_references_load_each_report_once(tmp_path: Path) -> None:- (tmp_path / "report.json").write_text(json.dumps(_report()))+ (tmp_path / "report.json").write_text(json.dumps(_report()), encoding="utf-8")reference = {"report_dir": ".", "json_report": "report.json"}- (tmp_path / "child.json").write_text(json.dumps({"skills": [reference] * 10}))+ (tmp_path / "child.json").write_text(json.dumps({"skills": [reference] * 10}), encoding="utf-8")child = {"report_dir": ".", "json_report": "child.json"}index = tmp_path / "catalog.json"- index.write_text(json.dumps({"skills": [child] * 10}))+ index.write_text(json.dumps({"skills": [child] * 10}), encoding="utf-8")data = load_dashboard_path(index)assert len(data.rows) == 2assert comparison_rows(data.rows)[0]["comparison_status"] == "comparable"@@ -477,14 +894,15 @@ def test_catalog_file_budget_is_shared_across_nested_references(tmp_path: Path,monkeypatch.setattr(dashboard_data, "_MAX_FILES", 3)index = tmp_path / "catalog.json"- (tmp_path / "one.json").write_text(json.dumps(_report()))+ (tmp_path / "one.json").write_text(json.dumps(_report()), encoding="utf-8")second = _report()second["skill_name"] = "second-skill"- (tmp_path / "two.json").write_text(json.dumps(second))+ (tmp_path / "two.json").write_text(json.dumps(second), encoding="utf-8")(tmp_path / "child.json").write_text(- json.dumps({"skills": [{"report_dir": ".", "json_report": name} for name in ("one.json", "two.json")]})+ json.dumps({"skills": [{"report_dir": ".", "json_report": name} for name in ("one.json", "two.json")]}),+ encoding="utf-8",)- index.write_text(json.dumps({"skills": [{"report_dir": ".", "json_report": "child.json"}]}))+ index.write_text(json.dumps({"skills": [{"report_dir": ".", "json_report": "child.json"}]}), encoding="utf-8")data = load_dashboard_path(index)assert len(data.rows) == 2assert any("catalog limit" in warning for warning in data.warnings)@@ -510,7 +928,7 @@ def test_directory_enumeration_stops_before_materializing_all_entries(tmp_path:yield entryfor index in range(20):- (tmp_path / f"report-{index}.json").write_text(json.dumps(_report()))+ (tmp_path / f"report-{index}.json").write_text(json.dumps(_report()), encoding="utf-8")monkeypatch.setattr(dashboard_data, "_MAX_PATHS", 4)monkeypatch.setattr(dashboard_data.os, "scandir", lambda _path: CountingScanner())data = load_dashboard_path(tmp_path)@@ -519,12 +937,65 @@ def test_directory_enumeration_stops_before_materializing_all_entries(tmp_path:assert any("discovery limit" in warning for warning in data.warnings)+def test_native_trial_bounds_mirror_report_data() -> None:+ from skillevaluator.reporting import dashboard_data+ from skillevaluator.tier3.harbor import report_data++ assert dashboard_data._MAX_NATIVE_TRIALS == report_data._MAX_TRIALS_PER_CONDITION+ assert dashboard_data._MAX_NATIVE_TRIAL_ENTRIES == report_data._MAX_TRIAL_PATHS_SCANNED+++def test_native_trial_discovery_limit_keeps_usage_unavailable(tmp_path: Path, monkeypatch) -> None:+ from skillevaluator.reporting import dashboard_data++ run = tmp_path / "run"+ _native_run(run)+ condition = run / "opencode" / "with-skill"+ original = condition / "trials" / "case-a__random"+ for index in (1, 2):+ trial_dir = shutil.copytree(original, original.with_name(f"case-a__trial{index}"))+ reward_path = trial_dir / "reward.json"+ reward = json.loads(reward_path.read_text(encoding="utf-8"))+ reward["trial_id"] = trial_dir.name+ reward_path.write_text(json.dumps(reward), encoding="utf-8")+ summary_path = condition / "summary.json"+ summary = json.loads(summary_path.read_text(encoding="utf-8"))+ summary.update(num_trials=3, expected_attempts=3, scored_attempts=3)+ summary_path.write_text(json.dumps(summary), encoding="utf-8")+ # Within the bound all three trials are measured.+ assert next(row for row in load_dashboard_path(run).rows if row["condition"] == "with_skill")["total_tokens"] == 630+ monkeypatch.setattr(dashboard_data, "_MAX_NATIVE_TRIALS", 2)+ data = load_dashboard_path(run)+ row = next(row for row in data.rows if row["condition"] == "with_skill")+ assert any("trial discovery limit reached; usage totals are unavailable." in warning for warning in data.warnings)+ assert row["total_tokens"] is None+ assert row["duration_seconds"] is None+++def test_local_json_over_limit_is_a_notice_and_other_reports_load(tmp_path: Path, monkeypatch) -> None:+ from skillevaluator.reporting import dashboard_data++ monkeypatch.setattr(dashboard_data, "_MAX_FILE_BYTES", 2048)+ content = json.dumps(_report())+ exact = tmp_path / "exact.json"+ exact.write_text(content.ljust(2048), encoding="utf-8")+ oversized = tmp_path / "oversized.json"+ oversized.write_text(content.ljust(2049), encoding="utf-8")+ assert (exact.stat().st_size, oversized.stat().st_size) == (2048, 2049)+ assert len(load_dashboard_path(exact).rows) == 2+ assert any("16 MiB" in warning for warning in load_dashboard_path(oversized).warnings)+ data = load_dashboard_path(tmp_path)+ assert {row["source"] for row in data.rows} == {str(exact.resolve())}+ assert len(data.rows) == 2+ assert any(str(oversized.resolve()) in warning and "16 MiB" in warning for warning in data.warnings)++def test_directory_file_budget_counts_malformed_reports(tmp_path: Path, monkeypatch) -> None:from skillevaluator.reporting import dashboard_datamonkeypatch.setattr(dashboard_data, "_MAX_FILES", 3)for index in range(20):- (tmp_path / f"broken-{index}.json").write_text("{")+ (tmp_path / f"broken-{index}.json").write_text("{", encoding="utf-8")data = load_dashboard_path(tmp_path)assert sum("cannot read report" in warning for warning in data.warnings) == 3assert any("file limit" in warning for warning in data.warnings)@@ -538,7 +1009,7 @@ def test_cyclic_input_catalog_and_native_symlinks_are_diagnostics(tmp_path: Pathpytest.skip("Symlinks are unavailable on this host")assert load_dashboard_path(loop).warningsindex = tmp_path / "catalog.json"- index.write_text(json.dumps({"skills": [{"report_dir": ".", "json_report": "loop.json"}]}))+ index.write_text(json.dumps({"skills": [{"report_dir": ".", "json_report": "loop.json"}]}), encoding="utf-8")assert load_dashboard_path(index).warningsrun = tmp_path / "run"_native_run(run)@@ -575,13 +1046,13 @@ def test_malformed_native_schema_is_diagnostic_and_preserves_other_sources(tmp_p_native_run(run)if malformation == "summary_status":path = run / "opencode" / "with-skill" / "summary.json"- summary = json.loads(path.read_text())+ summary = json.loads(path.read_text(encoding="utf-8"))summary["execution_status"] = ["succeeded"]- path.write_text(json.dumps(summary))+ path.write_text(json.dumps(summary), encoding="utf-8")else:- (run / "run_config.json").write_text(json.dumps({"agents": [{"agent": "opencode"}]}))+ (run / "run_config.json").write_text(json.dumps({"agents": [{"agent": "opencode"}]}), encoding="utf-8")assert load_dashboard_path(run / "result.json").warnings- (tmp_path / "valid.json").write_text(json.dumps(_report()))+ (tmp_path / "valid.json").write_text(json.dumps(_report()), encoding="utf-8")data = load_dashboard_path(tmp_path)assert len(data.rows) == 2assert {row["skill"] for row in data.rows} == {"calculator"}@@ -593,9 +1064,9 @@ def test_native_missing_attempt_does_not_publish_partial_usage_as_total(tmp_path_native_run(run)for variant in ("with-skill", "without-skill"):summary_path = run / "opencode" / variant / "summary.json"- summary = json.loads(summary_path.read_text())+ summary = json.loads(summary_path.read_text(encoding="utf-8"))summary.update(num_trials=2, expected_attempts=2, scored_attempts=2)- summary_path.write_text(json.dumps(summary))+ summary_path.write_text(json.dumps(summary), encoding="utf-8")data = load_dashboard_path(run)assert all(row["score"] is None and row["total_tokens"] is None for row in data.rows)assert all(row["duration_seconds"] is None for row in data.rows)@@ -615,13 +1086,13 @@ def test_native_generated_stop_on_pass_names_preserve_attempt_sets(tmp_path: Patshutil.copytree(first, second)for trial_dir in (first, second):reward_path = trial_dir / "reward.json"- reward = json.loads(reward_path.read_text())+ reward = json.loads(reward_path.read_text(encoding="utf-8"))reward["trial_id"] = trial_dir.name- reward_path.write_text(json.dumps(reward))+ reward_path.write_text(json.dumps(reward), encoding="utf-8")summary_path = condition / "summary.json"- summary = json.loads(summary_path.read_text())+ summary = json.loads(summary_path.read_text(encoding="utf-8"))summary.update(num_trials=2, expected_attempts=2, scored_attempts=2)- summary_path.write_text(json.dumps(summary))+ summary_path.write_text(json.dumps(summary), encoding="utf-8")data = load_dashboard_path(run)assert all(row["trials"] == 2 and row["score"] == 0.8 for row in data.rows)assert comparison_rows(data.rows)[0]["comparison_status"] == "comparable"@@ -631,15 +1102,15 @@ def test_native_different_condition_models_cannot_pair(tmp_path: Path) -> None:run = tmp_path / "run"_native_run(run)baseline_summary = run / "opencode" / "without-skill" / "summary.json"- summary = json.loads(baseline_summary.read_text())+ summary = json.loads(baseline_summary.read_text(encoding="utf-8"))summary["model"] = "different-model"- baseline_summary.write_text(json.dumps(summary))+ baseline_summary.write_text(json.dumps(summary), encoding="utf-8")comparisons = comparison_rows(load_dashboard_path(run).rows)assert len(comparisons) == 2assert all(row["comparison_status"] == "missing condition" for row in comparisons)[email protected]("value", [True, -1, 1.5, "1"])[email protected]("value", [True, -1, 1.5, "1", 10**30])@pytest.mark.parametrize("field", ["num_trials", "expected_attempts", "scored_attempts"])def test_native_invalid_coverage_counts_cannot_be_hidden_by_canonical_normalization(tmp_path: Path, field: str, value: object@@ -647,9 +1118,9 @@ def test_native_invalid_coverage_counts_cannot_be_hidden_by_canonical_normalizatrun = tmp_path / "run"_native_run(run)summary_path = run / "opencode" / "with-skill" / "summary.json"- summary = json.loads(summary_path.read_text())+ summary = json.loads(summary_path.read_text(encoding="utf-8"))summary[field] = value- summary_path.write_text(json.dumps(summary))+ summary_path.write_text(json.dumps(summary), encoding="utf-8")data = load_dashboard_path(run)row = next(row for row in data.rows if row["condition"] == "with_skill")assert row["score"] is None@@ -659,6 +1130,22 @@ def test_native_invalid_coverage_counts_cannot_be_hidden_by_canonical_normalizatassert comparison_rows(data.rows)[0]["score_delta"] is None[email protected](os.name == "nt" or os.geteuid() == 0, reason="Needs POSIX permissions enforced for this user")+def test_unreadable_condition_trials_keep_other_rows(tmp_path: Path) -> None:+ run = tmp_path / "run"+ _native_run(run)+ trials = run / "opencode" / "with-skill" / "trials"+ trials.chmod(0)+ try:+ data = load_dashboard_path(run)+ finally:+ trials.chmod(0o755)+ assert len(data.rows) == 2+ assert next(row for row in data.rows if row["condition"] == "with_skill")["total_tokens"] is None+ assert next(row for row in data.rows if row["condition"] == "without_skill")["total_tokens"] == 110+ assert any("cannot read retained trials" in warning for warning in data.warnings)++@pytest.mark.parametrize("artifact", ["trials", "trajectory.json", "result.json", "reward.json"])def test_native_usage_rejects_intermediate_directory_and_file_symlinks(tmp_path: Path, artifact: str) -> None:run = tmp_path / "run"@@ -673,8 +1160,13 @@ def test_native_usage_rejects_intermediate_directory_and_file_symlinks(tmp_path:pytest.skip("Symlinks are unavailable on this host")data = load_dashboard_path(run)row = next(row for row in data.rows if row["condition"] == "with_skill")- assert row["total_tokens"] is None- assert row["duration_seconds"] is None+ if artifact == "trajectory.json":+ # A skipped trajectory loses only its own totals; result.json still measures the trial.+ assert row["total_tokens"] == 210+ assert row["duration_seconds"] == 10+ else:+ assert row["total_tokens"] is None+ assert row["duration_seconds"] is Noneassert any("symlink" in warning for warning in data.warnings)@@ -682,20 +1174,20 @@ def test_multistep_native_cost_time_and_tokens_require_every_step(tmp_path: Pathrun = tmp_path / "run"_native_run(run)trial_result = run / "opencode" / "with-skill" / "trials" / "case-a__random" / "result.json"- result = json.loads(trial_result.read_text())+ result = json.loads(trial_result.read_text(encoding="utf-8"))step = {"agent_result": {"n_input_tokens": 50, "n_output_tokens": 5, "n_cache_tokens": 0, "cost_usd": 0.1},"agent_execution": {"started_at": "2026-10-01T00:00:00Z", "finished_at": "2026-10-01T00:00:03Z"},}result["step_results"] = [step, copy.deepcopy(step)]- trial_result.write_text(json.dumps(result))+ trial_result.write_text(json.dumps(result), encoding="utf-8")row = load_dashboard_path(run).rows[0]assert row["prompt_tokens"] == 100assert row["duration_seconds"] == 6assert row["cost_usd"] == pytest.approx(0.2)result["step_results"][1]["agent_result"].pop("n_input_tokens")result["step_results"][1]["agent_result"].pop("cost_usd")- trial_result.write_text(json.dumps(result))+ trial_result.write_text(json.dumps(result), encoding="utf-8")row = load_dashboard_path(run).rows[0]assert row["prompt_tokens"] is Noneassert row["total_tokens"] is None@@ -710,31 +1202,79 @@ def test_native_duplicate_reward_rows_require_authoritative_step_scope(tmp_path:second = first.with_name("case-a__random__finish")shutil.copytree(first, second)summary_path = condition / "summary.json"- summary = json.loads(summary_path.read_text())- summary["num_trials"] = 2- summary_path.write_text(json.dumps(summary))+ summary = json.loads(summary_path.read_text(encoding="utf-8"))+ # The collector counts one logical trial and records its two reward rows.+ summary["num_reward_rows"] = 2+ summary_path.write_text(json.dumps(summary), encoding="utf-8")for trial_dir in (first, second):result_path = trial_dir / "result.json"- result = json.loads(result_path.read_text())+ result = json.loads(result_path.read_text(encoding="utf-8"))step = {"step_name": "prepare","agent_result": {"n_input_tokens": 50, "n_output_tokens": 5, "n_cache_tokens": 0, "cost_usd": 0.1},"agent_execution": result["agent_execution"],}result["step_results"] = [step, {**copy.deepcopy(step), "step_name": "finish"}]- result_path.write_text(json.dumps(result))- row = load_dashboard_path(run).rows[0]+ result_path.write_text(json.dumps(result), encoding="utf-8")+ data = load_dashboard_path(run)+ assert not any("resource ownership is ambiguous" in warning for warning in data.warnings)+ row = data.rows[0]assert row["score"] == 0.8assert row["total_tokens"] == 110assert row["duration_seconds"] == 20assert row["cost_usd"] == pytest.approx(0.2)for trial_dir in (first, second):result_path = trial_dir / "result.json"- result = json.loads(result_path.read_text())+ result = json.loads(result_path.read_text(encoding="utf-8"))result.pop("step_results")- result_path.write_text(json.dumps(result))+ result_path.write_text(json.dumps(result), encoding="utf-8")data = load_dashboard_path(run)assert data.rows[0]["score"] == 0.8assert data.rows[0]["total_tokens"] is Noneassert data.rows[0]["duration_seconds"] is Noneassert any("resource ownership is ambiguous" in warning for warning in data.warnings)+++def test_retained_feature_metadata_precedence_summary_over_agent_over_root(tmp_path: Path) -> None:+ run = tmp_path / "run"+ _native_run(run)+ (run / "run_config.json").write_text(+ json.dumps(+ {+ "dashboard_metadata": {"harness_features": ["root"]},+ "agents": {+ "opencode": {+ "agent": "opencode",+ "model": "m",+ "dashboard_metadata": {"harness_features": ["agent"]},+ }+ },+ }+ ),+ encoding="utf-8",+ )+ summary_path = run / "opencode" / "without-skill" / "summary.json"+ summary = json.loads(summary_path.read_text(encoding="utf-8"))+ summary["dashboard_metadata"] = {"harness_features": ["summary"]}+ summary_path.write_text(json.dumps(summary), encoding="utf-8")++ rows = {row["condition"]: row for row in load_dashboard_path(run).rows}++ assert rows["with_skill"]["features"] == "agent"+ assert rows["without_skill"]["features"] == "summary"+ assert all(item["comparison_status"] == "missing condition" for item in comparison_rows(list(rows.values())))+ # An unreadable run_config.json falls back to the copy in result.json.+ (run / "run_config.json").write_text("{", encoding="utf-8")+ data = load_dashboard_path(run)+ assert [row["features"] for row in data.rows] == ["unknown", "summary"]+ assert any("cannot read run config" in warning for warning in data.warnings)+++def test_empty_feature_list_and_string_and_mapping_forms() -> None:+ labels = []+ for value in ([], "rules on", {"memory": False, "rules": True}):+ report = _report()+ report["dashboard_metadata"] = {"harness_features": value}+ labels.append(parse_dashboard_report(report, "r.json").rows[0]["features"])++ assert labels == ["none (recorded)", "rules on", '{"memory": false, "rules": true}']diff --git a/tests/reporting/test_dashboard_entry.py b/tests/reporting/test_dashboard_entry.pynew file mode 100644index 00000000..547a5152--- /dev/null+++ b/tests/reporting/test_dashboard_entry.py@@ -0,0 +1,29 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Run the packaged Streamlit entry script that the dashboard launcher starts."""++from __future__ import annotations++import sys++import pytest++pytest.importorskip("streamlit")+# st.dataframe otherwise imports these inside AppTest's timed script thread.+pytest.importorskip("pandas")+pytest.importorskip("pyarrow")+from streamlit.testing.v1 import AppTest++from skillevaluator.reporting import dashboard++# A ceiling for cold imports under xdist and coverage; passing runs finish far sooner.+_APP_TIMEOUT_SECONDS = 60+++def test_entry_script_runs_dashboard_main(monkeypatch) -> None:+ # AppTest runs the script in this process, so it would otherwise read pytest's argv.+ monkeypatch.setattr(sys, "argv", [str(dashboard._APP)])+ app = AppTest.from_file(str(dashboard._APP), default_timeout=_APP_TIMEOUT_SECONDS).run()+ assert not app.exception+ assert [message.value for message in app.info] == ["Upload JSON evaluation reports to start comparing results."]diff --git a/tests/test_cli.py b/tests/test_cli.pyindex b0d634a4..551ff06a 100644--- a/tests/test_cli.py+++ b/tests/test_cli.py@@ -64,7 +64,7 @@ def test_top_level_help_groups_commands_by_workflow() -> None:assert positions == sorted(positions)core = result.output.split("Core workflows:", 1)[1].split("Tier 1 · Static and security:", 1)[0]- assert all(command in core for command in ("validate", "health-check", "doctor", "models"))+ assert all(command in core for command in ("validate", "health-check", "doctor", "models", "dashboard"))tier3 = result.output.split("Tier 3 · Live evaluation:", 1)[1]assert all(command in tier3 for command in ("create-eval-dataset", "compare", "view", "harbor-view"))assert "Other commands:" not in result.outputdiff --git a/tests/test_oss_packaging.py b/tests/test_oss_packaging.pyindex 089cf73e..ab93b27a 100644--- a/tests/test_oss_packaging.py+++ b/tests/test_oss_packaging.py@@ -501,7 +501,9 @@ def test_release_lock_enforces_nspect_remediation_floors_without_removed_telemetif any(dependency["name"] == "protobuf" for dependency in package.get("dependencies", []))}assert protobuf_consumers == {"streamlit"}- assert extras["dashboard"] == ["streamlit>=1.49,<2"]+ # Older Streamlit releases silently ignore the launcher's server.allowedHosts setting.+ assert extras["dashboard"] == ["streamlit>=1.61,<2"]+ assert all(version >= Version("1.61") for version in all_lock_versions["streamlit"])assert not any("streamlit" in dependency for dependency in project["project"]["dependencies"])assert all(version >= Version("4.0.0") for version in all_lock_versions["oauthlib"])assert not any(name.startswith("opentelemetry") for name in all_lock_versions)diff --git a/uv.lock b/uv.lockindex 33ea5bc0..9b57dfe0 100644--- a/uv.lock+++ b/uv.lock@@ -2183,7 +2183,7 @@ requires-dist = [{ name = "skillevaluator", extras = ["llm"], marker = "extra == 'tier2'" },{ name = "skillevaluator", extras = ["llm"], marker = "extra == 'tier3'" },{ name = "skillevaluator", extras = ["tier2", "tier3", "security"], marker = "extra == 'all'" },- { name = "streamlit", marker = "extra == 'dashboard'", specifier = ">=1.49,<2" },+ { name = "streamlit", marker = "extra == 'dashboard'", specifier = ">=1.61,<2" },]provides-extras = ["llm", "tier2", "tier3", "security", "dashboard", "all", "dev"]