Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(tier3): add --skip-with-skill for baseline-only runs (#284)
Narendran Raghavanbfd477b3d20a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex da64cd8f..0ceae299 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -119,6 +119,17 @@ All notable changes to SkillEvaluator are documented in this file.was attempted. Records that do not match SkillSpector's provenance contract orthat contain endpoint- or credential-shaped values are left out, and olderSkillSpector releases produce no record.+- `--skip-with-skill` on `tier3 evaluate` and the direct `tier3` workflow runs+ only the without-skill baseline arm, for example to calibrate a task set or+ re-measure a baseline after a model change without paying for with-skill+ trials. Only baseline tasks are staged and launched. The run is reported as+ a baseline calibration: `mode` is `baseline_only`, the overall score and best+ agent come from the baseline arm, the with-skill arm is recorded as+ `skipped`, with-skill scores and lift stay unavailable instead of becoming+ 0.0 (report tables print `skipped`), and the verdict stays NEUTRAL. The flag+ is mutually exclusive with `--skip-baseline`, keeps the baseline isolation+ rules, and is not accepted by `validate`, so a baseline-only run cannot pass+ a Tier 3 gate.### Changeddiff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex 94f12cde..e57b134e 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -432,6 +432,7 @@ Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing| `--environment-kwarg`, `--ek` | none | Operator-only `KEY=VALUE` constructor option for Harbor-native non-Docker backends (repeatable). Values accept JSON-compatible types; skill-owned `evals/config.yml` cannot set them. Never pass secrets. Docker and local reject all kwargs, Kata accepts only `kata_runtime` and `kata_dns`, and Harbor runtime-policy fields for overrides, mounts, networks, extra Compose, lifecycle, and pod security are reserved, as are `stream`, `enable_environment_dir_upload`, and `auth` for `cwsandbox`/`wandb`. Names the selected backend does not accept in Harbor 0.24.0 are rejected. See [Native backend constructor options](agents-and-sandboxes.mdx#native-backend-constructor-options). || `--autopilot` | off | Create one eval case when no dataset/task source exists, then evaluate. The case is LLM-generated with the configured provider, with a deterministic keyless template fallback; an existing source is never overwritten. || `--skip-baseline` | off | Skip the without-skill baseline (no lift analysis, faster). |+| `--skip-with-skill` | off | Run only the without-skill baseline, for example to calibrate a task set or measure a model change before running the skill. The report is marked baseline-only: the overall score is the baseline score, with-skill and lift are unavailable (shown as `skipped` or N/A, never 0.0), and no skill verdict is given. Mutually exclusive with `--skip-baseline`. Not available on `validate`. || `--n-attempts INTEGER` | unset | Attempts per eval case (pass@k). || `--pass-threshold FLOAT` | unset | Score threshold (0.0–1.0) for a case to count as passed. || `--stop-on-pass / --no-stop-on-pass` | unset | Stop a case's remaining attempts once one passes. |diff --git a/docs/reports.mdx b/docs/reports.mdxindex 0b33a924..4cf054ad 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -350,7 +350,7 @@ completed run directly:you ran more than one agent.- **`<agent>/lift.json`** and **`<agent>/pass_at_k_lift.json`** — per-metricdeltas between the two arms; written only when both arms produced scores- (a `--skip-baseline` run has neither).+ (a `--skip-baseline` or `--skip-with-skill` run has neither).- **`<agent>/{with-skill,without-skill}/`** — each arm's `summary.json`(scores, dimensions, pass@k, trial counts) and a `trials/` directory ofper-trial artifacts: the trial's `reward.json`, Harbor's `trajectory.json`@@ -387,7 +387,9 @@ A finished `tier3 evaluate` run prints two tables and closes with a compactwith its per-arm scores and Skill Lift; **Results by Dimension** rolls thosesignals up into the five dimensions below. A metric the run couldn't scorerenders as `NO SCORE` — it is never coerced to 0.0 — and unscored metrics are-omitted from the averages, so `overall_score` can be `null` in the JSON. The+omitted from the averages, so `overall_score` can be `null` in the JSON. In a+baseline-only run (`--skip-with-skill`) the with-skill and Skill Lift cells read+`skipped` instead, because that arm never ran. TheArtifacts panel shows the HTML report path, the output directory, and — onlywhen a retained Harbor jobs directory exists — a ready-to-paste `harbor-view`command.diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex cf3dbf9f..416a97a6 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -559,6 +559,7 @@ The high-signal flags, with defaults:| `--max-agents` | selected-agent count | Maximum agents run in parallel || `--max-cases` | every case | Run only the first N eval cases; a run that skips cases is not publication evidence || `--skip-baseline` | off | Skip the without-skill arm; faster, but no Skill Lift |+| `--skip-with-skill` | off | Run only the without-skill arm, for example to calibrate a task set; the report shows baseline scores with no Skill Lift or skill verdict || `--stop-on-pass` | off | Stop a case's remaining attempts once one passes || `--agent-runtime-preflight` | off | Add one agent-only execution of the first staged task per agent before the full A/B matrix; `--no-agent-runtime-preflight` overrides a config value that enables it || `--progress` | `auto` | Progress rendering: `auto`, `rich` (live TTY view with secret redaction), `plain`, or `off` |@@ -578,6 +579,28 @@ While iterating on a dataset, `--skip-baseline` roughly halves the runtime.Re-enable the baseline arm for any run whose Skill Lift you intend to trust.</Tip>+### Baseline-only runs++`--skip-with-skill` is the counterpart of `--skip-baseline`: it stages and runs+only the without-skill arm. Use it to measure how agents do on a task set+before the skill exists, or to re-measure a baseline after a model or harness+change without paying for with-skill trials.++```bash title="Baseline-only calibration run"+skillevaluator tier3 evaluate ./my-skill --agents codex --skip-with-skill+```++The run is reported as a baseline calibration, not a skill evaluation. The+overall score and best agent come from the baseline arm, the with-skill arm is+recorded as `skipped`, with-skill scores and Skill Lift are unavailable (shown+as `skipped` or N/A, never 0.0), and the verdict stays NEUTRAL because no skill+was measured. The two flags are mutually+exclusive, and baseline isolation rules still apply: a config that sets+`harbor.pre_agent_setup` (or `setup_commands`) is rejected, because the+baseline arm must stay isolated from skill-owned setup commands. `validate`+does not accept `--skip-with-skill`, so a baseline-only run can never satisfy a+Tier 3 gate.+## How standard grading judges a trialWith `default` or `default_plus_custom` grading, SkillEvaluator's verifierdiff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex 83a04678..a260b412 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -2840,6 +2840,11 @@ def _tier2_workflow(help="Create one eval case when no dataset/task source exists, then evaluate.",)@click.option("--skip-baseline", is_flag=True, help="Skip without-skill baseline.")+ "--skip-with-skill",+ is_flag=True,+ help="Skip the with-skill arm and run only the without-skill baseline (no lift analysis).",+)@click.option("--n-attempts", type=int, default=None)@click.option("--pass-threshold", type=float, default=None)@click.option(@@ -2909,6 +2914,7 @@ def evaluate(environment_kwarg: tuple[str, ...],autopilot: bool,skip_baseline: bool,+ skip_with_skill: bool,n_attempts: int | None,pass_threshold: float | None,stop_on_pass: bool | None,@@ -2939,6 +2945,9 @@ def evaluate(from skillevaluator.evaluation import EvaluationOptions, EvaluationServicefrom skillevaluator.tier3.harbor.progress import create_progress_reporter+ if skip_baseline and skip_with_skill:+ # Reject before --autopilot can spend a generation call on a run that cannot start.+ raise click.UsageError("--skip-baseline and --skip-with-skill are mutually exclusive (no arm would run)")service = EvaluationService()# Resolved before the service is built so a non-canonical value is refused# at the boundary rather than after a long live run has already started.@@ -2958,6 +2967,7 @@ def evaluate(env_mode=env_mode,environment_kwarg=environment_kwarg,skip_baseline=skip_baseline,+ skip_with_skill=skip_with_skill,n_attempts=n_attempts,pass_threshold=pass_threshold,stop_on_pass=stop_on_pass,diff --git a/src/skillevaluator/evaluation/options.py b/src/skillevaluator/evaluation/options.pyindex c25ab78f..ad1345a4 100644--- a/src/skillevaluator/evaluation/options.py+++ b/src/skillevaluator/evaluation/options.py@@ -27,6 +27,9 @@ class EvaluationOptions:agents: str | None = Noneenv_mode: str = "docker"skip_baseline: bool = False+ # Baseline-only runs (for example harness calibration) skip the with-skill+ # arm. Mutually exclusive with ``skip_baseline``.+ skip_with_skill: bool = Falsen_attempts: int | None = Nonepass_threshold: float | None = Nonestop_on_pass: bool | None = Nonediff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.pyindex 5386454a..ec17bda9 100644--- a/src/skillevaluator/evaluation/tier3_report.py+++ b/src/skillevaluator/evaluation/tier3_report.py@@ -564,10 +564,18 @@ def _validation_result_from_payload(payload: dict[str, Any] | None) -> Validatioresult.metadata["run_id"] = run_idbest = payload.get("best_agent") or "n/a"if payload.get("execution_status") == "succeeded" and _finite_float(payload.get("overall_score")) is not None:- result.add_success(- "agent_eval",- f"Tier 3 evaluation complete: verdict {str(payload.get('verdict', 'neutral')).upper()}; best agent {best}",- )+ if payload.get("mode") == "baseline_only":+ result.add_success(+ "agent_eval",+ f"Tier 3 baseline calibration complete: best agent {best} "+ "(no with-skill arm run; no skill lift measured)",+ )+ else:+ result.add_success(+ "agent_eval",+ f"Tier 3 evaluation complete: verdict {str(payload.get('verdict', 'neutral')).upper()}; "+ f"best agent {best}",+ )result.passed = Trueelse:errors = payload.get("execution_errors") or ["Tier 3 evaluation did not produce a complete scored run"]@@ -689,13 +697,19 @@ def build_agent_eval_payload(metrics_for_condition,)+ mode = _evaluation_mode(agents)+ baseline_only = mode == "baseline_only"+ # A baseline-only run (--skip-with-skill) scores only its without-skill arm,+ # which then supplies the metric contract, ranking and overall score.+ scored_condition = "without_skill" if baseline_only else "with_skill"+ score_field = "baseline" if baseline_only else "with_skill"# Custom-only agents publish their grader's own overall under the custom# policy and rank against standard agents. The run-level policy, its# conflict checks and the metric-set comparison cover standard agents.custom_only_agents = {name for name, info in agents.items() if _is_custom_only_agent(info)}policy_agents = {name: info for name, info in agents.items() if name not in custom_only_agents} or agents- metrics = metrics_for_agents(policy_agents)- policy_metrics = _policy_metrics_from_agents(policy_agents, metrics)+ metrics = metrics_for_agents(policy_agents, condition=scored_condition)+ policy_metrics = _policy_metrics_from_agents(policy_agents, metrics, condition=scored_condition)from skillevaluator.tier3.harbor.metrics import (CUSTOM_SCORE_POLICY,@@ -740,28 +754,34 @@ def build_agent_eval_payload(metrics_for_condition(info, "without_skill"),model,CUSTOM_SCORE_POLICY if name in custom_only_agents else policy["score_policy"],+ baseline_only=baseline_only,)if not agent_payloads:return Nonescored_metric_sets = {- frozenset(agent["metrics_with_skill"])+ frozenset(agent[f"metrics_{scored_condition}"])for name, agent in agent_payloads.items()if name not in custom_only_agentsand agent["execution_status"] == "succeeded"- and agent["with_skill"] is not None+ and agent[score_field] is not None}conflicting_metric_sets = len(scored_metric_sets) > 1if conflicting_metric_sets and not conflicting_policies:policy["score_definition"] = "overall unavailable: incomparable metric sets across agents"- best_agent = "" if conflicting_policies or conflicting_metric_sets else _pick_best_agent(agent_payloads)+ best_agent = (+ ""+ if conflicting_policies or conflicting_metric_sets+ else _pick_best_agent(agent_payloads, score_field=score_field)+ )detail_priority = ([best_agent] if best_agent else []) + [name for name in agent_payloads if name != best_agent]for name in detail_priority:_attach_agent_report_details(agent_payloads[name],agents.get(name, {}),report_budget,+ baseline_only=baseline_only,)best = agent_payloads.get(best_agent, {})if suggestions_v2 is None and run_dir is not None:@@ -793,7 +813,7 @@ def build_agent_eval_payload(execution_status = "failed"execution_errors.append("Incomparable metric sets across scored agents; overall ranking is unavailable")elif policy["score_policy"] not in SUPPORTED_SCORE_POLICIES and any(- agent.get("execution_status") == "succeeded" and agent.get("with_skill") is None+ agent.get("execution_status") == "succeeded" and agent.get(score_field) is Nonefor agent in agent_payloads.values()):execution_status = "failed"@@ -801,10 +821,15 @@ def build_agent_eval_payload(# Count score-policy errors appended above as displayed diagnostics.execution_error_details = _aggregate_execution_error_details(agent_payloads, len(execution_errors))- raw_overall_score = best.get("with_skill")+ raw_overall_score = best.get(score_field)overall_score = _finite_float(raw_overall_score) if execution_status == "succeeded" else Noneoverall_lift = _finite_float(best.get("lift"))- verdict = _overall_verdict_from_agents(agent_payloads) if overall_score is not None else VERDICT_NEUTRAL+ # Baseline calibration scores measure no skill, so they never set a skill verdict.+ verdict = (+ _overall_verdict_from_agents(agent_payloads)+ if overall_score is not None and not baseline_only+ else VERDICT_NEUTRAL+ )metric_ids = list(best.get("evaluators", {}).keys())metric_labels = _metric_labels(metric_ids)@@ -843,6 +868,7 @@ def build_agent_eval_payload("verdict": verdict,"skill_activation": best.get("skill_activation", {}),"skill_name": skill_name,+ "mode": mode,"best_agent": best_agent,"agents_run": list(agent_payloads.keys()),"overall_score": round(overall_score, 4) if overall_score is not None else None,@@ -883,6 +909,13 @@ def build_agent_eval_payload(failure_message = "; ".join(execution_errors) or "Tier 3 evaluation did not produce a complete scored run"deterministic_conclusions = [{"severity": "fail", "title": "Evaluation incomplete", "message": failure_message}]deterministic_suggestions = [failure_message]+ elif baseline_only:+ # A deliberate baseline-only run is complete, not failed: state the+ # calibration result so it is never read as a skill-lift measurement.+ deterministic_conclusions = _build_baseline_only_conclusions(best_agent, overall_score, best_dimensions)+ deterministic_suggestions = [+ "Run without --skip-with-skill to evaluate the with-skill arm and measure skill lift against this baseline."+ ]elif verdict == VERDICT_INCONCLUSIVE:deterministic_conclusions = [{@@ -917,6 +950,7 @@ def build_agent_eval_payload("schema_version": _SCHEMA_VERSION,"summary": summary,"skill_name": skill_name,+ "mode": mode,"verdict": verdict,"skill_activation": summary["skill_activation"],"best_agent": best_agent,@@ -956,7 +990,7 @@ def build_agent_eval_payload("recommendations": recommendations,"suggestions": list(deterministic_suggestions),"suggestions_v2": _attach_harbor_evidence_to_suggestions_v2(- [] if _never_activated(best) else (suggestions_v2 or []),+ [] if _never_activated(best) or baseline_only else (suggestions_v2 or []),best_evidence_links,),"metric_ids": metric_ids,@@ -984,7 +1018,9 @@ def build_agent_eval_payload(payload,deterministic_conclusions=deterministic_conclusions,deterministic_suggestions=deterministic_suggestions,- use_llm_judge=use_llm_judge and overall_score is not None and not _never_activated(best),+ # The insights judge reasons about with-skill quality, which a+ # baseline-only run never measured.+ use_llm_judge=use_llm_judge and overall_score is not None and not _never_activated(best) and not baseline_only,)if evidence_links:payload["conclusions"] = _attach_harbor_evidence_to_conclusions(@@ -1493,6 +1529,8 @@ def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _Repordef _policy_metrics_from_agents(agents: dict[str, dict[str, Any]],metrics: list[str],+ *,+ condition: str = "with_skill",) -> tuple[str, ...]:from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS@@ -1500,10 +1538,10 @@ def _policy_metrics_from_agents(metricfor metric in (metrics or DEFAULT_METRICS)if any(- _condition_quality_available(info, "with_skill")- and (_finite_float(scores.get(metric)) is not None or metric in _arm_not_applicable(info, "with_skill"))+ _condition_quality_available(info, condition)+ and (_finite_float(scores.get(metric)) is not None or metric in _arm_not_applicable(info, condition))for info in agents.values()- for scores in (info.get("with_skill"),)+ for scores in (info.get(condition),)if isinstance(scores, dict)))@@ -1829,6 +1867,8 @@ def _build_agent(baseline_metrics: list[str],model: str | None,score_policy: str,+ *,+ baseline_only: bool = False,) -> dict[str, Any]:from skillevaluator.tier3.harbor.metrics import (CUSTOM_SCORE_POLICY,@@ -1869,12 +1909,13 @@ def _build_agent(with_not_applicable = _arm_not_applicable(info, "with_skill") if with_quality_available else []baseline_not_applicable = _arm_not_applicable(info, "without_skill") if baseline_quality_available else []evaluators = _build_evaluators(- with_metrics,+ baseline_metrics if baseline_only else with_metrics,with_scores,without_scores,lift_data,with_not_applicable=with_not_applicable,baseline_not_applicable=baseline_not_applicable,+ baseline_only=baseline_only,)dimensions = _build_dimensions(with_scores,@@ -1886,9 +1927,17 @@ def _build_agent(baseline_metrics=baseline_metrics,with_not_applicable=with_not_applicable,baseline_not_applicable=baseline_not_applicable,+ baseline_only=baseline_only,)- not_applicable_evaluators = _not_applicable_evaluators(with_metrics, with_scores, with_not_applicable)- not_applicable_dimensions = _not_applicable_dimensions(dimensions, with_not_applicable)+ # N/A notes describe the scored arm: the baseline in a baseline-only run.+ if baseline_only:+ not_applicable_evaluators = _not_applicable_evaluators(+ baseline_metrics, without_scores, baseline_not_applicable+ )+ not_applicable_dimensions = _not_applicable_dimensions(dimensions, baseline_not_applicable)+ else:+ not_applicable_evaluators = _not_applicable_evaluators(with_metrics, with_scores, with_not_applicable)+ not_applicable_dimensions = _not_applicable_dimensions(dimensions, with_not_applicable)if never_activated:for evaluator in evaluators.values():evaluator["lift"] = None@@ -2065,6 +2114,8 @@ def _attach_agent_report_details(agent_payload: dict[str, Any],info: dict[str, Any],report_budget: _ReportBudget,+ *,+ baseline_only: bool = False,) -> None:"""Populate bounded diagnostic details after the best agent is known.@@ -2078,11 +2129,13 @@ def _attach_agent_report_details(custom_lift = info.get("custom_lift")agent_payload["evaluator_cards"] = _evaluator_cards(agent_payload.get("evaluators", {}),- rewards=info.get("rewards") or [],+ # Baseline-only cards draw their evidence from the only arm that ran.+ rewards=(info.get("rewards_baseline") if baseline_only else info.get("rewards")) or [],custom_with_skill=custom_with_skill if isinstance(custom_with_skill, dict) else {},custom_without_skill=custom_without_skill if isinstance(custom_without_skill, dict) else {},custom_lift=custom_lift if isinstance(custom_lift, dict) else {},report_budget=report_budget,+ baseline_only=baseline_only,)if _never_activated(agent_payload):for card in agent_payload["evaluator_cards"]:@@ -2097,12 +2150,27 @@ def _build_evaluators(*,with_not_applicable: list[str] = (),baseline_not_applicable: list[str] = (),+ baseline_only: bool = False,) -> dict[str, dict[str, Any]]:evaluators = {}mixed = lift_data.get("judge_mixed_cases") or {}paired = isinstance(lift_data.get("basis"), dict) or lift_data.get("comparison_unavailable") is Truefor metric in metrics:ws, bl = _finite_float(with_scores.get(metric)), _finite_float(without_scores.get(metric))+ baseline_status = (+ "scored" if bl is not None else ("not_applicable" if metric in baseline_not_applicable else "missing")+ )+ if baseline_only:+ # No with-skill arm ran: the baseline score is the calibration row,+ # and with-skill/lift stay unavailable rather than becoming 0.0.+ evaluators[metric] = {+ "with_skill": None,+ "baseline": bl,+ "lift": None,+ "with_skill_status": "skipped",+ "baseline_status": baseline_status,+ }+ continuelift = _lift_value(metric, lift_data)if lift is None and ws is not None and bl is not None and metric not in mixed and not paired:lift = round(ws - bl, 4)@@ -2113,9 +2181,7 @@ def _build_evaluators("with_skill_status": "scored"if ws is not Noneelse ("not_applicable" if metric in with_not_applicable else "missing"),- "baseline_status": "scored"- if bl is not None- else ("not_applicable" if metric in baseline_not_applicable else "missing"),+ "baseline_status": baseline_status,}return evaluators@@ -2131,6 +2197,7 @@ def _build_dimensions(baseline_metrics: list[str] | None = None,with_not_applicable: list[str] = (),baseline_not_applicable: list[str] = (),+ baseline_only: bool = False,) -> list[dict[str, Any]]:dimensions: list[dict[str, Any]] = []for dim_id in _DIMENSION_IDS:@@ -2143,6 +2210,10 @@ def _build_dimensions()if ws is None and bl is None:continue+ if baseline_only:+ if bl is not None:+ dimensions.append(_baseline_only_dimension(dim_id, cfg, bl, without_scores, precomputed_without))+ continuelift = round(ws - bl, 4) if ws is not None and bl is not None else Noneentry = (precomputed_with.get(dim_id) if ws is not None and isinstance(precomputed_with.get(dim_id), dict) else {}@@ -2215,6 +2286,39 @@ def _build_dimensions(return dimensions+def _baseline_only_dimension(+ dim_id: str,+ cfg: dict[str, Any],+ bl: float,+ without_scores: dict[str, Any],+ precomputed_without: dict[str, Any],+) -> dict[str, Any]:+ """Describe one dimension of a baseline-only run (--skip-with-skill).++ The baseline is the measurement here, not a comparison point: it drives the+ reasoning and band, while with-skill and lift stay unavailable.+ """+ entry = precomputed_without.get(dim_id) if isinstance(precomputed_without.get(dim_id), dict) else {}+ signals = _dimension_signals(entry, without_scores, cfg)+ explanation = entry.get("explanation")+ reasoning_bullets = entry.get("reasoning_bullets")+ if not reasoning_bullets and not explanation:+ reasoning_bullets, explanation = _deterministic_reasoning(+ bl, None, None, signals, without_scores, baseline_only=True+ )+ return {+ "id": dim_id,+ "with_skill": None,+ "score": None,+ "baseline": round(bl, 4),+ "lift": None,+ "explanation": explanation,+ "verdict": entry.get("verdict") or _deterministic_verdict(bl),+ "evaluators": signals,+ "reasoning_bullets": reasoning_bullets or [],+ }++def _dimension_signals(entry: dict[str, Any], with_scores: dict[str, Any], cfg: dict[str, Any]) -> list[str]:"""Return the evaluator signals that feed a dimension (Signals column).@@ -2238,6 +2342,8 @@ def _deterministic_reasoning(lift: float | None,signals: list[str],with_scores: dict[str, Any],+ *,+ baseline_only: bool = False,) -> tuple[list[str], str]:"""Build deterministic reasoning bullets for a dimension (SkillEvaluator parity).@@ -2268,6 +2374,9 @@ def _deterministic_reasoning(lift=lift,parts=parts,)+ if baseline_only:+ # ``ws`` carries the baseline score; replace the "no baseline" lift note.+ bullets[1] = "Baseline-only run: no with-skill arm ran, so no skill lift was measured."return bullets, " ".join(bullets)@@ -2588,8 +2697,14 @@ def _evaluator_card(label: str,rewards: list[dict[str, Any]],report_budget: _ReportBudget,+ baseline_only: bool = False,) -> dict[str, Any]:- ws = _as_float(scores.get("with_skill"))+ if baseline_only:+ # No with-skill arm ran: the baseline score sets the card's status band.+ ws = None+ headline = _as_float(scores.get("baseline"))+ else:+ ws = headline = _as_float(scores.get("with_skill"))evidence_sampling: dict[str, Any] = {}card = {"id": metric,@@ -2597,7 +2712,7 @@ def _evaluator_card("with_skill": ws,"baseline": scores.get("baseline"),"lift": scores.get("lift"),- "status": "pass" if ws >= 0.8 else ("warn" if ws >= 0.6 else "fail"),+ "status": "pass" if headline >= 0.8 else ("warn" if headline >= 0.6 else "fail"),"evidence": _metric_evidence(metric, rewards, report_budget, evidence_sampling),}if evidence_sampling:@@ -2613,12 +2728,17 @@ def _evaluator_cards(custom_without_skill: dict[str, Any],custom_lift: dict[str, Any],report_budget: _ReportBudget,+ baseline_only: bool = False,) -> list[dict[str, Any]]:from skillevaluator.tier3.harbor.metrics import METRIC_DISPLAYcards: list[dict[str, Any]] = []+ # Cards cover scored evaluators; a baseline-only run is scored by its baseline arm.+ headline_field = "baseline" if baseline_only else "with_skill"evaluator_items = [- (metric, values) for metric, values in evaluators.items() if _finite_float(values.get("with_skill")) is not None+ (metric, values)+ for metric, values in evaluators.items()+ if _finite_float(values.get(headline_field)) is not None]for evaluator_index, (metric, scores) in enumerate(evaluator_items):if report_budget.cards_remaining <= 0:@@ -2632,18 +2752,32 @@ def _evaluator_cards(label=METRIC_DISPLAY.get(metric, metric.replace("_", " ").title()),rewards=rewards,report_budget=report_budget,+ baseline_only=baseline_only,))custom_scores = _discover_custom_metric_scores(- custom_with_skill,+ custom_without_skill if baseline_only else custom_with_skill,rewards,set(evaluators),report_budget.cards_remaining,report_budget,)- for metric, with_skill in custom_scores.items():+ for metric, score in custom_scores.items():report_budget.cards_remaining -= 1+ if baseline_only:+ cards.append(+ _evaluator_card(+ metric,+ {"with_skill": None, "baseline": score, "lift": None},+ label=f"Custom: {metric}",+ rewards=rewards,+ report_budget=report_budget,+ baseline_only=True,+ )+ )+ continue+ with_skill = scorebaseline = _custom_metric_score(metric, custom_without_skill, [])lift = _lift_value(metric, custom_lift)if lift is None and baseline is not None:@@ -3566,11 +3700,11 @@ def _overall_verdict_from_agents(agents: dict[str, dict[str, Any]]) -> str:return VERDICT_FAIL-def _canonical_agent_rank(agent: dict[str, Any]) -> tuple[float, float] | None:+def _canonical_agent_rank(agent: dict[str, Any], *, score_field: str = "with_skill") -> tuple[float, float] | None:"""Return the score/lift tuple used to rank one canonical agent payload."""if agent.get("execution_status") != "succeeded":return None- score = _finite_float(agent.get("with_skill"))+ score = _finite_float(agent.get(score_field))if score is None:return Nonereturn score, _as_float(agent.get("lift"))@@ -3610,8 +3744,68 @@ def _canonical_agent_rank_from_info(return _canonical_agent_rank(agent)-def _pick_best_agent(agents: dict[str, dict[str, Any]]) -> str:- eligible = {name: rank for name, agent in agents.items() if (rank := _canonical_agent_rank(agent)) is not None}+def _evaluation_mode(agents: dict[str, dict[str, Any]]) -> str:+ """Classify the arms a run executed: ``both``, ``with_skill_only`` or ``baseline_only``.++ Read from the collector's deliberate per-arm ``execution_status="skipped"``+ markers, never inferred from empty scores, which can also mean a failed arm.+ """++ def arm_statuses(condition: str) -> list[object]:+ statuses: list[object] = []+ for info in agents.values():+ conditions = info.get("conditions")+ arm = conditions.get(condition) if isinstance(conditions, dict) else None+ statuses.append(arm.get("execution_status") if isinstance(arm, dict) else None)+ return statuses++ with_statuses = arm_statuses("with_skill")+ without_statuses = arm_statuses("without_skill")+ if with_statuses and all(status == "skipped" for status in with_statuses):+ return "baseline_only"+ if without_statuses and all(status in {None, "skipped"} for status in without_statuses):+ return "with_skill_only"+ return "both"+++def _build_baseline_only_conclusions(+ best_agent: str,+ overall_score: float,+ dimensions: list[dict[str, Any]],+) -> list[dict[str, str]]:+ """Insights for a baseline-only run: the calibration result, never a lift claim."""+ conclusions = [+ {+ "severity": "pass",+ "title": "Baseline calibration complete",+ "message": (+ f"Baseline-only run (--skip-with-skill): {best_agent} scores {overall_score:.2f} without the "+ "skill. No with-skill arm was run, so these are baseline calibration scores and no skill lift "+ "was measured."+ ),+ }+ ]+ numeric_dims = [d for d in dimensions if _finite_float(d.get("baseline")) is not None]+ if numeric_dims:+ weakest = min(numeric_dims, key=lambda d: d["baseline"])+ conclusions.append(+ {+ "severity": "fail" if weakest["baseline"] < 0.4 else "warn",+ "title": "Weakest baseline dimension",+ "message": (+ f"{weakest.get('id', 'unknown').title()} is lowest at {weakest['baseline']:.2f} without the skill."+ ),+ }+ )+ return conclusions+++def _pick_best_agent(agents: dict[str, dict[str, Any]], *, score_field: str = "with_skill") -> str:+ eligible = {+ name: rank+ for name, agent in agents.items()+ if (rank := _canonical_agent_rank(agent, score_field=score_field)) is not None+ }activated = {name: rank for name, rank in eligible.items() if not _never_activated(agents[name])}eligible = activated or eligibleif not eligible:diff --git a/src/skillevaluator/reporting/cli.py b/src/skillevaluator/reporting/cli.pyindex 831b23f3..e52cf543 100644--- a/src/skillevaluator/reporting/cli.py+++ b/src/skillevaluator/reporting/cli.py@@ -388,10 +388,14 @@ class CLIReporter(ReporterBase):ws = scores.get("with_skill")bl = scores.get("baseline")lift = scores.get("lift")+ # A baseline-only run (--skip-with-skill) never ran the with-skill arm.+ with_skipped = scores.get("with_skill_status") == "skipped"if lift is None:lift_text = ("N/A"if "not_applicable" in (scores.get("with_skill_status"), scores.get("baseline_status"))+ else "skipped"+ if with_skippedelse "NO SCORE")else:@@ -403,6 +407,8 @@ class CLIReporter(ReporterBase):if ws is not Noneelse "N/A"if scores.get("with_skill_status") == "not_applicable"+ else "skipped"+ if with_skippedelse "NO SCORE",f"{bl:.2f}"if bl is not Nonediff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.pyindex 7c28901d..c3d838a9 100644--- a/src/skillevaluator/reporting/html.py+++ b/src/skillevaluator/reporting/html.py@@ -967,6 +967,10 @@ def _sanitize_tier3_display_payload(value: dict[str, Any]) -> dict[str, Any]:"baseline": _finite_number(scores.get("baseline"), minimum=0.0, maximum=1.0),"lift": _finite_number(scores.get("lift"), minimum=-1.0, maximum=1.0),}+ if scores.get("with_skill_status") == "skipped":+ # A baseline-only run (--skip-with-skill) never ran this arm;+ # the table says so instead of reporting a missing score.+ evaluators[name]["with_skill_status"] = "skipped"payload["evaluators"] = evaluatorsfor key in ("insights",@@ -2488,6 +2492,9 @@ class HTMLReporter(ReporterBase):score = _finite_number(agent.get("with_skill"), minimum=0.0, maximum=1.0)if score is None:score = _finite_number(agent.get("overall_score"), minimum=0.0, maximum=1.0)+ if score is None and candidate.get("mode") == "baseline_only":+ # A baseline-only run (--skip-with-skill) is scored by its baseline arm.+ score = _finite_number(agent.get("baseline"), minimum=0.0, maximum=1.0)attempts = max(payload_attempts, _nonnegative_count(agent.get("scored_attempts")))if score is not None and attempts > 0:return Truediff --git a/src/skillevaluator/reporting/markdown.py b/src/skillevaluator/reporting/markdown.pyindex 3d8dcdcf..2efdcda4 100644--- a/src/skillevaluator/reporting/markdown.py+++ b/src/skillevaluator/reporting/markdown.py@@ -352,9 +352,13 @@ class MarkdownReporter(ReporterBase):ws = scores.get("with_skill")bl = scores.get("baseline")lift = scores.get("lift")+ # A baseline-only run (--skip-with-skill) never ran the with-skill arm.+ with_skipped = scores.get("with_skill_status") == "skipped"ws_text = ("N/A"if scores.get("with_skill_status") == "not_applicable" and ws is None+ else "skipped"+ if with_skipped and ws is Noneelse "NO SCORE"if ws is Noneelse _format_report_number(ws)@@ -371,6 +375,8 @@ class MarkdownReporter(ReporterBase):if lift is not Noneelse "N/A"if "not_applicable" in (scores.get("with_skill_status"), scores.get("baseline_status"))+ else "skipped"+ if with_skippedelse "NO SCORE")lines.append(diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index ef205e34..f8452197 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -1348,9 +1348,10 @@{% set t3_key = tier3_verdict if tier3_verdict in ['pass', 'fail', 'neutral'] else 'incomplete' %}<span class="verdict-chip {{ {'pass': 'pass', 'fail': 'fail'}.get(t3_key, 'warn') }}"><span class="sr-only">Tier 3 verdict: </span>{{ advisory_fail_word if tier3_advisory_fail else verdict_words[t3_key] }}</span>{% if t3_key == 'incomplete' and tier3_status_reason %}<p class="hero-reason">{{ tier3_status_reason }}</p>{% endif %}- <div class="agent-hero-score {{ 'score-pass' if hero_score is not none and hero_score >= 0.8 else ('score-warn' if hero_score is not none and hero_score >= 0.6 else 'score-fail') }}">{{ "%.2f" | format(hero_score) if hero_score is not none else "N/A" }}<span class="hero-score-label">overall score</span></div>+ <div class="agent-hero-score {{ 'score-pass' if hero_score is not none and hero_score >= 0.8 else ('score-warn' if hero_score is not none and hero_score >= 0.6 else 'score-fail') }}">{{ "%.2f" | format(hero_score) if hero_score is not none else "N/A" }}<span class="hero-score-label">{{ 'baseline score' if tier3.get('mode') == 'baseline_only' else 'overall score' }}</span></div><div class="agent-hero-detail"><div><strong>Best performing agent:</strong> {{ tier3.best_agent or 'N/A' }}</div>+ {% if tier3.get('mode') == 'baseline_only' %}<div>Baseline-only run (<code>--skip-with-skill</code>): no with-skill arm, so no skill lift was measured.</div>{% endif %}{% if completed_agent and completed_agent.needs_attention %}<div class="score-warn">Completed-agent result. Other execution failures need attention; see agent details.</div>{% endif %}{% if completed_agent and completed_agent.recovered_attempts %}<div>{{ completed_agent.recovered_attempts }} timeout attempt(s) recovered; failed history retained.</div>{% endif %}<div><strong>Trials:</strong> {{ hero_trials }} across {{ (tier3.agents_run or []) | length }} agent(s)</div>@@ -2465,6 +2466,12 @@{% elif tier3_verdict == 'inconclusive' %}<strong>Inconclusive: skill never activated.</strong>Scores are diagnostic only; no lift verdict was computed. Check skill loading and agent support, then rerun the evaluation.+ {% elif tier3.get('mode') == 'baseline_only' %}+ <strong>Baseline-only calibration:</strong>+ without-skill scores only, with a baseline calibration score of+ {{ "%.2f" | format(tier3.overall_score) if tier3.overall_score is not none else "N/A" }}.+ {% if t3_best_name %}Best performing agent is <strong>{{ t3_best_name }}</strong>.{% else %}No successfully scored agent is available.{% endif %}+ No with-skill arm was run (<code>--skip-with-skill</code>), so no skill lift was measured.{% else %}{% if completed_agent and completed_agent.needs_attention %}<strong>Completed-agent result: {{ completed_agent.verdict | upper }}.</strong>@@ -2496,17 +2503,18 @@<div class="dashboard-card {{ 'success' if tier3_verdict == 'pass' else ('danger' if tier3_verdict == 'fail' else '') }}"><h3 class="dashboard-card-title">{{ 'Quality Verdict' if report_scope == 'tier3' and completed_agent else 'Verdict' }}</h3><p class="dashboard-card-value">{{ verdict_words['skipped'] if tier3_is_skipped else (advisory_fail_word if tier3_advisory_fail else (verdict_words.get(tier3_verdict) or tier3_verdict | title)) }}</p>- <p class="dashboard-card-subtitle">{{ 'Completed-agent quality; lift shown separately' if report_scope == 'tier3' and completed_agent else 'Evaluation outcome' }}</p>+ <p class="dashboard-card-subtitle">{{ "No skill verdict: baseline-only run" if tier3.get('mode') == 'baseline_only' else ('Completed-agent quality; lift shown separately' if report_scope == 'tier3' and completed_agent else 'Evaluation outcome') }}</p></div><div class="dashboard-card"><h3 class="dashboard-card-title">{{ 'Completed Agent Score' if completed_agent and completed_agent.needs_attention else 'Overall Score' }}</h3><p class="dashboard-card-value">{{ "%.2f" | format(hero_score) if hero_score is not none else "N/A" }}</p>- <p class="dashboard-card-subtitle">Best run across evaluated dimensions</p>+ <p class="dashboard-card-subtitle">{{ "Baseline calibration (no with-skill arm)" if tier3.get('mode') == 'baseline_only' else "Best run across evaluated dimensions" }}</p></div><div class="dashboard-card"><h3 class="dashboard-card-title">Best Performing Agent</h3><p class="dashboard-card-value" style="font-size:1.35rem;">{{ t3_best_name or 'N/A' }}</p>- <p class="dashboard-card-subtitle">Score {{ "%.2f" | format(t3_best.with_skill) if t3_best and t3_best.with_skill is not none else "N/A" }}</p>+ {% set t3_best_score = (t3_best.baseline if tier3.get('mode') == 'baseline_only' else t3_best.with_skill) if t3_best else none %}+ <p class="dashboard-card-subtitle">Score {{ "%.2f" | format(t3_best_score) if t3_best_score is not none else "N/A" }}</p></div><div class="dashboard-card"><h3 class="dashboard-card-title">Skill Lift</h3>@@ -3653,10 +3661,10 @@{% for name, scores in tier3.evaluators.items() %}<tr><td><strong>{{ name | replace("_", " ") | title }}</strong></td>- <td>{{ "%.2f" | format(scores.with_skill) if scores.with_skill is not none else ("N/A" if scores.with_skill_status == "not_applicable" else "NO SCORE") }}</td>+ <td>{{ "%.2f" | format(scores.with_skill) if scores.with_skill is not none else ("N/A" if scores.with_skill_status == "not_applicable" else ("skipped" if scores.with_skill_status == "skipped" else "NO SCORE")) }}</td><td>{{ "%.2f" | format(scores.baseline) if scores.baseline is not none else ("N/A" if scores.baseline_status == "not_applicable" else "NO SCORE") }}</td><td style="color:{{ 'var(--success-color)' if scores.lift is not none and scores.lift >= tier3_lift_pass_threshold else ('var(--danger-color)' if scores.lift is not none and scores.lift <= tier3_lift_fail_threshold else 'var(--text-secondary)') }};font-weight:600;">- {{ "%+.2f" | format(scores.lift) if scores.lift is not none else ("N/A" if "not_applicable" in [scores.with_skill_status, scores.baseline_status] else "NO SCORE") }}+ {{ "%+.2f" | format(scores.lift) if scores.lift is not none else ("N/A" if "not_applicable" in [scores.with_skill_status, scores.baseline_status] else ("skipped" if scores.with_skill_status == "skipped" else "NO SCORE")) }}</td></tr>{% endfor %}diff --git a/src/skillevaluator/tier3/case_summary.py b/src/skillevaluator/tier3/case_summary.pyindex dd3ad3f2..748a05f1 100644--- a/src/skillevaluator/tier3/case_summary.py+++ b/src/skillevaluator/tier3/case_summary.py@@ -70,6 +70,13 @@ def _never_activated(agent: Mapping[str, Any]) -> bool:return isinstance(activation, Mapping) and activation.get("status") == "never_activated"+def _with_skill_skipped(agent: Mapping[str, Any]) -> bool:+ """A baseline-only run (--skip-with-skill) deliberately never ran the with-skill arm."""+ conditions = agent.get("conditions")+ arm = conditions.get("with_skill") if isinstance(conditions, Mapping) else None+ return isinstance(arm, Mapping) and arm.get("execution_status") == "skipped"++def _details_truncated(agent: Mapping[str, Any]) -> bool:branches = agent.get("pass_at_k")for key, _ in _VARIANTS:@@ -226,6 +233,10 @@ def _pct(passed: int, total: int) -> float | None:def verdict(agents: Mapping[str, Any]) -> tuple[str, str]:"""Return (label, per-agent detail) using the raw per-attempt success rate."""+ scored = [agent for agent in agents.values() if isinstance(agent, Mapping)]+ if scored and all(_with_skill_skipped(agent) for agent in scored):+ # Baseline calibration: no with-skill arm ran, so there is no effect to call.+ return "baseline only", "no with-skill arm ran (--skip-with-skill), so no skill effect was measured"deltas: dict[str, float] = {}not_run: dict[str, int] = {}unavailable: list[str] = []@@ -377,6 +388,7 @@ def _condition_grid("""One agent: a row per case with both conditions side by side and the effect."""[agent] = agents.values()comparable = _comparable(agent)+ with_skill_skipped = _with_skill_skipped(agent)table = _table(title, "with skill", "without skill", "effect")for case_id in case_ids:with_case, without_case = _cases(agent, "with_skill").get(case_id), _cases(agent, "without_skill").get(case_id)@@ -384,7 +396,10 @@ def _condition_grid(Text(safe(case_id)),_condition_cell(with_case),_condition_cell(without_case),- _effect(with_case, without_case, comparable=_case_comparable(agent, with_case, without_case)),+ # No with-skill arm ran (--skip-with-skill), so there is no effect to call.+ Text("-", style="dim")+ if with_skill_skipped+ else _effect(with_case, without_case, comparable=_case_comparable(agent, with_case, without_case)),)table.add_section()rates = _agent_rates(agent)diff --git a/src/skillevaluator/tier3/commands.py b/src/skillevaluator/tier3/commands.pyindex e84b68dd..a965f2d5 100644--- a/src/skillevaluator/tier3/commands.py+++ b/src/skillevaluator/tier3/commands.py@@ -662,6 +662,7 @@ def evaluate(env_mode: str,environment_kwarg: tuple[str, ...] = (),skip_baseline: bool,+ skip_with_skill: bool = False,n_attempts: int | None,pass_threshold: float | None,stop_on_pass: bool | None = None,@@ -710,6 +711,7 @@ def evaluate(environment=env_mode,agents=tuple(agent_list),baseline=not skip_baseline,+ with_skill=not skip_with_skill,attempts=n_attempts,concurrency=n_concurrent,max_agents=max_agents,@@ -717,6 +719,8 @@ def evaluate())reporter.emit(ProgressEvent(stage="configuration", state="running"))+ if skip_baseline and skip_with_skill:+ raise ValueError("--skip-baseline and --skip-with-skill are mutually exclusive (no arm would run)")unknown = validate_agents(agent_list)if unknown:supported = ", ".join(sorted(HARBOR_AGENTS_SUPPORTED))@@ -745,6 +749,7 @@ def evaluate(skill_path=skill_path.resolve(),agents=agent_list,skip_baseline=skip_baseline,+ skip_with_skill=skip_with_skill,n_attempts=n_attempts,pass_threshold=pass_threshold,stop_on_pass=stop_on_pass,diff --git a/src/skillevaluator/tier3/harbor/collector.py b/src/skillevaluator/tier3/harbor/collector.pyindex d1efd1d9..e5d00d33 100644--- a/src/skillevaluator/tier3/harbor/collector.py+++ b/src/skillevaluator/tier3/harbor/collector.py@@ -302,6 +302,17 @@ def _reset_agent_generated_outputs(agent_dir: Path, output_root: Path) -> None:_remove_generated_output_path(path, output_root)+def _skipped_with_skill_activation() -> dict[str, Any]:+ """Activation summary for a baseline-only run, which never ran the skill."""+ return {+ "status": "not_applicable",+ "positive_trials": 0,+ "activated_trials": 0,+ "unknown_trials": 0,+ "reason": "With-skill arm was skipped; skill activation was not measured",+ }++def _find_job_dir(jobs_dir: Path, job_name: str) -> Path | None:"""Find the exact Harbor job directory produced for ``job_name``."""candidate = jobs_dir / job_name@@ -6851,6 +6862,7 @@ def collect_harbor_results(jobs_dir: Path,*,skip_baseline: bool = False,+ skip_with_skill: bool = False,n_attempts: int = 1,pass_threshold: float = 0.50,stop_on_pass: bool = False,@@ -6866,7 +6878,11 @@ def collect_harbor_results("""Collect results from Harbor jobs into evals/results/<agent>/ structure.Returns a dict with per-agent scores, lift, and a cross-agent comparison.+ ``skip_with_skill`` collects only the baseline arm of a baseline-only run:+ the with-skill arm is recorded as skipped and no lift is computed."""+ if skip_baseline and skip_with_skill:+ raise ValueError("skip_baseline and skip_with_skill are mutually exclusive (no arm would run)")if expected_trials is not None and expected_total_trials is not None and expected_trials != expected_total_trials:raise ValueError("Conflicting expected trial counts were provided")if expected_trials is None:@@ -6900,7 +6916,8 @@ def collect_harbor_results(agent_dir = output_dir / agentwith_job_name = f"{skill_name}-{agent}-with"- with_job_dir = _find_job_dir(jobs_dir, with_job_name)+ # A skipped arm never adopts a stale job left in a reused jobs directory.+ with_job_dir = None if skip_with_skill else _find_job_dir(jobs_dir, with_job_name)with_collected_rewards: list[dict[str, Any]] = []with_rewards: list[dict[str, Any]] = []@@ -7054,7 +7071,7 @@ def collect_harbor_results(len(with_logical_rewards),with_scores,)- else:+ elif not skip_with_skill:with_job_failure = f"No Harbor job found for {with_job_name}"logger.warning("No Harbor job found for %s (with-skill)", with_job_name)prefix = f"{agent} with-skill Harbor run failed: "@@ -7095,9 +7112,12 @@ def collect_harbor_results(n_attempts=n_attempts,job_failure=with_job_failure,runtime_failures=with_runtime_failures,+ skipped=skip_with_skill,stop_on_pass=stop_on_pass,pass_threshold=pass_threshold,)+ # A skipped with-skill arm still writes its summary: report loading keys+ # agent presence on it, and its "skipped" status marks a baseline-only run.if with_job_dir is None:summary_dir = agent_dir / "with-skill"summary_dir.mkdir(parents=True, exist_ok=True)@@ -7120,6 +7140,7 @@ def collect_harbor_results(**with_execution,"job_failure": with_job_failure,"trial_failures": [],+ **({"skill_activation": _skipped_with_skill_activation()} if skip_with_skill else {}),},)@@ -7156,8 +7177,10 @@ def collect_harbor_results(without_runtime_failures = _extract_agent_runtime_failures(without_job_dir)without_trial_failures = _extract_trial_failures(without_job_dir)preserve_partial = _can_preserve_partial_rewards(without_job_dir, without_trial_failures)+ # The runner stages unsuffixed task names when only one arm runs.+ without_arm_suffix = "-without-skill" if not skip_with_skill else ""without_task_entry_ids = (- _staged_task_entry_id_map(output_dir, agent, "without", arm_suffix="-without-skill")+ _staged_task_entry_id_map(output_dir, agent, "without", arm_suffix=without_arm_suffix)if case_id_by_task_selector is Noneelse None)@@ -7165,7 +7188,7 @@ def collect_harbor_results(_extract_rewards(without_job_dir,case_id_by_task_selector,- arm_suffix="-without-skill",+ arm_suffix=without_arm_suffix,task_entry_id_map=without_task_entry_ids,)if without_job_ok or preserve_partial@@ -7188,6 +7211,13 @@ def collect_harbor_results(without_policy, without_policy_ambiguous = _score_policy_for_rewards(without_logical_rewards, without_metrics)+ if skip_with_skill:+ # The baseline is the only scored arm, so it owns the run's metric contract.+ all_results["metric_set"] = without_metric_set+ all_results["metrics"] = list(without_metrics)+ all_results["attempt_policy"]["score_definition"] = _score_definition_for_policy(+ without_metrics, without_policy+ )without_custom_scores = average_custom_metrics(without_logical_rewards)without_pass = _pass_summary(without_logical_rewards,@@ -7373,8 +7403,9 @@ def collect_harbor_results(output_dir,without_summary,)- all_results["score_policy"] = with_policy- all_results["attempt_policy"]["score_policy"] = with_policy+ scored_policy = without_policy if skip_with_skill else with_policy+ all_results["score_policy"] = scored_policy+ all_results["attempt_policy"]["score_policy"] = scored_policyif with_job_dir:all_results["attempt_policy"]["score_definition"] = _score_definition_for_policy(with_metrics, with_policy)policy_conflict = (@@ -7403,7 +7434,9 @@ def collect_harbor_results(with_overall_score = Nonewithout_overall_score = None- skill_activation = summarize_skill_activation(with_rewards)+ skill_activation = (+ _skipped_with_skill_activation() if skip_with_skill else summarize_skill_activation(with_rewards)+ )lift: dict[str, Any] = {}paired_execution_succeeded = (with_execution.get("execution_status") == "succeeded"@@ -7525,7 +7558,7 @@ def collect_harbor_results("overall_without_skill": without_overall_score,"metrics_with_skill": list(with_metrics),"metrics_without_skill": list(without_metrics),- "score_policy_with_skill": with_policy,+ "score_policy_with_skill": with_policy if not skip_with_skill else None,"score_policy_without_skill": without_policy if not skip_baseline else None,"custom_with_skill": with_custom_scores,"custom_without_skill": without_custom_scores,@@ -7577,10 +7610,12 @@ def collect_harbor_results("output_dir": str(agent_dir.resolve()),}+ # A baseline-only run scores only the without-skill arm.+ scored_arm = "without_skill" if skip_with_skill else "with_skill"scored_policies = {- info["score_policy_with_skill"]+ info[f"score_policy_{scored_arm}"]for info in all_results["agents"].values()- if info.get("execution_status") == "succeeded" and info.get("score_policy_with_skill")+ if info.get("execution_status") == "succeeded" and info.get(f"score_policy_{scored_arm}")}# Custom-only agents publish their grader's own overall and rank against# standard agents, so only standard score policies can conflict.@@ -7597,13 +7632,13 @@ def collect_harbor_results(scored_agent = next(infofor info in all_results["agents"].values()- if info.get("execution_status") == "succeeded" and info.get("score_policy_with_skill") == score_policy+ if info.get("execution_status") == "succeeded" and info.get(f"score_policy_{scored_arm}") == score_policy)selected_metrics = tuple(metricfor metric in DEFAULT_METRICS- if metric in scored_agent["with_skill"]- or metric in scored_agent.get("not_applicable_metrics", {}).get("with_skill", ())+ if metric in scored_agent[scored_arm]+ or metric in scored_agent.get("not_applicable_metrics", {}).get(scored_arm, ()))all_results["metric_set"] = (DEFAULT_METRIC_SETdiff --git a/src/skillevaluator/tier3/harbor/progress.py b/src/skillevaluator/tier3/harbor/progress.pyindex 470ce074..cdcca6c7 100644--- a/src/skillevaluator/tier3/harbor/progress.py+++ b/src/skillevaluator/tier3/harbor/progress.py@@ -60,6 +60,7 @@ class Tier3RunPlan:case_count: int | None = Noneattempts: int | None = Nonebaseline: bool | None = None+ with_skill: bool | None = Noneconcurrency: int | None = Nonemax_agents: int | None = Nonetimeout_multiplier: float | None = None@@ -239,6 +240,7 @@ class PlainProgressReporter:("tasks", plan.task_count),("cases", plan.case_count),("attempts", plan.attempts),+ ("with-skill", None if plan.with_skill is None else "yes" if plan.with_skill else "no"),("baseline", None if plan.baseline is None else "yes" if plan.baseline else "no"),("concurrency", plan.concurrency),("max-agents", plan.max_agents),@@ -355,7 +357,10 @@ class RichProgressReporter(PlainProgressReporter):plan = self._live_planif plan is None:return Table(title="Harbor Run Configuration")- mode_label = "Skill-lift evaluation" if plan.baseline is not False else "Agent evaluation"+ if plan.with_skill is False:+ mode_label = "Baseline-only evaluation"+ else:+ mode_label = "Skill-lift evaluation" if plan.baseline is not False else "Agent evaluation"title = Text.assemble(("Harbor Run Configuration", "bold cyan")," · ",@@ -389,6 +394,7 @@ class RichProgressReporter(PlainProgressReporter):("Tasks", plan.task_count),("Cases", plan.case_count),("Attempts", plan.attempts),+ ("With skill", None if plan.with_skill is None else "yes" if plan.with_skill else "no"),("Baseline", None if plan.baseline is None else "yes" if plan.baseline else "no"),("Concurrency", plan.concurrency),("Max agents", plan.max_agents),diff --git a/src/skillevaluator/tier3/harbor/report_data.py b/src/skillevaluator/tier3/harbor/report_data.pyindex 5c8be085..cdb16eb9 100644--- a/src/skillevaluator/tier3/harbor/report_data.py+++ b/src/skillevaluator/tier3/harbor/report_data.py@@ -545,11 +545,11 @@ def metrics_for_condition(agent_info: dict[str, Any], condition: str) -> list[streturn _metrics_for_rewards(rewards) if isinstance(rewards, list) else []-def metrics_for_agents(agents: dict[str, dict[str, Any]]) -> list[str]:+def metrics_for_agents(agents: dict[str, dict[str, Any]], *, condition: str = "with_skill") -> list[str]:"""Return the canonical default or legacy metric set represented by agents."""saw_metrics = Falsefor info in agents.values():- metrics = metrics_for_condition(info, "with_skill")+ metrics = metrics_for_condition(info, condition)if metrics:saw_metrics = Trueif "security" in metrics:diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.pyindex 56079543..c73f0e5d 100644--- a/src/skillevaluator/tier3/harbor/runner.py+++ b/src/skillevaluator/tier3/harbor/runner.py@@ -4364,7 +4364,7 @@ def _run_agent_pair(agent: str,model: str,env_mode: str,- with_skill: Path,+ with_skill: Path | None,baseline: Path | None,jobs_dir: Path,run_env: dict[str, str],@@ -4382,9 +4382,12 @@ def _run_agent_pair(verifier_env: Mapping[str, str] | None = None,environment_kwargs: Mapping[str, Any] | None = None,) -> list[str]:- jobs = [("with", with_skill)]+ # A baseline-only run (``skip_with_skill``) stages no with-skill dataset.+ jobs = [] if with_skill is None else [("with", with_skill)]if baseline is not None:jobs.append(("without", baseline))+ if not jobs:+ return [f"{agent} has no staged Harbor dataset to run"]if stop_on_pass:# A later attempt is launched only after the previous one scored, so# stop-on-pass runs each condition sequentially, one attempt at a time.@@ -4580,6 +4583,7 @@ def _run_harbor_eval_impl(agents: list[str],*,skip_baseline: bool = False,+ skip_with_skill: bool = False,n_attempts: int | None = None,pass_threshold: float | None = None,stop_on_pass: bool | None = None,@@ -4621,6 +4625,10 @@ def _run_harbor_eval_impl(if not agents:reporter.emit(ProgressEvent(stage="configuration", state="failed", detail="no agents selected"))return {"error": ["At least one Harbor agent is required."]}+ if skip_baseline and skip_with_skill:+ detail = "skip_baseline and skip_with_skill are mutually exclusive (no arm would run)"+ reporter.emit(ProgressEvent(stage="configuration", state="failed", detail=detail))+ return {"error": [detail]}if env_mode == ENV_MODE_LOCAL:from skillevaluator.tier3.harbor import local_sandbox@@ -4729,10 +4737,18 @@ def _run_harbor_eval_impl(reporter.emit(ProgressEvent(stage="configuration", state="failed", detail="invalid workspace mode"))return {"error": ["skill_workspace.mode must be isolated or group"]}if not skip_baseline and harbor_config.get("pre_agent_setup"):- detail = (- "harbor.pre_agent_setup/setup_commands cannot run in a paired evaluation; "- "use --skip-baseline for a with-skill-only run"- )+ if skip_with_skill:+ # --skip-baseline cannot be combined with --skip-with-skill, so it+ # is not offered as a way out of a baseline-only run.+ detail = (+ "harbor.pre_agent_setup/setup_commands cannot run in a baseline-only run (--skip-with-skill); "+ "the baseline arm must stay isolated from skill-owned setup commands"+ )+ else:+ detail = (+ "harbor.pre_agent_setup/setup_commands cannot run in a paired evaluation; "+ "use --skip-baseline for a with-skill-only run"+ )reporter.emit(ProgressEvent(stage="configuration", state="failed", detail=detail))return {"error": [detail]}@@ -4770,6 +4786,7 @@ def _run_harbor_eval_impl(provider=provider.provider,attempts=n_attempts,baseline=not skip_baseline,+ with_skill=not skip_with_skill,concurrency=n_concurrent,max_agents=max_agents,timeout_multiplier=float(timeout_multiplier),@@ -5220,26 +5237,35 @@ def _run_harbor_eval_impl(detail="base image build failed; falling back to per-task Dockerfiles",))- agent_task_dirs: dict[str, tuple[Path, Path | None]] = {}+ agent_task_dirs: dict[str, tuple[Path | None, Path | None]] = {}+ # The first staged arm per agent: with-skill tasks, or the baseline tasks+ # of a baseline-only run. Timeout planning and the runtime preflight use it.+ primary_task_dirs: dict[str, Path] = {}staged_task_paths: list[Path] = []expected_task_selectors: list[str] | None = Noneexpected_case_id_by_task_selector: dict[str, str] | None = None+ staging_failure_stage = "baseline-tasks" if skip_with_skill else "with-skill-tasks"+ if skip_with_skill:+ reporter.emit(ProgressEvent(stage="with-skill-tasks", state="skipped", detail="with-skill arm disabled"))reporter.emit(ProgressEvent(- stage="with-skill-tasks",+ stage=staging_failure_stage,state="running",output_dir=str(run_dir),result_path=str(result_path),))- staging_failure_stage = "with-skill-tasks"try:- is_dual_arm = not skip_baseline+ is_dual_arm = not skip_baseline and not skip_with_skillwith_arm_suffix = "-with-skill" if is_dual_arm else ""without_arm_suffix = "-without-skill" if is_dual_arm else ""for agent in agents:- with_dir = tasks_dir / agent / "with"+ with_dir = None if skip_with_skill else tasks_dir / agent / "with"without_dir = None if skip_baseline else tasks_dir / agent / "without"+ agent_task_dirs[agent] = (with_dir, without_dir)+ if with_dir is None:+ continue+ primary_task_dirs[agent] = with_dirtask_paths = emitter(skill_path,with_dir,@@ -5274,10 +5300,11 @@ def _run_harbor_eval_impl(or case_id_by_task_selector != expected_case_id_by_task_selector):raise ValueError(f"Generated task identities differ for agent {agent}")- agent_task_dirs[agent] = (with_dir, without_dir)- reporter.emit(ProgressEvent(stage="with-skill-tasks", state="ready", detail="task inputs staged"))+ if not skip_with_skill:+ reporter.emit(ProgressEvent(stage="with-skill-tasks", state="ready", detail="task inputs staged"))if not skip_baseline:- reporter.emit(ProgressEvent(stage="baseline-tasks", state="running"))+ if not skip_with_skill:+ reporter.emit(ProgressEvent(stage="baseline-tasks", state="running"))staging_failure_stage = "baseline-tasks"baseline_alias_validation = _prevalidate_baseline_skill_candidates(skill_path,@@ -5290,6 +5317,7 @@ def _run_harbor_eval_impl(for agent in agents:without_dir = agent_task_dirs[agent][1]if without_dir is not None:+ primary_task_dirs.setdefault(agent, without_dir)baseline_task_paths = emitter(skill_path,without_dir,@@ -5317,7 +5345,14 @@ def _run_harbor_eval_impl(zip(baseline_task_selectors, baseline_logical_case_ids, strict=True))staged_task_paths.extend(baseline_task_paths)- if (+ if staged_environment_error := _staged_task_environment_error(env_mode, baseline_task_paths):+ raise ValueError(staged_environment_error)+ if expected_task_selectors is None:+ # A baseline-only run stages no with-skill tasks, so the+ # baseline arm defines the task identities.+ expected_task_selectors = baseline_task_selectors+ expected_case_id_by_task_selector = baseline_case_id_by_task_selector+ elif (baseline_task_selectors != expected_task_selectorsor baseline_case_id_by_task_selector != expected_case_id_by_task_selector):@@ -5378,12 +5413,12 @@ def _run_harbor_eval_impl(except DatasetSnapshotContractError as exc:return _persist_pre_execution_failure([str(exc)])expected_trials = len(task_selectors) * n_attempts- variants = 1 if skip_baseline else 2+ variants = 2 - int(skip_baseline) - int(skip_with_skill)matrix_trials = expected_trials * len(agents) * variantspreflight_trials = len(agents) if agent_runtime_preflight else 0try:task_timeout_seconds = _task_timeout_plan(- [paths[0] for paths in agent_task_dirs.values()],+ list(primary_task_dirs.values()),float(timeout_multiplier),)except ValueError as exc:@@ -5400,6 +5435,7 @@ def _run_harbor_eval_impl(case_count=len(case_ids),attempts=n_attempts,baseline=not skip_baseline,+ with_skill=not skip_with_skill,concurrency=n_concurrent,max_agents=max_agents,timeout_multiplier=float(timeout_multiplier),@@ -5429,7 +5465,7 @@ def _run_harbor_eval_impl(preflight_errors: list[str] = []for agent in agents:preflight = run_agent_runtime_preflight(- dataset=agent_task_dirs[agent][0],+ dataset=primary_task_dirs[agent],agent=agent,model=model_resolution[agent]["model"],env_mode=env_mode,@@ -5504,7 +5540,10 @@ def _run_harbor_eval_impl(except Empty:returnactive_agents.add(agent)- variants = "with-skill" if skip_baseline else "with-skill + baseline"+ if skip_with_skill:+ variants = "baseline only"+ else:+ variants = "with-skill" if skip_baseline else "with-skill + baseline"reporter.emit(ProgressEvent(stage=f"agent:{agent}", state="running", detail=variants))with ThreadPoolExecutor(max_workers=min(max_agents, len(agents))) as executor:@@ -5555,6 +5594,7 @@ def _run_harbor_eval_impl(output_dir=run_dir,jobs_dir=jobs_dir,skip_baseline=skip_baseline,+ skip_with_skill=skip_with_skill,n_attempts=n_attempts,pass_threshold=float(pass_threshold),stop_on_pass=bool(stop_on_pass),@@ -5764,6 +5804,7 @@ def run_harbor_eval(*args: Any, **kwargs: Any) -> dict[str, Any]:environment=kwargs.get("env_mode", DEFAULT_ENV_MODE),agents=(),baseline=not kwargs.get("skip_baseline", False),+ with_skill=not kwargs.get("skip_with_skill", False),attempts=kwargs.get("n_attempts"),concurrency=kwargs.get("n_concurrent"),max_agents=kwargs.get("max_agents"),diff --git a/src/skillevaluator/tier3/result_display.py b/src/skillevaluator/tier3/result_display.pyindex 6e9b4855..715722de 100644--- a/src/skillevaluator/tier3/result_display.py+++ b/src/skillevaluator/tier3/result_display.py@@ -415,6 +415,8 @@ def _render_agent_scores(safe: Any,) -> None:with_usable = _condition_usable(data, "with_skill")+ # A baseline-only run (--skip-with-skill) deliberately leaves the with-skill arm unscored.+ with_unavailable = "skipped" if _condition_status(data, "with_skill") == "skipped" else "NO SCORE"baseline_status = _condition_status(data, "without_skill")baseline_skipped = baseline_status == "skipped"baseline_usable = _condition_usable(data, "without_skill") and not baseline_skipped@@ -454,7 +456,9 @@ def _render_agent_scores(baseline_na = (metric in _not_applicable_metrics(data, "without_skill") and _finite_number(baseline_value) is None)- with_score, with_bar = _not_applicable_cell() if with_na else _score_cell(with_value)+ with_score, with_bar = (+ _not_applicable_cell() if with_na else _score_cell(with_value, unavailable=with_unavailable)+ )row: list[Text] = [label, with_score, with_bar]if show_baseline:persisted = lift.get(metric) if isinstance(lift.get(metric), Mapping) else {}@@ -479,7 +483,7 @@ def _render_agent_scores(for metric in custom_metrics:with_value = custom_with.get(metric) if with_usable else Nonebaseline_value = custom_without.get(metric) if baseline_usable else None- with_score, with_bar = _score_cell(with_value)+ with_score, with_bar = _score_cell(with_value, unavailable=with_unavailable)row = [Text(f"custom: {safe(metric)}", style="cyan"), with_score, with_bar]if show_baseline:persisted = custom_lift.get(metric) if isinstance(custom_lift.get(metric), Mapping) else {}@@ -642,7 +646,12 @@ def _render_dimensions(baseline_na = baseline_numeric is None and dimension_is_not_applicable(dimension, _not_applicable_metrics(data, "without_skill"))- with_cell, with_bar = _not_applicable_cell() if with_na else _score_cell(with_numeric)+ with_skipped = _condition_status(data, "with_skill") == "skipped"+ with_cell, with_bar = (+ _not_applicable_cell()+ if with_na+ else _score_cell(with_numeric, unavailable="skipped" if with_skipped else "NO SCORE")+ )baseline_skipped = _condition_status(data, "without_skill") == "skipped"if baseline_skipped:baseline_cell, baseline_bar = Text("skipped", style="dim"), Text("")diff --git a/src/skillevaluator/tier3/workflow.py b/src/skillevaluator/tier3/workflow.pyindex 34b3f7cc..bed5ee92 100644--- a/src/skillevaluator/tier3/workflow.py+++ b/src/skillevaluator/tier3/workflow.py@@ -118,6 +118,8 @@ def _preflight_options(skill_path: Path, params: dict[str, Any]) -> dict[str, An_workspace_skills,)+ if params.get("skip_baseline") and params.get("skip_with_skill"):+ raise ValueError("--skip-baseline and --skip-with-skill are mutually exclusive (no arm would run)")provider = resolve_llm_provider()config, config_path = load_evals_config(skill_path)effective = deepcopy(config)diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex 64fa19fe..7388736d 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -465,6 +465,16 @@"param_type": "option","type": "boolean"},+ {+ "default": "False",+ "is_flag": true,+ "name": "skip_with_skill",+ "opts": [+ "--skip-with-skill"+ ],+ "param_type": "option",+ "type": "boolean"+ },{"name": "n_attempts","opts": [@@ -2408,6 +2418,16 @@"param_type": "option","type": "boolean"},+ {+ "default": "False",+ "is_flag": true,+ "name": "skip_with_skill",+ "opts": [+ "--skip-with-skill"+ ],+ "param_type": "option",+ "type": "boolean"+ },{"name": "n_attempts","opts": [@@ -2932,6 +2952,16 @@"param_type": "option","type": "boolean"},+ {+ "default": "False",+ "is_flag": true,+ "name": "skip_with_skill",+ "opts": [+ "--skip-with-skill"+ ],+ "param_type": "option",+ "type": "boolean"+ },{"name": "n_attempts","opts": [diff --git a/tests/tier3/test_skip_with_skill.py b/tests/tier3/test_skip_with_skill.pynew file mode 100644index 00000000..3db422a6--- /dev/null+++ b/tests/tier3/test_skip_with_skill.py@@ -0,0 +1,618 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Baseline-only Tier 3 runs (``--skip-with-skill``) stay truthful end to end."""++from __future__ import annotations++import io+import json+import re+from pathlib import Path+from types import SimpleNamespace+from typing import Any++import pytest+from click.testing import CliRunner++from skillevaluator.cli import cli, evaluate+from skillevaluator.evaluation import EvaluationService+from skillevaluator.evaluation.tier3_report import agent_eval_result_from_directory, render_agent_eval_html_report+from skillevaluator.tier3 import commands+from skillevaluator.tier3.harbor import progress, runner+from skillevaluator.tier3.harbor.collector import collect_harbor_results+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRIC_SET, DEFAULT_METRICS+from skillevaluator.tier3.result_display import render_result+from skillevaluator.tier3.workflow import build_tier3_workflow++CASE_IDS = ["case-001", "case-002"]+++def skill(tmp_path: Path) -> Path:+ path = tmp_path / "demo"+ path.mkdir()+ (path / "SKILL.md").write_text(+ "---\nname: demo\ndescription: Explain how to validate a configuration.\n---\n# Demo\n",+ encoding="utf-8",+ )+ return path+++def _reward(score: float, **identity: str) -> dict[str, Any]:+ return {**identity, "metric_set": DEFAULT_METRIC_SET, **dict.fromkeys(DEFAULT_METRICS, score), "overall": score}+++def _write_job(job_dir: Path, scores: dict[str, float]) -> None:+ for case_id, score in scores.items():+ verifier = job_dir / f"{case_id}__attempt" / "verifier"+ verifier.mkdir(parents=True)+ (verifier / "reward.json").write_text(json.dumps(_reward(score, entry_id=case_id)), encoding="utf-8")+ _write_job_result(job_dir, [f"{case_id}__attempt" for case_id in scores])+++def _write_job_result(job_dir: Path, trial_names: list[str]) -> None:+ (job_dir / "result.json").write_text(+ json.dumps(+ {+ "n_total_trials": len(trial_names),+ "stats": {+ "n_completed_trials": len(trial_names),+ "n_errored_trials": 0,+ "n_running_trials": 0,+ "n_pending_trials": 0,+ "n_cancelled_trials": 0,+ "n_retries": 0,+ "evals": {+ "agent__model___harbor-tasks": {+ "n_trials": len(trial_names),+ "n_errors": 0,+ "reward_stats": {"reward": {"0.1": trial_names}},+ }+ },+ },+ }+ ),+ encoding="utf-8",+ )+++def _collect_baseline_only(tmp_path: Path) -> tuple[dict[str, Any], Path]:+ jobs_dir = tmp_path / "jobs"+ # A with-skill job left in a reused jobs directory must never be adopted.+ _write_job(jobs_dir / "demo-opencode-with", dict.fromkeys(CASE_IDS, 0.99))+ _write_job(jobs_dir / "demo-opencode-without", {"case-001": 0.62, "case-002": 0.72})+ run_dir = tmp_path / "results" / "20260101_000000"+ results = collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=run_dir,+ jobs_dir=jobs_dir,+ skip_with_skill=True,+ expected_cases=len(CASE_IDS),+ expected_case_ids=CASE_IDS,+ expected_trials=len(CASE_IDS),+ )+ return results, run_dir+++# ---------------------------------------------------------------------------+# CLI, service and engine entry points+# ---------------------------------------------------------------------------+++def captured_options(monkeypatch: pytest.MonkeyPatch) -> list[Any]:+ captured: list[Any] = []+ monkeypatch.setattr(+ EvaluationService,+ "evaluate",+ lambda _self, options, **_kwargs: captured.append(options) or {"execution_status": "succeeded"},+ )+ return captured+++def test_cli_forwards_skip_with_skill(skill: Path, captured_options: list[Any]) -> None:+ result = CliRunner().invoke(+ cli,+ ["tier3", "evaluate", str(skill), "--agents", "opencode", "--skip-with-skill", "--progress", "off"],+ )++ assert result.exit_code == 0, result.output+ assert captured_options[0].skip_with_skill is True+ assert captured_options[0].skip_baseline is False+++def test_cli_rejects_skipping_both_arms_before_any_work(+ skill: Path, captured_options: list[Any], monkeypatch: pytest.MonkeyPatch+) -> None:+ monkeypatch.setattr(+ EvaluationService, "create_autopilot_dataset", lambda *_args, **_kwargs: pytest.fail("must not generate")+ )+ result = CliRunner().invoke(+ cli,+ ["tier3", "evaluate", str(skill), "--autopilot", "--skip-baseline", "--skip-with-skill", "--progress", "off"],+ )++ assert result.exit_code == 2+ assert "mutually exclusive" in result.output+ assert captured_options == []+++def test_direct_workflow_rejects_skipping_both_arms_before_dataset_generation(+ skill: Path, captured_options: list[Any], monkeypatch: pytest.MonkeyPatch+) -> None:+ monkeypatch.setattr(+ EvaluationService, "create_autopilot_dataset", lambda *_args, **_kwargs: pytest.fail("must not generate")+ )+ result = CliRunner().invoke(+ build_tier3_workflow(evaluate),+ [str(skill), "--skip-baseline", "--skip-with-skill", "--progress", "off"],+ )++ assert result.exit_code != 0+ assert "mutually exclusive" in result.output+ assert captured_options == []+++def _commands_kwargs(**overrides: Any) -> dict[str, Any]:+ kwargs: dict[str, Any] = {+ "agents": "opencode",+ "env_mode": "docker",+ "skip_baseline": False,+ "n_attempts": None,+ "pass_threshold": None,+ "n_concurrent": None,+ "max_agents": None,+ "model": None,+ "agent_model": (),+ "custom_dockerfile_mode": None,+ "skill_workspace_mode": None,+ "include_skills": (),+ "copy_repo": False,+ "grading_mode": None,+ "results_dir": None,+ "harbor_keep_jobs": False,+ "timeout_multiplier": None,+ "override_cpus": None,+ "override_memory_mb": None,+ "override_storage_mb": None,+ }+ kwargs.update(overrides)+ return kwargs+++def test_commands_forward_skip_with_skill_and_reject_both_arms(skill: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[dict[str, Any]] = []+ monkeypatch.setattr(commands, "resolve_llm_provider", lambda: SimpleNamespace(provider="openai"))+ monkeypatch.setattr(commands, "run_harbor_eval", lambda **kwargs: calls.append(kwargs) or {})++ commands.evaluate(skill, **_commands_kwargs(skip_with_skill=True))+ assert calls[0]["skip_with_skill"] is True+ assert calls[0]["skip_baseline"] is False++ with pytest.raises(ValueError, match="mutually exclusive"):+ commands.evaluate(skill, **_commands_kwargs(skip_baseline=True, skip_with_skill=True))+ assert len(calls) == 1+++def test_engine_rejects_skipping_both_arms(skill: Path) -> None:+ result = runner.run_harbor_eval(skill, ["opencode"], skip_baseline=True, skip_with_skill=True)++ assert result == {"error": ["skip_baseline and skip_with_skill are mutually exclusive (no arm would run)"]}+++# ---------------------------------------------------------------------------+# Runner staging and launch+# ---------------------------------------------------------------------------+++class _Reporter:+ def __init__(self) -> None:+ self.plans: list[Any] = []+ self.events: list[Any] = []+ self.closed = False++ @property+ def is_active(self) -> bool:+ return bool(self.plans) and not self.closed++ def start(self, plan: Any) -> None:+ self.plans.append(plan)++ def set_secret_values(self, _values: Any) -> None:+ return None++ def emit(self, event: Any) -> None:+ self.events.append(event)++ def heartbeat(self) -> None:+ return None++ def close(self) -> None:+ self.closed = True+++def _stub_engine(+ monkeypatch: pytest.MonkeyPatch,+ skill: Path,+ *,+ harbor_config: dict[str, Any] | None = None,+) -> dict[str, list[Any]]:+ (skill / "evals").mkdir(exist_ok=True)+ evals_file = skill / "evals" / "evals.json"+ evals_file.write_text("{}", encoding="utf-8")+ provider = SimpleNamespace(+ provider="openai", model="gpt-5", api_key="test-provider-key", base_url="https://api.openai.com/v1"+ )+ calls: dict[str, list[Any]] = {"emitted": [], "agent_runs": [], "collected": []}++ def emit_tasks(_skill: Path, output: Path, **kwargs: Any) -> list[Path]:+ calls["emitted"].append({"output": output, **kwargs})+ tasks = [output / case_id for case_id in CASE_IDS]+ for task in tasks:+ task.mkdir(parents=True, exist_ok=True)+ return tasks++ def collect(**kwargs: Any) -> dict[str, Any]:+ calls["collected"].append(kwargs)+ return {"execution_status": "succeeded", "execution_errors": [], "metrics": []}++ def write_report(_skill_path: Path, run_dir: Path, **_kwargs: Any) -> Path:+ path = run_dir / "report.html"+ path.write_text("<html></html>", encoding="utf-8")+ return path++ config = {"task_source": "evals_json", "n_attempts": 1, "n_concurrent": 1, "max_agents": 1}+ config.update(harbor_config or {})+ monkeypatch.setattr(runner, "resolve_llm_provider", lambda: provider)+ monkeypatch.setattr(runner, "load_evals_config", lambda _path: ({"harbor": config}, None))+ monkeypatch.setattr(runner, "_check_prerequisites", lambda **_kwargs: [])+ monkeypatch.setattr(runner, "find_evals_file", lambda _path: evals_file)+ monkeypatch.setattr(runner, "generate_harbor_tasks", emit_tasks)+ monkeypatch.setattr(runner, "_provider_environment", lambda _provider: {"OPENAI_API_KEY": provider.api_key})+ monkeypatch.setattr(runner, "_resolve_runtime_env", lambda _templates, **_kwargs: ({}, []))+ monkeypatch.setattr(runner, "_harbor_subprocess_environment", lambda **_kwargs: {})+ monkeypatch.setattr(runner, "_run_agent_pair", lambda **kwargs: calls["agent_runs"].append(kwargs) or [])+ monkeypatch.setattr(runner, "collect_harbor_results", collect)+ monkeypatch.setattr(runner, "render_agent_eval_html_report", write_report)+ from skillevaluator.tier3.harbor import runtime_preflight++ monkeypatch.setattr(+ runtime_preflight,+ "probe_model",+ lambda selected: SimpleNamespace(+ ok=True,+ provider=selected.provider,+ model=selected.model,+ detail="model is available",+ failure_kind=None,+ http_status=None,+ ),+ )+ return calls+++def test_runner_stages_and_runs_only_the_baseline_arm(skill: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ calls = _stub_engine(monkeypatch, skill)+ reporter = _Reporter()++ result = runner.run_harbor_eval(+ skill,+ ["codex"],+ skip_with_skill=True,+ output_dir=skill.parent / "results",+ progress_reporter=reporter,+ )++ assert result["execution_status"] == "succeeded", result+ # Only the baseline arm is staged, with single-arm (unsuffixed) task names+ # that the collector resolves with the same rule.+ assert [call["with_skill"] for call in calls["emitted"]] == [False]+ assert calls["emitted"][0]["output"].name == "without"+ assert calls["emitted"][0]["arm_suffix"] == ""+ [agent_run] = calls["agent_runs"]+ assert agent_run["with_skill"] is None+ assert agent_run["baseline"] == calls["emitted"][0]["output"]+ assert agent_run["task_names"] == CASE_IDS+ [collected] = calls["collected"]+ assert collected["skip_with_skill"] is True+ assert collected["skip_baseline"] is False+ assert collected["expected_case_ids"] == CASE_IDS++ final_plan = reporter.plans[-1]+ assert final_plan.with_skill is False+ assert final_plan.baseline is True+ assert final_plan.matrix_trials == len(CASE_IDS)+ transitions = [(event.stage, event.state) for event in reporter.events]+ assert ("with-skill-tasks", "skipped") in transitions+ assert ("with-skill-tasks", "running") not in transitions+ assert transitions.index(("baseline-tasks", "running")) < transitions.index(("baseline-tasks", "ready"))+ agent_event = next(event for event in reporter.events if event.stage == "agent:codex" and event.state == "running")+ assert agent_event.detail == "baseline only"+++def test_paired_runner_still_suffixes_both_arms(skill: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ calls = _stub_engine(monkeypatch, skill)++ result = runner.run_harbor_eval(skill, ["codex"], output_dir=skill.parent / "results")++ assert result["execution_status"] == "succeeded", result+ assert [(call["with_skill"], call["arm_suffix"]) for call in calls["emitted"]] == [+ (True, "-with-skill"),+ (False, "-without-skill"),+ ]+ assert calls["agent_runs"][0]["with_skill"] is not None+ assert calls["collected"][0]["skip_with_skill"] is False+++def test_baseline_only_keeps_the_baseline_isolation_guard(skill: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ calls = _stub_engine(monkeypatch, skill, harbor_config={"pre_agent_setup": ["echo setup"]})++ result = runner.run_harbor_eval(skill, ["codex"], skip_with_skill=True, output_dir=skill.parent / "results")++ assert "pre_agent_setup" in result["error"][0]+ assert calls["emitted"] == []+++def test_baseline_only_setup_rejection_does_not_suggest_skip_baseline(+ skill: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ # --skip-baseline cannot be combined with --skip-with-skill, so the+ # paired-run advice would be a dead end for a baseline-only run.+ calls = _stub_engine(monkeypatch, skill, harbor_config={"pre_agent_setup": ["echo setup"]})+ reporter = _Reporter()++ result = runner.run_harbor_eval(+ skill,+ ["codex"],+ skip_with_skill=True,+ output_dir=skill.parent / "results",+ progress_reporter=reporter,+ )++ expected = (+ "harbor.pre_agent_setup/setup_commands cannot run in a baseline-only run (--skip-with-skill); "+ "the baseline arm must stay isolated from skill-owned setup commands"+ )+ assert result == {"error": [expected]}+ assert "--skip-baseline" not in result["error"][0]+ failed = [event for event in reporter.events if event.stage == "configuration" and event.state == "failed"]+ assert [event.detail for event in failed] == [expected]+ assert calls["emitted"] == []++[email protected]("stop_on_pass", [False, True])+def test_agent_pair_launches_only_the_baseline_job(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, stop_on_pass: bool+) -> None:+ launched: list[str] = []+ monkeypatch.setattr(+ runner,+ "_run_harbor",+ lambda **kwargs: launched.append(kwargs["job_name"]) or (True, ""),+ )+ monkeypatch.setattr(+ runner,+ "_run_stop_on_pass_variant",+ lambda **kwargs: launched.append(f"{kwargs['skill_name']}-{kwargs['agent']}-{kwargs['variant']}") or [],+ )++ errors = runner._run_agent_pair(+ skill_name="demo",+ agent="codex",+ model="gpt-5",+ env_mode="docker",+ with_skill=None,+ baseline=tmp_path / "without",+ jobs_dir=tmp_path / "jobs",+ run_env={},+ n_attempts=2,+ n_concurrent=2,+ timeout_multiplier=1.0,+ override_cpus=None,+ override_memory_mb=None,+ override_storage_mb=None,+ expected_trials=4,+ stop_on_pass=stop_on_pass,+ task_names=CASE_IDS,+ )++ assert errors == []+ assert launched == ["demo-codex-without"]+++# ---------------------------------------------------------------------------+# Collection and reporting+# ---------------------------------------------------------------------------+++def test_collector_records_a_skipped_with_skill_arm_and_ignores_stale_jobs(tmp_path: Path) -> None:+ results, run_dir = _collect_baseline_only(tmp_path)++ assert results["execution_status"] == "succeeded", results.get("execution_errors")+ agent = results["agents"]["opencode"]+ assert agent["conditions"]["with_skill"]["execution_status"] == "skipped"+ assert agent["conditions"]["without_skill"]["execution_status"] == "succeeded"+ assert agent["with_skill"] == {}+ assert agent["without_skill"]["goal_accuracy"] == pytest.approx(0.67)+ assert agent["num_trials_with"] == 0+ assert agent["num_trials_without"] == 2+ assert agent["lift"] == {}+ assert agent["pass_at_k"]["lift"] == {}+ assert agent["skill_activation"]["status"] == "not_applicable"+ assert agent["score_policy_with_skill"] is None+ assert results["metric_set"] == DEFAULT_METRIC_SET+ assert results["score_policy"] == agent["score_policy_without_skill"]+ assert not (run_dir / "opencode" / "with-skill" / "trials").exists()+ summary = json.loads((run_dir / "opencode" / "with-skill" / "summary.json").read_text(encoding="utf-8"))+ assert summary["execution_status"] == "skipped"+ assert summary["scores"] == {}+ assert summary["skill_activation"]["status"] == "not_applicable"+++def test_collector_resolves_unsuffixed_baseline_only_task_names(tmp_path: Path) -> None:+ # Single-arm runs stage unsuffixed task names. A case ID that happens to end+ # in the paired-run arm suffix must survive collection intact.+ case_id = "probe-without-skill"+ job_dir = tmp_path / "jobs" / "demo-opencode-without"+ trial_dir = job_dir / f"{case_id}__attempt"+ (trial_dir / "verifier").mkdir(parents=True)+ # No entry_id or task path: identity comes from Harbor's persisted task name.+ (trial_dir / "verifier" / "reward.json").write_text(json.dumps(_reward(0.7)), encoding="utf-8")+ (trial_dir / "result.json").write_text(json.dumps({"task_name": case_id}), encoding="utf-8")+ _write_job_result(job_dir, [trial_dir.name])++ results = collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ skip_with_skill=True,+ expected_cases=1,+ expected_case_ids=[case_id],+ expected_trials=1,+ )++ assert results["execution_status"] == "succeeded", results.get("execution_errors")+ assert list(results["agents"]["opencode"]["pass_at_k"]["without_skill"]["cases"]) == [case_id]+++def test_collector_rejects_skipping_both_arms(tmp_path: Path) -> None:+ with pytest.raises(ValueError, match="mutually exclusive"):+ collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ skip_with_skill=True,+ )+++def test_report_presents_baseline_calibration_without_a_skill_verdict(tmp_path: Path, skill: Path) -> None:+ results, run_dir = _collect_baseline_only(tmp_path)++ result = agent_eval_result_from_directory(skill, run_dir, engine_result=results, use_llm_judge=False)++ assert result is not None+ assert result.passed is True+ assert "baseline calibration complete" in result.success_details[0].message+ payload = result.metadata["agent_eval"]+ assert payload["mode"] == payload["summary"]["mode"] == "baseline_only"+ assert payload["execution_status"] == "succeeded"+ assert payload["verdict"] == "neutral"+ assert payload["best_agent"] == "opencode"+ assert payload["overall_score"] == pytest.approx(0.67)+ assert payload["overall_lift"] is None+ agent = payload["agents"]["opencode"]+ assert agent["with_skill"] is None+ assert agent["baseline"] == pytest.approx(0.67)+ assert agent["lift"] is None+ # Unmeasured with-skill values stay unavailable; they never become 0.0.+ assert payload["evaluators"]["goal_accuracy"] == {+ "with_skill": None,+ "baseline": pytest.approx(0.67),+ "lift": None,+ "with_skill_status": "skipped",+ "baseline_status": "scored",+ }+ assert {card["with_skill"] for card in payload["evaluator_cards"]} == {None}+ assert {card["status"] for card in payload["evaluator_cards"]} == {"warn"}+ assert all(dimension["with_skill"] is None and dimension["lift"] is None for dimension in payload["dimensions"])+ assert all(dimension["baseline"] == pytest.approx(0.67) for dimension in payload["dimensions"])+ assert "no skill lift was measured" in payload["dimensions"][0]["explanation"]+ assert payload["conclusions"][0]["title"] == "Baseline calibration complete"+ assert "--skip-with-skill" in payload["suggestions"][0]+++def test_paired_report_mode_is_both(tmp_path: Path, skill: Path) -> None:+ jobs_dir = tmp_path / "jobs"+ _write_job(jobs_dir / "demo-opencode-with", {"case-001": 0.8, "case-002": 0.9})+ _write_job(jobs_dir / "demo-opencode-without", {"case-001": 0.6, "case-002": 0.7})+ run_dir = tmp_path / "results" / "20260101_000000"+ results = collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=run_dir,+ jobs_dir=jobs_dir,+ expected_cases=len(CASE_IDS),+ expected_case_ids=CASE_IDS,+ expected_trials=len(CASE_IDS),+ )++ result = agent_eval_result_from_directory(skill, run_dir, engine_result=results, use_llm_judge=False)++ assert result is not None+ payload = result.metadata["agent_eval"]+ assert payload["mode"] == "both"+ assert payload["overall_score"] == pytest.approx(0.85)+ assert payload["overall_lift"] == pytest.approx(0.2)+ assert "Tier 3 evaluation complete" in result.success_details[0].message+++def test_html_report_explains_the_baseline_only_run(tmp_path: Path, skill: Path) -> None:+ results, run_dir = _collect_baseline_only(tmp_path)++ report = render_agent_eval_html_report(skill, run_dir, engine_result=results, use_llm_judge=False)+ html = report.read_text(encoding="utf-8")++ assert "Baseline-only calibration:" in html+ assert "Baseline calibration (no with-skill arm)" in html+ assert "No skill verdict: baseline-only run" in html+ assert "Verdict at a glance:" not in html+ assert '<span class="hero-score-label">baseline score</span>' in html+ assert re.search(r"<td><strong>Goal Accuracy</strong></td>\s*<td>skipped</td>\s*<td>0\.67</td>", html)+++def test_report_tables_mark_the_with_skill_arm_skipped(tmp_path: Path, skill: Path) -> None:+ from skillevaluator.reporting.cli import CLIReporter+ from skillevaluator.reporting.markdown import MarkdownReporter++ results, run_dir = _collect_baseline_only(tmp_path)+ result = agent_eval_result_from_directory(skill, run_dir, engine_result=results, use_llm_judge=False)+ assert result is not None++ terminal = re.sub(r"\x1b\[[0-9;]*m", "", CLIReporter().render(result))+ assert re.search(r"Goal Accuracy\s*│\s*skipped\s*│\s*0\.67\s*│\s*skipped", terminal)+ assert "NO SCORE" not in terminal+ markdown = MarkdownReporter().render_all([result])+ assert "| Goal Accuracy | skipped | 0.67 | skipped |" in markdown+++def test_cli_summary_marks_the_with_skill_arm_skipped(tmp_path: Path) -> None:+ results, _run_dir = _collect_baseline_only(tmp_path)++ rendered = render_result(results)++ assert re.search(r"Goal Accuracy\s+skipped\s+0\.67", rendered)+ # The case-first verdict line names the calibration instead of an unmeasured skill effect.+ verdict_line = rendered.splitlines()[0]+ assert "baseline only" in verdict_line+ assert "no skill effect was measured" in verdict_line+ assert "contracts differ" not in rendered+ # Per-case rows show no effect rather than "inconclusive" for an arm that never ran.+ assert "inconclusive" not in rendered+ assert re.search(r"case-001\s+-\s+1/1 .*\s+-\s*$", rendered, re.MULTILINE)+ security_dimension = next(line for line in rendered.splitlines() if line.strip("│ ").startswith("Security"))+ assert "skipped" in security_dimension+++def test_progress_plan_names_the_baseline_only_mode() -> None:+ plan = progress.Tier3RunPlan(+ skill_name="demo", environment="docker", agents=("codex",), baseline=True, with_skill=False+ )+ plain = io.StringIO()+ reporter = progress.PlainProgressReporter(stream=plain, refresh_interval=60)+ reporter.start(plan)+ reporter.close()+ assert "with-skill=no, baseline=yes" in plain.getvalue()++ rich_reporter = progress.RichProgressReporter(stream=io.StringIO(), refresh_interval=60)+ rich_reporter._live_plan = plan+ from rich.console import Console++ rendered = io.StringIO()+ Console(file=rendered, force_terminal=False, width=180).print(rich_reporter._build_live_table())+ assert "Baseline-only evaluation" in rendered.getvalue()+ assert "Skill-lift evaluation" not in rendered.getvalue()