Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(tier3): add fixed-source skill interaction evaluation (#392)
YMuskrat6eb75632e409
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 57720591..5efeafbe 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -69,6 +69,15 @@ All notable changes to SkillEvaluator are documented in this file.compatible retained skill versions, report paired per-agent deltas andregressions, and keep version comparisons separate from no-skill Skill Lift.+- Add opt-in `tier3 interact` experiments for two or more skills. The command+ uses one reviewed task suite and a fixed attempt budget to compare baseline,+ individual, and combined performance. It reports the departure from additive+ gains and retains source identities and trial evidence. Invalid inputs exit+ with code 2. Incomplete or incomparable experiments and output I/O failures+ exit with code 1. Configuration failures identify their arm, trial progress+ uses stderr, and output roots honor `SKILLEVALUATOR_RESULTS_DIR` with canonical+ paths. Documentation describes trial noise and sequential configuration order.+- Read-only Tier 3 `history` lists retained Skill Lift runs chronologically,separates incompatible configurations into segments, preserves unavailablelift as missing, and exports JSON, CSV, or Markdown summaries.@@ -502,6 +511,9 @@ All notable changes to SkillEvaluator are documented in this file.the PR or merge request's target revision. Policy files from the proposedchanges cannot approve their own exceptions.+- The `ci/gitlab/skill-evaluation.yml` template honors `TMPDIR` for temporary+ scan reports and defaults to `/tmp`.+- `skillevaluator validate` on a target with no skill, rules, workflows, orplugin, such as an ordinary source repository, no longer fails Tier 1 withthree `No skills found` errors and exit 1 when every content scan passes.diff --git a/ci/gitlab/skill-evaluation.yml b/ci/gitlab/skill-evaluation.ymlindex ca9e5443..f05cfa51 100644--- a/ci/gitlab/skill-evaluation.yml+++ b/ci/gitlab/skill-evaluation.yml@@ -24,7 +24,7 @@ skill-evaluation:echo "skill-evaluator-report already exists; choose a clean checkout for the report." >&2exit 1fi- reports_dir="$(mktemp -d /tmp/skillevaluator-reports.XXXXXX)"+ reports_dir="$(mktemp -d "${TMPDIR:-/tmp}/skillevaluator-reports.XXXXXX")"trap 'rm -rf "$reports_dir"' EXITexport SKILL_PATHS="$SKILL_EVAL_PATHS"export MIN_SCORE="$SKILL_EVAL_MIN_SCORE"diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex 2553354f..b32d798b 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -32,6 +32,7 @@ commands referenced below.| [similarity-check](#similarity-check) | Detect duplicate content across a collection | Embeddings provider key || [tier3 evaluate](#tier3-evaluate) | Run Tier 3 live agent evaluation | Provider key plus a credential the selected agents can use (one `NVIDIA_API_KEY` covers both on NVIDIA Build), backend (Docker by default) || [tier3 select-decoys](#tier3-select-decoys) | Save a reproducible set of neighboring skills for group evaluation | Local description-embedding catalog, skill collection, embeddings provider key |+| [tier3 interact](#tier3-interact) | Measure departure from additive skill gains | At least two skills, one reviewed evaluation source, Tier 3 runtime and credentials || [tier3 compare-versions](#tier3-compare-versions) | Compare current and prior retained with-skill scores without writes | Two compatible retained runs || [create-eval-dataset](#create-eval-dataset) | Generate synthetic eval datasets | LLM provider key (none with `--no-llm`) || [init-custom-grader](#init-custom-grader) | Scaffold a bring-your-own-grader starter | Nothing |@@ -181,7 +182,7 @@ pipeline.| --- | --- || `tier1` | `validate`, `quality-check`, `rubric-eval`, `security-scan`, `pii-scan`, `lint-scripts` || `tier2` | `context-optimization-check`, `dedup-scan`, `similarity-check` |-| `tier3` | `evaluate`, `select-decoys`, `create-eval-dataset`, `migrate-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view`, `import-metrics`, `import-traces`, `accept-traces` |+| `tier3` | `evaluate`, `interact`, `select-decoys`, `create-eval-dataset`, `migrate-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view`, `import-metrics`, `import-traces`, `accept-traces` |`health-check` and [models](#models) are top-level only. The expert commands[tier3 evaluate](#tier3-evaluate), [tier3 select-decoys](#tier3-select-decoys),@@ -524,6 +525,38 @@ without contacting the embedding provider. See[Select a reproducible decoy set](agents-and-sandboxes.mdx#select-a-reproducible-decoy-set)for a complete example and the limits of similarity-based selection.+## tier3 interact++Run a fixed-source skill interaction experiment:++```bash+skillevaluator tier3 interact ./skills/a ./skills/b \+ --eval-source ./reviewed-suite --n-attempts 3 --agents codex --json+```++| Option | Default | Purpose |+| --- | --- | --- |+| `SKILLS...` | Required | At least two distinct skill directories. |+| `--eval-source` | Required | Skill directory containing the reviewed `evals/` source reused for every configuration. |+| `-a`, `--agents` | Provider-native agent | Comma-separated Harbor agents. |+| `--model` | Provider/config model | Global agent model override. |+| `--env-mode` | `docker` | Existing Harbor environment backend. |+| `--environment-kwarg`, `--ek` | None | Repeatable backend `KEY=VALUE`; never pass secrets. |+| `--n-attempts` | `1` | Positive fixed attempt budget for every case; disables early stopping. |+| `--n-concurrent` | Source config or `4` | Concurrent trials within a configuration. |+| `--metric` | `goal_accuracy` | Outcome metric: `goal_accuracy`, `accuracy`, or `overall` for `custom_only` grading. |+| `--results-dir` | `SKILLEVALUATOR_RESULTS_DIR` or `interaction-results` beside the source | Output root outside all inputs; home shortcuts and symlinked parents are resolved. |+| `--harbor-keep-jobs` | Off | Retain raw Harbor jobs alongside collected evidence. |+| `--json` | Off | Print the complete JSON report on stdout; progress uses stderr. |++Runs baseline, each individual skill, and the combined selection. Saves+`interaction.json`, fixed-source evidence, and separate Tier 3 results for every+configuration. Exit 0 means a comparable experiment completed; positive,+negative, and zero interaction all use exit 0. Exit 1 means incomplete or+incomparable evidence. Invalid command arguments use Click's usage-error exit.+See [Skill interaction experiments](/skills/skillevaluator/tier3-live-evaluation#skill-interaction-experiments)+for the calculation, input restrictions, attempt budget, and interpretation.+## tier3 compare-versions```bashdiff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex acb4d178..26b2c4d5 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -68,6 +68,93 @@ When at least one positive trial activates the skill, lift remains eligiblesubject to the comparison requirements below. This also makes activationfailures visible with `--skip-baseline`.+## Skill interaction experiments++Use the opt-in `tier3 interact` command to measure whether selected skills+produce more or less improvement together than the sum of their individual+improvements. Choose one reviewed evaluation source before running:++```bash+skillevaluator tier3 interact ./skills/research ./skills/summarize \+ --eval-source ./skills/research --agents codex --n-attempts 3 \+ --results-dir ./interaction-results+```++`--eval-source` is a skill directory containing an existing valid `evals/`+dataset or native Harbor task suite. The command never generates or refines+tasks. It freezes that source and the selected skills once, then reuses the+existing Harbor evaluator and skill staging to run:++| Configuration | Installed skills |+| --- | --- |+| Baseline | None |+| Individual | One run for each selected skill alone |+| Combined | Every selected skill together |++For two skills this is four configurations. For a group of N skills it is+N + 2 configurations, each with the full task suite, selected agents, and the+same `--n-attempts` budget. Early stopping is disabled, including when the source+config enables it. Source-config companion selections, partial task suites, and+`pre_agent_setup` are rejected before execution. Skill roots must have distinct+names and be self-contained: runtime snapshots include each skill's files,+while common task inputs belong in the fixed source's `evals/files/`. Results+must be outside the selected skills and evaluation source. The output root uses+`--results-dir`, then `SKILLEVALUATOR_RESULTS_DIR`, then `interaction-results`+beside the source. User-home shortcuts and symlinked parents are resolved before+checking that the output stays outside the inputs.++The default outcome metric is `goal_accuracy`; `--metric accuracy` is also+available. These compare task outcomes on the same cases instead of the+standard overall score, whose discovery and skill-usage components differ+between installed-skill and baseline configurations. For `custom_only` grading,+use `--metric overall` with one shared task-outcome grader.++For two skills, the interaction is:++```text+I(A, B) = P(A, B) - P(A) - P(B) + P(empty)+```++For a selected group S:++```text+I(S) = [P(S) - P(empty)] - sum(P(skill) - P(empty) for skill in S)+```++P is the mean per-case outcome, averaging the fixed number of attempts within+each case first. A positive interaction means more than additive improvement;+a negative interaction means less; zero means additive. For example, baseline+0.50, individual scores 0.75 and 0.70, and combined score 0.65 yield interaction+-0.30 and combined lift +0.15. Negative interaction does not imply performance+below baseline. For more than two skills this is total departure from additive+individual gains, rather than an isolated higher-order interaction.++These signs describe the observed estimate, without a confidence interval or+statistical significance test. A small interaction can reflect trial noise;+with one attempt on ten binary-outcome cases, one outcome changing in an+individual or combined configuration moves the interaction by 0.10. Configurations+run sequentially: baseline, each individual, then combined. Provider changes,+rate limiting, or timeouts during a long experiment can therefore align with+configuration order. Repeat the experiment before interpreting a small+interaction as a reliable effect.++The terminal prints baseline, individual and combined scores, combined lift,+and signed interaction separately for each agent/model. Add `--json` for a+machine-readable report on stdout; each configuration's trial progress goes to+stderr. Every experiment+also saves `interaction.json`, its fixed evaluation source, and independent+Tier 3 run directories containing reports, dataset snapshots, and trial+evidence. The interaction report records original source paths, content+identities, per-case calculations, and the reward artifacts used.++Missing trials, N/A outcomes, changed cases, unequal attempt budgets, or+different agent models, evaluator versions, or outcome scorers produce an+explicit `incomplete` or `incomparable` result with no interaction estimate and+exit code 1. Judge fallback to a different model is incomparable. Multiple+retained reward rows for one logical attempt are also incomparable rather than+being counted as extra attempts. Other completed configuration evidence remains+available for inspection.+## Repeatable activation validationUse the existing dataset generator, runner and trusted trajectory collector todiff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex 4ede3925..c33c5eba 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -4199,6 +4199,10 @@ from skillevaluator.tier3.grader_validation import validate_gradertier3.add_command(select_decoys_command, "select-decoys")tier3.add_command(validate_grader, "validate-grader")+from skillevaluator.tier3.interaction_commands import interact_command++tier3.add_command(interact_command, "interact")+from skillevaluator.tier3.workflow import build_tier3_workflowtier3.workflow = build_tier3_workflow(evaluate)diff --git a/src/skillevaluator/tier3/interaction_commands.py b/src/skillevaluator/tier3/interaction_commands.pynew file mode 100644index 00000000..1245b872--- /dev/null+++ b/src/skillevaluator/tier3/interaction_commands.py@@ -0,0 +1,129 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Command surface for explicit skill-interaction experiments."""++from __future__ import annotations++from pathlib import Path+from typing import Any++import click++from skillevaluator.cli_help import RichCommand+from skillevaluator.tier3_environments import all_env_modes++[email protected]("interact", cls=RichCommand, context_settings={"help_option_names": ["-h", "--help"]})[email protected]("skills", nargs=-1, required=True, type=click.Path(exists=True, file_okay=False, path_type=Path))+ "--eval-source",+ required=True,+ type=click.Path(exists=True, file_okay=False, path_type=Path),+ help="Skill directory containing the single reviewed evals/ source used for every configuration.",+)+ "-a", "--agents", default=None, help="Comma-separated Harbor agents; defaults to the provider-native agent."+)[email protected]("--model", default=None, help="Global agent model override.")[email protected]("--env-mode", default="docker", show_default=True, type=click.Choice(all_env_modes()))[email protected]("--environment-kwarg", "--ek", multiple=True, help="Harbor backend kwarg KEY=VALUE; never pass secrets.")+ "--n-attempts",+ type=click.IntRange(min=1),+ default=1,+ show_default=True,+ help="Fixed attempts per case in every configuration; early stopping is disabled.",+)[email protected]("--n-concurrent", type=click.IntRange(min=1), default=None)+ "--metric",+ type=click.Choice(["goal_accuracy", "accuracy", "overall"]),+ default="goal_accuracy",+ show_default=True,+ help="Task outcome to compare; overall requires custom_only grading.",+)+ "--results-dir",+ type=click.Path(file_okay=False, path_type=Path),+ default=None,+ help="Output root outside the inputs; defaults to SKILLEVALUATOR_RESULTS_DIR or interaction-results beside --eval-source.",+)[email protected]("--harbor-keep-jobs", is_flag=True, help="Keep raw Harbor jobs as well as the collected evidence.")[email protected]("--json", "as_json", is_flag=True, help="Print the complete interaction report as JSON.")+def interact_command(+ skills: tuple[Path, ...],+ eval_source: Path,+ agents: str | None,+ model: str | None,+ env_mode: str,+ environment_kwarg: tuple[str, ...],+ n_attempts: int,+ n_concurrent: int | None,+ metric: str,+ results_dir: Path | None,+ harbor_keep_jobs: bool,+ as_json: bool,+) -> None:+ """Measure interaction among at least two skills on one fixed task suite.++ Run no skills, each skill alone, and all skills together. Positive means+ more than additive improvement; negative means less; zero means additive.+ """+ import json+ from contextlib import redirect_stdout++ from skillevaluator.evaluation import EvaluationOptions+ from skillevaluator.tier3.harbor.progress import create_progress_reporter+ from skillevaluator.tier3.interactions import run_interaction++ try:+ with redirect_stdout(click.get_text_stream("stderr")):+ report = run_interaction(+ skills,+ eval_source=eval_source,+ options=EvaluationOptions(+ skill_path=eval_source,+ agents=agents,+ model=model,+ env_mode=env_mode,+ environment_kwarg=environment_kwarg,+ n_attempts=n_attempts,+ n_concurrent=n_concurrent,+ results_dir=results_dir,+ harbor_keep_jobs=harbor_keep_jobs,+ ),+ metric=metric,+ on_configuration=lambda name: click.echo(f"Tier 3 interaction: running {name}", err=True),+ progress_reporter=create_progress_reporter("auto", stream=click.get_text_stream("stderr")),+ )+ except ValueError as exc:+ raise click.UsageError(str(exc)) from exc+ except OSError as exc:+ raise click.ClickException(str(exc)) from exc+ if as_json:+ click.echo(json.dumps(report, indent=2, allow_nan=False))+ else:+ _print_report(report)+ if report["status"] != "complete":+ raise click.exceptions.Exit(1)+++def _print_report(report: dict[str, Any]) -> None:+ click.echo(f"Interaction evaluation: {report['status']} ({report['metric']})")+ for reason in dict.fromkeys(report.get("reasons", [])):+ click.echo(f" {reason}")+ names = [skill["name"] for skill in report["skills"]]+ for agent, score in report.get("agents", {}).items():+ click.echo(f"{agent}: {score['status']}")+ if score["status"] != "complete":+ for reason in dict.fromkeys(score.get("reasons", [])):+ click.echo(f" {reason}")+ continue+ click.echo(f" Baseline: {score['baseline']:.4f}")+ for name, value in zip(names, score["individuals"], strict=True):+ click.echo(f" {name}: {value:.4f}")+ click.echo(f" Combined: {score['combined']:.4f}; combined lift: {score['combined_lift']:+.4f}")+ click.echo(f" Interaction: {score['interaction']:+.4f} ({score['classification']})")+ click.echo("Negative interaction means less than additive gain; combined performance may still exceed baseline.")+ click.echo(f"Evidence: {Path(report['run_dir']) / 'interaction.json'}")diff --git a/src/skillevaluator/tier3/interactions.py b/src/skillevaluator/tier3/interactions.pynew file mode 100644index 00000000..ec748693--- /dev/null+++ b/src/skillevaluator/tier3/interactions.py@@ -0,0 +1,418 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Opt-in, fixed-source experiments measuring departure from additive skill gains."""++from __future__ import annotations++import json+import math+import os+import tempfile+from collections.abc import Callable, Mapping, Sequence+from dataclasses import replace+from datetime import UTC, datetime+from pathlib import Path+from typing import TYPE_CHECKING, Any++from skillevaluator.evaluation import EvaluationOptions, EvaluationService+from skillevaluator.tier3.decoy_identity import (+ DECOY_SOURCE_DIGEST_ALGORITHM,+ decoy_source_copy_ignore,+ decoy_source_digest,+)+from skillevaluator.tier3.harbor.secure_copy import copytree_secure, tree_content_fingerprint_secure++if TYPE_CHECKING:+ from skillevaluator.tier3.harbor.progress import ProgressReporter++OUTCOME_METRICS = ("goal_accuracy", "accuracy", "overall")+_RUNTIME_ENTRY_FIELDS = frozenset({"has_skill", "evaluated_skill", "skill_workspace_mode", "workspace_skill_names"})+++class IncomparableInteraction(ValueError):+ """Evidence cannot support an interaction estimate."""+++def interaction_scores(baseline: float, individuals: Sequence[float], combined: float) -> dict[str, Any]:+ """Calculate the signed interaction from comparable outcome means."""+ if len(individuals) < 2:+ raise ValueError("Interaction evaluation requires at least two skills")+ for score in (baseline, *individuals, combined):+ if (+ isinstance(score, bool)+ or not isinstance(score, (int, float))+ or not math.isfinite(score)+ or not 0 <= score <= 1+ ):+ raise ValueError("Outcome scores must be finite numbers between 0 and 1")+ lift = combined - baseline+ individual_lifts = [score - baseline for score in individuals]+ interaction = lift - math.fsum(individual_lifts)+ if math.isclose(interaction, 0.0, abs_tol=1e-12):+ interaction = 0.0+ return {+ "baseline": baseline,+ "individuals": list(individuals),+ "combined": combined,+ "individual_lifts": individual_lifts,+ "combined_lift": lift,+ "interaction": interaction,+ "classification": "positive" if interaction > 0 else "negative" if interaction < 0 else "zero",+ }+++def _read_json(path: Path, root: Path) -> Any:+ from skillevaluator.tier3.harbor.adapter import _read_regular_evals_file++ return json.loads(_read_regular_evals_file(path, allowed_root=root, max_bytes=16 * 1024 * 1024))+++def _case_contract(result: Mapping[str, Any], run_dir: Path) -> dict[str, Any]:+ snapshot = _read_json(run_dir / "dataset_snapshot.json", run_dir)+ entries = snapshot.get("dataset") if isinstance(snapshot, dict) else None+ if not isinstance(entries, list) or not entries:+ raise IncomparableInteraction("Missing retained dataset cases")+ cases = {}+ for entry in entries:+ case_id = entry.get("id") if isinstance(entry, dict) else None+ if not isinstance(case_id, str) or not case_id or case_id in cases:+ raise IncomparableInteraction("Missing or duplicate dataset case identity")+ cases[case_id] = {key: value for key, value in entry.items() if key not in _RUNTIME_ENTRY_FIELDS}+ selection = result.get("run_config", {}).get("case_selection", {})+ if not isinstance(selection, Mapping) or selection.get("skipped", 0):+ raise IncomparableInteraction("Partial task suites cannot support an interaction estimate")+ return cases+++def _outcome_rows(+ run_dir: Path, agent: str, condition: str, metric: str, *, custom_only: bool+) -> dict[str, list[dict[str, Any]]]:+ from skillevaluator.tier3.harbor.collector import _reward_judge_models+ from skillevaluator.tier3.harbor.metrics import metric_is_not_applicable, metric_value, overall_score++ trials = run_dir / agent / condition / "trials"+ rows: dict[str, list[dict[str, Any]]] = {}+ seen_trials: set[str] = set()+ for path in sorted(trials.glob("*/reward.json")):+ reward = _read_json(path, run_dir)+ if not isinstance(reward, dict) or reward.get("evaluation_status") in {"error", "failed"}:+ raise IncomparableInteraction("Unscored trial in retained evidence")+ case_id, trial_id = reward.get("entry_id"), reward.get("trial_id")+ if not isinstance(case_id, str) or not isinstance(trial_id, str) or not case_id or not trial_id:+ raise IncomparableInteraction("Missing trial or case identity")+ if trial_id in seen_trials:+ raise IncomparableInteraction("Multiple reward rows for one logical attempt; no comparable outcome")+ seen_trials.add(trial_id)+ value = overall_score(reward) if metric == "overall" else metric_value(reward, metric)+ if (+ value is None+ or isinstance(value, bool)+ or not math.isfinite(value)+ or not 0 <= value <= 1+ or (metric != "overall" and metric_is_not_applicable(reward, metric))+ ):+ raise IncomparableInteraction(f"Missing, N/A, or invalid {metric} outcome for {case_id}")+ models = sorted(_reward_judge_models(reward, metric))+ if not custom_only and len(models) != 1:+ raise IncomparableInteraction(f"Unknown or mixed outcome scorer for {case_id}")+ details = reward.get("details")+ detail = details.get(metric) if isinstance(details, dict) else None+ scorer = {+ "judge_models": models,+ "method": detail.get("method") if isinstance(detail, dict) else None,+ "metric_set": reward.get("metric_set"),+ "score_policy": reward.get("score_policy"),+ }+ rows.setdefault(case_id, []).append(+ {"score": value, "trial_id": trial_id, "reward_path": str(path), "scorer": scorer}+ )+ return rows+++def compare_configurations(+ configurations: Sequence[Mapping[str, Any]], *, metric: str, n_attempts: int+) -> dict[str, Any]:+ """Require complete per-case, per-agent evidence before calculating a score.++ Every case has equal weight; attempts are averaged within a case first.+ Configuration order is baseline, singleton skills, combined skills.+ """+ if len(configurations) < 4:+ raise ValueError("Expected baseline, at least two individuals, and combined evidence")+ failures = [+ f"{config['id']}: {config.get('error') or 'Configuration failed or returned no result'}"+ for config in configurations+ if "error" in config or "result" not in config+ ]+ if failures:+ return {"status": "incomplete", "reasons": failures, "agents": {}}+ results = [config["result"] for config in configurations]+ failures = [+ f"{config['id']}: {failure}"+ for config, result in zip(configurations, results, strict=True)+ if (failure := EvaluationService.failure_reason(result))+ ]+ if failures:+ return {"status": "incomplete", "reasons": failures, "agents": {}}+ if any(not isinstance(result.get("agents"), Mapping) for result in results):+ return {"status": "incomplete", "reasons": ["Missing agent results"], "agents": {}}+ agent_names = sorted(set().union(*(set(result.get("agents", {})) for result in results)))+ compared = {}+ for agent in agent_names:+ try:+ signatures, case_contracts, outcomes = [], [], []+ for config, result in zip(configurations, results, strict=True):+ failure = EvaluationService.failure_reason(result)+ if failure:+ raise IncomparableInteraction(failure)+ info = result.get("agents", {}).get(agent)+ if not isinstance(info, dict) or not info.get("model"):+ raise IncomparableInteraction(f"Missing agent/model identity: {agent}")+ run_dir = Path(result["run_dir"])+ run_config = result.get("run_config", {})+ if not isinstance(run_config, Mapping):+ raise IncomparableInteraction("Missing run configuration identity")+ harbor = run_config.get("harbor", {})+ if not isinstance(harbor, Mapping):+ raise IncomparableInteraction("Missing attempt configuration")+ if harbor.get("n_attempts") != n_attempts or harbor.get("stop_on_pass") is not False:+ raise IncomparableInteraction("Unequal attempt budgets or early stopping")+ grading = run_config.get("grading", {})+ if not isinstance(grading, Mapping):+ raise IncomparableInteraction("Missing grading configuration")+ custom_only = grading.get("mode") == "custom_only"+ if (metric == "overall") != custom_only:+ raise IncomparableInteraction(+ "Use overall only for custom_only grading; otherwise select an outcome metric"+ )+ judge_identity = run_config.get("judge")+ if isinstance(judge_identity, Mapping):+ # Each arm probes the model catalog independently.+ judge_identity = {k: v for k, v in judge_identity.items() if k != "catalog_verification"}+ if not judge_identity or not harbor.get("environment") or not result.get("evaluator_version"):+ raise IncomparableInteraction("Missing scorer, environment, or evaluator identity")+ signatures.append(+ (+ info["model"],+ judge_identity,+ grading,+ harbor.get("environment"),+ result.get("evaluator_version"),+ )+ )+ cases = _case_contract(result, run_dir)+ rows = _outcome_rows(run_dir, agent, config["condition"], metric, custom_only=custom_only)+ if rows.keys() != cases.keys() or any(len(attempts) != n_attempts for attempts in rows.values()):+ raise IncomparableInteraction("Incomplete or unequal per-case attempt coverage")+ case_contracts.append(cases)+ outcomes.append(rows)+ if any(signature != signatures[0] for signature in signatures[1:]):+ raise IncomparableInteraction("Agent model, scorer, environment, or evaluator identities differ")+ if any(cases != case_contracts[0] for cases in case_contracts[1:]):+ raise IncomparableInteraction("Task suite identities or case contents differ")+ per_case = {}+ for case_id in sorted(case_contracts[0]):+ scorers = {json.dumps(row["scorer"], sort_keys=True) for rows in outcomes for row in rows[case_id]}+ if len(scorers) != 1:+ raise IncomparableInteraction(+ f"Outcome scorer differs across configurations or attempts: {case_id}"+ )+ means = [math.fsum(row["score"] for row in rows[case_id]) / n_attempts for rows in outcomes]+ per_case[case_id] = interaction_scores(means[0], means[1:-1], means[-1])+ means = [+ math.fsum(math.fsum(row["score"] for row in rows[case]) / n_attempts for case in per_case)+ / len(per_case)+ for rows in outcomes+ ]+ compared[agent] = {+ "status": "complete",+ **interaction_scores(means[0], means[1:-1], means[-1]),+ "cases": per_case,+ "evidence": {config["id"]: rows for config, rows in zip(configurations, outcomes, strict=True)},+ }+ except (OSError, ValueError, KeyError, TypeError) as exc:+ compared[agent] = {"status": "incomparable", "reasons": [str(exc)]}+ if not compared:+ return {"status": "incomplete", "reasons": ["No agent results were retained"], "agents": {}}+ complete = all(agent["status"] == "complete" for agent in compared.values())+ return {"status": "complete" if complete else "incomparable", "agents": compared}+++def _selected_skills(skills: Sequence[Path]) -> list[Path]:+ from skillevaluator.tier3.harbor.runner import _workspace_skill_name+ from skillevaluator.tier3.workflow import _validate_skill_root++ if len(skills) < 2:+ raise ValueError("Interaction evaluation requires at least two selected skills")+ selected, paths, names = [], set(), set()+ for path in skills:+ _validate_skill_root(path)+ path = path.resolve()+ name = _workspace_skill_name(path).casefold()+ if path in paths or name in names or path.name.casefold() in {item.name.casefold() for item in selected}:+ raise ValueError("Selected skills must have distinct paths, directory names, and skill names")+ if any(path.is_relative_to(other) or other.is_relative_to(path) for other in paths):+ raise ValueError("Selected skill roots must not overlap")+ selected.append(path)+ paths.add(path)+ names.add(name)+ return selected+++def _validate_source(source: Path, metric: str) -> None:+ from skillevaluator.tier3.commands import _validate_existing_scaffold_state+ from skillevaluator.tier3.evals_config import load_evals_config+ from skillevaluator.tier3.evals_spec import validate_tier3_source++ _validate_existing_scaffold_state(source, source / "evals")+ _, checks = validate_tier3_source(source)+ errors = [check.message for check in checks if check.status in {"error", "missing"}]+ if errors:+ raise ValueError("Invalid fixed evaluation source: " + "; ".join(errors))+ config, _ = load_evals_config(source)+ workspace = config.get("skill_workspace", {})+ if workspace.get("include") or "decoy_manifest" in workspace:+ raise ValueError(+ "Interaction skills are explicit; remove skill_workspace.include/decoy_manifest from the source"+ )+ if config.get("harbor", {}).get("max_cases"):+ raise ValueError("Interaction evaluation requires the full reviewed suite; remove harbor.max_cases")+ if config.get("harbor", {}).get("pre_agent_setup"):+ raise ValueError("Interaction evaluation requires a no-skill baseline; remove harbor.pre_agent_setup")+ custom_only = config.get("grading", {}).get("mode") == "custom_only"+ if metric not in OUTCOME_METRICS or (metric == "overall") != custom_only:+ raise ValueError("Select goal_accuracy or accuracy for standard grading, or overall for custom_only grading")+++def run_interaction(+ skills: Sequence[Path],+ *,+ eval_source: Path,+ options: EvaluationOptions,+ metric: str = "goal_accuracy",+ on_configuration: Callable[[str], None] | None = None,+ progress_reporter: ProgressReporter | None = None,+) -> dict[str, Any]:+ """Freeze reviewed inputs once, then delegate every arm to the Tier 3 service."""+ from skillevaluator.tier3.evals_config import parse_environment_kwarg_overrides+ from skillevaluator.tier3.harbor.adapter import (+ private_evaluator_skill_snapshot,+ validate_output_provenance_key_location,+ )+ from skillevaluator.tier3.harbor.runner import _reserve_run_dir+ from skillevaluator.tier3.results_location import ENV_RESULTS_DIR+ from skillevaluator.tier3.workflow import _validate_skill_root++ selected = _selected_skills(skills)+ _validate_skill_root(eval_source)+ source = eval_source.resolve()+ if not (source / "evals").is_dir():+ raise ValueError(f"No evaluator source directory found at {source / 'evals'}")+ parse_environment_kwarg_overrides(options.environment_kwarg, env_mode=options.env_mode)+ if (+ options.include_skills+ or options.copy_repo+ or options.skip_baseline+ or options.skip_with_skill+ or options.max_cases+ ):+ raise ValueError("Interaction evaluation controls skill selection and requires full, uncopied task suites")+ attempts = options.n_attempts if options.n_attempts is not None else 1+ if isinstance(attempts, bool) or not isinstance(attempts, int) or attempts < 1:+ raise ValueError("n_attempts must be a positive integer")+ root = Path(options.results_dir or os.environ.get(ENV_RESULTS_DIR) or source.parent / "interaction-results")+ root = root.expanduser().resolve()+ if any(root.is_relative_to(path) or path.is_relative_to(root) for path in [source, *selected]):+ raise ValueError("Save interaction results outside the evaluation source and selected skill roots")+ validate_output_provenance_key_location(source, root, reference_skills_dir=None, workspace_skill_paths=selected)+ service = EvaluationService()+ with (+ private_evaluator_skill_snapshot(source) as evaluator,+ tempfile.TemporaryDirectory(prefix="skillevaluator-interaction-") as directory,+ ):+ _validate_source(evaluator, metric)+ frozen, identities = [], []+ for index, path in enumerate(selected):+ isolated_root = Path(directory) / f"skill-{index + 1}"+ isolated_root.mkdir()+ destination = isolated_root / path.name+ digest = decoy_source_digest(path)+ if digest is None:+ raise ValueError(f"Unsafe or unstable selected skill: {path}")+ copytree_secure(path, destination, allowed_root=path, ignore=decoy_source_copy_ignore(path))+ if decoy_source_digest(destination) != digest:+ raise ValueError(f"Skill source changed while freezing: {path}")+ identities.append({"path": str(path), "name": path.name, "source_digest": digest})+ copytree_secure(evaluator / "evals", destination / "evals", allowed_root=evaluator)+ frozen.append(destination)+ run_dir = _reserve_run_dir(root, datetime.now(UTC).strftime("interaction_%Y%m%d_%H%M%S"))+ copytree_secure(evaluator / "evals", run_dir / "source" / "evals", allowed_root=evaluator)+ report: dict[str, Any] = {+ "schema_version": 1,+ "kind": "skill_interaction",+ "definition": "[P(S) - P(empty)] - sum(P(skill) - P(empty) for skill in S)",+ "interpretation": "Total departure from additive individual gains; not an isolated higher-order interaction.",+ "metric": metric,+ "n_attempts": attempts,+ "source": {"path": str(source), "fingerprint": tree_content_fingerprint_secure(evaluator / "evals")},+ "skill_digest_algorithm": DECOY_SOURCE_DIGEST_ALGORITHM,+ "skills": identities,+ "run_dir": str(run_dir),+ "configurations": [],+ "status": "incomplete",+ }+ report_path = run_dir / "interaction.json"+ plan = [("baseline", 0, (), "without-skill")]+ plan.extend((f"individual-{index + 1}", index, (), "with-skill") for index in range(len(frozen)))+ plan.append(("combined", 0, tuple(frozen[1:]), "with-skill"))+ try:+ for config_id, index, companions, condition in plan:+ config: dict[str, Any] = {+ "id": config_id,+ "condition": condition,+ "skills": []+ if config_id == "baseline"+ else [index]+ if config_id != "combined"+ else list(range(len(frozen))),+ }+ report["configurations"].append(config)+ report_path.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")+ if on_configuration:+ on_configuration(config_id)+ try:+ result = service.evaluate(+ replace(+ options,+ skill_path=frozen[index],+ results_dir=run_dir / config_id,+ skip_baseline=condition == "with-skill",+ skip_with_skill=condition == "without-skill",+ n_attempts=attempts,+ stop_on_pass=False,+ skill_workspace_mode="group" if companions else "isolated",+ include_skills=companions,+ ),+ progress_reporter=progress_reporter,+ )+ config["result"] = result+ if failure := service.failure_reason(result):+ config["error"] = failure+ except Exception as exc:+ from skillevaluator.tier3.harbor.progress import (+ redact_progress_detail,+ secret_values_from_environment,+ )++ redacted = redact_progress_detail(+ str(exc), secret_values=secret_values_from_environment(os.environ)+ )+ config["error"] = redacted or type(exc).__name__+ report.update(compare_configurations(report["configurations"], metric=metric, n_attempts=attempts))+ except KeyboardInterrupt:+ report["reasons"] = ["Interaction evaluation interrupted before all configurations completed"]+ raise+ finally:+ report_path.write_text(json.dumps(report, indent=2, allow_nan=False) + "\n", encoding="utf-8")+ return reportdiff --git a/tests/ci/test_gitlab_template.py b/tests/ci/test_gitlab_template.pyindex 9cd30be6..279a8dce 100644--- a/tests/ci/test_gitlab_template.py+++ b/tests/ci/test_gitlab_template.py@@ -40,9 +40,12 @@ def _run_template(tmp_path, extra_env, *, scanner_exit=0, existing_report=False)"done\n"'printf \'%s\\n%s\\n%s\\n%s\\n\' "$SCAN_MODE" "${DIFF_BASE_SHA:-}" "${DIFF_HEAD_SHA:-}" "$MIN_SCORE" > "$report_dir/evidence.txt"\n''printf \'%s\' "${REPORT_RUN_URL:-}" > "$report_dir/run_url.txt"\n'+ 'printf \'%s\' "$report_dir" > "$report_dir/temp_location.txt"\n'f"exit {scanner_exit}\n")stub.chmod(0o755)+ temporary = tmp_path / "temporary reports"+ temporary.mkdir()# Start from a clean environment so a suite run inside GitLab CI cannot leak its own job variables.outer = {key: value for key, value in os.environ.items() if not key.startswith(("CI_", "GITLAB_", "SKILL_EVAL_"))}env = {@@ -55,6 +58,7 @@ def _run_template(tmp_path, extra_env, *, scanner_exit=0, existing_report=False)"SKILL_EVAL_PATHS": "skills","SKILL_EVAL_MIN_SCORE": "80","SKILL_EVAL_SCAN_MODE": "auto",+ "TMPDIR": str(temporary),**extra_env,}result = subprocess.run(["bash", "-c", script], env=env, capture_output=True, text=True, check=False)@@ -68,6 +72,17 @@ def test_gitlab_job_runs_all_skills_on_push_and_keeps_output(tmp_path):assert evidence == ["all", "", "", "80"]+def test_gitlab_job_uses_and_cleans_configured_temporary_storage(tmp_path):+ temporary = tmp_path / "custom temporary storage"+ temporary.mkdir()+ result, evidence = _run_template(tmp_path, {"TMPDIR": str(temporary)})+ assert result.returncode == 0, result.stderr+ assert evidence == ["all", "", "", "80"]+ location = Path((tmp_path / "workspace/skill-evaluator-report/temp_location.txt").read_text())+ assert location.parent == temporary+ assert not location.exists()++def test_gitlab_baseline_opt_in_rejects_noncanonical_boolean_and_inherited_policy(tmp_path):result, evidence = _run_template(tmp_path,diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex 469ba6c5..7d879ddb 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -3735,6 +3735,150 @@}]},+ "interact": {+ "params": [+ {+ "name": "skills",+ "opts": [+ "skills"+ ],+ "param_type": "argument",+ "required": true,+ "type": "directory"+ },+ {+ "name": "eval_source",+ "opts": [+ "--eval-source"+ ],+ "param_type": "option",+ "required": true,+ "type": "directory"+ },+ {+ "name": "agents",+ "opts": [+ "--agents",+ "-a"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "model",+ "opts": [+ "--model"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "choices": [+ "docker",+ "kata",+ "daytona",+ "e2b",+ "modal",+ "runloop",+ "langsmith",+ "ec2",+ "gke",+ "ack",+ "openshift",+ "novita",+ "apple-container",+ "singularity",+ "islo",+ "tensorlake",+ "cwsandbox",+ "wandb",+ "use-computer",+ "blaxel",+ "beam",+ "skypilot",+ "hyperbrowser",+ "vercel",+ "runta",+ "mosaic",+ "local"+ ],+ "default": "docker",+ "name": "env_mode",+ "opts": [+ "--env-mode"+ ],+ "param_type": "option"+ },+ {+ "multiple": true,+ "name": "environment_kwarg",+ "opts": [+ "--ek",+ "--environment-kwarg"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "default": "1",+ "name": "n_attempts",+ "opts": [+ "--n-attempts"+ ],+ "param_type": "option",+ "type": "integer range"+ },+ {+ "name": "n_concurrent",+ "opts": [+ "--n-concurrent"+ ],+ "param_type": "option",+ "type": "integer range"+ },+ {+ "choices": [+ "goal_accuracy",+ "accuracy",+ "overall"+ ],+ "default": "goal_accuracy",+ "name": "metric",+ "opts": [+ "--metric"+ ],+ "param_type": "option"+ },+ {+ "name": "results_dir",+ "opts": [+ "--results-dir"+ ],+ "param_type": "option",+ "type": "directory"+ },+ {+ "default": "False",+ "is_flag": true,+ "name": "harbor_keep_jobs",+ "opts": [+ "--harbor-keep-jobs"+ ],+ "param_type": "option",+ "type": "boolean"+ },+ {+ "default": "False",+ "is_flag": true,+ "name": "as_json",+ "opts": [+ "--json"+ ],+ "param_type": "option",+ "type": "boolean"+ }+ ]+ },"migrate-dataset": {"params": [{diff --git a/tests/inference/test_llm_client.py b/tests/inference/test_llm_client.pyindex 79090c22..abed12d1 100644--- a/tests/inference/test_llm_client.py+++ b/tests/inference/test_llm_client.py@@ -11,6 +11,7 @@ from pathlib import Pathfrom types import SimpleNamespacefrom unittest.mock import MagicMock, patch+import httpximport pytestfrom skillevaluator.constants import RUBRIC_CRITERIA@@ -200,17 +201,21 @@ class TestLLMClientGetClient:],)def test_rejects_noncanonical_aliases_normalized_to_native_openai_before_request(self, base_url: str) -> None:- client = LLMClient(- model="gpt-5.4-mini",- api_key="test-key",- base_url=base_url,- max_tokens=512,- )+ # This exercises SDK URL normalization without using a host proxy or making requests.+ transport = httpx.MockTransport(lambda _request: pytest.fail("Endpoint validation must precede requests"))+ with httpx.Client(transport=transport, trust_env=False) as http_client:+ client = LLMClient(+ model="gpt-5.4-mini",+ api_key="test-key",+ base_url=base_url,+ max_tokens=512,+ http_client=http_client,+ )- with pytest.raises(LLMClientError, match="noncanonical alias for the native OpenAI endpoint"):- client._get_client()+ with pytest.raises(LLMClientError, match="noncanonical alias for the native OpenAI endpoint"):+ client._get_client()- assert client._client is None+ assert client._client is None@pytest.mark.parametrize(("provider", "api_key_env", "base_url_env", "base_url"),diff --git a/tests/test_benchmark_source_provenance.py b/tests/test_benchmark_source_provenance.pyindex a6f05fd6..b73f9788 100644--- a/tests/test_benchmark_source_provenance.py+++ b/tests/test_benchmark_source_provenance.py@@ -1258,13 +1258,17 @@ class TestIdentityReachesTheRunArtifact:than the ``null`` a dropped option leaves behind."""import json+ import subprocessfrom skillevaluator.provider_config import ProviderConfigfrom skillevaluator.tier3 import commands as tier3_commandsfrom skillevaluator.tier3.harbor import runner, runtime_preflight- skill = tmp_path / "demo"+ skill = tmp_path / "fixture-repo" / "demo"(skill / "evals").mkdir(parents=True)+ # An explicit repository keeps ambient parent checkouts out of provenance checks.+ # The autouse fixture's private key remains outside this repository.+ subprocess.run(["git", "init", "-q", str(skill.parent)], check=True)(skill / "evals" / "evals.json").write_text("[]\n", encoding="utf-8")provider = ProviderConfig(provider="nv_build",@@ -1341,6 +1345,7 @@ class TestIdentityReachesTheRunArtifact:evaluated_source=identity,)+ assert "run_dir" in result, result.get("error")run_config_path = Path(result["run_dir"]) / "run_config.json"persisted = json.loads(run_config_path.read_text(encoding="utf-8"))assert persisted["evaluated_source"] == identitydiff --git a/tests/test_harbor_adapter_safety.py b/tests/test_harbor_adapter_safety.pyindex 56cd5441..5de4b96b 100644--- a/tests/test_harbor_adapter_safety.py+++ b/tests/test_harbor_adapter_safety.py@@ -143,9 +143,11 @@ def test_copy_repo_excludes_tracked_sensitive_files(tmp_path: Path) -> None:_assert_sensitive_repo_files_are_excluded(repo, skill, tmp_path / "tracked-task" / "environment")-def test_copy_repo_excludes_sensitive_files_without_git(tmp_path: Path) -> None:+def test_copy_repo_excludes_sensitive_files_without_git(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:repo = tmp_path / "plain-repo"skill = _write_repo_context_fixture(repo)+ # Keep this no-Git fixture independent of any checkout containing pytest's temp directory.+ monkeypatch.setattr("skillevaluator.utils.path_security.find_git_repo_root", lambda _path: None)_assert_sensitive_repo_files_are_excluded(repo, skill, tmp_path / "plain-task" / "environment")diff --git a/tests/tier3/test_interactions.py b/tests/tier3/test_interactions.pynew file mode 100644index 00000000..e2072a8d--- /dev/null+++ b/tests/tier3/test_interactions.py@@ -0,0 +1,754 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Numerical, evidence, orchestration, and CLI contracts for skill interactions."""++from __future__ import annotations++import copy+import json+import subprocess+from pathlib import Path+from typing import Any++import pytest+from click.testing import CliRunner++from skillevaluator.cli import cli+from skillevaluator.evaluation import EvaluationOptions, EvaluationService+from skillevaluator.tier3.harbor.metrics import CUSTOM_SCORE_POLICY, DEFAULT_METRIC_SET, DEFAULT_SCORE_POLICY+from skillevaluator.tier3.interactions import compare_configurations, interaction_scores, run_interaction+++def _skill(root: Path, name: str) -> Path:+ skill = root / name+ skill.mkdir(parents=True)+ (skill / "SKILL.md").write_text(+ f"---\nname: {name}\ndescription: Help with {name}.\n---\n# {name}\n", encoding="utf-8"+ )+ return skill+++def _source(root: Path, config: str = "") -> Path:+ # Keep the source checkout separate from the test operator's key, even with an in-repo TMPDIR.+ skill = _skill(root / "suite-repo", "reviewed-suite")+ subprocess.run(["git", "init", "-q", str(skill.parent)], check=True)+ (skill / "evals").mkdir()+ (skill / "evals" / "evals.json").write_text(+ json.dumps(+ {+ "skill_name": skill.name,+ "evals": [{"id": "case-1", "prompt": "Solve the task", "expected_output": "A correct answer"}],+ }+ ),+ encoding="utf-8",+ )+ if config:+ (skill / "evals" / "config.yml").write_text("schema_version: 1\n" + config, encoding="utf-8")+ return skill+++def _evidence(+ root: Path,+ values: dict[str, list[float]],+ *,+ n_attempts: int = 1,+ condition: str = "with-skill",+ custom_only: bool = False,+) -> dict[str, Any]:+ root.mkdir(parents=True)+ entries = [+ {+ "id": case_id,+ "question": f"Solve {case_id}",+ "ground_truth": "answer",+ "has_skill": condition == "with-skill",+ }+ for case_id in values+ ]+ (root / "dataset_snapshot.json").write_text(json.dumps({"dataset": entries}), encoding="utf-8")+ for case_id, scores in values.items():+ for ordinal, value in enumerate(scores):+ trial = f"{case_id}-{ordinal}"+ directory = root / "codex" / condition / "trials" / trial+ directory.mkdir(parents=True)+ reward = {+ "entry_id": case_id,+ "trial_id": trial,+ "goal_accuracy": value,+ "accuracy": value,+ "metric_set": "custom-only" if custom_only else DEFAULT_METRIC_SET,+ "score_policy": CUSTOM_SCORE_POLICY if custom_only else DEFAULT_SCORE_POLICY,+ "details": {+ "goal_accuracy": {"judge_model_used": "same-judge", "method": "custom"},+ "accuracy": {"judge_model_used": "same-judge", "method": "custom"},+ },+ **({"overall": value} if custom_only else {}),+ }+ (directory / "reward.json").write_text(json.dumps(reward), encoding="utf-8")+ return {+ "execution_status": "succeeded",+ "run_dir": str(root),+ "evaluator_version": "test-version",+ "agents": {"codex": {"model": "same-agent-model"}},+ "run_config": {+ "harbor": {+ "n_attempts": n_attempts,+ "stop_on_pass": False,+ "environment": {"value": "docker", "source": "CLI"},+ },+ "grading": {"mode": "custom_only" if custom_only else "default"},+ "judge": {"enabled": not custom_only, "model": "same-judge"},+ },+ }+++def _matrix(tmp_path: Path, scores=(0.50, 0.75, 0.70, 0.65), *, custom_only=False) -> list[dict[str, Any]]:+ configurations = []+ for index, score in enumerate(scores):+ condition = "without-skill" if index == 0 else "with-skill"+ configurations.append(+ {+ "id": f"config-{index}",+ "condition": condition,+ "result": _evidence(+ tmp_path / f"config-{index}", {"case-1": [score]}, condition=condition, custom_only=custom_only+ ),+ }+ )+ return configurations+++ ("baseline", "individuals", "combined", "interaction", "classification"),+ [+ (0.50, [0.75, 0.70], 0.65, -0.30, "negative"),+ (0.20, [0.30, 0.35], 0.60, 0.15, "positive"),+ (0.20, [0.30, 0.35], 0.45, 0.0, "zero"),+ (0.20, [0.30, 0.35, 0.40], 0.80, 0.15, "positive"),+ (0.50, [0.20, 0.30], 0.10, 0.10, "positive"),+ ],+)+def test_signed_interaction_equation(baseline, individuals, combined, interaction, classification):+ result = interaction_scores(baseline, individuals, combined)+ assert result["interaction"] == pytest.approx(interaction)+ assert result["classification"] == classification+ assert result["combined_lift"] == pytest.approx(combined - baseline)++[email protected]("invalid", [True, float("nan"), float("inf"), -0.1, 1.1, "0.5"])+def test_rejects_invalid_outcome_scores(invalid):+ with pytest.raises(ValueError, match="finite numbers"):+ interaction_scores(0.1, [0.2, invalid], 0.3)+++def test_requires_two_individuals():+ with pytest.raises(ValueError, match="at least two"):+ interaction_scores(0.1, [0.2], 0.3)+++def test_compares_paired_outcomes_and_keeps_trial_evidence(tmp_path):+ report = compare_configurations(_matrix(tmp_path), metric="goal_accuracy", n_attempts=1)+ assert report["status"] == "complete"+ score = report["agents"]["codex"]+ assert score["combined_lift"] == pytest.approx(0.15)+ assert score["interaction"] == pytest.approx(-0.30)+ assert score["classification"] == "negative"+ assert score["cases"]["case-1"]["interaction"] == pytest.approx(-0.30)+ assert Path(score["evidence"]["config-0"]["case-1"][0]["reward_path"]).is_file()+++def test_catalog_probe_status_does_not_change_judge_identity(tmp_path):+ matrix = _matrix(tmp_path)+ for arm in matrix:+ arm["result"]["run_config"]["judge"]["catalog_verification"] = "verified"+ judge = matrix[-1]["result"]["run_config"]["judge"]+ judge["catalog_verification"] = "degraded"++ report = compare_configurations(matrix, metric="goal_accuracy", n_attempts=1)+ assert report["status"] == "complete"+ assert judge["catalog_verification"] == "degraded"++ judge["override_applied"] = True+ report = compare_configurations(matrix, metric="goal_accuracy", n_attempts=1)+ assert report["status"] == "incomparable"+++def _edit_reward(config, change):+ run_dir = Path(config["result"]["run_dir"])+ path = next(run_dir.glob("codex/*/trials/*/reward.json"))+ data = json.loads(path.read_text())+ change(data)+ path.write_text(json.dumps(data), encoding="utf-8")+++ "change",+ [+ "budget",+ "early_stop",+ "model",+ "judge",+ "missing_judge",+ "case_contents",+ "missing_trial",+ "extra_trial",+ "duplicate_trial",+ "unknown_case",+ "nan",+ "not_applicable",+ "fallback",+ "method",+ "missing_identity",+ "partial",+ "metric_set",+ "multi_step",+ ],+)+def test_refuses_incomparable_evidence(tmp_path, change):+ matrix = _matrix(tmp_path)+ config = matrix[-1]+ result = config["result"]+ run_dir = Path(result["run_dir"])+ if change == "budget":+ result["run_config"]["harbor"]["n_attempts"] = 2+ elif change == "early_stop":+ result["run_config"]["harbor"]["stop_on_pass"] = True+ elif change == "model":+ result["agents"]["codex"]["model"] = "different-model"+ elif change == "judge":+ result["run_config"]["judge"]["model"] = "different-judge"+ elif change == "missing_judge":+ _edit_reward(config, lambda reward: reward.pop("details"))+ elif change == "case_contents":+ path = run_dir / "dataset_snapshot.json"+ path.write_text(json.dumps({"dataset": [{"id": "case-1", "question": "Changed task"}]}), encoding="utf-8")+ elif change == "missing_trial":+ next(run_dir.glob("codex/*/trials/*/reward.json")).unlink()+ elif change in {"extra_trial", "duplicate_trial", "multi_step"}:+ source = next(run_dir.glob("codex/*/trials/*/reward.json"))+ dest = source.parent.parent / "another-trial" / "reward.json"+ dest.parent.mkdir()+ data = json.loads(source.read_text())+ if change == "extra_trial":+ data["trial_id"] = "another-trial"+ dest.write_text(json.dumps(data), encoding="utf-8")+ elif change == "unknown_case":+ _edit_reward(config, lambda reward: reward.update(entry_id="unknown"))+ elif change == "nan":+ _edit_reward(config, lambda reward: reward.update(goal_accuracy=float("nan")))+ elif change == "not_applicable":+ _edit_reward(config, lambda reward: reward.update(goal_accuracy=None))+ _edit_reward(config, lambda reward: reward["details"]["goal_accuracy"].update(status="not_applicable"))+ elif change == "fallback":+ _edit_reward(+ config, lambda reward: reward["details"]["goal_accuracy"].update(judge_model_used="fallback-judge")+ )+ elif change == "method":+ _edit_reward(config, lambda reward: reward["details"]["goal_accuracy"].update(method="ragas"))+ elif change == "missing_identity":+ _edit_reward(config, lambda reward: reward.pop("trial_id"))+ elif change == "partial":+ result["run_config"]["case_selection"] = {"skipped": 1}+ elif change == "metric_set":+ _edit_reward(config, lambda reward: reward.update(metric_set="different-contract"))+ report = compare_configurations(matrix, metric="goal_accuracy", n_attempts=1)+ assert report["status"] == "incomparable"+ assert report["agents"]["codex"]["reasons"]+ assert "interaction" not in report["agents"]["codex"]+++def test_requires_identical_case_coverage_not_equal_total_counts(tmp_path):+ matrix = _matrix(tmp_path)+ _edit_reward(matrix[-1], lambda reward: reward.update(entry_id="different-case"))+ report = compare_configurations(matrix, metric="goal_accuracy", n_attempts=1)+ assert report["status"] == "incomparable"+++def test_equal_attempt_budgets_average_attempts_within_each_case(tmp_path):+ matrix = []+ for index, values in enumerate(+ [+ {"one": [0.2, 0.8], "two": [0.0, 0.0]},+ {"one": [0.5, 0.5], "two": [0.5, 0.5]},+ {"one": [0.5, 0.5], "two": [0.5, 0.5]},+ {"one": [1.0, 0.0], "two": [1.0, 1.0]},+ ]+ ):+ condition = "without-skill" if index == 0 else "with-skill"+ result = _evidence(tmp_path / str(index), values, n_attempts=2, condition=condition)+ matrix.append({"id": str(index), "condition": condition, "result": result})+ score = compare_configurations(matrix, metric="goal_accuracy", n_attempts=2)["agents"]["codex"]+ assert score["baseline"] == pytest.approx(0.25)+ assert score["combined"] == pytest.approx(0.75)+ assert score["interaction"] == pytest.approx(0.0)+++def test_custom_only_compares_shared_grader_overall(tmp_path):+ matrix = _matrix(tmp_path, custom_only=True)+ report = compare_configurations(matrix, metric="overall", n_attempts=1)+ assert report["status"] == "complete"+ assert report["agents"]["codex"]["interaction"] == pytest.approx(-0.3)+++def test_standard_overall_is_not_a_comparable_task_outcome(tmp_path):+ report = compare_configurations(_matrix(tmp_path), metric="overall", n_attempts=1)+ assert report["status"] == "incomparable"+++ "failure",+ [+ {"error": "launch failed"},+ {"execution_status": "failed"},+ {},+ {"execution_status": "succeeded", "execution_errors": ["unscored trial"]},+ ],+)+def test_failed_runs_never_produce_zero_or_numeric_interaction(tmp_path, failure):+ matrix = _matrix(tmp_path)+ matrix[-1]["result"] = failure+ report = compare_configurations(matrix, metric="goal_accuracy", n_attempts=1)+ assert report["status"] != "complete"+ assert all("interaction" not in agent for agent in report["agents"].values())++[email protected]("failure", ["missing_result", "empty_error", "empty_error_with_result"])+def test_missing_results_and_empty_errors_are_incomplete(tmp_path, failure):+ matrix = _matrix(tmp_path)+ if failure != "empty_error_with_result":+ del matrix[-1]["result"]+ if failure != "missing_result":+ matrix[-1]["error"] = ""+ report = compare_configurations(matrix, metric="goal_accuracy", n_attempts=1)+ assert report["status"] == "incomplete"+ assert report["reasons"] and all(report["reasons"])+ assert report["agents"] == {}+ assert "interaction" not in report+++def test_failed_configuration_reasons_identify_each_arm(tmp_path):+ matrix = _matrix(tmp_path)+ for config in matrix[1:]:+ config["error"] = "same setup failure"+ report = compare_configurations(matrix, metric="goal_accuracy", n_attempts=1)+ assert report["reasons"] == [f"config-{index}: same setup failure" for index in (1, 2, 3)]+++def test_terminal_deduplicates_failure_reasons(capsys):+ from skillevaluator.tier3.interaction_commands import _print_report++ _print_report(+ {+ "status": "incomplete",+ "metric": "goal_accuracy",+ "skills": [],+ "run_dir": "/results/experiment",+ "reasons": ["baseline: setup failed"] * 2,+ "agents": {"codex": {"status": "incomparable", "reasons": ["missing evidence"] * 2}},+ }+ )+ output = capsys.readouterr().out+ assert output.count("baseline: setup failed") == 1+ assert output.count("missing evidence") == 1+++def _fake_engine(monkeypatch, scores=(0.50, 0.75, 0.70, 0.65), *, on_run=None):+ calls = []++ def evaluate(self, options, **kwargs):+ index = len(calls)+ calls.append(copy.deepcopy(options))+ assert (options.skill_path / "evals" / "evals.json").is_file()+ staged_entry = on_run(index, options) if on_run else None+ condition = "without-skill" if options.skip_with_skill else "with-skill"+ result = _evidence(+ options.results_dir / "run",+ {"case-1": [scores[index]] * options.n_attempts},+ condition=condition,+ n_attempts=options.n_attempts,+ )+ if staged_entry:+ (Path(result["run_dir"]) / "dataset_snapshot.json").write_text(+ json.dumps({"dataset": [staged_entry]}), encoding="utf-8"+ )+ return result++ monkeypatch.setattr(EvaluationService, "evaluate", evaluate)+ return calls+++def test_orchestrates_frozen_inputs_and_only_selected_skills(monkeypatch, tmp_path):+ source = _source(tmp_path, "harbor:\n n_attempts: 7\n stop_on_pass: true\n")+ skills = [_skill(tmp_path, name) for name in ("a-skill", "b-skill")]+ original_evals = (source / "evals" / "evals.json").read_bytes()+ original_skill = (skills[1] / "SKILL.md").read_text()+ source_copies = []++ def inspect(index, options):+ source_copies.append((options.skill_path / "evals" / "evals.json").read_bytes())+ if index == 0:+ (skills[1] / "SKILL.md").write_text("changed after freezing", encoding="utf-8")+ if index == 2:+ assert (options.skill_path / "SKILL.md").read_text() == original_skill+ assert not (options.skill_path / "evals" / "results").exists()++ calls = _fake_engine(monkeypatch, on_run=inspect)+ report = run_interaction(skills, eval_source=source, options=EvaluationOptions(source, n_attempts=2))+ assert report["status"] == "complete"+ assert len(calls) == 4+ assert calls[0].skip_with_skill and not calls[0].skip_baseline+ assert all(call.skip_baseline and not call.skip_with_skill for call in calls[1:])+ assert [call.skill_path.name for call in calls] == ["a-skill", "a-skill", "b-skill", "a-skill"]+ assert [len(call.include_skills) for call in calls] == [0, 0, 0, 1]+ assert calls[-1].include_skills[0].name == "b-skill"+ assert [call.skill_workspace_mode for call in calls] == ["isolated", "isolated", "isolated", "group"]+ assert all(call.n_attempts == 2 and call.stop_on_pass is False for call in calls)+ assert source_copies == [original_evals] * 4+ assert len({str(call.results_dir) for call in calls}) == 4+ assert (source / "evals" / "evals.json").read_bytes() == original_evals+ saved = json.loads((Path(report["run_dir"]) / "interaction.json").read_text())+ assert saved == report+ assert len(saved["skills"]) == 2+ assert saved["source"]["fingerprint"]+ assert (Path(report["run_dir"]) / "source" / "evals" / "evals.json").read_bytes() == original_evals+++def test_group_runs_n_plus_two_configurations(monkeypatch, tmp_path):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b", "c")]+ calls = _fake_engine(monkeypatch, (0.2, 0.3, 0.35, 0.4, 0.8))+ report = run_interaction(skills, eval_source=source, options=EvaluationOptions(source))+ assert len(calls) == 5+ assert len(calls[-1].include_skills) == 2+ assert report["agents"]["codex"]["interaction"] == pytest.approx(0.15)++[email protected]("use_cli_root", [False, True])+def test_results_root_honors_environment_and_explicit_precedence(monkeypatch, tmp_path, use_cli_root):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ env_root = tmp_path / "env-results"+ cli_root = tmp_path / "cli-results" if use_cli_root else None+ monkeypatch.setenv("SKILLEVALUATOR_RESULTS_DIR", str(env_root))+ _fake_engine(monkeypatch)+ report = run_interaction(skills, eval_source=source, options=EvaluationOptions(source, results_dir=cli_root))+ assert Path(report["run_dir"]).parent == (cli_root or env_root).resolve()+++def test_results_root_resolves_symlinked_parent(monkeypatch, tmp_path):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ real_root = tmp_path / "real-results"+ real_root.mkdir()+ linked_root = tmp_path / "linked-results"+ linked_root.symlink_to(real_root, target_is_directory=True)+ _fake_engine(monkeypatch)+ report = run_interaction(+ skills, eval_source=source, options=EvaluationOptions(source, results_dir=linked_root / "runs")+ )+ assert Path(report["run_dir"]).parent == real_root / "runs"+++def test_results_root_symlink_into_selected_skill_is_rejected(monkeypatch, tmp_path):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ linked_root = tmp_path / "linked-results"+ linked_root.symlink_to(skills[1], target_is_directory=True)+ calls = _fake_engine(monkeypatch)+ with pytest.raises(ValueError, match="Save interaction results outside"):+ run_interaction(skills, eval_source=source, options=EvaluationOptions(source, results_dir=linked_root / "runs"))+ assert calls == []+++def test_existing_harbor_staging_installs_the_configuration_and_keeps_tasks_fixed(monkeypatch, tmp_path):+ from skillevaluator.tier3.harbor.adapter import generate_harbor_tasks++ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ staged = []++ def inspect(index, options):+ # Bound the fixture's repository context even when TMPDIR is inside a checkout.+ subprocess.run(["git", "init", "-q", str(options.skill_path.parent)], check=True)+ task = generate_harbor_tasks(+ options.skill_path,+ tmp_path / f"staged-{index}",+ with_skill=not options.skip_with_skill,+ workspace_skill_paths=list(options.include_skills),+ workspace_mode=options.skill_workspace_mode,+ )[0]+ installed = {path.parent.name for path in (task / "environment" / "skills").glob("*/SKILL.md")}+ entry = json.loads((task / "tests" / "entry.json").read_text())+ staged.append((installed, entry, (task / "instruction.md").read_text()))+ return entry++ _fake_engine(monkeypatch, on_run=inspect)+ report = run_interaction(skills, eval_source=source, options=EvaluationOptions(source))+ assert report["status"] == "complete", report.get("reasons")+ assert [row[0] for row in staged] == [set(), {"a"}, {"b"}, {"a", "b"}]+ assert len({row[2] for row in staged}) == 1+ assert len({row[1]["ground_truth"] for row in staged}) == 1+ assert [row[1]["has_skill"] for row in staged] == [False, True, True, True]+++def test_native_harbor_source_reuses_the_reviewed_tasks(monkeypatch, tmp_path):+ from skillevaluator.tier3.commands import init_harbor_task+ from skillevaluator.tier3.harbor.adapter import stage_native_harbor_tasks++ source = _skill(tmp_path, "reviewed-native")+ subprocess.run(["git", "init", "-q", str(source)], check=True)+ init_harbor_task(source, force=False, case_id="case-1", mode="default", language="python", with_config=True)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ staged = []++ def inspect(index, options):+ subprocess.run(["git", "init", "-q", str(options.skill_path.parent)], check=True)+ task = stage_native_harbor_tasks(+ options.skill_path,+ tmp_path / f"native-staged-{index}",+ with_skill=not options.skip_with_skill,+ workspace_skill_paths=list(options.include_skills),+ workspace_mode=options.skill_workspace_mode,+ )[0]+ staged.append((task / "instruction.md").read_bytes())+ return json.loads((task / "tests" / "entry.json").read_text())++ _fake_engine(monkeypatch, on_run=inspect)+ report = run_interaction(skills, eval_source=source, options=EvaluationOptions(source))+ assert report["status"] == "complete", report.get("reasons")+ assert len(staged) == 4 and all(instruction == staged[0] for instruction in staged)+++def test_interrupted_experiment_retains_explicit_incomplete_report(monkeypatch, tmp_path):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]++ def interrupt(self, options, **kwargs):+ raise KeyboardInterrupt++ monkeypatch.setattr(EvaluationService, "evaluate", interrupt)+ with pytest.raises(KeyboardInterrupt):+ run_interaction(skills, eval_source=source, options=EvaluationOptions(source))+ saved = json.loads(next((source.parent / "interaction-results").glob("*/interaction.json")).read_text())+ assert saved["status"] == "incomplete"+ assert "interrupted" in saved["reasons"][0]+ assert "interaction" not in saved++[email protected]("location", ["source", "repo", "second_skill", "results"])+def test_checks_original_provenance_key_location_before_snapshot(monkeypatch, tmp_path, location):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ results = tmp_path / "results"+ roots = {"source": source, "repo": source.parent, "second_skill": skills[1], "results": results}+ monkeypatch.setenv("SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE", str(roots[location] / "marker.key"))++ def unexpected_snapshot(*args, **kwargs):+ pytest.fail("Input validation must finish before snapshotting")++ monkeypatch.setattr("skillevaluator.tier3.harbor.adapter.private_evaluator_skill_snapshot", unexpected_snapshot)+ with pytest.raises(ValueError, match="Output provenance key must be outside"):+ run_interaction(skills, eval_source=source, options=EvaluationOptions(source, results_dir=results))+ assert not results.exists()+++ "invalid",+ [+ "one_skill",+ "duplicate",+ "same_name",+ "overlap",+ "inside_output",+ "source_selection",+ "partial",+ "standard_overall",+ "missing_source",+ "missing_evals_dir",+ "malformed_environment_kwarg",+ ],+)[email protected]("via_cli", [False, True])+def test_rejects_invalid_experiments_before_execution(monkeypatch, tmp_path, invalid, via_cli):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ options = EvaluationOptions(source)+ metric = "goal_accuracy"+ if invalid == "one_skill":+ skills.pop()+ elif invalid == "duplicate":+ skills[1] = skills[0]+ elif invalid == "same_name":+ (skills[1] / "SKILL.md").write_text("---\nname: a\n---\n", encoding="utf-8")+ elif invalid == "overlap":+ skills[1] = _skill(skills[0], "nested")+ elif invalid == "inside_output":+ options.results_dir = skills[0] / "evals" / "results"+ elif invalid == "source_selection":+ (source / "evals" / "config.yml").write_text(+ "schema_version: 1\nskill_workspace:\n mode: group\n include: [a]\n", encoding="utf-8"+ )+ elif invalid == "partial":+ (source / "evals" / "config.yml").write_text("schema_version: 1\nharbor:\n max_cases: 1\n", encoding="utf-8")+ elif invalid == "standard_overall":+ metric = "overall"+ elif invalid == "missing_source":+ (source / "evals" / "evals.json").unlink()+ elif invalid == "missing_evals_dir":+ (source / "evals").rename(source / "unused")+ elif invalid == "malformed_environment_kwarg":+ options.environment_kwarg = ("malformed",)+ calls = _fake_engine(monkeypatch)+ if via_cli:+ args = ["tier3", "interact", *(str(path) for path in skills), "--eval-source", str(source), "--metric", metric]+ if options.results_dir:+ args.extend(["--results-dir", str(options.results_dir)])+ for value in options.environment_kwarg:+ args.extend(["--ek", value])+ result = CliRunner().invoke(cli, args)+ assert result.exit_code == 2, result.output+ assert "Usage:" in result.stderr+ else:+ with pytest.raises((OSError, ValueError)):+ run_interaction(skills, eval_source=source, options=options, metric=metric)+ assert calls == []+ assert not (source.parent / "interaction-results").exists()+++ ("failure", "expected_error"),+ [+ (RuntimeError("a configuration failed"), "a configuration failed"),+ (TimeoutError(), "TimeoutError"),+ ],+)+def test_preserves_other_configuration_evidence_after_failure(monkeypatch, tmp_path, failure, expected_error):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ counter = []++ def failing(self, options, **kwargs):+ counter.append(options)+ if len(counter) == 2:+ raise failure+ return _evidence(+ options.results_dir / "run",+ {"case-1": [0.5]},+ condition="without-skill" if options.skip_with_skill else "with-skill",+ )++ monkeypatch.setattr(EvaluationService, "evaluate", failing)+ report = run_interaction(skills, eval_source=source, options=EvaluationOptions(source))+ assert report["status"] == "incomplete"+ assert len(counter) == 4+ assert report["configurations"][1]["error"] == expected_error+ assert all("result" in report["configurations"][index] for index in (0, 2, 3))+ assert "interaction" not in report+ assert json.loads((Path(report["run_dir"]) / "interaction.json").read_text()) == report++[email protected]("as_json", [False, True])+def test_cli_reports_signed_interaction_and_keeps_json_clean(monkeypatch, tmp_path, as_json):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]+ calls = _fake_engine(monkeypatch)+ args = [+ "tier3",+ "interact",+ *(str(path) for path in skills),+ "--eval-source",+ str(source),+ "--n-attempts",+ "2",+ "--agents",+ "codex",+ "--model",+ "test-model",+ ]+ if as_json:+ args.append("--json")+ result = CliRunner().invoke(cli, args)+ assert result.exit_code == 0, result.output+ assert len(calls) == 4 and calls[0].agents == "codex" and calls[0].model == "test-model"+ if as_json:+ report = json.loads(result.stdout)+ assert report["agents"]["codex"]["classification"] == "negative"+ assert "running baseline" in result.stderr+ else:+ assert "Interaction: -0.3000 (negative)" in result.stdout+ assert "combined lift: +0.1500" in result.stdout+++def test_cli_sends_each_arms_progress_to_stderr(monkeypatch, tmp_path):+ from skillevaluator.tier3.harbor.progress import ProgressEvent, Tier3RunPlan++ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]++ def evaluate(self, options, *, progress_reporter=None):+ if progress_reporter is not None:+ progress_reporter.start(+ Tier3RunPlan(skill_name=options.skill_path.name, environment=options.env_mode, agents=("codex",))+ )+ progress_reporter.emit(ProgressEvent(stage="fixture-trial", state="running"))+ progress_reporter.emit(ProgressEvent(stage="fixture-trial", state="complete"))+ progress_reporter.close()+ return _evidence(+ options.results_dir / "run",+ {"case-1": [0.5]},+ condition="without-skill" if options.skip_with_skill else "with-skill",+ )++ monkeypatch.setattr(EvaluationService, "evaluate", evaluate)+ result = CliRunner().invoke(+ cli, ["tier3", "interact", *(str(path) for path in skills), "--eval-source", str(source), "--json"]+ )+ assert result.exit_code == 0, result.output+ assert json.loads(result.stdout)["status"] == "complete"+ assert result.stderr.count("fixture-trial: running") == 4++[email protected]("failure", [None, TimeoutError(), ValueError("evaluation failed")])+def test_cli_incomplete_is_nonzero_and_machine_readable(monkeypatch, tmp_path, failure):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]++ def failing(self, options, **kwargs):+ if failure is not None:+ raise failure+ return {"error": ["launch failed"]}++ monkeypatch.setattr(EvaluationService, "evaluate", failing)+ result = CliRunner().invoke(+ cli, ["tier3", "interact", *(str(path) for path in skills), "--eval-source", str(source), "--json"]+ )+ assert result.exit_code == 1+ report = json.loads(result.stdout)+ assert report["status"] == "incomplete"+ assert "interaction" not in report+ assert all(config["error"] for config in report["configurations"])+++def test_cli_output_io_failure_uses_exit_one(monkeypatch, tmp_path):+ source = _source(tmp_path)+ skills = [_skill(tmp_path, name) for name in ("a", "b")]++ def fail_to_write(*args, **kwargs):+ raise OSError("Cannot write interaction results")++ monkeypatch.setattr("skillevaluator.tier3.interactions.run_interaction", fail_to_write)+ result = CliRunner().invoke(+ cli, ["tier3", "interact", *(str(path) for path in skills), "--eval-source", str(source)]+ )+ assert result.exit_code == 1+ assert "Cannot write interaction results" in result.stderr+++def test_cli_requires_two_skills_and_never_generates_a_source(monkeypatch, tmp_path):+ skill = _skill(tmp_path, "a")+ calls = _fake_engine(monkeypatch)+ result = CliRunner().invoke(cli, ["tier3", "interact", str(skill), "--eval-source", str(skill)])+ assert result.exit_code == 2+ assert "at least two" in result.output+ assert calls == []