Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(tier3): import production trace exports as reviewed eval cases (#378)
Christopher Kevinfe60fd3fbe15
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex eee51e90..1bece060 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -27,6 +27,10 @@ All notable changes to SkillEvaluator are documented in this file.versioned labels, with offline suite checks, repeated-grading statistics,provenance, optional rate gates and a bundled public reference suite.+- Import production trace exports into redacted review candidates with+ `tier3 import-traces`, then promote only explicitly human-reviewed expectations+ and replay fixtures with `tier3 accept-traces`.+- Tier 3 evaluations support token, judge-call, attempt, and priced-cost budgets.- Tier 3 reports list up to three run-level **Next steps** under the per-casediff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex b1d6cd90..e0e0bbee 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -45,6 +45,8 @@ commands referenced below.| [health-check](#health-check) | Quick CLI + backend readiness check, or another tier with `--tier` | Nothing || [models](#models) | List models visible through the selected provider's catalog | Provider configuration/key || [tier3 validate](#tier3-validate) | Validate `evals/` and the optional Harbor contract | Nothing |+| [tier3 import-traces](#tier3-import-traces) | Import redacted trace review candidates | Nothing |+| [tier3 accept-traces](#tier3-accept-traces) | Promote explicitly reviewed trace candidates | Nothing || [tier3 import-metrics](#tier3-import-metrics) | Report precomputed custom scores as external observations | A versioned metrics JSON file; no runtime or provider key |## tier3 import-metrics@@ -57,7 +59,7 @@ skillevaluator tier3 import-metrics INPUT.json --output-dir NEW_DIRECTORYnot exist. The command validates the complete input before writing`external_metrics.json` and `external_metrics.md`. It never executes tests orassigns canonical scores; exit 0 means a valid import, not passing tests.-See the [input contract and simulator example](custom-graders.mdx#import-precomputed-custom-metrics).+See the [input contract and simulator example](/skills/skillevaluator/custom-graders#import-precomputed-custom-metrics).## Global conventions@@ -178,12 +180,47 @@ pipeline.| --- | --- || `tier1` | `validate`, `quality-check`, `rubric-eval`, `security-scan`, `pii-scan`, `lint-scripts` || `tier2` | `context-optimization-check`, `dedup-scan`, `similarity-check` |-| `tier3` | `evaluate`, `select-decoys`, `create-eval-dataset`, `migrate-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view`, `import-metrics` |+| `tier3` | `evaluate`, `select-decoys`, `create-eval-dataset`, `migrate-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view`, `import-metrics`, `import-traces`, `accept-traces` |`health-check` and [models](#models) are top-level only. The expert commands[tier3 evaluate](#tier3-evaluate), [tier3 select-decoys](#tier3-select-decoys),-[tier3 feedback](#tier3-feedback), and `tier3 validate` are advertised **only**-inside the `tier3` group:+[tier3 feedback](#tier3-feedback), [tier3 import-metrics](#tier3-import-metrics),+[tier3 import-traces](#tier3-import-traces),+[tier3 accept-traces](#tier3-accept-traces), and `tier3 validate` are advertised+**only** inside the `tier3` group:++### tier3 import-traces++Import a production trace export into redacted candidates outside the skill.+See [Trace Datasets](trace-datasets.mdx) for the v1 JSON/JSONL contract and redaction limits.++```bash+skillevaluator tier3 import-traces ./my-skill ./sessions.jsonl -o ./review+```++| Option | Default | Purpose |+| --- | --- | --- |+| `-o`, `--output DIR` | `./<skill-dir-name>-trace-review` | Review bundle directory, outside the skill. |+| `--force` | off | Overwrite an existing candidates file. |+| `--dry-run` | off | Print classifications, warnings, and the next-step hint; write nothing. |++### tier3 accept-traces++Promote only explicitly reviewed candidates into `evals/evals.json`. Requires+`SKILL_PATH` and `REVIEW_DIR`. `accept` requires a reviewer-written expectation;+`accept_observed` is allowed only for successful sessions with non-empty output.+Pending and rejected candidates are skipped. All decision/redaction errors abort+before writing; dataset validation errors after writing return a non-zero status.++```bash+skillevaluator tier3 accept-traces ./my-skill ./review --dry-run+skillevaluator tier3 accept-traces ./my-skill ./review+```++| Option | Default | Purpose |+| --- | --- | --- |+| `--dry-run` | off | Report additions and already-present ids without writing. |+| `--allow-native-harbor` | off | Explicitly accept traces when native Harbor tasks are selected. |### tier3 validatediff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdxindex d6d8a48a..21db8b51 100644--- a/docs/eval-datasets.mdx+++ b/docs/eval-datasets.mdx@@ -25,6 +25,13 @@ Tier 3 executes these cases throughevaluation framework. Native Harbor task bundles are the bring-your-own-taskalternative to the `evals.json` dataset described below.+## Candidates from production trace exports++Use `tier3 import-traces` to create a redacted review bundle outside the skill,+then `tier3 accept-traces` to promote only explicitly reviewed expectations.+Observed production behavior never automatically becomes an expected answer.+See [Trace Datasets](trace-datasets.mdx) for the export contract and review decisions.+## The evals/ directorySkillEvaluator recognizes this tree inside your skill:diff --git a/docs/trace-datasets.mdx b/docs/trace-datasets.mdxnew file mode 100644index 00000000..0b94b4c6--- /dev/null+++ b/docs/trace-datasets.mdx@@ -0,0 +1,121 @@+---+title: "Trace Datasets"+description: "Import a production trace export as redacted candidates, then promote only human-reviewed expectations into an eval dataset."+---++Production behavior is evidence, **not an expected answer**. Import creates a+review bundle outside the skill; nothing is promoted until a reviewer explicitly+chooses a decision. This workflow is local and does not call an LLM.++## Import and review++```bash+skillevaluator tier3 import-traces ./my-skill ./sessions.jsonl+skillevaluator tier3 import-traces ./my-skill ./sessions.json -o ./review --dry-run+```++The default output is `./my-skill-trace-review`, relative to the current working+directory. Output inside the skill is refused. Existing `candidates.json` requires+`--force` to replace it. `--dry-run` prints classifications without writing.+Keep the original export and review bundle private; do not ship either with a skill.++The bundle contains `candidates.json` and redacted replay inputs under+`fixtures/<candidate-id>/`. Each candidate has observed evidence, provenance,+redaction counts, warnings, and a pending review. Observed read/open calls for+files outside the skill that were not exported as fixtures produce+`missing replay input` warnings (deduplicated, capped at 20 per candidate).+Tool-derived+`suggested_assertions` are hints only, never promoted automatically.++## Export contract v1++Accepts JSONL (one session object per line), or a JSON wrapper:++```json+{+ "format": "skillevaluator.trace-export",+ "version": 1,+ "sessions": [+ {"session_id": "session-1", "prompt": "Summarize the input", "outcome": "unknown"}+ ]+}+```++Other wrapper formats and versions are rejected. JSONL uses the v1 session+contract without a wrapper. Records are numbered from 1 (physical line numbers+for JSONL).++| Field | Contract |+| --- | --- |+| `session_id` | Required non-empty string; hashed into a stable `trace-` id. |+| `prompt` | User request; falls back to the first user trajectory message. |+| `outcome` | `success`, `failure`, or `unknown` (default). Other values are unsupported. |+| `skill` | Optional skill name, matched against the frontmatter name or skill directory name. Non-matches are skipped. |+| `trajectory` | ATIF object containing `steps`. |+| `agent`, `agent_log` | Alternative raw log input; agents: `claude-code`, `cursor-cli`, `cline`, `opencode`, `codex`. Unknown formats retain a candidate with a warning and no tool summary. |+| `final_output` | Optional string; otherwise the last agent trajectory message. |+| `files` | List of objects with relative `path` and string `content`. Absolute/traversing paths and invalid content are dropped with incomplete warnings. |+| `source`, `started_at`, `tags` | Optional provenance, redacted before writing. |+| Other fields | Dropped, with unsupported-field warnings. |++The summary and output explicitly report candidates and skipped records:+`duplicate` (normalized prompt and identical fixture contents, pointing to the+earlier candidate), `incomplete` (missing id or prompt), `unsupported` (including+invalid JSON and non-object records), and `other skill`.++## Review decisions and promotion++Edit each candidate's `review` in `candidates.json`:++| Decision | Effect |+| --- | --- |+| `pending` | Skip; the default. |+| `reject` | Skip. |+| `accept` | Requires a non-empty, reviewer-written `expected_output`. Optional `assertions` is a list of strings. |+| `accept_observed` | Explicit opt-in to use the observed output, only for `success` with non-empty output. Failure/unknown outcomes require an independent expectation instead. |++Unknown decisions and missing expectations abort the whole operation before+writing. Write expectations that define correct behavior, not merely what a+production session happened to do. Move only independently reviewed suggestions+into `assertions`; `suggested_assertions` and review notes are not eval criteria.++```bash+skillevaluator tier3 accept-traces ./my-skill ./my-skill-trace-review --dry-run+skillevaluator tier3 accept-traces ./my-skill ./my-skill-trace-review+```++Accepted entries append to agentskills-shaped `evals/evals.json`. Legacy flat+lists or other dataset extensions require manual migration first. Fixtures copy+to `evals/trace-files/<candidate-id>/`; refs use+`trace-files/<candidate-id>/...`, outside the implicit `evals/files/` pool, so+they are not staged into unrelated entries. Fixture-free entries explicitly use `[]`.+Before accepting, the command checks which task source is selected. If native+Harbor tasks are selected, use `--allow-native-harbor` to explicitly accept the+source change: with `task_source: auto`, the new eval dataset takes precedence;+with `task_source: native_harbor`, the trace cases are added but are not run.+`trace_source` records the redacted session id, export hash, outcome, and review+decision. Re-running skips ids already present for the same session; a different+session colliding with an id is an error. Dataset validation runs after writing;+validation errors are printed and return a non-zero exit status.++## Redaction guarantees and limits++Before writing, import applies the shared credential redactor and configured PII+patterns for email addresses, phone numbers, SSNs, personal macOS/Windows paths,+and Linux `/home/<name>` paths (replaced by `/home/<user>`). Placeholder+exceptions in matching patterns are respected. Prompts, observed outputs, tool+summaries, fixtures, and provenance strings are redacted; raw logs and unknown+fields are not copied. Per-candidate counts record introduced secret-redaction markers and PII+matches. The export hash is calculated from the original bytes, not redacted text.++Promotion rechecks prompts, expectations, assertions, provenance, and fixtures;+if redaction would still change them, it refuses the candidate rather than+silently editing the reviewed expectation. Reviewers must sanitize newly pasted+material before trying again.++Pattern redaction is **best effort**, not a guarantee that every secret or person+is detected. Reviewers must still inspect all bundle files for sensitive data,+missing replay inputs, and suitability for publication. Review decisions are+explicit local metadata, not authenticated signatures or proof of who reviewed.++See [Eval Datasets](eval-datasets.mdx) for the promoted dataset contract.diff --git a/fern/docs.yml b/fern/docs.ymlindex b62df7c7..977d01b5 100644--- a/fern/docs.yml+++ b/fern/docs.yml@@ -62,6 +62,9 @@ navigation:- page: Eval Datasetspath: ../docs/eval-datasets.mdxslug: eval-datasets+ - page: Trace Datasets+ path: ../docs/trace-datasets.mdx+ slug: trace-datasets- page: Agents & Sandboxespath: ../docs/agents-and-sandboxes.mdxslug: agents-and-sandboxesdiff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex cba97bac..ed045d37 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -306,6 +306,63 @@ def tier3() -> None:"""[email protected]("import-traces")[email protected]("skill_path", type=click.Path(exists=True, file_okay=False, path_type=Path))[email protected]("export", type=click.Path(exists=True, dir_okay=False, path_type=Path))+ "--output",+ "-o",+ type=click.Path(file_okay=False, path_type=Path),+ help="Review bundle directory outside the skill (default: ./<skill-dir-name>-trace-review).",+)[email protected]("--force", is_flag=True, help="Overwrite an existing candidates file.")[email protected]("--dry-run", is_flag=True, help="Print the summary without writing files.")+def tier3_import_traces(skill_path: Path, export: Path, output: Path | None, force: bool, dry_run: bool) -> None:+ """Import a production trace export into redacted review candidates."""+ from skillevaluator.tier3.trace_import import import_traces++ output = output or Path.cwd() / f"{skill_path.resolve().name}-trace-review"+ try:+ document = import_traces(skill_path, export, output, force=force, dry_run=dry_run)+ except (ValueError, OSError) as exc:+ raise click.ClickException(str(exc)) from exc+ click.echo(f"candidate: {len(document['candidates'])}")+ for reason in ("duplicate", "incomplete", "unsupported", "other skill"):+ click.echo(f"{reason}: {sum(s['reason'] == reason for s in document['skipped'])}")+ for item in document["skipped"]:+ click.echo(json.dumps(item, ensure_ascii=False))+ for candidate in document["candidates"]:+ click.echo(f"{candidate['id']}: redactions={candidate['redactions']} warnings={candidate['warnings']}")+ click.echo(+ f"Next: review {output / 'candidates.json'}, then run skillevaluator tier3 accept-traces {skill_path} {output}"+ )++[email protected]("accept-traces")[email protected]("skill_path", type=click.Path(exists=True, file_okay=False, path_type=Path))[email protected]("review_dir", type=click.Path(exists=True, file_okay=False, path_type=Path))[email protected]("--dry-run", is_flag=True, help="Report additions without writing files.")+ "--allow-native-harbor",+ is_flag=True,+ help="Explicitly accept traces when native Harbor tasks are selected.",+)+def tier3_accept_traces(skill_path: Path, review_dir: Path, dry_run: bool, allow_native_harbor: bool) -> None:+ """Promote only human-reviewed trace candidates into evals/evals.json."""+ from skillevaluator.tier3.trace_import import accept_traces++ try:+ result = accept_traces(skill_path, review_dir, dry_run=dry_run, allow_native_harbor=allow_native_harbor)+ except (ValueError, OSError, KeyError, TypeError) as exc:+ raise click.ClickException(str(exc)) from exc+ for key in ("added", "already present", "skipped"):+ click.echo(f"{'would add' if dry_run and key == 'added' else key}: {len(result[key])} {', '.join(result[key])}")+ for check in result["validation"]:+ click.echo(str(check))+ if any(check.status == "error" for check in result["validation"]):+ raise click.ClickException("Dataset validation failed after writing")++def _target_argument(func):return click.argument("target_path", type=click.Path(exists=True, resolve_path=True, path_type=Path))(func)diff --git a/src/skillevaluator/tier3/trace_import.py b/src/skillevaluator/tier3/trace_import.pynew file mode 100644index 00000000..3e7864b6--- /dev/null+++ b/src/skillevaluator/tier3/trace_import.py@@ -0,0 +1,564 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Import production trace exports as redacted, explicitly reviewed candidates."""++from __future__ import annotations++import hashlib+import json+import re+from functools import lru_cache+from pathlib import Path, PureWindowsPath+from typing import Any++from skillevaluator.config import load_pii_patterns+from skillevaluator.tier3.dataset_utils import find_eval_file+from skillevaluator.tier3.eval_core import log_converters+from skillevaluator.tier3.eval_core.codex_tool_call_normalizer import atif_content_text+from skillevaluator.tier3.evals_spec import _validate_dataset_contents, validate_tier3_source+from skillevaluator.tier3.generate_dataset import _parse_skill+from skillevaluator.utils.redaction import redact_sensitive_data, redact_sensitive_text++_CONVERTERS = {+ "claude-code": log_converters.synthetic_trajectory_from_claude_stream_jsonl,+ "cursor-cli": log_converters.synthetic_trajectory_from_cursor_cli,+ "cline": log_converters.synthetic_trajectory_from_cline_cli,+ "opencode": log_converters.synthetic_trajectory_from_opencode_json,+ "codex": log_converters.synthetic_trajectory_from_codex_json,+}+_FIELDS = {+ "session_id",+ "prompt",+ "outcome",+ "skill",+ "trajectory",+ "agent",+ "agent_log",+ "final_output",+ "files",+ "source",+ "started_at",+ "tags",+}+_SCIENTIFIC_LITERAL_PATTERN = re.compile(r"(?<![\w.])[-+]?(?:\d+(?:\.\d*)?|\.\d+)[eE][-+]?\d+")+_SIGNED_DECIMAL_PATTERN = re.compile(r"(?<![\w.])[+-]\d+\.\d+(?![\w.])")+_HHMM_OFFSET_PATTERN = re.compile(r"\b\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?P<offset>[+-]\d{4})\b")+++@lru_cache(maxsize=1)+def _pii_patterns() -> tuple[tuple[str, re.Pattern[str], tuple[str, ...]], ...]:+ configured = load_pii_patterns()+ patterns = [+ (category, re.compile(spec["pattern"]), tuple(spec.get("exceptions", [])))+ for category in ("personal_paths", "home_paths", "emails", "phone_numbers", "ssn")+ for spec in configured.get(category, [])+ ]+ patterns.extend(+ (+ ("home_paths", re.compile(r"/home/(?!<user>)[^/\s\"'<>]+/?"), ()),+ ("personal_paths", re.compile(r"/Users/(?!(?:<user>|\{[^/{}]+\}))[^/\s\"'<>]+/?"), ()),+ )+ )+ return tuple(patterns)+++def _redaction_markers(value: Any) -> int:+ if isinstance(value, str):+ return value.count("<redacted>")+ if isinstance(value, list):+ return sum(_redaction_markers(item) for item in value)+ if isinstance(value, dict):+ return sum(_redaction_markers(key) + _redaction_markers(item) for key, item in value.items())+ return 0+++def redact(value: Any, *, structured: bool = False) -> tuple[Any, dict[str, int]]:+ """Redact string leaves, counting introduced secret markers and PII matches."""+ counts = {"secrets": 0, "pii": 0}+ patterns = _pii_patterns()+ if structured:+ sanitized = redact_sensitive_data(value)+ counts["secrets"] += max(0, _redaction_markers(sanitized) - _redaction_markers(value))+ value = sanitized++ def clean(item: Any) -> Any:+ if isinstance(item, list):+ return [clean(v) for v in item]+ if isinstance(item, dict):+ cleaned = {}+ for key, value in item.items():+ cleaned_key = clean(key)+ if cleaned_key in cleaned:+ base = str(cleaned_key)+ suffix = 2+ while f"{base}-{suffix}" in cleaned:+ suffix += 1+ cleaned_key = f"{base}-{suffix}"+ cleaned[cleaned_key] = clean(value)+ return cleaned+ if not isinstance(item, str):+ return item+ text = redact_sensitive_text(item)+ counts["secrets"] += max(0, text.count("<redacted>") - item.count("<redacted>"))+ for category, pattern, exceptions in patterns:++ def replace(+ match: re.Match[str],+ category: str = category,+ exceptions: tuple[str, ...] = exceptions,+ text: str = text,+ ) -> str:+ found = match.group()+ if category == "home_paths":+ replacement = "/home/<user>" + ("/" if found.endswith("/") else "")+ elif category == "personal_paths":+ replacement = (+ "/Users/<user>" + ("/" if found.endswith("/") else "")+ if found.startswith("/")+ else "C:\\Users\\<user>"+ )+ else:+ replacement = "[redacted-pii]"+ if found == replacement or any(e in found for e in exceptions):+ return found+ if category == "phone_numbers":+ if "\n" in found or "\r" in found:+ return found+ window = (max(0, match.start() - 64), match.start() + 64)+ if any(+ literal.start() <= match.start() < literal.end()+ for literal in _SCIENTIFIC_LITERAL_PATTERN.finditer(text, *window)+ ):+ return found+ if any(+ number.start() <= match.start() and match.end() <= number.end()+ for number in _SIGNED_DECIMAL_PATTERN.finditer(text)+ ):+ return found+ if any(+ offset.start("offset") <= match.start() and match.end() <= offset.end("offset")+ for offset in _HHMM_OFFSET_PATTERN.finditer(text)+ ):+ return found+ counts["pii"] += 1+ return replacement++ text = pattern.sub(replace, text)+ return text++ return clean(value), counts+++def parse_export(raw: bytes) -> list[tuple[int, Any]]:+ """Parse a v1 wrapper or JSONL; malformed lines become unsupported records."""+ text = raw.decode("utf-8-sig")+ try:+ obj = json.loads(text)+ except json.JSONDecodeError:+ obj = None+ if isinstance(obj, dict) and "session_id" not in obj and ("format" in obj or "sessions" in obj):+ if (+ obj.get("format") != "skillevaluator.trace-export"+ or type(obj.get("version")) is not int+ or obj["version"] != 1+ ):+ raise ValueError("Expected skillevaluator.trace-export version 1")+ if not isinstance(obj.get("sessions"), list):+ raise ValueError("Export sessions must be a list")+ return list(enumerate(obj["sessions"], 1))+ records = []+ for number, line in enumerate(text.split("\n"), 1):+ if not line.strip():+ continue+ try:+ records.append((number, json.loads(line)))+ except json.JSONDecodeError:+ records.append((number, None))+ return records+++def safe_relative(value: Any) -> Path:+ if not isinstance(value, str) or not value or "\x00" in value or "\\" in value:+ raise ValueError("Fixture path must be a relative POSIX file path")+ path = Path(value)+ if path.is_absolute() or PureWindowsPath(value).drive or ".." in path.parts or not path.parts:+ raise ValueError("Fixture path must not be absolute or traverse parents")+ return path+++def contained(root: Path, relative: str) -> Path:+ path = root / safe_relative(relative)+ if not path.resolve().is_relative_to(root.resolve()):+ raise ValueError("Fixture path resolves outside its directory")+ return path+++def _missing_replay_path(call: dict, fixtures: dict[str, str], skill_name: str, skill_path: Path | None) -> str | None:+ if str(call.get("function_name", "")).casefold() not in ("read", "read_file", "open", "open_file"):+ return None+ args = call.get("arguments", {})+ if not isinstance(args, dict):+ return None+ path = args.get("path", args.get("file_path"))+ if not isinstance(path, str) or not path:+ return None+ relative = Path(path).as_posix()+ if relative in fixtures:+ return None+ target = Path(path)+ if target.is_absolute():+ if (+ skill_path is not None and target.resolve().is_relative_to(skill_path.resolve())+ ) or _is_installed_skill_path(target.parts, skill_name):+ return None+ else:+ parts = target.parts+ if parts and parts[0] == skill_name:+ parts = parts[1:]+ if ".." not in parts and parts and parts[0] in ("SKILL.md", "scripts", "references", "assets", "tools"):+ return None+ if (+ skill_path is not None+ and (skill_path / target).is_file()+ and (skill_path / target).resolve().is_relative_to(skill_path.resolve())+ ):+ return None+ return path+++def _is_installed_skill_path(parts: tuple[str, ...], skill_name: str) -> bool:+ normalized = [part.casefold() for part in parts]+ skill = skill_name.casefold()+ allowed = {"skill.md", "scripts", "references", "assets", "tools"}+ return any(+ normalized[index] == "skills"+ and normalized[index + 1] == skill+ and index + 2 < len(parts)+ and normalized[index + 2] in allowed+ for index in range(len(parts) - 2)+ )+++def build_candidates(+ raw: bytes,+ export_name: str,+ skill_name: str,+ *,+ skill_path: Path | None = None,+ skill_directory_name: str | None = None,+) -> tuple[dict, dict[str, str]]:+ """Return only redacted review metadata and fixture contents, without writing."""+ candidates: list[dict] = []+ skipped: list[dict] = []+ fixtures: dict[str, str] = {}+ fingerprints: dict[str, str] = {}+ ids: set[str] = set()+ for record, session in parse_export(raw):+ sid = session.get("session_id") if isinstance(session, dict) else None++ def skip(reason: str, sid: Any = sid, record: int = record, **extra: Any) -> None:+ skipped.append(+ {"session_id": sid if isinstance(sid, str) else None, "record": record, "reason": reason, **extra}+ )++ if not isinstance(session, dict):+ skip("unsupported")+ continue+ try:+ json.dumps(session, ensure_ascii=False).encode("utf-8")+ except UnicodeEncodeError:+ skip("unsupported")+ continue+ accepted_skill_names = (skill_name, skill_directory_name)+ if "skill" in session and session["skill"] not in accepted_skill_names:+ skip("other skill")+ continue+ outcome = session.get("outcome", "unknown")+ if outcome not in ("success", "failure", "unknown"):+ skip("unsupported")+ continue+ warnings = [f"unsupported field: {k}" for k in session if k not in _FIELDS]+ trajectory = session.get("trajectory")+ source = "trajectory" if isinstance(trajectory, dict) else None+ if source is None and "agent_log" in session:+ agent = session.get("agent")+ converter = _CONVERTERS.get(agent) if isinstance(agent, str) else None+ if converter and isinstance(session["agent_log"], str):+ try:+ trajectory = converter(session["agent_log"])+ if not trajectory and agent == "codex":+ trajectory = log_converters.synthetic_trajectory_from_codex_txt(session["agent_log"])+ except (ValueError, TypeError, AttributeError):+ trajectory = None+ source = f"{agent} log" if trajectory else None+ if source is None:+ warnings.append("unsupported agent log format")+ steps = trajectory.get("steps", []) if isinstance(trajectory, dict) else []+ steps = [s for s in steps if isinstance(s, dict)] if isinstance(steps, list) else []+ prompt = session.get("prompt")+ if not isinstance(prompt, str) or not prompt.strip():+ prompt = next((atif_content_text(s.get("message")) for s in steps if s.get("source") == "user"), "")+ if not isinstance(sid, str) or not sid.strip() or not prompt.strip():+ skip("incomplete")+ continue+ final = session.get("final_output")+ if not isinstance(final, str):+ final = next(+ (atif_content_text(s.get("message")) for s in reversed(steps) if s.get("source") == "agent"), ""+ )+ local_files = {}+ inputs = session.get("files", [])+ if not isinstance(inputs, list):+ warnings.append("incomplete: invalid files list")+ inputs = []+ for item in inputs:+ try:+ if not isinstance(item, dict) or not isinstance(item.get("content"), str):+ raise ValueError("invalid fixture content")+ path = safe_relative(item.get("path")).as_posix()+ local_files[path] = item["content"]+ except ValueError:+ warnings.append("incomplete: unsafe or invalid fixture dropped")+ redacted_prompt, prompt_counts = redact(prompt)+ redacted_files, fixture_counts = redact(local_files)+ fingerprint = hashlib.sha256(+ json.dumps([" ".join(redacted_prompt.split()), sorted(redacted_files.items())], ensure_ascii=False).encode(+ "utf-8"+ )+ ).hexdigest()+ if fingerprint in fingerprints:+ skip("duplicate", duplicate_of=fingerprints[fingerprint])+ continue+ cid = "trace-" + hashlib.sha256(sid.encode()).hexdigest()[:10]+ if cid in ids:+ raise ValueError("Repeated session_id with different replay inputs")+ ids.add(cid)+ fingerprints[fingerprint] = cid+ tools = []+ suggestions = []+ missing_paths: set[str] = set()+ structured_counts = {"secrets": 0, "pii": 0}+ for step in steps:+ calls = step.get("tool_calls", [])+ for call in calls if isinstance(calls, list) else []:+ if not isinstance(call, dict):+ continue+ missing = _missing_replay_path(call, local_files, skill_name, skill_path)+ if missing is not None and missing not in missing_paths and len(missing_paths) < 20:+ warnings.append(f"missing replay input: {missing}")+ missing_paths.add(missing)+ call, call_counts = redact(call, structured=True)+ for category in structured_counts:+ structured_counts[category] += call_counts[category]+ summary = f"{call.get('function_name', 'unknown')}: {json.dumps(call.get('arguments', {}), ensure_ascii=False)}"+ tools.append(summary)+ if "SKILL.md" in summary or "scripts/" in summary:+ suggestions.append(f"Review whether the replay should use: {summary}")+ provenance, provenance_counts = redact(+ {+ "session_id": sid,+ "source": session.get("source"),+ "started_at": session.get("started_at"),+ "tags": session.get("tags"),+ "record": record,+ },+ structured=True,+ )+ candidate = {+ "id": cid,+ "prompt": redacted_prompt,+ "files": [],+ "observed": {"outcome": outcome, "final_output": final, "tool_calls": tools, "trajectory_source": source},+ "provenance": provenance,+ "warnings": warnings,+ "review": {+ "decision": "pending",+ "expected_output": "",+ "assertions": [],+ "suggested_assertions": suggestions,+ "notes": "",+ },+ }+ clean, counts = redact({"candidate": candidate, "fixtures": redacted_files})+ for category in counts:+ counts[category] += (+ prompt_counts[category]+ + fixture_counts[category]+ + structured_counts[category]+ + provenance_counts[category]+ )+ candidate = clean["candidate"]+ candidate["redactions"] = counts+ for path, content in clean["fixtures"].items():+ ref = f"fixtures/{cid}/{safe_relative(path).as_posix()}"+ fixtures[ref] = content+ candidate["files"].append(ref)+ candidate["observed"]["tool_calls"] = [s[:250] for s in candidate["observed"]["tool_calls"]]+ candidates.append(candidate)+ document = {+ "format": "skillevaluator.trace-candidates",+ "version": 1,+ "skill_name": skill_name,+ "export": {"name": export_name, "sha256": hashlib.sha256(raw).hexdigest()},+ "candidates": candidates,+ "skipped": skipped,+ }+ return redact(document)[0], fixtures+++def import_traces(skill: Path, export: Path, output: Path, *, force: bool = False, dry_run: bool = False) -> dict:+ if output.resolve().is_relative_to(skill.resolve()):+ raise ValueError("Review output must be outside the skill directory")+ target = output / "candidates.json"+ if target.exists() and not force:+ raise ValueError("candidates.json already exists; use --force to overwrite")+ skill_data = _parse_skill(skill)+ document, fixtures = build_candidates(+ export.read_bytes(),+ export.name,+ skill_data["name"],+ skill_path=skill,+ skill_directory_name=skill.name,+ )+ destinations = [(contained(output, ref), content) for ref, content in fixtures.items()]+ contained(output, "candidates.json")+ fixture_bytes = [(path, content.encode("utf-8")) for path, content in destinations]+ document_bytes = (json.dumps(document, indent=2, ensure_ascii=False) + "\n").encode("utf-8")+ if not dry_run:+ output.mkdir(parents=True, exist_ok=True)+ for path, content in fixture_bytes:+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_bytes(content)+ target.write_bytes(document_bytes)+ return document+++def accept_traces(skill: Path, review_dir: Path, *, dry_run: bool = False, allow_native_harbor: bool = False) -> dict:+ document = json.loads((review_dir / "candidates.json").read_text(encoding="utf-8"))+ if (+ not isinstance(document, dict)+ or document.get("format") != "skillevaluator.trace-candidates"+ or type(document.get("version")) is not int+ or document["version"] != 1+ ):+ raise ValueError("Expected skillevaluator.trace-candidates version 1")+ name = _parse_skill(skill)["name"]+ accepted_skill_names = (name, skill.name)+ if document.get("skill_name") not in accepted_skill_names:+ raise ValueError("Review skill_name does not match skill")+ source, _ = validate_tier3_source(skill)+ if source == "native_harbor" and not allow_native_harbor:+ raise ValueError(+ "Native Harbor tasks are selected for this skill; pass --allow-native-harbor to explicitly accept traces"+ )+ target = skill / "evals" / "evals.json"+ existing = find_eval_file(skill)+ if existing and existing != target:+ raise ValueError("Existing dataset uses another location or extension; migrate to evals/evals.json first")+ dataset = json.loads(target.read_text(encoding="utf-8")) if target.exists() else {"skill_name": name, "evals": []}+ if (+ not isinstance(dataset, dict)+ or not isinstance(dataset.get("skill_name"), str)+ or not isinstance(dataset.get("evals"), list)+ ):+ raise ValueError("Existing dataset must use agentskills shape, not a legacy flat list")+ dataset_name = dataset["skill_name"]+ if dataset_name not in accepted_skill_names:+ raise ValueError("Existing dataset skill_name must match the frontmatter or directory name")+ errors = []+ additions = []+ copies = []+ present = []+ skipped = []+ by_id = {e.get("id"): e for e in dataset["evals"]}+ for candidate in document["candidates"]:+ cid = candidate["id"]+ safe_relative(cid)+ if not re.fullmatch(r"trace-[0-9a-f]{10}", cid):+ raise ValueError("Invalid candidate id")+ review = candidate["review"]+ decision = review.get("decision")+ if decision in ("pending", "reject"):+ skipped.append(cid)+ continue+ expected = review.get("expected_output", "")+ observed = candidate["observed"]+ if decision == "accept_observed":+ expected = observed.get("final_output", "")+ if observed.get("outcome") != "success" or not isinstance(expected, str) or not expected.strip():+ errors.append(+ f"{cid}: write an independent expectation; accept_observed requires success and non-empty output"+ )+ continue+ elif decision != "accept":+ errors.append(f"{cid}: unknown review decision")+ continue+ if not isinstance(expected, str) or not expected.strip():+ errors.append(f"{cid}: accept requires non-empty reviewer-written expected_output")+ continue+ trace_source = {+ "session_id": candidate["provenance"]["session_id"],+ "export_sha256": document["export"]["sha256"],+ "observed_outcome": observed["outcome"],+ "decision": decision,+ }+ if cid in by_id:+ if by_id[cid].get("trace_source", {}).get("session_id") == trace_source["session_id"]:+ present.append(cid)+ else:+ errors.append(f"{cid}: id collision with a different session")+ continue+ entry = {+ "id": cid,+ "prompt": candidate["prompt"],+ "expected_output": expected,+ "files": [],+ "expected_skill": dataset_name,+ "trace_source": trace_source,+ }+ assertions = review.get("assertions", [])+ if not isinstance(assertions, list) or any(not isinstance(a, str) for a in assertions):+ errors.append(f"{cid}: assertions must be a list of strings")+ continue+ if assertions:+ entry["assertions"] = assertions+ contents = []+ for ref in candidate.get("files", []):+ prefix = f"fixtures/{cid}/"+ if not isinstance(ref, str) or not ref.startswith(prefix):+ raise ValueError(f"{cid}: invalid fixture reference")+ source = contained(review_dir, ref)+ rel = f"trace-files/{cid}/{safe_relative(ref[len(prefix) :]).as_posix()}"+ dest = contained(skill / "evals", rel)+ content = source.read_bytes().decode("utf-8")+ contents.append(content)+ copies.append((dest, content))+ entry["files"].append(rel)+ payload = [entry, contents]+ if redact(payload)[0] != payload:+ errors.append(f"{cid}: secret/PII redaction required; sanitize the review before accepting")+ continue+ additions.append(entry)+ by_id[cid] = entry+ if errors:+ raise ValueError("\n".join(errors))+ validation = []+ if additions:+ contained(skill, "evals/evals.json")+ updated_dataset = {**dataset, "evals": [*dataset["evals"], *additions]}+ copy_bytes = [(dest, content.encode("utf-8")) for dest, content in copies]+ dataset_bytes = (json.dumps(updated_dataset, indent=2, ensure_ascii=False) + "\n").encode("utf-8")+ if not dry_run and additions:+ for dest, content in copy_bytes:+ dest.parent.mkdir(parents=True, exist_ok=True)+ dest.write_bytes(content)+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_bytes(dataset_bytes)+ _validate_dataset_contents(target, validation)+ return {+ "added": [e["id"] for e in additions],+ "already present": present,+ "skipped": skipped,+ "validation": validation,+ }diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex ff4b9f02..6cc97f9b 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -2628,6 +2628,48 @@},"tier3": {"commands": {+ "accept-traces": {+ "params": [+ {+ "name": "skill_path",+ "opts": [+ "skill_path"+ ],+ "param_type": "argument",+ "required": true,+ "type": "directory"+ },+ {+ "name": "review_dir",+ "opts": [+ "review_dir"+ ],+ "param_type": "argument",+ "required": true,+ "type": "directory"+ },+ {+ "default": "False",+ "is_flag": true,+ "name": "dry_run",+ "opts": [+ "--dry-run"+ ],+ "param_type": "option",+ "type": "boolean"+ },+ {+ "default": "False",+ "is_flag": true,+ "name": "allow_native_harbor",+ "opts": [+ "--allow-native-harbor"+ ],+ "param_type": "option",+ "type": "boolean"+ }+ ]+ },"compare": {"params": [{@@ -3319,6 +3361,57 @@}]},+ "import-traces": {+ "params": [+ {+ "name": "skill_path",+ "opts": [+ "skill_path"+ ],+ "param_type": "argument",+ "required": true,+ "type": "directory"+ },+ {+ "name": "export",+ "opts": [+ "export"+ ],+ "param_type": "argument",+ "required": true,+ "type": "file"+ },+ {+ "name": "output",+ "opts": [+ "--output",+ "-o"+ ],+ "param_type": "option",+ "type": "directory"+ },+ {+ "default": "False",+ "is_flag": true,+ "name": "force",+ "opts": [+ "--force"+ ],+ "param_type": "option",+ "type": "boolean"+ },+ {+ "default": "False",+ "is_flag": true,+ "name": "dry_run",+ "opts": [+ "--dry-run"+ ],+ "param_type": "option",+ "type": "boolean"+ }+ ]+ },"init-custom-grader": {"params": [{diff --git a/tests/test_trace_import.py b/tests/test_trace_import.pynew file mode 100644index 00000000..c3183fb7--- /dev/null+++ b/tests/test_trace_import.py@@ -0,0 +1,577 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Production trace export import and human review regression tests."""++import json+from time import perf_counter++import pytest+from click.testing import CliRunner++from skillevaluator.cli import cli+from skillevaluator.tier3.dataset_utils import load_dataset_entries_with_format+from skillevaluator.tier3.harbor.adapter import _entry_file_refs, _resolve_entry_file_ref+from skillevaluator.tier3.trace_import import (+ _pii_patterns,+ accept_traces,+ build_candidates,+ import_traces,+ parse_export,+ redact,+)+++def workspace(tmp_path):+ skill = tmp_path / "skill"+ skill.mkdir()+ (skill / "SKILL.md").write_text("---\nname: sample\ndescription: Example\n---\nInstructions\n")+ return skill, tmp_path / "export.jsonl", tmp_path / "review"+++def session(**kwargs):+ return {"session_id": "session-one", "prompt": "Analyze the input", **kwargs}+++def write_export(path, sessions):+ path.write_text("\n".join(json.dumps(s) for s in sessions))+++def reviewed(workspace, decision="accept", **kwargs):+ skill, export, review = workspace+ write_export(export, [session(**kwargs)])+ doc = import_traces(skill, export, review)+ doc["candidates"][0]["review"].update(decision=decision, expected_output="Independent expectation")+ save_review(review, doc)+ return doc+++def save_review(review, doc):+ (review / "candidates.json").write_text(json.dumps(doc))+++def test_parse_jsonl_and_wrapper():+ records = [session(), session(session_id="two")]+ assert [s for _, s in parse_export("\n".join(map(json.dumps, records)).encode())] == records+ assert [+ s+ for _, s in parse_export(+ json.dumps({"format": "skillevaluator.trace-export", "version": 1, "sessions": records}).encode()+ )+ ] == records+ bom_jsonl = b"\xef\xbb\xbf" + (json.dumps(records[0]) + "\n").encode()+ assert [s for _, s in parse_export(bom_jsonl)] == [records[0]]+ versioned_session = {"session_id": "v", "prompt": "Do it", "version": "2.1.0"}+ assert [s for _, s in parse_export((json.dumps(versioned_session) + "\n").encode())] == [versioned_session]+ unicode_separators = {+ "session_id": "unicode",+ "prompt": "first\u2028second\u2029third\u0085fourth",+ }+ assert [s for _, s in parse_export((json.dumps(unicode_separators, ensure_ascii=False) + "\n").encode())] == [+ unicode_separators+ ]+++ ("fmt", "version"), [("other", 1), ("skillevaluator.trace-export", 2), ("skillevaluator.trace-export", True)]+)+def test_invalid_contract(fmt, version):+ with pytest.raises(ValueError, match="version 1"):+ parse_export(json.dumps({"format": fmt, "version": version, "sessions": []}).encode())+++def test_classifications_and_warnings():+ records = [+ session(extra="ignored", files=[{"path": "../escape", "content": "x"}]),+ session(session_id="duplicate", prompt=" Analyze the input "),+ {"prompt": "Missing identity"},+ session(session_id="missing", prompt=""),+ session(outcome="invalid"),+ session(skill="different"),+ 4,+ ]+ raw = ("\n".join(map(json.dumps, records)) + "\ninvalid json").encode()+ doc, fixtures = build_candidates(raw, "export.jsonl", "sample")+ assert len(doc["candidates"]) == 1+ assert not fixtures+ assert [s["reason"] for s in doc["skipped"]] == [+ "duplicate",+ "incomplete",+ "incomplete",+ "unsupported",+ "other skill",+ "unsupported",+ "unsupported",+ ]+ assert doc["skipped"][0]["duplicate_of"] == doc["candidates"][0]["id"]+ assert doc["skipped"][-1]["record"] == 8+ assert "unsupported field: extra" in doc["candidates"][0]["warnings"]+ assert any("incomplete" in w for w in doc["candidates"][0]["warnings"])+++def test_skill_filter_accepts_directory_name(tmp_path):+ skill = tmp_path / "pdf-tools-v2"+ skill.mkdir()+ (skill / "SKILL.md").write_text("---\nname: pdf-tools\ndescription: Example\n---\nInstructions\n")+ doc, _ = build_candidates(+ json.dumps(session(skill="pdf-tools-v2")).encode(),+ "export",+ "pdf-tools",+ skill_path=skill,+ skill_directory_name=skill.name,+ )+ assert len(doc["candidates"]) == 1+++def test_redaction_every_written_file(workspace):+ skill, export, review = workspace+ sensitive = "sk-abcdefghijklmno [email protected] /home/privateperson/input"+ write_export(+ export,+ [+ session(+ prompt=sensitive,+ final_output=sensitive,+ source=sensitive,+ files=[{"path": "input.txt", "content": sensitive}],+ trajectory={+ "steps": [{"tool_calls": [{"function_name": "shell", "arguments": {"command": sensitive}}]}]+ },+ )+ ],+ )+ doc = import_traces(skill, export, review)+ for path in review.rglob("*"):+ if path.is_file():+ content = path.read_text()+ for raw in ("sk-abcdefghijklmno", "[email protected]", "privateperson"):+ assert raw not in content+ counts = doc["candidates"][0]["redactions"]+ assert counts["secrets"] >= 4+ assert counts["pii"] >= 8+ assert "/home/<user>/" in doc["candidates"][0]["prompt"]+ assert redact(doc)[0] == doc+++ "path", ["/absolute", "../escape", "nested/../../escape", "C:\\Users\\someone\\file", "ok\x00bad"]+)+def test_bad_fixture_paths(path):+ doc, fixtures = build_candidates(+ json.dumps(session(files=[{"path": path, "content": "text"}])).encode(), "export", "sample"+ )+ assert not fixtures+ assert "incomplete" in doc["candidates"][0]["warnings"][0]+++def test_reconstruct_log_and_unknown_agent():+ log = json.dumps(+ {+ "type": "assistant",+ "message": {+ "content": [+ {"type": "text", "text": "Finished"},+ {"type": "tool_use", "id": "call", "name": "Read", "input": {"file_path": "SKILL.md"}},+ ]+ },+ }+ )+ doc, _ = build_candidates(json.dumps(session(agent="claude-code", agent_log=log)).encode(), "export", "sample")+ c = doc["candidates"][0]+ assert c["observed"]["trajectory_source"] == "claude-code log"+ assert c["observed"]["final_output"] == "Finished"+ assert c["review"]["suggested_assertions"]+ assert not c["review"]["assertions"]+ doc, _ = build_candidates(json.dumps(session(agent="unrecognized", agent_log="raw")).encode(), "export", "sample")+ assert "unsupported agent log format" in doc["candidates"][0]["warnings"]+ assert doc["candidates"][0]["observed"]["tool_calls"] == []+++def test_prompt_recovered_from_trajectory():+ doc, _ = build_candidates(+ json.dumps(+ {+ "session_id": "recovered",+ "trajectory": {+ "steps": [+ {"source": "user", "message": "Original request"},+ {"source": "agent", "message": "Response"},+ ]+ },+ }+ ).encode(),+ "export",+ "sample",+ )+ assert doc["candidates"][0]["prompt"] == "Original request"+++def test_output_refusal_force_and_dry_run(workspace):+ skill, export, review = workspace+ write_export(export, [session()])+ with pytest.raises(ValueError, match="outside"):+ import_traces(skill, export, skill / "review")+ import_traces(skill, export, review, dry_run=True)+ assert not review.exists()+ import_traces(skill, export, review)+ with pytest.raises(ValueError, match="already exists"):+ import_traces(skill, export, review)+ import_traces(skill, export, review, force=True)++[email protected]("decision", ["pending", "reject"])+def test_pending_and_reject(workspace, decision):+ reviewed(workspace, decision)+ result = accept_traces(workspace[0], workspace[2])+ assert len(result["skipped"]) == 1+ assert not (workspace[0] / "evals").exists()++[email protected]("decision", ["accept", "not-a-decision"])+def test_invalid_review(workspace, decision):+ doc = reviewed(workspace, decision)+ doc["candidates"][0]["review"]["expected_output"] = ""+ save_review(workspace[2], doc)+ with pytest.raises(ValueError, match=doc["candidates"][0]["id"]):+ accept_traces(workspace[0], workspace[2])+ assert not (workspace[0] / "evals").exists()++[email protected]("outcome", ["failure", "unknown"])+def test_refuse_observed_failure(workspace, outcome):+ reviewed(workspace, "accept_observed", outcome=outcome, final_output="Observed response")+ with pytest.raises(ValueError, match="independent expectation"):+ accept_traces(workspace[0], workspace[2])+++def test_allow_observed_success(workspace):+ reviewed(workspace, "accept_observed", outcome="success", final_output="Observed response")+ result = accept_traces(workspace[0], workspace[2])+ assert result["added"]+ entry = json.loads((workspace[0] / "evals/evals.json").read_text())["evals"][0]+ assert entry["expected_output"] == "Observed response"+ assert entry["files"] == []++[email protected]("field", ["expected_output", "assertions"])+def test_reviewer_pasted_secret(workspace, field):+ doc = reviewed(workspace)+ value = "sk-abcdefghijklmno"+ doc["candidates"][0]["review"][field] = [value] if field == "assertions" else value+ save_review(workspace[2], doc)+ with pytest.raises(ValueError, match="redaction required"):+ accept_traces(workspace[0], workspace[2])+ assert not (workspace[0] / "evals").exists()++[email protected]("surface", ["prompt", "fixture", "expected_output"])+def test_reviewer_pasted_pii(workspace, surface):+ skill, _, review = workspace+ doc = reviewed(workspace, files=[{"path": "input.txt", "content": "Safe fixture"}])+ candidate = doc["candidates"][0]+ pasted = "Contact [email protected] at /home/privateperson/input"+ if surface == "prompt":+ candidate["prompt"] = pasted+ elif surface == "fixture":+ (review / candidate["files"][0]).write_text(pasted)+ else:+ candidate["review"]["expected_output"] = pasted+ save_review(review, doc)+ with pytest.raises(ValueError, match=candidate["id"]):+ accept_traces(skill, review)+ assert not (skill / "evals").exists()+++def test_errors_abort_all_candidates_before_copying(workspace):+ skill, export, review = workspace+ write_export(+ export,+ [+ session(files=[{"path": "input.txt", "content": "Safe fixture"}]),+ session(session_id="second", prompt="Another request"),+ ],+ )+ doc = import_traces(skill, export, review)+ first, second = doc["candidates"]+ first["review"].update(decision="accept", expected_output="Independent expectation")+ second["review"]["decision"] = "accept"+ save_review(review, doc)+ with pytest.raises(ValueError, match=second["id"]):+ accept_traces(skill, review)+ assert not (skill / "evals").exists()+++def test_invalid_fixture_content_dropped():+ doc, fixtures = build_candidates(+ json.dumps(session(files=[{"path": "input.txt", "content": 123}])).encode(), "export", "sample"+ )+ assert not fixtures+ assert "incomplete" in doc["candidates"][0]["warnings"][0]+++def test_append_fixtures_validation_adapter_and_idempotency(workspace):+ skill, _, review = workspace+ reviewed(workspace, files=[{"path": "nested/input.txt", "content": "Replay\r\nfixture\r\n"}])+ (skill / "evals").mkdir()+ target = skill / "evals/evals.json"+ original = {"id": "original", "prompt": "Original", "expected_output": "Expected"}+ target.write_text(json.dumps({"skill_name": "sample", "evals": [original]}))+ assert accept_traces(skill, review, dry_run=True)["added"]+ assert not (skill / "evals/files").exists()+ result = accept_traces(skill, review)+ assert result["added"]+ assert all(c.status != "error" for c in result["validation"])+ assert (skill / "evals/trace-files").is_dir()+ assert not (skill / "evals/files").exists()+ entries, fmt = load_dataset_entries_with_format(target)+ assert fmt == "agentskills"+ assert json.loads(target.read_text())["evals"][0] == original+ entry = entries[1]+ assert entry["trace_source"]["session_id"] == "session-one"+ refs = _entry_file_refs(entry)+ assert refs == [f"trace-files/{entry['id']}/nested/input.txt"]+ source, staged_path = _resolve_entry_file_ref(+ refs[0], skill_path=skill, evals_dir=skill / "evals", input_files_dir=skill / "evals/files"+ )+ assert staged_path.as_posix() == refs[0]+ assert source.read_bytes() == b"Replay\r\nfixture\r\n"+ assert accept_traces(skill, review)["already present"] == [entry["id"]]+ assert len(json.loads(target.read_text())["evals"]) == 2+ entries[1]["trace_source"]["session_id"] = "different"+ target.write_text(json.dumps({"skill_name": "sample", "evals": entries}))+ with pytest.raises(ValueError, match="collision"):+ accept_traces(skill, review)++[email protected]("extension", ["json", "yaml"])+def test_legacy_dataset_refused(workspace, extension):+ skill, _, review = workspace+ reviewed(workspace)+ (skill / "evals").mkdir()+ (skill / "evals" / f"evals.{extension}").write_text("[]")+ with pytest.raises(ValueError, match=r"agentskills|extension"):+ accept_traces(skill, review)+++def test_cli_smoke(workspace):+ skill, export, review = workspace+ write_export(export, [session()])+ runner = CliRunner()+ result = runner.invoke(cli, ["tier3", "import-traces", str(skill), str(export), "-o", str(review)])+ assert result.exit_code == 0, result.output+ assert "candidate: 1" in result.output+ assert "accept-traces" in result.output+ result = runner.invoke(cli, ["tier3", "accept-traces", str(skill), str(review), "--dry-run"])+ assert result.exit_code == 0, result.output+ assert "would add: 0" in result.output+++def test_empty_export_and_bad_review_contract(workspace):+ skill, export, review = workspace+ export.write_text("")+ assert import_traces(skill, export, review)["candidates"] == []+ save_review(review, {"format": "wrong", "version": 1})+ with pytest.raises(ValueError, match="trace-candidates"):+ accept_traces(skill, review)+++def test_accepting_into_native_harbor_requires_explicit_flag(workspace):+ skill, _, review = workspace+ reviewed(workspace)+ (skill / "evals/harbor/task-a").mkdir(parents=True)+ with pytest.raises(ValueError, match="allow-native-harbor"):+ accept_traces(skill, review)+ assert accept_traces(skill, review, allow_native_harbor=True)["added"]+++def test_accept_traces_preserves_directory_named_dataset_skill(workspace, tmp_path):+ _, _, review = workspace+ skill = tmp_path / "pdf-tools"+ skill.mkdir()+ (skill / "SKILL.md").write_text("---\nname: pdf\ndescription: Example\n---\nInstructions\n")+ write_export(workspace[1], [session()])+ doc = import_traces(skill, workspace[1], review)+ doc["candidates"][0]["review"].update(decision="accept", expected_output="Independent expectation")+ save_review(review, doc)+ (skill / "evals").mkdir()+ target = skill / "evals/evals.json"+ target.write_text(json.dumps({"skill_name": "pdf-tools", "evals": []}))++ accept_traces(skill, review)++ entry = json.loads(target.read_text())["evals"][0]+ assert entry["expected_skill"] == "pdf-tools"+++def test_trace_fingerprints_use_redacted_prompt_and_fixtures():+ records = [+ session(+ prompt="Summarize /Users/alice/q3/report.csv", files=[{"path": "report.txt", "content": "[email protected]"}]+ ),+ session(+ session_id="other",+ prompt="Summarize /Users/bob/q3/report.csv",+ files=[{"path": "report.txt", "content": "[email protected]"}],+ ),+ ]+ doc, _ = build_candidates(("\n".join(json.dumps(item) for item in records)).encode(), "export", "sample")+ assert len(doc["candidates"]) == 1+ assert doc["skipped"][0]["reason"] == "duplicate"+++def test_surrogate_records_are_skipped_before_artifacts_are_written(workspace):+ skill, export, review = workspace+ write_export(+ export,+ [+ session(files=[{"path": "good.txt", "content": "good"}]),+ session(session_id="broken", prompt="broken \ud83d", files=[{"path": "bad.txt", "content": "bad"}]),+ ],+ )++ doc = import_traces(skill, export, review)++ assert len(doc["candidates"]) == 1+ assert doc["skipped"][0]["reason"] == "unsupported"+ assert (review / "fixtures" / doc["candidates"][0]["id"] / "good.txt").read_bytes() == b"good"+ assert json.loads((review / "candidates.json").read_bytes())["candidates"]+++def test_pii_categories_and_stability():+ original = (+ "email: [email protected] /Users/someone/file C:\\Users\\someone\\file /home/root/file 212-345-6789 123-45-6789"+ )+ clean, counts = redact(original)+ assert counts["pii"] == 6+ assert "someone" not in clean and "/home/root/" not in clean+ assert redact(clean)[0] == clean+ assert redact("[email protected] /Users/{id}/")[0] == "[email protected] /Users/{id}/"+++def test_bare_home_paths_are_redacted_idempotently():+ clean, counts = redact("cwd /home/jdoe and /Users/jdoe")+ assert clean == "cwd /home/<user> and /Users/<user>"+ assert counts["pii"] == 2+ assert redact(clean)[0] == clean+++def test_numeric_phone_patterns_do_not_change_literals_or_csv_rows():+ text = "1.000000e+01 2.500000e+00 delta +15.25 pts at 2026-11-07T10:00:00+0000\nid,score\n1,+12.5\n2,-3.0"+ clean, counts = redact(text)+ assert clean == text+ assert counts["pii"] == 0+++def test_redaction_disambiguates_fixture_keys_after_pii_replacement():+ doc, fixtures = build_candidates(+ json.dumps(+ session(+ files=[+ {"path": "exports/[email protected]", "content": "alice rows"},+ {"path": "exports/[email protected]", "content": "bob rows"},+ ]+ )+ ).encode(),+ "export",+ "sample",+ )+ candidate = doc["candidates"][0]+ assert len(fixtures) == 2+ assert len(candidate["files"]) == 2+ assert set(fixtures.values()) == {"alice rows", "bob rows"}+++def test_structured_tool_and_provenance_secrets():+ raw = json.dumps(+ session(+ source={"password": "opaque-value"},+ trajectory={+ "steps": [+ {"tool_calls": [{"function_name": "request", "arguments": {"password": "another-opaque-value"}}]}+ ]+ },+ )+ ).encode()+ doc, _ = build_candidates(raw, "export", "sample")+ assert "opaque-value" not in json.dumps(doc)+ assert doc["candidates"][0]["redactions"]["secrets"] >= 2+++def test_large_text_redaction_is_fast():+ chunk = "Observed tool output with ordinary words and whitespace. "+ text = (+ chunk * (2 * 1024 * 1024 // len(chunk)) + " sk-abcdefghijklmno [email protected] /home/privateperson/input"+ )+ start = perf_counter()+ clean, counts = redact(text)+ elapsed = perf_counter() - start+ assert elapsed < 30, f"Redacting 2 MiB took {elapsed:.2f}s"+ assert "sk-abcdefghijklmno" not in clean+ assert "[email protected]" not in clean+ assert "privateperson" not in clean+ assert counts == {"secrets": 1, "pii": 2}+++def test_pii_patterns_are_cached():+ assert _pii_patterns() is _pii_patterns()+ assert all(hasattr(pattern, "sub") for _, pattern, _ in _pii_patterns())+++def test_marker_counts_ignore_existing_markers():+ assert redact("<redacted> sk-abcdefghijklmno")[1]["secrets"] == 1+ assert redact({"password": "opaque", "message": "<redacted>"}, structured=True)[1]["secrets"] == 1+++def test_missing_replay_inputs_redacted_deduplicated_and_capped(workspace):+ skill, _, _ = workspace+ paths = [+ "input.txt",+ "/workspace/input/input.txt",+ "/workspace/skills/sample/SKILL.md",+ "/home/alice/.claude/skills/sample/references/guide.md",+ "/workspace/skills/other/SKILL.md",+ "SKILL.md",+ "scripts/run.py",+ "references/guide.md",+ str(skill / "SKILL.md"),+ "missing.csv",+ "missing.csv",+ "/home/privateperson/report.csv",+ ]+ paths += [f"missing-{i}.csv" for i in range(30)]+ calls = [{"function_name": "Read", "arguments": {"file_path": path}} for path in paths]+ doc, _ = build_candidates(+ json.dumps(+ session(files=[{"path": "input.txt", "content": "fixture"}], trajectory={"steps": [{"tool_calls": calls}]})+ ).encode(),+ "export",+ "sample",+ skill_path=skill,+ )+ warnings = doc["candidates"][0]["warnings"]+ assert len(warnings) == 20+ assert warnings.count("missing replay input: missing.csv") == 1+ assert "missing replay input: /workspace/input/input.txt" in warnings+ assert "missing replay input: /workspace/skills/other/SKILL.md" in warnings+ assert "missing replay input: /home/<user>/report.csv" in warnings+ assert "privateperson" not in json.dumps(doc)+ assert "missing replay input: /workspace/skills/sample/SKILL.md" not in warnings+ assert "missing replay input: /home/<user>/.claude/skills/sample/references/guide.md" not in warnings+ assert "missing replay input: SKILL.md" not in warnings+ assert "missing replay input: scripts/run.py" not in warnings+ assert "missing replay input: references/guide.md" not in warnings++[email protected]("tool", ["read", "read_file", "open", "open_file"])+def test_read_and_open_tools_report_missing_inputs(tool):+ doc, _ = build_candidates(+ json.dumps(+ session(+ trajectory={"steps": [{"tool_calls": [{"function_name": tool, "arguments": {"path": "missing.txt"}}]}]}+ )+ ).encode(),+ "export",+ "sample",+ )+ assert "missing replay input: missing.txt" in doc["candidates"][0]["warnings"]