Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
feat(skillopt): one-shot rewrite by default, benchmark hooks, held-out/criteria/edit-attribution fixes
garrytan7b31e68c90ba
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4b6906fc4f..66a64be75c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,23 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++**`gbrain skillopt` now makes one gated rewrite by default; the iterative edit loop is opt-in with `--loop`, and benchmark harnesses can drive every SkillOpt rollout.**++In gbrain-evals' Cat 31 re-run the full loop and a one-shot rewrite both scored 0.54 on held-out tasks (paired bootstrap p = 1.00), and the loop cost about 1.8x as much. Until a benchmark shows the loop wins, the cheaper method is the default. The default is reversible: `--loop` on the CLI, or `optimizer_mode: "reflect"` on the `run_skillopt` MCP operation and queued jobs, runs the loop as before. The one-shot rewrite now reflects on training tasks only and must beat the baseline on the selection gate before anything is written, so a worse rewrite leaves the skill unchanged. Four bugs that made the loop look worse than it is are fixed too.++### Itemized changes++- One-shot rewrite is the default for the CLI, `run_skillopt`, `--all`, queued jobs and the dream-cycle phase. A job queued before this release without `optimizer_mode` keeps the loop it was queued for, and `--resume` continues in the mode its checkpoint ran (a changed mode is refused). Receipts carry `optimizer_mode`.+- The optimizer's success criteria come from training tasks only, and a rule argument or LLM rubric is quoted only when every training task shares it; a per-task answer is described ("a task-specific required text"), never quoted. Before, every selection and test answer was shown to the optimizer. `criteriaText` replaces the generated criteria.+- Held-out results never reach the optimizer: the held-out check runs once, on the final champion, before anything is written, and a refused champion is not filed in the rejected-edit buffer. Legacy `held_out_regression` rows on disk are skipped when the buffer loads.+- Only edits that were applied are filed as rejected or recorded as accepted; edits the learning-rate budget clipped stay free to propose again.+- The success-reflect prompt states the edit schema, so its edits are no longer all dropped (#7257; carried from #8324).+- Library surface for benchmark harnesses (`src/core/skillopt/index.ts`): explicit `tasks: {train, sel, test?}` lists instead of a benchmark file; a `rolloutHook` that receives each scoring site's whole batch (baseline, training forward pass, selection gate, held-out, test) and whose budget refusal or malformed reply aborts the run without writing; `scoreSkillPerTask`; and receipt counts (`rollouts` by phase, `candidates_scored_on_sel`, `optimizer_calls`, `cost_scope`).+- A skill can add the reviewed read tools `recall` and `get_timeline` to its rollouts by naming them in its frontmatter `tools:`; rollouts never offer `add_timeline_entry` or any other write.+- Tests: `test/e2e/skillopt-wave-e1b1.serial.test.ts` (15 cases, each red on master), `test/skillopt/benchmark-hooks.test.ts`, and a mode case in `test/skillopt/models-four-paths.serial.test.ts`.+## [0.60.539.0] - 2026-11-01**A remote `put_page` of a page an agent read whole now keeps its frontmatter and the code blocks it added, and timeline sources and ontology values are quoted below "your notes" like the rest of their rows.**diff --git a/docs/guides/skillopt.md b/docs/guides/skillopt.mdindex 3276f2a826..80635db39e 100644--- a/docs/guides/skillopt.md+++ b/docs/guides/skillopt.md@@ -49,7 +49,28 @@ skills/my-skill/The audit trail lives at `~/.gbrain/audit/skillopt-YYYY-Www.jsonl`(ISO-week rotated; honors `GBRAIN_AUDIT_DIR`).-## How the loop works+## One-shot rewrite (the default)++`gbrain skillopt <skill>` runs one gated rewrite:++1. **Forward pass.** Run the current skill on one batch (`--batch-size`) of+ `D_train` tasks.+2. **Rewrite.** One optimizer call rewrites the whole skill body from those+ rollouts (frontmatter untouched).+3. **Validation gate.** The same D12 gate a loop step faces: median-of-3 per+ `D_sel` task, accepted only above the baseline by more than 0.05. A+ rejected rewrite leaves the skill unchanged (`no_improvement`).+4. **Held-out check** (with `--held-out`), then the D8 commit.++**Why one-shot is the default.** In gbrain-evals' Cat 31 re-baseline+(2026-11-01), the iterative loop and a one-shot rewrite both scored 0.54 on+held-out tasks (paired bootstrap p = 1.00) and the loop cost about 1.8x as+much. Every edit the loop kept came at its first step. `--loop` (MCP and job+`optimizer_mode: "reflect"`) still runs the iterative loop. This default is+reversible: it flips back if gbrain-evals' SkillOpt method benchmark shows the+loop winning. A resume continues in the mode its run started in.++## How the loop works (`--loop`)For each step:@@ -87,7 +108,7 @@ the audit event only; the full meta-edit proposal is a tracked follow-up.| `--judge-model MODEL` | tier.reasoning | Scores rollouts || `--reflect-max-tokens N` | `skillopt.reflect_max_tokens`, else 32000 (thinking optimizer) / 4096 | Output cap for every optimizer call. See [Optimizer output cap](#optimizer-output-cap) || `--models-strict` | `skillopt.models_strict`, else off | Abort before any spend unless every active model was explicitly configured. See [Strict mode](#strict-mode) |-| `--patch \| --rewrite` | patch | Edit ops only vs. full rewrites (both go through the validation gate) |+| `--loop` | off | Run the iterative reflect loop instead of the default one gated rewrite. See [One-shot rewrite](#one-shot-rewrite-the-default) || `--dry-run` | off | Models banner, strict verdict and cost preview; zero model calls || `--no-mutate` | off | Write proposed.md, don't replace SKILL.md (no held-out needed) || `--allow-mutate-bundled` | off | Required to mutate gbrain-bundled skills in place — ALSO requires `--held-out` (>=5 rows) or the run hard-refuses |@@ -144,10 +165,10 @@ early. `stop_reason` records why the loop ended: `completed`,### Optimizer output cap-Every optimizer call has one output cap: both reflect calls in patch mode and-in `--rewrite` mode, plus the eval-internal one-shot rewrite ablation.-`--rewrite` itself works the same as before. Its edits still go through the-validation gate. The CLI does not expose the ungated one-shot path.+Every optimizer call has one output cap: the one-shot rewrite and both+`--loop` reflect calls. A one-shot rewrite needs room for the whole body, so a+cap smaller than the body refuses before the call+(`one_shot_rewrite_output_cap_too_small`).Where the cap comes from, highest precedence first:@@ -182,6 +203,13 @@ truncated: sent X of Y chars)` and the receipt records`skill_body_truncated: {sent_chars, total_chars}`. The one-shot rewriterefuses to run on a truncated body.+The success criteria in the prompt are built from the **train** tasks' judges+only. A rule check or LLM rubric is written out only when every train task+carries it (a format rule such as "include a Confidence: line"); one that+varies by task is an answer key and is described without its argument.+Selection, test and held-out judges never reach the optimizer, and neither do+held-out verdicts.+### Unusable replies fail loudEach optimizer call records at most one error. When the reply hit the cap, its@@ -419,7 +447,7 @@ refuses to start when the estimate exceeds `--max-cost-usd`.| Frontmatter mutation forbidden | D5 | Routing surface drift (`check-resolvable` regression) || Per-skill DB lock | D14 | Two concurrent runs corrupting history/versions || Bundled-skill gate | D16 | Auto-mutating skills shipped with gbrain (in-place mutation requires `--allow-mutate-bundled` + a `--held-out` set of >=5 benchmark-disjoint tasks; else hard-refuse + proposed.md) |-| Held-out gate | F11 | Accepting a candidate that overfits its own benchmark — `--held-out` refuses a candidate whose held-out score regresses below baseline |+| Held-out gate | F11 | Accepting a candidate that overfits its own benchmark — with `--held-out`, nothing is written until the final champion is checked once against the baseline; a regression leaves the skill unchanged and the optimizer never sees the verdict || Bootstrap review sentinel | D15 | Self-referential benchmark gaming || Read-only tool sandbox in rollouts | D13 | Optimization runs writing junk pages to your brain || History-intent-first atomic commit | D8 | Half-written SKILL.md on crash |diff --git a/skills/skill-optimizer/SKILL.md b/skills/skill-optimizer/SKILL.mdindex c03910d5b4..e5a22ae995 100644--- a/skills/skill-optimizer/SKILL.md+++ b/skills/skill-optimizer/SKILL.md@@ -57,7 +57,9 @@ gbrain skillopt <skill-name> [flags]│├── Baseline eval on D_sel (sets best_sel_score)│- ├── for epoch in 1..N:+ ├── default: one rewrite of the body from a D_train forward pass,+ │ accepted only through the same D12 validation gate+ ├── --loop (opt-in; has not beaten one-shot on benchmarks): for epoch in 1..N:│ for step in 1..steps_per_epoch:│ ├── forward pass: rollouts on D_train batch│ ├── backward pass: reflect × 2 (failures + successes per D7)@@ -143,7 +145,8 @@ https://github.com/garrytan/gbrain/blob/master/docs/tutorials/improving-skills-w| Costly run, want preview | Add `--dry-run` || Bundled skill (skills/ in gbrain repo) | Default writes proposed.md; to commit in place add `--allow-mutate-bundled` AND `--held-out <path>` (>=5 benchmark-disjoint tasks) — else it hard-refuses. Plugin installs cannot mutate bundled skills in place (the snapshot is read-only): copy the SKILL.md into your own skills dir to iterate, or stick with the default proposed.md review flow || Want to review changes before applying | Add `--no-mutate` (writes proposed.md, no held-out needed) |-| Guard against benchmark overfitting | Add `--held-out <path>` — a candidate that beats the benchmark but regresses on the held-out set is refused |+| Guard against benchmark overfitting | Add `--held-out <path>` — checked once on the final candidate; one that beats the benchmark but regresses on the held-out set is refused |+| Want the iterative edit loop | Add `--loop` (about 1.8x the cost; it tied one-shot on gbrain-evals Cat 31) || Mid-run crash | `gbrain skillopt foo --resume <run-id>` |## Output Formatdiff --git a/skills/skills.lock.json b/skills/skills.lock.jsonindex ad82dfcdd5..643d8bcb06 100644--- a/skills/skills.lock.json+++ b/skills/skills.lock.json@@ -216,7 +216,7 @@"skill-autobench/SKILL.md": "29f5760dcfd823c5f7d4fb50a3cf4e36601b1b8125e57d6bf83645a41f97f613","skill-autobench/routing-eval.jsonl": "8d961ed6403b7e2f690948e4c18529d40f26f6f21064befc56d465966b1a9ec0","skill-creator/SKILL.md": "ae8119e6a28faacdc48b13a51c89ff752e9d1b698a4ed0b49f380eb1de16e322",- "skill-optimizer/SKILL.md": "1dea9a5257a60fb29a4c7701c92bce428025fb2321e950533f1367edfca1b8b0",+ "skill-optimizer/SKILL.md": "8f0dea0c50e180619f732841a5a23faee989506bab252a28b0fbd9aa13bf397f","skill-optimizer/routing-eval.jsonl": "48f7fc04414b194ee8674577c3e58e03ebd7f74d836766bd16cb5abfd4effb76","skill-optimizer/skillopt-benchmark.jsonl": "5552457d6eaa32486b79796d12fcbe2c078b0c53d7fbdaf582f0fd1a17e9a381","skillify/SKILL.md": "6fa84bd744b1082d6550def977b49ecaac7ada0abf1a04180a8e3b0d432fb20e",diff --git a/src/commands/skillopt.ts b/src/commands/skillopt.tsindex 4421f28e52..c8e54442cc 100644--- a/src/commands/skillopt.ts+++ b/src/commands/skillopt.ts@@ -20,12 +20,12 @@ import {} from '../core/skillopt/models-plan.ts';import { SKILLOPT_HELP_TEXT } from '../core/skillopt/help.ts';import { runSkillOpt, parseSplit } from '../core/skillopt/orchestrator.ts';-import { checkpointPath } from '../core/skillopt/checkpoint.ts';+import { checkpointOptimizerMode, checkpointPath, loadCheckpoint } from '../core/skillopt/checkpoint.ts';import { formatModelsUsedTable } from '../core/budget/models-used.ts';import { parsePositiveInt } from '../core/skillopt/output-cap.ts';import { serializeError, StructuredAgentError } from '../core/errors.ts';import type { BrainEngine } from '../core/engine.ts';-import type { RunReceipt, SkillOptOpts } from '../core/skillopt/types.ts';+import type { OptimizerMode, RunReceipt, SkillOptOpts } from '../core/skillopt/types.ts';import { capNotice, mergeCapFlag, parseCapFlag, type CapFlag } from '../core/budget/cap-flag.ts';/** Per-run cost cap when no cap flag is passed (a `default` cap: unpriced models warn and run). */@@ -52,6 +52,8 @@ interface ParsedFlags {/** Abort before spend unless every active model was chosen by explicit configuration. */modelsStrict: boolean;mode: 'patch' | 'rewrite';+ /** E1: `--loop` opts into the iterative reflect loop; one-shot rewrite is the default. */+ loop: boolean;dryRun: boolean;noMutate: boolean;allowMutateBundled: boolean;@@ -151,6 +153,11 @@ export async function runSkillOptCommand(engine: BrainEngine | null, args: strin}}+ // E1: one-shot rewrite unless --loop. A resume continues in the mode its+ // checkpoint ran (a checkpoint from before this default ran the loop).+ const resumed = parsed.resumeRunId && parsed.skillName ? loadCheckpoint(skillsDir, parsed.skillName, parsed.resumeRunId) : null;+ const optimizerMode: OptimizerMode = parsed.loop ? 'reflect' : resumed ? checkpointOptimizerMode(resumed) : 'one-shot-rewrite';+// ── F4: --all batch mode ────────────────────────────────────────────────if (parsed.all) {try {@@ -175,6 +182,7 @@ export async function runSkillOptCommand(engine: BrainEngine | null, args: strinlr: parsed.lr,lrSchedule: parsed.lrSchedule,split: parsed.split,+ optimizerMode,dryRun: parsed.dryRun,noMutate: parsed.noMutate,allowMutateBundled: parsed.allowMutateBundled,@@ -219,6 +227,7 @@ export async function runSkillOptCommand(engine: BrainEngine | null, args: strinlr: parsed.lr,lrSchedule: parsed.lrSchedule,split: parsed.split,+ optimizerMode,dryRun: parsed.dryRun,noMutate: parsed.noMutate,allowMutateBundled: parsed.allowMutateBundled,@@ -276,6 +285,7 @@ export async function runSkillOptCommand(engine: BrainEngine | null, args: strinmodelsStrict: parsed.modelsStrict,...(parsed.reflectMaxTokens !== undefined ? { reflectMaxTokens: parsed.reflectMaxTokens } : {}),mode: parsed.mode,+ optimizerMode,dryRun: parsed.dryRun,noMutate: parsed.noMutate,allowMutateBundled: parsed.allowMutateBundled,@@ -321,6 +331,7 @@ export async function runSkillOptCommand(engine: BrainEngine | null, args: strinmodelsStrict: parsed.modelsStrict,...(parsed.reflectMaxTokens !== undefined ? { reflectMaxTokens: parsed.reflectMaxTokens } : {}),mode: parsed.mode,+ optimizerMode,dryRun: parsed.dryRun,noMutate: parsed.noMutate,allowMutateBundled: parsed.allowMutateBundled,@@ -447,6 +458,7 @@ export function parseFlags(args: string[]): ParsedFlags {let judgeModel: string | undefined;let reflectMaxTokens: number | undefined;let mode: 'patch' | 'rewrite' = 'patch';+ let loop = false;let dryRun = false;let noMutate = false;let allowMutateBundled = false;@@ -500,6 +512,7 @@ export function parseFlags(args: string[]): ParsedFlags {}if (a === '--patch') { mode = 'patch'; i += 1; continue; }if (a === '--rewrite') { mode = 'rewrite'; i += 1; continue; }+ if (a === '--loop') { loop = true; i += 1; continue; }if (a === '--dry-run') { dryRun = true; i += 1; continue; }if (a === '--no-mutate') { noMutate = true; i += 1; continue; }if (a === '--allow-mutate-bundled') { allowMutateBundled = true; i += 1; continue; }@@ -584,6 +597,7 @@ export function parseFlags(args: string[]): ParsedFlags {...(reflectMaxTokens !== undefined ? { reflectMaxTokens } : {}),modelsStrict,mode,+ loop,dryRun,noMutate,allowMutateBundled,diff --git a/src/core/cli-flag-registry.generated.ts b/src/core/cli-flag-registry.generated.tsindex f2b80818f2..6d191ee1ae 100644--- a/src/core/cli-flag-registry.generated.ts+++ b/src/core/cli-flag-registry.generated.ts@@ -123,7 +123,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {'serve': ['--access', '--add', '--adopt-columns', '--adopt-permissions', '--advertise', '--all-new', '--apply-rewrites', '--bind', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--budget-usd-per-day', '--cli-socket', '--client', '--days', '--delegated-namespace', '--delegated-slug-prefixes', '--enable-dcr', '--enable-dcr-insecure', '--expect', '--fail-fast', '--federated-read', '--grant-types', '--harness', '--help', '--http', '--id', '--if-version', '--json', '--log-full-params', '--migrate-legacy', '--min-trust', '--no-embedding', '--no-pull', '--oauth-request', '--only', '--operations', '--path', '--pglite', '--port', '--prefer-postgres', '--print-admin-token', '--private-pages', '--profile', '--provenance', '--public-url', '--read-sources', '--redirect-uri', '--refresh-operations', '--repair', '--reset-default', '--retry-failed', '--scopes', '--source', '--source-guard', '--source-id', '--sources', '--stale', '--stdio-idle-timeout', '--suppress-bootstrap-token', '--surface', '--takes-holders', '--token', '--token-endpoint-auth-method', '--token-ttl', '--tools-json', '--url', '--usage', '--yes'],'setup': ['--accept-capabilities', '--advertise', '--agent', '--auto', '--bearer-token-env-var', '--brain', '--break-lock', '--capture', '--dry-run', '--env', '--force', '--gbrain-bin', '--harness', '--header', '--help', '--http', '--install', '--instructions', '--instructions-only', '--issuer-url', '--json', '--legacy-lock', '--link', '--mcp-only', '--mcp-url', '--name', '--no-capture', '--no-embedding', '--no-hooks', '--no-instructions', '--no-providers', '--oauth-client-id', '--oauth-client-secret', '--only', '--path', '--pglite', '--probe', '--providers', '--pure', '--remove', '--runtime', '--save', '--scope', '--source', '--source-ids', '--surface', '--token', '--url', '--workspace', '--yes'],'skillify': ['--brain', '--description', '--dry-run', '--force', '--help', '--json', '--mutating', '--recent', '--skills-dir', '--source', '--strict', '--triggers', '--writes-pages', '--writes-to'],- 'skillopt': ['--all', '--allow-mutate-bundled', '--background', '--batch-size', '--benchmark', '--bootstrap-from-routing', '--bootstrap-from-skill', '--bootstrap-reviewed', '--bootstrap-tasks', '--brain', '--brain-wide-max-cost-usd', '--dry-run', '--epochs', '--follow', '--force', '--held-out', '--help', '--json', '--judge-model', '--lr', '--lr-schedule', '--max-cost-usd', '--max-runtime-min', '--max-usd', '--models-strict', '--no-max-cost', '--no-mutate', '--optimizer-model', '--patch', '--reflect-max-tokens', '--resume', '--rewrite', '--skills-dir', '--source', '--split', '--status', '--target-model', '--target-models'],+ 'skillopt': ['--all', '--allow-mutate-bundled', '--background', '--batch-size', '--benchmark', '--bootstrap-from-routing', '--bootstrap-from-skill', '--bootstrap-reviewed', '--bootstrap-tasks', '--brain', '--brain-wide-max-cost-usd', '--dry-run', '--epochs', '--follow', '--force', '--held-out', '--help', '--json', '--judge-model', '--loop', '--lr', '--lr-schedule', '--max-cost-usd', '--max-runtime-min', '--max-usd', '--models-strict', '--no-max-cost', '--no-mutate', '--optimizer-model', '--patch', '--reflect-max-tokens', '--resume', '--rewrite', '--skills-dir', '--source', '--split', '--status', '--target-model', '--target-models'],'skillpack': ['--all', '--apply-clean-hunks', '--author', '--auto', '--brain', '--dest', '--dry-run', '--env', '--exit-code', '--fast', '--fix', '--force-unlock', '--format', '--from', '--full', '--harness', '--help', '--homepage', '--json', '--license', '--list', '--minimal', '--name-only', '--no-cache', '--no-lint', '--note', '--out', '--overwrite-local', '--persona', '--probe', '--pure', '--push', '--quick', '--quiet', '--refresh', '--repo', '--schema-pack', '--scope', '--short', '--since', '--skill', '--skills', '--skills-dir', '--skip-doctor', '--source', '--strict', '--stub', '--supabase', '--target', '--tier', '--trust', '--url', '--verify', '--version', '--workspace', '--yes'],'skillpack-check': ['--accept-pending', '--background', '--brain', '--explain', '--fast', '--follow', '--help', '--json', '--list', '--no-accept-pending', '--progress-interval', '--progress-json', '--quiet', '--source', '--strict', '--timeout', '--timings', '--wait', '--yes'],'smoke-test': ['--brain', '--help', '--json', '--source'],diff --git a/src/core/operation-manifest.generated.ts b/src/core/operation-manifest.generated.tsindex cca15f2d97..48715291a9 100644--- a/src/core/operation-manifest.generated.ts+++ b/src/core/operation-manifest.generated.ts@@ -6719,6 +6719,14 @@ export const OPERATION_MANIFEST: OperationMeta[] = ["type": "boolean","description": "Preview: models plan, strict verdict and cost estimate, no LLM calls"},+ "optimizer_mode": {+ "type": "string",+ "enum": [+ "one-shot-rewrite",+ "reflect"+ ],+ "description": "Default one-shot-rewrite: one gated rewrite. reflect opts into the iterative edit loop, which has not yet beaten one-shot on gbrain benchmarks."+ },"shared_skill": {"type": "boolean","description": "Optimize an exact shared catalog revision in a private proposal workspace and publish the accepted body through put_skill."diff --git a/src/core/ops/skillopt.ts b/src/core/ops/skillopt.tsindex 313a9cace3..f098a9295e 100644--- a/src/core/ops/skillopt.ts+++ b/src/core/ops/skillopt.ts@@ -144,6 +144,7 @@ const run_skillopt: Operation = {allow_mutate_bundled: { type: 'boolean', description: 'Required to mutate bundled skills' },held_out_path: { type: 'string', description: 'Path to a held-out test set (JSONL). REQUIRED (>=5 rows) to mutate a bundled skill in place — otherwise the run hard-refuses. Remote callers: must resolve within the skills directory.' },dry_run: { type: 'boolean', description: 'Preview: models plan, strict verdict and cost estimate, no LLM calls' },+ optimizer_mode: { type: 'string', enum: ['one-shot-rewrite', 'reflect'], description: 'Default one-shot-rewrite: one gated rewrite. reflect opts into the iterative edit loop, which has not yet beaten one-shot on gbrain benchmarks.' },shared_skill: { type: 'boolean', description: 'Optimize an exact shared catalog revision in a private proposal workspace and publish the accepted body through put_skill.' },source_id: { type: 'string', description: 'Required with shared_skill: canonical source.' },source_incarnation: { type: 'string', description: 'Required with shared_skill: exact catalog source incarnation.' },@@ -187,6 +188,10 @@ const run_skillopt: Operation = {'run_skillopt: max_cost_usd must be a positive number of USD for remote callers; 0 (uncapped) is a host-only setting',{ example: 5 });}+ const optimizerMode = p.optimizer_mode ?? 'one-shot-rewrite';+ if (optimizerMode !== 'one-shot-rewrite' && optimizerMode !== 'reflect') {+ throw invalidParam(ctx, 'run_skillopt', 'optimizer_mode', "run_skillopt: optimizer_mode must be 'one-shot-rewrite' or 'reflect'", { example: 'reflect' });+ }const { clampRemoteReflectMaxTokens } = await import('../skillopt/output-cap.ts');let reflectMaxTokens: number | undefined;try {@@ -269,6 +274,7 @@ const run_skillopt: Operation = {...skillOptModelOpts(models),...(reflectMaxTokens !== undefined ? { reflectMaxTokens } : {}),mode: 'patch',+ optimizerMode,dryRun: (p.dry_run as boolean) === true,noMutate: (p.no_mutate as boolean) === true,allowMutateBundled: (p.allow_mutate_bundled as boolean) === true,diff --git a/src/core/shared-skills/optimizer.ts b/src/core/shared-skills/optimizer.tsindex e91cb5db42..75eb817356 100644--- a/src/core/shared-skills/optimizer.ts+++ b/src/core/shared-skills/optimizer.ts@@ -97,10 +97,11 @@ export async function optimizeSharedSkill(opts: SkillOptOpts, run: (opts: SkillO}skillName(opts.skillName); skillName(target.pack_id, 'pack_id');requireUuid(target.source_incarnation); requireUuid(target.expected_revision); requireUuid(target.request_id);- if (opts.resumeRunId || opts.writeCapture || opts.disableValidationGate || opts.optimizerMode || opts.reflectMode) {- throw opError('invalid_params', 'Shared optimization does not permit legacy resume, write capture, or evaluation ablations.',- 'Drop the resume run id, write capture, validation-gate and optimizer/reflect mode overrides, then start the optimization again; shared skills always run the full gated evaluation.');+ if (opts.resumeRunId || opts.writeCapture || opts.disableValidationGate || opts.reflectMode || opts.tasks || opts.rolloutHook || opts.criteriaText !== undefined || !opts.benchmarkPath) {+ throw opError('invalid_params', 'Shared optimization does not permit legacy resume, write capture, evaluation ablations or harness inputs.',+ 'Drop the resume run id, write capture, validation-gate and reflect mode overrides, explicit tasks, criteria and rollout hook, and pass a benchmark file; then start the optimization again. Shared skills always run the full gated evaluation.');}+ const benchmarkInput = opts.benchmarkPath;await authorize(ctx, opts);if (await getWriteRequest(ctx.engine, await requestPrincipalForContext(ctx), target.request_id)) {throw new OperationError('request_already_submitted', 'This optimizer publication request has already been submitted.',@@ -132,7 +133,7 @@ export async function optimizeSharedSkill(opts: SkillOptOpts, run: (opts: SkillO}const files = normalizeSkillFiles(opts.skillName, stored.files.map(file => ({ ...file, encoding: 'base64' })));const evaluationInputs: Buffer[] = [];- for (const input of [opts.benchmarkPath, ...(opts.heldOutPath ? [opts.heldOutPath] : [])]) {+ for (const input of [benchmarkInput, ...(opts.heldOutPath ? [opts.heldOutPath] : [])]) {let bytes: Buffer;if (ctx.remote !== false) {const path = skillPath(relative(dirname(skillsDir), resolve(input)).split(sep).join('/'));diff --git a/src/core/skillopt/batch.ts b/src/core/skillopt/batch.tsindex 58ede129fd..0b614b0034 100644--- a/src/core/skillopt/batch.ts+++ b/src/core/skillopt/batch.ts@@ -19,7 +19,7 @@ import * as path from 'node:path';import type { BrainEngine } from '../engine.ts';import { sanitizeEcho, type ModelsPlanEntry, type SkillOptModels } from './models-plan.ts';import { runSkillOpt } from './orchestrator.ts';-import type { RunReceipt, SkillOptOpts } from './types.ts';+import type { OptimizerMode, RunReceipt, SkillOptOpts } from './types.ts';export interface BatchAllOpts {engine: BrainEngine;@@ -47,6 +47,8 @@ export interface BatchAllOpts {lr: number;lrSchedule: 'cosine' | 'linear' | 'constant';split: [number, number, number];+ /** E1: 'one-shot-rewrite' (default) or 'reflect' (--loop), applied to every run. */+ optimizerMode: OptimizerMode;dryRun: boolean;noMutate: boolean;allowMutateBundled: boolean;@@ -115,6 +117,7 @@ export async function runBatchAll(opts: BatchAllOpts): Promise<BatchAllResult> {...(opts.modelsBannerBaseline ? { modelsBannerBaseline: opts.modelsBannerBaseline } : {}),...(opts.reflectMaxTokens !== undefined ? { reflectMaxTokens: opts.reflectMaxTokens } : {}),mode: 'patch',+ optimizerMode: opts.optimizerMode,dryRun: opts.dryRun,noMutate: opts.noMutate,allowMutateBundled: opts.allowMutateBundled,@@ -170,6 +173,8 @@ export interface FleetOpts {lr: number;lrSchedule: 'cosine' | 'linear' | 'constant';split: [number, number, number];+ /** E1: 'one-shot-rewrite' (default) or 'reflect' (--loop), applied to every run. */+ optimizerMode: OptimizerMode;dryRun: boolean;noMutate: boolean;allowMutateBundled: boolean;@@ -257,6 +262,7 @@ export async function runFleet(opts: FleetOpts): Promise<FleetResult> {...(opts.modelsStrict ? { modelsStrict: true } : {}),...(opts.reflectMaxTokens !== undefined ? { reflectMaxTokens: opts.reflectMaxTokens } : {}),mode: 'patch',+ optimizerMode: opts.optimizerMode,dryRun: opts.dryRun,noMutate,allowMutateBundled: opts.allowMutateBundled,diff --git a/src/core/skillopt/benchmark.ts b/src/core/skillopt/benchmark.tsindex 263a94647b..ebd2949d8d 100644--- a/src/core/skillopt/benchmark.ts+++ b/src/core/skillopt/benchmark.ts@@ -334,6 +334,50 @@ export function splitBench(return { train, sel, test };}+/**+ * B1.2 — a run's tasks: explicit train/sel/test lists used exactly as given,+ * or the benchmark file cut by ratio. Explicit lists get the same row+ * validation as a file, must not share a task_id, and keep the D17 floor on+ * sel. Their sha hashes the three lists in order, so moving a task between+ * lists is a different benchmark.+ */+export function loadRunTasks(opts: {+ tasks?: { train: BenchmarkTask[]; sel: BenchmarkTask[]; test?: BenchmarkTask[] };+ benchmarkPath?: string;+ split?: [number, number, number];+ bootstrapReviewed?: boolean;+}): { bench: Benchmark; split: BenchmarkSplit } {+ if (!opts.tasks) {+ if (!opts.benchmarkPath) {+ throw errorFor({ class: 'BenchmarkNotFound', code: 'benchmark_not_found', message: 'No benchmark: pass benchmarkPath or explicit tasks.', hint: 'Pass --benchmark <path>.' });+ }+ const bench = loadBenchmark(opts.benchmarkPath, { bootstrapReviewed: opts.bootstrapReviewed });+ return { bench, split: splitBench(bench, opts.split ?? [4, 1, 5]) };+ }+ if (opts.benchmarkPath) {+ throw errorFor({ class: 'BadSplit', code: 'explicit_tasks_conflict', message: 'Explicit tasks and a benchmark file were both given.', hint: 'Pass either tasks or benchmarkPath, not both.' });+ }+ const label = '(explicit tasks)';+ const check = (list: BenchmarkTask[], name: string) => list.map((t, i) => validateRow(t, i + 1, `${label} ${name}`));+ const split: BenchmarkSplit = { train: check(opts.tasks.train, 'train'), sel: check(opts.tasks.sel, 'sel'), test: check(opts.tasks.test ?? [], 'test') };+ const all = [...split.train, ...split.sel, ...split.test];+ const seen = new Set<string>();+ for (const t of all) {+ if (seen.has(t.task_id)) {+ throw errorFor({ class: 'BenchmarkDuplicateId', code: 'benchmark_duplicate_task_id', message: `task_id '${t.task_id}' appears more than once across train, sel and test.`, hint: 'Each task belongs to exactly one list.' });+ }+ seen.add(t.task_id);+ }+ if (split.train.length === 0) {+ throw errorFor({ class: 'BenchmarkEmpty', code: 'benchmark_empty', message: 'Explicit train list is empty.', hint: 'Pass at least one train task.' });+ }+ if (split.sel.length < D_SEL_MIN_SIZE) {+ throw errorFor({ class: 'DSelTooSmall', code: 'd_sel_too_small', message: `Explicit sel list has ${split.sel.length} task(s) (need >=${D_SEL_MIN_SIZE} for meaningful validation).`, hint: `Pass at least ${D_SEL_MIN_SIZE} selection tasks.` });+ }+ const canonical = JSON.stringify({ train: split.train, sel: split.sel, test: split.test });+ return { bench: { source_path: label, tasks: all, benchmark_sha8: createHash('sha256').update(canonical).digest('hex').slice(0, 8) }, split };+}+/** Parse a split string like "4:1:5" into a tuple. */export function parseSplit(s: string): [number, number, number] {const parts = s.split(':').map((p) => Number(p.trim()));diff --git a/src/core/skillopt/checkpoint.ts b/src/core/skillopt/checkpoint.tsindex 34539e2222..117042cc66 100644--- a/src/core/skillopt/checkpoint.ts+++ b/src/core/skillopt/checkpoint.ts@@ -41,6 +41,7 @@ import { errorFor } from '../errors.ts';import { atomicWrite } from './apply-edits.ts';import type { BudgetSnapshot } from '../budget/budget-tracker.ts';import { buildModelsUsed, type ModelUsageRow } from '../budget/models-used.ts';+import type { EditOp, OptimizerMode } from './types.ts';const CHECKPOINT_SCHEMA = 1;@@ -65,6 +66,13 @@ export interface RunSpec {lr: number;lr_schedule: 'cosine' | 'linear' | 'constant';reflect_max_tokens: number;+ /** E1. Absent on checkpoints written before one-shot became the default: those ran the loop. */+ optimizer_mode?: OptimizerMode;+}++/** The mode a checkpoint's run used; a legacy checkpoint ran the loop. */+export function checkpointOptimizerMode(cp: RunCheckpoint): OptimizerMode {+ return cp.run_spec?.optimizer_mode ?? 'reflect';}/** Whole-run optimizer-reply accounting (survives resume). */@@ -110,6 +118,12 @@ export interface RunCheckpoint {next_step?: number;run_spec?: RunSpec;tally?: ReflectTally;+ /**+ * D2: with a held-out set, accepted steps only move best_skill_text; this+ * carries the champion's last accepted step, its applied edits and how many+ * steps it took until the single held-out check writes it.+ */+ champion?: { epoch: number; step: number; steps: number; edits: EditOp[] };cumulative_cost_usd: number;/** Ledger rows banked across segments. Absent on legacy checkpoints. */models_used?: ModelUsageRow[];@@ -205,6 +219,8 @@ export function assertResumeCompatible(cp: RunCheckpoint, current: RunSpec): voiif (JSON.stringify(stored[field]) !== JSON.stringify(current[field])) refuse(field, stored[field], current[field]);}if (!stored.mutate && current.mutate) refuse('mutate policy', 'no-mutate', 'mutate');+ const storedMode = checkpointOptimizerMode(cp);+ if (current.optimizer_mode !== undefined && current.optimizer_mode !== storedMode) refuse('optimizer_mode', storedMode, current.optimizer_mode);}export function checkpointPath(skillsDir: string, skillName: string, runId: string): string {diff --git a/src/core/skillopt/cycle-phase.ts b/src/core/skillopt/cycle-phase.tsindex 05fbb1c774..48a53a0eea 100644--- a/src/core/skillopt/cycle-phase.ts+++ b/src/core/skillopt/cycle-phase.ts@@ -210,6 +210,7 @@ export async function runPhaseSkillopt(opts: SkilloptPhaseOpts): Promise<Skillop...skillOptModelOpts(models),modelsBannerBaseline: basePlan,mode: 'patch',+ optimizerMode: 'one-shot-rewrite',dryRun: opts.dryRun ?? false,// Bundled-skill safety: dream-cycle NEVER auto-mutates bundled skills.// For bundled skills we set --no-mutate; the user reviews proposed.mddiff --git a/src/core/skillopt/held-out.ts b/src/core/skillopt/held-out.tsindex 0a23b2b8ca..fb6ad3367c 100644--- a/src/core/skillopt/held-out.ts+++ b/src/core/skillopt/held-out.ts@@ -33,7 +33,7 @@ import type { OperationContext } from '../ops/contract.ts';import { loadBenchmark } from './benchmark.ts';import { D_SEL_MIN_SIZE } from './types.ts';import { gbrainPath } from '../config.ts';-import type { BenchmarkTask } from './types.ts';+import type { BenchmarkTask, RolloutBatchHook } from './types.ts';import { scoreSkillOnTasks } from './validate-gate.ts';const CAPTURE_CONFIG_KEY = 'skillopt.capture_enabled';@@ -117,6 +117,8 @@ export interface HeldOutGateOpts {/** #4119 — in-rollout-loop deadline (epoch ms); see ValidateGateOpts. */deadlineMs?: number;abortSignal?: AbortSignal;+ /** B1.3: score both skills through the harness hook (phase 'held_out'). */+ rolloutHook?: RolloutBatchHook;}export interface HeldOutGateResult {@@ -145,8 +147,10 @@ export async function runHeldOutGate(opts: HeldOutGateOpts): Promise<HeldOutGatetasks: opts.heldOutTasks,targetModel: opts.targetModel,judgeModel: opts.judgeModel,+ phase: 'held_out' as const,...(opts.deadlineMs !== undefined ? { deadlineMs: opts.deadlineMs } : {}),...(opts.abortSignal ? { abortSignal: opts.abortSignal } : {}),+ ...(opts.rolloutHook ? { rolloutHook: opts.rolloutHook } : {}),};const baselineScore = await scoreSkillOnTasks({ ...scoreOpts, skillText: opts.baselineSkillText });const candidateScore = await scoreSkillOnTasks({ ...scoreOpts, skillText: opts.candidateSkillText });diff --git a/src/core/skillopt/help.ts b/src/core/skillopt/help.tsindex af50a7b3a2..37e44b2731 100644--- a/src/core/skillopt/help.ts+++ b/src/core/skillopt/help.ts@@ -7,6 +7,11 @@ export const SKILLOPT_HELP_TEXT = `gbrain skillopt <skill-name> [flags]Self-evolving skill optimization. Treats SKILL.md as the trainable parametersof a frozen agent. Validation-gated, budget-capped, atomic-versioned.+By default it runs ONE gated rewrite: a forward pass on one train batch, one+whole-body rewrite, then the selection gate. --loop runs the iterative edit+loop instead; on gbrain's benchmarks it has not yet beaten the one-shot rewrite+(same held-out score at ~1.8x the cost), so it is opt-in until it does.+Based on SkillOpt (arXiv 2605.23904, MSR May 2026).Required (one of):@@ -22,7 +27,8 @@ Required (one of):after human review)--bootstrap-reviewed Confirm bootstrap benchmark was hand-reviewed-Training knobs:+Training knobs (--epochs, --lr, --lr-schedule apply to --loop only;+--batch-size is also the one-shot forward-pass size):--epochs N Default 4--batch-size N Default 8--lr N Max edits per step. Default 4@@ -49,8 +55,8 @@ Models:Same as skillopt.models_strict=true.Modes:- --patch Edit ops only (default; safer)- --rewrite Allow full rewrites of sections+ --loop Iterative reflect loop (edit ops, epochs).+ Default is one gated one-shot rewrite.--dry-run Models plan, strict verdict + cost estimate,no LLM calls (exit 1 only on a strict failure)--no-mutate Write proposed.md without replacing SKILL.mddiff --git a/src/core/skillopt/index.ts b/src/core/skillopt/index.tsindex d4d8a54e8b..282a08033a 100644--- a/src/core/skillopt/index.ts+++ b/src/core/skillopt/index.ts@@ -1,16 +1,34 @@/*** Public skillopt surface for downstream eval harnesses (audit finding* skillopt-cats-11, closed by the 2026-08 fix wave): gbrain-evals cat30–33- * consume exactly these three entry points and previously reached them via- * physical `node_modules/gbrain/src/...` deep imports, which break under- * non-hoisting installs (Yarn PnP, relocated runners). The `./core/skillopt`- * package export maps here; keep this barrel to the externally-consumed- * surface — internal skillopt modules stay internal.+ * consume these entry points and previously reached them via physical+ * `node_modules/gbrain/src/...` deep imports, which break under non-hoisting+ * installs (Yarn PnP, relocated runners). The `./core/skillopt` package+ * export maps here; keep this barrel to the externally-consumed surface —+ * internal skillopt modules stay internal. B1.1 adds the rollout and scoring+ * primitives a method benchmark's rollout server runs, so every optimizer arm+ * runs the target with gbrain's own rollout.*/export { runSkillOpt } from './orchestrator.ts';export type { RunSkillOptResult } from './orchestrator.ts';-export type { SkillOptOpts, BenchmarkTask } from './types.ts';-export { scoreSkillOnTasks } from './validate-gate.ts';-export type { ScoreOnTasksOpts } from './validate-gate.ts';+export type {+ SkillOptOpts,+ BenchmarkTask,+ Judge,+ Trajectory,+ ScoredRollout,+ RunReceipt,+ OptimizerMode,+ RolloutPhase,+ RolloutBatchRequest,+ RolloutBatchHook,+} from './types.ts';+export { scoreSkillOnTasks, scoreSkillPerTask } from './validate-gate.ts';+export type { ScoreOnTasksOpts, PerTaskScore } from './validate-gate.ts';export { loadHeldOut } from './held-out.ts';+export { runRollout, READ_ONLY_BRAIN_TOOLS, rolloutToolsForSkill } from './rollout.ts';+export type { RolloutOpts } from './rollout.ts';+export { scoreTrajectory } from './score.ts';+export { describeJudge, describeJudges } from './reflect.ts';+export { SKILLOPT_HOOK_BUDGET_EXHAUSTED, SKILLOPT_HOOK_INFRA_ERROR } from './must-abort.ts';diff --git a/src/core/skillopt/job.ts b/src/core/skillopt/job.tsindex 1db22dc3ba..0cd3aa14bb 100644--- a/src/core/skillopt/job.ts+++ b/src/core/skillopt/job.ts@@ -26,9 +26,9 @@ import type { SkillOptOpts } from './types.ts';export type SkillOptJobInput = Pick<SkillOptOpts,| 'skillsDir' | 'skillName' | 'benchmarkPath' | 'epochs' | 'batchSize' | 'lr' | 'lrSchedule' | 'split'- | 'reflectMaxTokens' | 'mode' | 'dryRun' | 'noMutate' | 'allowMutateBundled' | 'heldOutPath'+ | 'reflectMaxTokens' | 'mode' | 'optimizerMode' | 'dryRun' | 'noMutate' | 'allowMutateBundled' | 'heldOutPath'| 'bootstrapReviewed' | 'maxCostUsd' | 'maxCostSource' | 'maxRuntimeMin' | 'force'-> & { models: SkillOptModels; modelFlags: SkillOptModelFlags; modelsStrict: boolean };+> & { benchmarkPath: string; models: SkillOptModels; modelFlags: SkillOptModelFlags; modelsStrict: boolean };export function buildSkillOptJobData(input: SkillOptJobInput): Record<string, unknown> {const flags = input.modelFlags;@@ -57,6 +57,7 @@ export function buildSkillOptJobData(input: SkillOptJobInput): Record<string, un...(input.modelsStrict ? { models_strict: true } : {}),...(input.reflectMaxTokens !== undefined ? { reflect_max_tokens: input.reflectMaxTokens } : {}),mode: input.mode,+ optimizer_mode: input.optimizerMode,dry_run: input.dryRun,no_mutate: input.noMutate,allow_mutate_bundled: input.allowMutateBundled,@@ -112,6 +113,8 @@ export async function runSkillOptJob(engine: BrainEngine, rawData: unknown): PromodelsStrict: data.models_strict === true,reflectMaxTokens: clampRemoteReflectMaxTokens(data.reflect_max_tokens),mode: (data.mode as 'patch' | 'rewrite') ?? 'patch',+ // A job queued before one-shot became the default carries no mode: it was queued for the loop.+ optimizerMode: data.optimizer_mode === 'one-shot-rewrite' ? 'one-shot-rewrite' : 'reflect',dryRun: Boolean(data.dry_run),noMutate: Boolean(data.no_mutate),allowMutateBundled: Boolean(data.allow_mutate_bundled),diff --git a/src/core/skillopt/must-abort.ts b/src/core/skillopt/must-abort.tsindex 8ac483f91b..ff3b7a7d6c 100644--- a/src/core/skillopt/must-abort.ts+++ b/src/core/skillopt/must-abort.ts@@ -14,8 +14,13 @@ import { isMustAbortError } from '../worker-pool.ts';/** #4119 — the runtime-deadline breach error, shared with validate-gate + orchestrator. */export const SKILLOPT_RUNTIME_EXCEEDED = 'skillopt_runtime_exceeded';+/** B1.3 — a harness rollout hook refused a batch over its shared budget. */+export const SKILLOPT_HOOK_BUDGET_EXHAUSTED = 'skillopt_hook_budget_exhausted';+/** B1.3 — a harness rollout hook failed (already retried by the caller), or returned a malformed batch. */+export const SKILLOPT_HOOK_INFRA_ERROR = 'skillopt_hook_infra_error';+export function isSkilloptMustAbort(err: unknown): boolean {if (isMustAbortError(err) || isAIInvocationPolicyError(err)) return true;const msg = err instanceof Error ? err.message : String(err);- return msg.includes(SKILLOPT_RUNTIME_EXCEEDED) || msg.includes('SIGINT');+ return [SKILLOPT_RUNTIME_EXCEEDED, SKILLOPT_HOOK_BUDGET_EXHAUSTED, SKILLOPT_HOOK_INFRA_ERROR, 'SIGINT'].some((tag) => msg.includes(tag));}diff --git a/src/core/skillopt/orchestrator.ts b/src/core/skillopt/orchestrator.tsindex 0dd550bd72..2a83211921 100644--- a/src/core/skillopt/orchestrator.ts+++ b/src/core/skillopt/orchestrator.ts@@ -97,7 +97,7 @@*/import { assertLegacySkillWriter } from '../skillpack/writer-guard.ts';-import { randomUUID } from 'node:crypto';+import { createHash, randomUUID } from 'node:crypto';import * as fs from 'node:fs';import { BudgetExhausted, BudgetTracker, loadPricingOverrides, type NoPricingGuidance } from '../budget/budget-tracker.ts';import type { PricingOverrides } from '../budget/reservation-cost.ts';@@ -106,7 +106,7 @@ import { withBudgetTracker } from '../ai/gateway.ts';import { errorFor } from '../errors.ts';import { applyEditBatch, getWorkingTreeStatusForFile, splitFrontmatter } from './apply-edits.ts';import { logEvent, sha8 } from './audit.ts';-import { loadBenchmark, splitBench, parseSplit } from './benchmark.ts';+import { loadRunTasks, parseSplit } from './benchmark.ts';import { getBundledSkillContext, shouldMutateSkillFile, assertBundledMutationHeldOut } from './bundled-skill-gate.ts';import {advanceCursor,@@ -126,7 +126,7 @@ import {import { loadHeldOut, runHeldOutGate } from './held-out.ts';import { withSkilloptLock } from './lock.ts';import { resolveLrSchedule } from './lr-schedule.ts';-import { SKILLOPT_RUNTIME_EXCEEDED } from './must-abort.ts';+import { SKILLOPT_HOOK_BUDGET_EXHAUSTED, SKILLOPT_HOOK_INFRA_ERROR, SKILLOPT_RUNTIME_EXCEEDED } from './must-abort.ts';import {buildModelsPlan,formatModelsBanner,@@ -157,9 +157,9 @@ import {skillPath,writeProposed,} from './version-store.ts';-import { runValidationGate, scoreSkillOnTasks } from './validate-gate.ts';-import { ROLLOUT_SUCCESS_THRESHOLD } from './types.ts';-import type { SkillOptOpts, EditOp, RunReceipt, BenchmarkTask } from './types.ts';+import { runValidationGate } from './validate-gate.ts';+import { ROLLOUT_SUCCESS_THRESHOLD, VALIDATION_RUNS_PER_TASK } from './types.ts';+import type { SkillOptOpts, EditOp, RunReceipt, BenchmarkTask, RolloutPhase } from './types.ts';export interface RunSkillOptResult {outcome: 'accepted' | 'no_improvement' | 'aborted' | 'errored';@@ -227,9 +227,8 @@ export async function runSkillOpt(opts: SkillOptOpts): Promise<RunSkillOptResultskillName,});- // Load + validate benchmark (D17 floor enforcement, D15 sentinel check).- const bench = loadBenchmark(opts.benchmarkPath, { bootstrapReviewed: opts.bootstrapReviewed });- const split = splitBench(bench, opts.split);+ // Load + validate the tasks (D17 floor enforcement, D15 sentinel check, B1.2 explicit lists).+ const { bench, split } = loadRunTasks(opts);// Held-out must be INDEPENDENT of the benchmark. If a held-out file shares// task_ids with the benchmark, an overfit candidate passes the held-out gate@@ -277,6 +276,8 @@ export async function runSkillOpt(opts: SkillOptOpts): Promise<RunSkillOptResultinteractive: process.stderr.isTTY === true,reflectMaxTokens: reflectCap.maxTokens,judgeModels: modelsPlan.filter((e) => e.touchpoint === 'judge' && e.active).map((e) => e.model),+ optimizerMode: opts.optimizerMode,+ rolloutsExternal: opts.rolloutHook !== undefined,...(pricingOverrides ? { pricingOverrides } : {}),});if (opts.json !== true) {@@ -286,6 +287,7 @@ export async function runSkillOpt(opts: SkillOptOpts): Promise<RunSkillOptResultoptimizerModel: opts.optimizerModel, targetModel: opts.targetModel, judgeModel: opts.judgeModel,maxCostUsd: opts.maxCostUsd,maxCostSource: opts.maxCostSource ?? 'user',+ optimizerMode: opts.optimizerMode,}) + '\n');}if (preflightResult.abort_code === 'no_pricing') {@@ -327,6 +329,8 @@ export async function runSkillOpt(opts: SkillOptOpts): Promise<RunSkillOptResultreflect_max_tokens_source: reflectCap.source,models_plan: modelsPlan,models_strict: strict,+ optimizer_mode: opts.optimizerMode,+ cost_scope: opts.rolloutHook ? 'optimizer_only' : 'all',};return { outcome: 'aborted', receipt, finalText: fs.readFileSync(skillFile, 'utf8'), mutatedSkillFile: false };}@@ -392,6 +396,10 @@ export function classifyAbortError(abortDetail: msg,...(err.pricing ? { pricing: err.pricing } : {}),};+ } else if (msg.includes(SKILLOPT_HOOK_BUDGET_EXHAUSTED)) {+ return { outcome: 'aborted', abortReason: 'budget_exhausted', abortDetail: msg };+ } else if (msg.includes(SKILLOPT_HOOK_INFRA_ERROR)) {+ return { outcome: 'aborted', abortReason: 'error', abortDetail: msg };} else if (msg.includes(SKILLOPT_RUNTIME_EXCEEDED)) {return {outcome: 'aborted',@@ -413,8 +421,8 @@ const BUDGET_ADMISSION_HINT = ' (raise --max-cost-usd; each optimizer call reserasync function runOptimizationLoop(opts: SkillOptOpts,- bench: ReturnType<typeof loadBenchmark>,- split: ReturnType<typeof splitBench>,+ bench: ReturnType<typeof loadRunTasks>['bench'],+ split: ReturnType<typeof loadRunTasks>['split'],bundledCtx: ReturnType<typeof getBundledSkillContext>,mutateDecision: ReturnType<typeof shouldMutateSkillFile>,heldOutTasks: BenchmarkTask[],@@ -429,14 +437,13 @@ async function runOptimizationLoop(const { reflectCap, modelsPlan } = resolved;const skillFile = skillPath(skillsDir, skillName);- // Plain-English success criteria from the benchmark judges, fed to the- // optimizer's reflect step so it targets WHAT the scorer rewards instead of- // guessing from a bare pass/fail score. Without this, rule-judged benchmarks- // (e.g. "must include a Confidence: line") are near-unoptimizable: the- // optimizer proposes plausible-but-off edits, the candidate scores 0, the gate- // rejects it, and the skill never changes. The held-out gate defends against- // the optimizer gaming these criteria at the expense of real quality.- const benchmarkCriteria = describeJudges([...split.train, ...split.sel, ...split.test]);+ // Plain-English success criteria fed to the optimizer so it targets WHAT the+ // scorer rewards instead of guessing from a bare pass/fail score. Without+ // this, rule-judged benchmarks (e.g. "must include a Confidence: line") are+ // near-unoptimizable. Built from the train split only, with per-task+ // arguments (answer keys) redacted (B1.4, defect 3): selection and test+ // judges never reach the proposer. A caller's criteriaText replaces it.+ const benchmarkCriteria = opts.criteriaText ?? describeJudges(split.train);// Crash-recovery sweep (D8): revert any pending rows from a prior crashed run.revertAllPending(skillsDir, skillName);@@ -447,11 +454,11 @@ async function runOptimizationLoop(const runSpec: RunSpec = {skills_dir: skillsDir,- benchmark_path: opts.benchmarkPath,+ benchmark_path: opts.benchmarkPath ?? bench.source_path,benchmark_sha8: bench.benchmark_sha8,held_out_path: opts.heldOutPath ?? null,held_out_sha8: opts.heldOutPath ? sha8(fs.readFileSync(opts.heldOutPath, 'utf8')) : null,- split: opts.split,+ split: opts.tasks ? [split.train.length, split.sel.length, split.test.length] : opts.split ?? [4, 1, 5],mutate: mutateDecision.mutate,no_mutate: opts.noMutate,allow_mutate_bundled: opts.allowMutateBundled,@@ -464,6 +471,7 @@ async function runOptimizationLoop(lr: opts.lr,lr_schedule: opts.lrSchedule,reflect_max_tokens: reflectCap.maxTokens,+ optimizer_mode: opts.optimizerMode,};// Resume or init checkpoint. A resume must match what the checkpoint measured.@@ -521,7 +529,7 @@ async function runOptimizationLoop(max_cost_usd: opts.maxCostUsd,reflect_mode: opts.reflectMode ?? 'both',validation_gate_disabled: opts.disableValidationGate === true,- optimizer_mode: opts.optimizerMode ?? 'reflect',+ optimizer_mode: opts.optimizerMode,held_out_size: heldOutTasks.length,models_plan: modelsPlan,} as never);@@ -583,20 +591,62 @@ async function runOptimizationLoop(// (no partial commit beyond whatever already passed every gate).const deadline = Date.now() + opts.maxRuntimeMin * 60_000;+ // Scoring sites (B1.3 / B1.6): every rollout goes through runValidationGate+ // (with the caller's hook when set) and is counted by phase.+ const hook = opts.rolloutHook;+ const rollouts: Record<RolloutPhase, number> = { baseline_sel: 0, train_forward: 0, sel_gate: 0, held_out: 0, test: 0 };+ const selCandidates = new Set<string>();+ const score = async (phase: RolloutPhase, skillText: string, tasks: BenchmarkTask[], o: { bestScore?: number; runsPerTask?: number; epoch?: number; step?: number } = {}) => {+ const runsPerTask = o.runsPerTask ?? VALIDATION_RUNS_PER_TASK;+ const result = await runValidationGate({+ engine: opts.engine,+ operationContext: opts.operationContext,+ candidateSkillText: skillText,+ selSet: tasks,+ bestScore: o.bestScore ?? -1,+ targetModel: opts.targetModel,+ judgeModel: opts.judgeModel,+ runsPerTask,+ deadlineMs: deadline,+ phase,+ epoch: o.epoch ?? 0,+ step: o.step ?? 0,+ ...(hook ? { rolloutHook: hook } : {}),+ });+ rollouts[phase] += tasks.length * runsPerTask;+ if (phase === 'sel_gate') selCandidates.add(createHash('sha256').update(skillText).digest('hex'));+ return result;+ };+ // D2: with a held-out set, nothing is written until the single held-out+ // check on the final champion passes; without one, each accepted step is+ // written as before (so an abort keeps the best skill so far on disk).+ const deferWrites = heldOutTasks.length > 0;+ const writeCandidate = (text: string, selScore: number, epoch: number, step: number, edits: EditOp[]) => {+ if (mutateDecision.mutate) {+ acceptCandidate({ skillsDir, skillName, runId, epoch, step, edits, candidateText: text, selScore, delta: selScore - baselineSelScore });+ } else {+ writeProposed(skillsDir, skillName, text);+ }+ finalText = text;+ };+ const promote = (text: string, selScore: number, epoch: number, step: number, edits: EditOp[]) => {+ if (deferWrites) {+ const prev = checkpoint!.champion;+ checkpoint!.champion = { epoch, step, steps: (prev?.steps ?? 0) + 1, edits: [...(prev?.edits ?? []), ...edits] };+ } else {+ writeCandidate(text, selScore, epoch, step, edits);+ tally.accepted_steps += 1;+ }+ checkpoint!.best_sel_score = selScore;+ checkpoint!.best_skill_text = text;+ };+ let heldOut: RunReceipt['held_out'];+try {await withBudgetTracker(tracker, async () => {// Baseline eval: score the baseline skill on D_sel to set best_sel_score.// We use the FULL validation gate with median-of-3 for a stable baseline.- const baselineGate = await runValidationGate({- engine: opts.engine,- operationContext: opts.operationContext,- candidateSkillText: baselineText,- selSet: split.sel,- bestScore: -1, // any score > -1 + 0.05 accepts; we just want the score.- targetModel: opts.targetModel,- judgeModel: opts.judgeModel,- deadlineMs: deadline,- });+ const baselineGate = await score('baseline_sel', baselineText, split.sel);if (checkpoint!.best_sel_score === 0) {// Fresh run; set baseline.checkpoint!.best_sel_score = baselineGate.selScore;@@ -605,28 +655,19 @@ async function runOptimizationLoop(}baselineSelScore = baselineGate.selScore;- // ── Ablation (cat31 config C): one-shot rewrite ─────────────────────- // ONE rewrite call, no optimization loop, no D12 gate. Forward-pass on- // D_sel for signal, rewrite the body once, score the result, promote it.- // Sets last_completed_epoch = epochs so the epoch loop below no-ops, then- // falls through to the shared final-test + receipt path.- if (opts.optimizerMode === 'one-shot-rewrite') {+ // ── E1 one-shot rewrite (the default) ─────────────────────────────────+ // The loop's first step with a whole-body rewrite instead of edits:+ // forward pass on one train batch, one rewrite call, then the same D12+ // selection gate a loop step faces. A rejected rewrite leaves the skill+ // unchanged. Sets the cursor past the last epoch so the loop below no-ops.+ if (opts.optimizerMode === 'one-shot-rewrite' && resumeCursor(checkpoint!, stepsPerEpoch).epoch <= opts.epochs) {stepStartTally = structuredClone(tally);- const { body: baselineBody, bodyStart } = splitFrontmatter(baselineText);- const fmBlock = baselineText.slice(0, bodyStart);- const fwd = await runValidationGate({- engine: opts.engine,- operationContext: opts.operationContext,- candidateSkillText: baselineText,- selSet: split.sel,- bestScore: -1,- targetModel: opts.targetModel,- judgeModel: opts.judgeModel,- runsPerTask: 1,- deadlineMs: deadline,- });+ totalStepsRun = 1;+ const incumbent = checkpoint!.best_skill_text;+ const { body: incumbentBody, bodyStart } = splitFrontmatter(incumbent);+ const fwd = await score('train_forward', incumbent, split.train.slice(0, opts.batchSize), { runsPerTask: 1, epoch: 1, step: 1 });const rewrite = await runOneShotRewrite({- skillBodyText: baselineBody,+ skillBodyText: incumbentBody,successes: fwd.scoredRollouts.filter((r) => r.score >= ROLLOUT_SUCCESS_THRESHOLD),failures: fwd.scoredRollouts.filter((r) => r.score < ROLLOUT_SUCCESS_THRESHOLD),rejected: [],@@ -642,49 +683,28 @@ async function runOptimizationLoop(});if (rewrite.error) process.stderr.write(`[skillopt] one-shot rewrite produced no candidate (${rewrite.error})\n`);if (rewrite.newBody) {- const candidate = fmBlock + rewrite.newBody;- // Held-out gate still applies (independent signal); skipped when unset.- let promote = true;- if (heldOutTasks.length > 0) {- const ho = await runHeldOutGate({- engine: opts.engine,- operationContext: opts.operationContext,- candidateSkillText: candidate,- baselineSkillText: baselineText,- heldOutTasks,- deadlineMs: deadline,- targetModel: opts.targetModel,- judgeModel: opts.judgeModel,- });- promote = ho.passed;- }- if (promote) {- const score = await scoreSkillOnTasks({- engine: opts.engine,- operationContext: opts.operationContext,- skillText: candidate,- tasks: split.sel,- targetModel: opts.targetModel,- judgeModel: opts.judgeModel,- deadlineMs: deadline,- });- checkpoint!.best_sel_score = score;- checkpoint!.best_skill_text = candidate;- totalStepsRun = 1;- if (mutateDecision.mutate) {- acceptCandidate({- skillsDir, skillName, runId, epoch: 1, step: 1,- edits: [], // whole-body rewrite — no granular edits to record- candidateText: candidate, selScore: score, delta: score - baselineSelScore,- });- } else {- writeProposed(skillsDir, skillName, candidate);- }- tally.accepted_steps += 1;- finalText = candidate;- }+ const candidate = incumbent.slice(0, bodyStart) + rewrite.newBody;+ const gate = await score('sel_gate', candidate, split.sel, { bestScore: checkpoint!.best_sel_score, epoch: 1, step: 1 });+ // Ablation (cat31 config D): disableValidationGate promotes regardless.+ const accepted = opts.disableValidationGate === true || gate.accepted;+ logEvent({+ kind: 'step',+ run_id: runId,+ skill: skillName,+ epoch: 1,+ step: 1,+ sel_score_median: gate.selScore,+ sel_score_runs: gate.perTaskMedians.map((t) => t.median),+ accepted,+ edits_attempted: 1,+ edits_applied: 1,+ delta: gate.selScore - checkpoint!.best_sel_score,+ ...(accepted ? {} : { reason: gate.reason ?? 'rejected' }),+ cumulative_cost_usd: tracker.snapshot().cumulativeCostUsd,+ } as never);+ if (accepted) promote(candidate, gate.selScore, 1, 1, []);}- advanceCursor(checkpoint!, opts.epochs, 1, 1); // no-op the epoch loop below+ advanceCursor(checkpoint!, opts.epochs, stepsPerEpoch, stepsPerEpoch); // no-op the epoch loop belowbankAccounting(checkpoint!, prior, tracker.snapshot());saveCheckpoint(skillsDir, skillName, checkpoint!);stepStartTally = undefined;@@ -711,17 +731,7 @@ async function runOptimizationLoop(// runsPerTask=1 — for the train batch we only need a rough partition// into successes/failures, not the median-of-3 noise rejection the// sel-side gate uses. ScoredRollouts come back via GateResult.- const forwardGate = await runValidationGate({- engine: opts.engine,- operationContext: opts.operationContext,- candidateSkillText: checkpoint!.best_skill_text,- selSet: batch,- bestScore: -1,- targetModel: opts.targetModel,- judgeModel: opts.judgeModel,- runsPerTask: 1,- deadlineMs: deadline,- });+ const forwardGate = await score('train_forward', checkpoint!.best_skill_text, batch, { runsPerTask: 1, epoch, step });// Partition into successes vs failures (>= 0.5 threshold). Reflect// gets the actual scored trajectories so failure-mode + success-mode// analysis can ground in real agent behavior (D7).@@ -752,6 +762,15 @@ async function runOptimizationLoop(const fresh = allEdits.filter((e) => !isRejected(rejected, checkpoint!.best_skill_text, [e]));// Apply under LR budget.const applied = applyEditBatch(checkpoint!.best_skill_text, fresh, lrBudget);+ // Defect 4: only edits that were actually applied are attributed to+ // the gate's verdict. Edits that failed to apply are filed as+ // apply_failed; edits the LR budget clipped were never tried and+ // stay free to be proposed again.+ const appliedEdits = applied.results.flatMap((r) => (r.outcome === 'applied' ? [r.edit] : []));+ const applyFailed = applied.results.flatMap((r) =>+ r.outcome === 'rejected' && r.detail !== 'lr_budget_exhausted'+ ? [makeRejectedEntry(checkpoint!.best_skill_text, [r.edit], 'apply_failed')]+ : []);if (fresh.length === 0) {// #4741: the optimizer proposed NOTHING (reflect returned/parsed no@@ -784,12 +803,9 @@ async function runOptimizationLoop(break stepBody;}- if (applied.results.every((r) => r.outcome === 'rejected')) {- // Nothing applied; record rejected entries + skip gate.- const newRejections = fresh.map((e) =>- makeRejectedEntry(checkpoint!.best_skill_text, [e], 'apply_failed'),- );- saveRejectedBuffer(skillsDir, skillName, newRejections);+ if (appliedEdits.length === 0) {+ // Nothing applied; record the apply failures + skip gate.+ saveRejectedBuffer(skillsDir, skillName, applyFailed);logEvent({kind: 'step',run_id: runId,@@ -810,128 +826,42 @@ async function runOptimizationLoop(}// VALIDATION GATE (D12 median-of-3 + epsilon=0.05, D4 parallel).- const gate = await runValidationGate({- engine: opts.engine,- operationContext: opts.operationContext,- candidateSkillText: applied.newText,- selSet: split.sel,- bestScore: checkpoint!.best_sel_score,- targetModel: opts.targetModel,- judgeModel: opts.judgeModel,- deadlineMs: deadline,- });+ const gate = await score('sel_gate', applied.newText, split.sel, { bestScore: checkpoint!.best_sel_score, epoch, step });// Ablation (cat31 config D): disableValidationGate greedy-accepts any// applied edit, bypassing the D12 median+epsilon check. We still ran// the gate above to GET the score (so sel_score tracking is honest),- // but ignore its accept verdict. We only reach here when at least one- // edit applied (the all-rejected case exits via `break stepBody` earlier,- // which still advances the cursor).+ // but ignore its accept verdict.const stepAccepted = opts.disableValidationGate === true ? true : gate.accepted;+ const delta = gate.selScore - checkpoint!.best_sel_score;if (stepAccepted) {- // F11 HELD-OUT GATE — guards CHECKPOINT ACCEPTANCE (not just file- // mutation), so the no-mutate / proposed.md paths can't promote a- // held-out-failing candidate either. Independent signal: a candidate- // that improved D_sel but regresses on the held-out set is refused- // (benchmark-gaming defense). Skipped when no held-out is configured.- if (heldOutTasks.length > 0) {- const ho = await runHeldOutGate({- engine: opts.engine,- operationContext: opts.operationContext,- candidateSkillText: applied.newText,- baselineSkillText: baselineText,- heldOutTasks,- deadlineMs: deadline,- targetModel: opts.targetModel,- judgeModel: opts.judgeModel,- });- if (!ho.passed) {- const newRejections = fresh.map((e) =>- makeRejectedEntry(checkpoint!.best_skill_text, [e], 'held_out_regression'),- );- saveRejectedBuffer(skillsDir, skillName, newRejections);- logEvent({- kind: 'step',- run_id: runId,- skill: skillName,- epoch,- step,- sel_score_median: gate.selScore,- sel_score_runs: gate.perTaskMedians.map((t) => t.median),- accepted: false,- edits_attempted: fresh.length,- edits_applied: applied.results.filter((r) => r.outcome === 'applied').length,- delta: gate.selScore - checkpoint!.best_sel_score,- reason: 'held_out_regression',- held_out_baseline: ho.baselineScore,- held_out_candidate: ho.candidateScore,- cumulative_cost_usd: tracker.snapshot().cumulativeCostUsd,- } as never);- break stepBody; // do NOT promote- }- }-- const delta = gate.selScore - checkpoint!.best_sel_score;// ACCEPT: D8 commit via version-store (mutate) OR write proposed.md- // for the --no-mutate / bundled-without-allow paths.- if (mutateDecision.mutate) {- acceptCandidate({- skillsDir,- skillName,- runId,- epoch,- step,- edits: fresh,- candidateText: applied.newText,- selScore: gate.selScore,- delta,- });- } else {- writeProposed(skillsDir, skillName, applied.newText);- }- checkpoint!.best_sel_score = gate.selScore;- checkpoint!.best_skill_text = applied.newText;- tally.accepted_steps += 1;-- logEvent({- kind: 'step',- run_id: runId,- skill: skillName,- epoch,- step,- sel_score_median: gate.selScore,- sel_score_runs: gate.perTaskMedians.map((t) => t.median),- accepted: true,- edits_attempted: fresh.length,- edits_applied: applied.results.filter((r) => r.outcome === 'applied').length,- delta,- cumulative_cost_usd: tracker.snapshot().cumulativeCostUsd,- ...(reflectResult.invalidEditsDropped > 0 ? { invalid_edits_dropped: reflectResult.invalidEditsDropped } : {}),- } as never);- finalText = applied.newText;+ // for the --no-mutate / bundled-without-allow paths; with a+ // held-out set the write waits for the final held-out check (D2).+ promote(applied.newText, gate.selScore, epoch, step, appliedEdits);} else {- // REJECT: push to rejected-buffer.- const newRejections = fresh.map((e) =>- makeRejectedEntry(checkpoint!.best_skill_text, [e], `validation_gate_${gate.reason ?? 'rejected'}`),- );- saveRejectedBuffer(skillsDir, skillName, newRejections);- logEvent({- kind: 'step',- run_id: runId,- skill: skillName,- epoch,- step,- sel_score_median: gate.selScore,- sel_score_runs: gate.perTaskMedians.map((t) => t.median),- accepted: false,- edits_attempted: fresh.length,- edits_applied: applied.results.filter((r) => r.outcome === 'applied').length,- delta: gate.selScore - checkpoint!.best_sel_score,- reason: gate.reason ?? 'rejected',- cumulative_cost_usd: tracker.snapshot().cumulativeCostUsd,- ...(reflectResult.invalidEditsDropped > 0 ? { invalid_edits_dropped: reflectResult.invalidEditsDropped } : {}),- } as never);+ // REJECT: file the applied edits (and apply failures) in the rejected buffer.+ saveRejectedBuffer(skillsDir, skillName, [+ ...appliedEdits.map((e) => makeRejectedEntry(checkpoint!.best_skill_text, [e], `validation_gate_${gate.reason ?? 'rejected'}`)),+ ...applyFailed,+ ]);}+ logEvent({+ kind: 'step',+ run_id: runId,+ skill: skillName,+ epoch,+ step,+ sel_score_median: gate.selScore,+ sel_score_runs: gate.perTaskMedians.map((t) => t.median),+ accepted: stepAccepted,+ edits_attempted: fresh.length,+ edits_applied: appliedEdits.length,+ delta,+ ...(stepAccepted ? {} : { reason: gate.reason ?? 'rejected' }),+ cumulative_cost_usd: tracker.snapshot().cumulativeCostUsd,+ ...(reflectResult.invalidEditsDropped > 0 ? { invalid_edits_dropped: reflectResult.invalidEditsDropped } : {}),+ } as never);}// Step completed (accepted, rejected or no candidate): advance the@@ -963,30 +893,45 @@ async function runOptimizationLoop(}- // FINAL TEST: score the best skill AND the baseline on D_test so the- // receipt carries an honest held-out generalization signal (the eval- // harnesses also compute this themselves, but a truthful receipt removes- // the prior hardcoded-0 lie). Skipped when D_test is empty, and after an- // early stop with nothing accepted (best === baseline; budget not spent).- if (split.test.length > 0 && !(earlyStopped && tally.accepted_steps === 0)) {- testScore = await scoreSkillOnTasks({+ // D2 HELD-OUT: one independent check of the final champion against the+ // baseline, before anything is written. The proposer never sees its+ // verdict, so the held-out set never becomes a second selection set.+ // A refused champion is dropped: the skill stays at the baseline.+ const champion = checkpoint!.champion;+ if (champion) {+ const ho = await runHeldOutGate({engine: opts.engine,operationContext: opts.operationContext,- skillText: checkpoint!.best_skill_text,- tasks: split.test,- targetModel: opts.targetModel,- judgeModel: opts.judgeModel,+ candidateSkillText: checkpoint!.best_skill_text,+ baselineSkillText: baselineText,+ heldOutTasks,deadlineMs: deadline,- });- baselineTestScore = await scoreSkillOnTasks({- engine: opts.engine,- operationContext: opts.operationContext,- skillText: baselineText,- tasks: split.test,targetModel: opts.targetModel,judgeModel: opts.judgeModel,- deadlineMs: deadline,+ ...(hook ? { rolloutHook: hook } : {}),});+ rollouts.held_out += heldOutTasks.length * VALIDATION_RUNS_PER_TASK * 2;+ heldOut = { baseline_score: ho.baselineScore, candidate_score: ho.candidateScore, passed: ho.passed };+ logEvent({ kind: 'held_out', run_id: runId, skill: skillName, ...heldOut } as never);+ if (ho.passed) {+ writeCandidate(checkpoint!.best_skill_text, checkpoint!.best_sel_score, champion.epoch, champion.step, champion.edits);+ tally.accepted_steps += champion.steps;+ } else {+ process.stderr.write(`[skillopt] held-out check refused the champion (${ho.candidateScore.toFixed(3)} < baseline ${ho.baselineScore.toFixed(3)}); skill unchanged\n`);+ checkpoint!.best_skill_text = baselineText;+ checkpoint!.best_sel_score = baselineSelScore;+ }+ delete checkpoint!.champion;+ saveCheckpoint(skillsDir, skillName, checkpoint!);+ }++ // FINAL TEST: score the best skill AND the baseline on D_test so the+ // receipt carries an honest generalization signal. Skipped when D_test+ // is empty, and after an early stop with nothing accepted (best ===+ // baseline; budget not spent).+ if (split.test.length > 0 && !(earlyStopped && tally.accepted_steps === 0)) {+ testScore = (await score('test', checkpoint!.best_skill_text, split.test)).selScore;+ baselineTestScore = (await score('test', baselineText, split.test)).selScore;}});} catch (err) {@@ -1084,7 +1029,12 @@ async function runOptimizationLoop(// Ablation provenance (cat31 replayability) — only when a non-default knob set....(opts.reflectMode ? { reflect_mode: opts.reflectMode } : {}),...(opts.disableValidationGate ? { validation_gate_disabled: true } : {}),- ...(opts.optimizerMode ? { optimizer_mode: opts.optimizerMode } : {}),+ optimizer_mode: opts.optimizerMode,+ rollouts,+ candidates_scored_on_sel: selCandidates.size,+ optimizer_calls: tally.reflect_calls,+ cost_scope: hook ? 'optimizer_only' : 'all',+ ...(heldOut ? { held_out: heldOut } : {}),};logEvent({diff --git a/src/core/skillopt/preflight.ts b/src/core/skillopt/preflight.tsindex 4198fee4ae..aedca38083 100644--- a/src/core/skillopt/preflight.ts+++ b/src/core/skillopt/preflight.ts@@ -43,7 +43,7 @@ import { peakRateNote, priceFor, reservationCostUsd, type PricingOverrides } froimport { noPricingGuidance, noPricingMessage, type NoPricingGuidance } from '../budget/no-pricing.ts';import { skilloptOutputCap } from './output-cap.ts';import { JUDGE_SITE_MAX_TOKENS } from './score.ts';-import { VALIDATION_RUNS_PER_TASK } from './types.ts';+import { VALIDATION_RUNS_PER_TASK, type OptimizerMode } from './types.ts';/** Conservative per-rollout token estimates (input + output). */const ROLLOUT_INPUT_TOKENS = 3000; // skill + task prompt + tool defs@@ -67,7 +67,7 @@ export interface PreflightOpts {maxCostSource?: 'user' | 'default';/*** Held-out task count (F11). When > 0, the held-out gate scores baseline +- * candidate on the held-out set at every accepted step. Conservatively+ * candidate on the held-out set once, on the final champion (D2). Priced* priced as if EVERY step accepts (upper bound) so the cap is honest.* 0 / omitted = no held-out gate.*/@@ -84,6 +84,10 @@ export interface PreflightOpts {judgeModels?: string[];/** Operator `pricing.overrides`, the same map the run's BudgetTracker uses. */pricingOverrides?: PricingOverrides;+ /** E1: one-shot is one step with one optimizer call; default (absent) prices the loop. */+ optimizerMode?: OptimizerMode;+ /** B1.3: a rollout hook runs (and pays for) every rollout, so only optimizer calls are priced here. */+ rolloutsExternal?: boolean;}export type ReservationRole = 'optimizer' | 'target' | 'judge';@@ -106,8 +110,10 @@ const RESERVATION_CONTROL: Record<ReservationRole, string> = {export function singleCallReservations(opts: PreflightOpts): CallReservation[] {const reserve = (role: ReservationRole, model: string, maxOut: number): CallReservation =>({ role, model, max_output_tokens: maxOut, reservation_usd: reservationCostUsd(model, 'chat', 0, maxOut, opts.pricingOverrides) });+ const optimizer = reserve('optimizer', opts.optimizerModel, opts.reflectMaxTokens ?? defaultMaxOutputTokens(opts.optimizerModel));+ if (opts.rolloutsExternal) return [optimizer];return [- reserve('optimizer', opts.optimizerModel, opts.reflectMaxTokens ?? defaultMaxOutputTokens(opts.optimizerModel)),+ optimizer,reserve('target', opts.targetModel, defaultMaxOutputTokens(opts.targetModel)),...(opts.judgeModels ?? [opts.judgeModel]).map((m) => reserve('judge', m, skilloptOutputCap(m, JUDGE_SITE_MAX_TOKENS))),];@@ -154,39 +160,37 @@ export interface PreflightResult {}export function estimateCost(opts: PreflightOpts): PreflightEstimate {- const stepsPerEpoch = Math.max(1, Math.floor(opts.trainSize / opts.batchSize));- const totalSteps = opts.epochs * stepsPerEpoch;+ const oneShot = opts.optimizerMode === 'one-shot-rewrite';+ const stepsPerEpoch = oneShot ? 1 : Math.max(1, Math.floor(opts.trainSize / opts.batchSize));+ const totalSteps = oneShot ? 1 : opts.epochs * stepsPerEpoch;// Per-step counts.- const rolloutsPerStep = opts.batchSize;- const reflectsPerStep = 2; // D7: two reflect calls+ const rolloutsPerStep = oneShot ? Math.min(opts.batchSize, opts.trainSize) : opts.batchSize;+ const reflectsPerStep = oneShot ? 1 : 2; // D7: two reflect calls; one-shot: one rewriteconst sel_runs_per_step = opts.selSize * VALIDATION_RUNS_PER_TASK;- // F11 held-out: baseline + candidate scored on the held-out set at every- // accepted step. Upper-bound: assume every step accepts (2 = baseline+candidate).+ // D2 held-out: baseline + candidate scored once, on the final champion.const heldOutSize = opts.heldOutSize ?? 0;- const heldOutRollouts = heldOutSize > 0- ? totalSteps * heldOutSize * VALIDATION_RUNS_PER_TASK * 2- : 0;+ const heldOutRollouts = heldOutSize > 0 ? heldOutSize * VALIDATION_RUNS_PER_TASK * 2 : 0;// Cumulative counts across the whole run.const rollout_calls = totalSteps * rolloutsPerStep+ opts.selSize * VALIDATION_RUNS_PER_TASK // baseline sel eval+ opts.selSize * VALIDATION_RUNS_PER_TASK * totalSteps // per-step sel validation+ opts.testSize * 2 // final test eval (best + baseline)- + heldOutRollouts; // F11 held-out gate (baseline+candidate per accepted step)+ + heldOutRollouts; // D2 held-out check (baseline+candidate, once)const reflect_calls = totalSteps * reflectsPerStep- + opts.epochs; // slow-update meta calls+ + (oneShot ? 0 : opts.epochs); // slow-update meta callsconst judgeModels = opts.judgeModels ?? [opts.judgeModel];const judgeCallsRaw = opts.selSize // baseline (1 per task; median-of-3 is in the rollout count already? — no, judge runs per rollout)+ opts.selSize * VALIDATION_RUNS_PER_TASK * totalSteps // per-step validation+ opts.testSize * 2 // final test judges (best + baseline)+ heldOutRollouts; // F11 held-out judges (1 per held-out rollout)- const judge_calls = judgeModels.length > 0 ? judgeCallsRaw : 0;+ const judge_calls = judgeModels.length > 0 && !opts.rolloutsExternal ? judgeCallsRaw : 0;// Cost per call type. Judge calls price at the priciest judge the run can reach.const lookup = (model: string) => priceFor(model, 'chat', opts.pricingOverrides)?.pricing ?? null;- const unpriced = [...new Set([opts.targetModel, opts.optimizerModel, ...(judge_calls > 0 ? judgeModels : [])])]+ const unpriced = [...new Set([...(opts.rolloutsExternal ? [] : [opts.targetModel]), opts.optimizerModel, ...(judge_calls > 0 ? judgeModels : [])])].filter((m) => lookup(m) === null);const zero = { input: 0, output: 0 };const targetPrice = lookup(opts.targetModel) ?? zero;@@ -197,7 +201,7 @@ export function estimateCost(opts: PreflightOpts): PreflightEstimate {);const unpricedJudge = judge_calls > 0 && judgeModels.some((m) => unpriced.includes(m));- const rolloutCost = rollout_calls * (+ const rolloutCost = opts.rolloutsExternal ? 0 : rollout_calls * ((ROLLOUT_INPUT_TOKENS * targetPrice.input) / 1_000_000+ (ROLLOUT_OUTPUT_TOKENS * targetPrice.output) / 1_000_000);@@ -246,7 +250,9 @@ export function estimateCost(opts: PreflightOpts): PreflightEstimate {*/export function formatPreflightReport(est: PreflightEstimate, opts: PreflightOpts): string {return [- `[skillopt] Cost estimate for ${opts.epochs} epochs × ${est.steps_per_epoch} steps × ${opts.batchSize} rollouts:`,+ opts.optimizerMode === 'one-shot-rewrite'+ ? `[skillopt] Cost estimate for one rewrite (one-shot; --loop runs the iterative optimizer):`+ : `[skillopt] Cost estimate for ${opts.epochs} epochs × ${est.steps_per_epoch} steps × ${opts.batchSize} rollouts:`,` Rollouts: ${est.rollout_calls.toLocaleString()} calls`,` Reflects: ${est.reflect_calls.toLocaleString()} calls`,` Judges: ${est.judge_calls.toLocaleString()} calls`,diff --git a/src/core/skillopt/reflect.ts b/src/core/skillopt/reflect.tsindex e807c52db1..8bb6c799a3 100644--- a/src/core/skillopt/reflect.ts+++ b/src/core/skillopt/reflect.ts@@ -64,19 +64,40 @@ export function describeJudge(judge: Judge): string {}}+const TASK_SPECIFIC_CHECK: Record<RuleCheck['op'], string> = {+ contains: 'the output must contain a task-specific required text',+ regex: 'the output must match a task-specific pattern',+ section_present: 'the output must include a task-specific markdown heading',+ max_chars: 'the output must stay under a task-specific length limit',+ min_citations: 'the output must include a task-specific number of citations',+ tool_called: 'the agent must call a task-specific tool',+ tool_not_called: 'the agent must avoid a task-specific tool',+};+/**- * Describe the DISTINCT success criteria across a set of benchmark tasks. Most- * benchmarks use one judge shape for every task, so this collapses to a single- * block; heterogeneous benchmarks list each distinct shape once.+ * Describe the success criteria the optimizer may see for a set of tasks (the+ * train split). A rule check or LLM rubric is written out verbatim only when+ * every task (at least two) carries it, so it is a format rule; one that varies+ * by task is an answer key and is described without its argument.*/export function describeJudges(tasks: ReadonlyArray<{ judge: Judge }>): string {- const seen = new Set<string>();- const blocks: string[] = [];- for (const t of tasks) {- const desc = describeJudge(t.judge);- if (!seen.has(desc)) { seen.add(desc); blocks.push(desc); }+ const lines: string[] = [];+ const add = (line: string) => { if (!lines.includes(line)) lines.push(line); };+ const everyTask = (has: (j: Judge) => boolean) => tasks.length >= 2 && tasks.every((t) => has(t.judge));+ for (const { judge } of tasks) {+ if (judge.kind === 'rule') {+ for (const c of judge.checks) {+ const shared = everyTask((j) => j.kind === 'rule' && j.checks.some((x) => x.op === c.op && x.arg === c.arg));+ add(`- ${shared ? describeCheck(c) : TASK_SPECIFIC_CHECK[c.op]}`);+ }+ } else if (judge.kind === 'llm') {+ const shared = everyTask((j) => j.kind === 'llm' && j.rubric === judge.rubric);+ add(shared ? describeJudge(judge) : '- the output is graded 0..1 by an LLM judge against a task-specific rubric');+ } else {+ add(describeJudge(judge));+ }}- return blocks.join('\n');+ return lines.join('\n');}/** The output format both reflect modes send (#7257): parseEdits accepts exactly these op and field names. */diff --git a/src/core/skillopt/rejected-buffer.ts b/src/core/skillopt/rejected-buffer.tsindex 250b149df8..b305e9ec5a 100644--- a/src/core/skillopt/rejected-buffer.ts+++ b/src/core/skillopt/rejected-buffer.ts@@ -23,6 +23,9 @@ import * as path from 'node:path';import { atomicWrite } from './apply-edits.ts';import type { EditOp } from './types.ts';+/** Legacy reason of a held-out refusal (no longer written; filtered on load). */+const HELD_OUT_REGRESSION = 'held_out_regression';+/** Bounded LRU cap. Older entries garbage-collect when this many accumulate. */export const REJECTED_BUFFER_CAP = 100;@@ -87,8 +90,9 @@ export function loadRejectedBuffer(skillsDir: string, skillName: string): Rejectif (obj.schema !== 1 || !Array.isArray(obj.entries)) return [];// Type-narrow without full validation — entries from our own writes// are trusted; corrupted entries simply replay as rejection attempts- // when their key fails to match.- return obj.entries as RejectedEntry[];+ // when their key fails to match. Held-out refusals written before D2 are+ // dropped: the proposer must never learn the held-out verdict.+ return (obj.entries as RejectedEntry[]).filter((e) => e.reason !== HELD_OUT_REGRESSION);} catch (err) {const msg = err instanceof Error ? err.message : String(err);process.stderr.write(`[skillopt] rejected.json unreadable for ${skillName} (${msg}); starting fresh\n`);diff --git a/src/core/skillopt/rollout.ts b/src/core/skillopt/rollout.tsindex 1668d45ec0..26d7b8d17e 100644--- a/src/core/skillopt/rollout.ts+++ b/src/core/skillopt/rollout.ts@@ -25,17 +25,33 @@ import type { BrainEngine } from '../engine.ts';import { SKILLOPT_PURPOSE, type BenchmarkTask, type Trajectory } from './types.ts';import { sharedSkillToolAccess } from '../shared-skills/tool-access.ts';import { authorizeSkillRead } from '../shared-skills/policy.ts';+import { parseSkillFrontmatter } from '../skill-frontmatter.ts';++/** An op a rollout may call: it exists, is not mutating and is not local-only. */+function isRolloutReadOp(name: string): boolean {+ const op = operations.find((o) => o.name === name);+ return op !== undefined && !op.mutating && !op.localOnly;+}/**- * D13: which tools SkillOpt rollouts are allowed to call.- *- * Derived from BRAIN_TOOL_ALLOWLIST minus `put_page` (only mutating op in- * the base set). New mutating ops MUST be added here AND BRAIN_TOOL_ALLOWLIST- * mustn't be silently widened; the rollout test pins zero-write invariant.+ * D13: which tools SkillOpt rollouts are allowed to call: BRAIN_TOOL_ALLOWLIST+ * minus every mutating or local-only op (put_page, add_timeline_entry). The+ * rollout test pins the zero-write invariant.+ */+export const READ_ONLY_BRAIN_TOOLS: ReadonlySet<string> = new Set([...BRAIN_TOOL_ALLOWLIST].filter(isRolloutReadOp));++/**+ * B1.5: read ops a skill may add to its rollouts by naming them in its+ * frontmatter `tools:`. Each was reviewed for side effects under remote=true;+ * add one only after the same review.*/-export const READ_ONLY_BRAIN_TOOLS: ReadonlySet<string> = new Set(- [...BRAIN_TOOL_ALLOWLIST].filter((name) => name !== 'put_page'),-);+const SKILL_DECLARABLE_TOOLS: ReadonlySet<string> = new Set(['recall', 'get_timeline']);++/** The tools a rollout of this skill offers: the read-only base plus the reviewed read ops its frontmatter declares. */+export function rolloutToolsForSkill(skillText: string): ReadonlySet<string> {+ const declared = (parseSkillFrontmatter(skillText)?.tools ?? []).filter((n) => SKILL_DECLARABLE_TOOLS.has(n) && isRolloutReadOp(n));+ return new Set([...READ_ONLY_BRAIN_TOOLS, ...declared]);+}export interface RolloutOpts {operationContext?: OperationContext;@@ -50,6 +66,8 @@ export interface RolloutOpts {abortSignal?: AbortSignal;/** F10: when true, enable write-capture (virtual put_page/submit_job/file_upload). */writeCapture?: boolean;+ /** B1.5: tool names to offer instead of rolloutToolsForSkill(skillText); mutating and local-only ops are still dropped. */+ allowedTools?: ReadonlySet<string>;/** Test seam — substitute the toolLoop call. */toolLoopFn?: typeof toolLoop;/** Test seam — substitute chat (currently unused; toolLoop wraps chat). */@@ -81,7 +99,7 @@ export async function runRollout(opts: RolloutOpts): Promise<Trajectory> {defs = registry.defs;handlers = registry.handlers;} else {- const r = buildReadOnlyToolRegistry(ctx, original ? new Set(await sharedSkillToolAccess(original)) : undefined, original);+ const r = buildReadOnlyToolRegistry(ctx, opts.allowedTools ?? rolloutToolsForSkill(skillText), original ? new Set(await sharedSkillToolAccess(original)) : undefined, original);defs = r.defs;handlers = r.handlers;}@@ -190,11 +208,11 @@ interface ToolRegistry {* Tool names are prefixed `brain_` for Anthropic-name compliance (same* convention as the subagent handler's buildBrainTools).*/-function buildReadOnlyToolRegistry(ctx: OperationContext, allowed?: ReadonlySet<string>, original?: OperationContext): ToolRegistry {+function buildReadOnlyToolRegistry(ctx: OperationContext, offered: ReadonlySet<string>, allowed?: ReadonlySet<string>, original?: OperationContext): ToolRegistry {const defs: ChatToolDef[] = [];const handlers = new Map<string, ToolHandler>();for (const op of operations) {- if (!READ_ONLY_BRAIN_TOOLS.has(op.name)) continue;+ if (!offered.has(op.name)) continue;if (op.mutating || op.localOnly) continue;if (allowed && !allowed.has(op.name)) continue;const toolName = `brain_${op.name}`;diff --git a/src/core/skillopt/run-outcome.ts b/src/core/skillopt/run-outcome.tsindex bc4119e303..03eb8f7456 100644--- a/src/core/skillopt/run-outcome.ts+++ b/src/core/skillopt/run-outcome.ts@@ -133,6 +133,7 @@ export function buildResumeCommand(if (spec.no_mutate) args.push('--no-mutate');if (spec.allow_mutate_bundled) args.push('--allow-mutate-bundled');if (spec.bootstrap_reviewed) args.push('--bootstrap-reviewed');+ if ((spec.optimizer_mode ?? 'reflect') === 'reflect') args.push('--loop');if (run) {if (run.mode === 'rewrite') args.push('--rewrite');if (run.maxCostUsd > 0) args.push('--max-cost-usd', run.maxCostUsd);diff --git a/src/core/skillopt/types.ts b/src/core/skillopt/types.tsindex 4de3f72f30..305f13d11e 100644--- a/src/core/skillopt/types.ts+++ b/src/core/skillopt/types.ts@@ -137,8 +137,14 @@ export interface SkillOptOpts {};/** Kebab-case skill name. Resolves to `skills/<name>/SKILL.md`. */skillName: string;- /** Absolute path to the benchmark JSONL file. */- benchmarkPath: string;+ /** Absolute path to the benchmark JSONL file. Exactly one of this and `tasks` is set. */+ benchmarkPath?: string;+ /**+ * B1.2 (library only, never on the CLI, MCP or job data): explicit task+ * lists used exactly as given. sel keeps the D_sel floor; an absent or empty+ * test skips final test scoring.+ */+ tasks?: { train: BenchmarkTask[]; sel: BenchmarkTask[]; test?: BenchmarkTask[] };/** Brain engine (required for D14 lock + read-only tool calls during rollouts). */engine: BrainEngine;/** Skills directory root (used for bundled-skill detection, install-path gate). */@@ -150,8 +156,8 @@ export interface SkillOptOpts {/** Max edits per step (the LR). */lr: number;lrSchedule: 'cosine' | 'linear' | 'constant';- /** Split ratio as 3-tuple, e.g. [4, 1, 5] for 4:1:5. */- split: [number, number, number];+ /** Split ratio as 3-tuple, e.g. [4, 1, 5] for 4:1:5. Default [4, 1, 5]; unused with `tasks`. */+ split?: [number, number, number];// Models.optimizerModel: string;@@ -197,8 +203,17 @@ export interface SkillOptOpts {reflectMode?: 'both' | 'failure-only';/** Ablation: greedy-accept every applied edit, skip the D12 median+epsilon gate. Default false. */disableValidationGate?: boolean;- /** Ablation: 'one-shot-rewrite' does ONE rewrite call, no loop, no gate. Default 'reflect'. */- optimizerMode?: 'reflect' | 'one-shot-rewrite';+ /**+ * E1: 'one-shot-rewrite' (the product default: one rewrite from a train+ * batch, gated on selection like a loop step) or 'reflect' (the iterative+ * edit loop, opt-in). Required so no programmatic caller changes behavior+ * silently.+ */+ optimizerMode: OptimizerMode;+ /** B1.4 (library only): replaces the generated success criteria in every optimizer prompt, verbatim. */+ criteriaText?: string;+ /** B1.3 (library only): every rollout the run makes goes through this hook; gbrain's budget then meters optimizer calls only. */+ rolloutHook?: RolloutBatchHook;// Safety./** USD cap for the run. #3516: 0 means UNCAPPED (--no-max-cost) — pricing@@ -215,6 +230,8 @@ export interface SkillOptOpts {resumeRunId?: string;}+export type OptimizerMode = 'reflect' | 'one-shot-rewrite';+export interface StepRecord {epoch: number;step: number;@@ -287,7 +304,18 @@ export interface RunReceipt {// ablation knob was set.reflect_mode?: 'both' | 'failure-only';validation_gate_disabled?: boolean;- optimizer_mode?: 'reflect' | 'one-shot-rewrite';+ /** E1: always present. */+ optimizer_mode?: OptimizerMode;+ /** B1.6: single rollouts this segment ran, by scoring site. */+ rollouts?: Record<RolloutPhase, number>;+ /** B1.6: distinct candidate skill texts (sha256, baseline excluded) that reached the selection gate. */+ candidates_scored_on_sel?: number;+ /** B1.6: optimizer (reflect + one-shot) calls across the run. */+ optimizer_calls?: number;+ /** B1.6: 'optimizer_only' when a rollout hook ran the rollouts, so final_cost_usd excludes them. */+ cost_scope?: 'all' | 'optimizer_only';+ /** D2: the single held-out check on the final champion, when one ran. */+ held_out?: { baseline_score: number; candidate_score: number; passed: boolean };}export interface HistoryRow {@@ -321,6 +349,34 @@ export const SKILLOPT_PURPOSE = {judge: 'skillopt.judge',} as const;+// ─── Rollout hook (B1.3) ──────────────────────────────────────────────────++/** Which scoring site a rollout batch belongs to. */+export type RolloutPhase = 'baseline_sel' | 'train_forward' | 'sel_gate' | 'held_out' | 'test';++export interface RolloutBatchRequest {+ phase: RolloutPhase;+ /** The full SKILL.md text the target runs with. */+ skillText: string;+ skillSha256: string;+ /** In the order gbrain would run them. */+ tasks: BenchmarkTask[];+ /** 1 for train_forward, VALIDATION_RUNS_PER_TASK otherwise. */+ runsPerTask: number;+ /** 0 for baseline and test. */+ epoch: number;+ step: number;+}++/**+ * Runs and scores one whole batch for a harness (shared rollout budget,+ * metering). Returns one row per task, each with runsPerTask ScoredRollouts,+ * in request order. Only train_forward trajectories reach an optimizer prompt;+ * other phases read score and judge_error only. A throw ends the run; tag it+ * with SKILLOPT_HOOK_BUDGET_EXHAUSTED or SKILLOPT_HOOK_INFRA_ERROR.+ */+export type RolloutBatchHook = (req: RolloutBatchRequest, signal?: AbortSignal) => Promise<ScoredRollout[][]>;+export interface GateInput {candidateSkillText: string;selSet: BenchmarkTask[];@@ -331,7 +387,7 @@ export interface GateInput {export interface GateResult {accepted: boolean;/** Per-task median across N runs. */- perTaskMedians: Array<{ task_id: string; median: number; runs: number[] }>;+ perTaskMedians: Array<{ task_id: string; median: number; runs: number[]; error?: string }>;/** Mean of per-task medians. */selScore: number;reason?: 'no_margin' | 'below_baseline' | 'all_judge_errors';diff --git a/src/core/skillopt/validate-gate.ts b/src/core/skillopt/validate-gate.tsindex af61250432..1484b23c62 100644--- a/src/core/skillopt/validate-gate.ts+++ b/src/core/skillopt/validate-gate.ts@@ -20,11 +20,12 @@* which is cached (D11), so the effective cost ~1.3x not 3x.*/-import { runWithLimit } from '../worker-pool.ts';-import { isSkilloptMustAbort, SKILLOPT_RUNTIME_EXCEEDED } from './must-abort.ts';+import { runWithLimit, type SettledItem } from '../worker-pool.ts';+import { createHash } from 'node:crypto';+import { isSkilloptMustAbort, SKILLOPT_HOOK_INFRA_ERROR, SKILLOPT_RUNTIME_EXCEEDED } from './must-abort.ts';import { runRollout, type RolloutOpts } from './rollout.ts';import { scoreTrajectory } from './score.ts';-import type { BenchmarkTask, GateInput, GateResult, ScoredRollout } from './types.ts';+import type { BenchmarkTask, GateInput, GateResult, RolloutBatchHook, RolloutPhase, ScoredRollout } from './types.ts';import { VALIDATION_EPSILON, VALIDATION_RUNS_PER_TASK } from './types.ts';import type { BrainEngine } from '../engine.ts';import type { OperationContext } from '../ops/contract.ts';@@ -57,9 +58,43 @@ export interface ValidateGateOpts extends Omit<GateInput, 'selSet'> {rolloutFn?: typeof runRollout;/** Test seam — substitute scoreTrajectory. */scoreFn?: typeof scoreTrajectory;+ /** B1.3: when set, the whole batch goes to the hook in one call; rolloutFn/scoreFn are unused. */+ rolloutHook?: RolloutBatchHook;+ /** Hook request labels. Default phase 'sel_gate', epoch/step 0. */+ phase?: RolloutPhase;+ epoch?: number;+ step?: number;}export { SKILLOPT_RUNTIME_EXCEEDED };++interface PerTask {+ task_id: string;+ median: number;+ runs: number[];+ rollouts: ScoredRollout[];+}++/** One hook call for the whole batch; a malformed reply is an infra error, never a score. */+async function hookBatch(opts: ValidateGateOpts, hook: RolloutBatchHook, runsPerTask: number): Promise<PerTask[]> {+ if (opts.deadlineMs !== undefined && Date.now() > opts.deadlineMs) throw new Error(SKILLOPT_RUNTIME_EXCEEDED);+ const rows = await hook({+ phase: opts.phase ?? 'sel_gate',+ skillText: opts.candidateSkillText,+ skillSha256: createHash('sha256').update(opts.candidateSkillText).digest('hex'),+ tasks: opts.selSet,+ runsPerTask,+ epoch: opts.epoch ?? 0,+ step: opts.step ?? 0,+ }, opts.abortSignal);+ if (!Array.isArray(rows) || rows.length !== opts.selSet.length || rows.some((r) => !Array.isArray(r) || r.length !== runsPerTask)) {+ throw new Error(`${SKILLOPT_HOOK_INFRA_ERROR}: rollout hook returned a malformed batch for ${opts.phase ?? 'sel_gate'} (want ${opts.selSet.length} rows of ${runsPerTask})`);+ }+ return opts.selSet.map((task, i) => {+ const runs = rows[i]!.map((r) => r.score);+ return { task_id: task.task_id, median: median(runs), runs, rollouts: rows[i]! };+ });+}function isRuntimeExceeded(e: unknown): boolean {return e instanceof Error && e.message === SKILLOPT_RUNTIME_EXCEEDED;}@@ -85,44 +120,39 @@ export async function runValidationGate(opts: ValidateGateOpts): Promise<GateRes// at 1 (a runsPerTask of 0 would silently produce an empty median).const runsPerTask = Math.max(1, opts.runsPerTask ?? VALIDATION_RUNS_PER_TASK);- interface PerTask {- task_id: string;- median: number;- runs: number[];- rollouts: ScoredRollout[];- }-// Per sel-task, run N rollouts + N judges, take median.- const settled = await runWithLimit({- items: opts.selSet,- limit: concurrency,- fn: async (task: BenchmarkTask): Promise<PerTask> => {- const runs: number[] = [];- const rollouts: ScoredRollout[] = [];- for (let i = 0; i < runsPerTask; i++) {- // #4119: in-rollout-loop deadline — see ValidateGateOpts.deadlineMs.- if (opts.deadlineMs !== undefined && Date.now() > opts.deadlineMs) {- throw new Error(SKILLOPT_RUNTIME_EXCEEDED);+ const settled: SettledItem<PerTask>[] = opts.rolloutHook+ ? (await hookBatch(opts, opts.rolloutHook, runsPerTask)).map((value, idx) => ({ ok: true, value, idx }))+ : await runWithLimit({+ items: opts.selSet,+ limit: concurrency,+ fn: async (task: BenchmarkTask): Promise<PerTask> => {+ const runs: number[] = [];+ const rollouts: ScoredRollout[] = [];+ for (let i = 0; i < runsPerTask; i++) {+ // #4119: in-rollout-loop deadline — see ValidateGateOpts.deadlineMs.+ if (opts.deadlineMs !== undefined && Date.now() > opts.deadlineMs) {+ throw new Error(SKILLOPT_RUNTIME_EXCEEDED);+ }+ const rolloutOpts: RolloutOpts = {+ operationContext: opts.operationContext,+ engine: opts.engine,+ skillText: opts.candidateSkillText,+ task,+ targetModel: opts.targetModel,+ abortSignal: opts.abortSignal,+ };+ const trajectory = await rolloutFn(rolloutOpts);+ const scored: ScoredRollout = await scoreFn(trajectory, task.judge, {+ judgeModel: opts.judgeModel,+ });+ runs.push(scored.score);+ rollouts.push(scored);}- const rolloutOpts: RolloutOpts = {- operationContext: opts.operationContext,- engine: opts.engine,- skillText: opts.candidateSkillText,- task,- targetModel: opts.targetModel,- abortSignal: opts.abortSignal,- };- const trajectory = await rolloutFn(rolloutOpts);- const scored: ScoredRollout = await scoreFn(trajectory, task.judge, {- judgeModel: opts.judgeModel,- });- runs.push(scored.score);- rollouts.push(scored);- }- return { task_id: task.task_id, median: median(runs), runs, rollouts };- },- signal: opts.abortSignal,- });+ return { task_id: task.task_id, median: median(runs), runs, rollouts };+ },+ signal: opts.abortSignal,+ });// MUST-ABORT errors (budget exhaustion / no-pricing / spend-policy refusals) are NOT scoring noise —// swallowing them as score=0 turns a pricing/cap crash into a fake "0/N" run@@ -156,7 +186,7 @@ export async function runValidationGate(opts: ValidateGateOpts): Promise<GateRes// trajectories rather than fabricated zero-score entries).const perTaskMedians = settled.map((s, idx) => {if (s && s.ok) return { task_id: s.value.task_id, median: s.value.median, runs: s.value.runs };- return { task_id: opts.selSet[idx]!.task_id, median: 0, runs: [] };+ return { task_id: opts.selSet[idx]!.task_id, median: 0, runs: [], error: errMsg(s.error) };});const scoredRollouts: ScoredRollout[] = settled.flatMap((s) => (s && s.ok) ? s.value.rollouts : []);// #4741: same rule for the judge side. scoreLlm fails open per rollout@@ -199,17 +229,20 @@ export interface ScoreOnTasksOpts {abortSignal?: AbortSignal;rolloutFn?: typeof runRollout;scoreFn?: typeof scoreTrajectory;+ /** B1.3: score through a harness hook (one call for the batch). */+ rolloutHook?: RolloutBatchHook;+ /** Hook request label. Default 'test'. */+ phase?: RolloutPhase;}-/**- * Score a skill on an arbitrary task set and return the mean-of-per-task-medians- * (`selScore`). The single primitive for "how good is this skill on these tasks?"- * — used by the orchestrator's baseline + final-test eval, the held-out gate, and- * the Track B eval harnesses, so they can't drift on scoring semantics. Thin- * wrapper over `runValidationGate({ bestScore: -1 })` (any score "accepts"; we- * only read `.selScore`).- */-export async function scoreSkillOnTasks(opts: ScoreOnTasksOpts): Promise<number> {+/** Per-task medians plus their mean; a task whose rollout errored has median 0, no runs and its error. */+export interface PerTaskScore {+ score: number;+ perTask: GateResult['perTaskMedians'];+}++/** scoreSkillOnTasks with the per-task rows (paired comparisons across arms need them). */+export async function scoreSkillPerTask(opts: ScoreOnTasksOpts): Promise<PerTaskScore> {const gate = await runValidationGate({operationContext: opts.operationContext,engine: opts.engine,@@ -217,14 +250,27 @@ export async function scoreSkillOnTasks(opts: ScoreOnTasksOpts): Promise<number>selSet: opts.tasks,bestScore: -1,targetModel: opts.targetModel,+ phase: opts.phase ?? 'test',...(opts.judgeModel !== undefined ? { judgeModel: opts.judgeModel } : {}),...(opts.runsPerTask !== undefined ? { runsPerTask: opts.runsPerTask } : {}),...(opts.deadlineMs !== undefined ? { deadlineMs: opts.deadlineMs } : {}),...(opts.abortSignal ? { abortSignal: opts.abortSignal } : {}),...(opts.rolloutFn ? { rolloutFn: opts.rolloutFn } : {}),...(opts.scoreFn ? { scoreFn: opts.scoreFn } : {}),+ ...(opts.rolloutHook ? { rolloutHook: opts.rolloutHook } : {}),});- return gate.selScore;+ return { score: gate.selScore, perTask: gate.perTaskMedians };+}++/**+ * Score a skill on an arbitrary task set and return the mean-of-per-task-medians+ * (`selScore`). The single primitive for "how good is this skill on these tasks?"+ * — used by the orchestrator's baseline + final-test eval, the held-out gate, and+ * the Track B eval harnesses, so they can't drift on scoring semantics. Thin+ * wrapper over `scoreSkillPerTask` (any score "accepts"; we only read the mean).+ */+export async function scoreSkillOnTasks(opts: ScoreOnTasksOpts): Promise<number> {+ return (await scoreSkillPerTask(opts)).score;}/** Pure median for an array of numbers. Returns 0 for empty array. */diff --git a/test/e2e/skillopt-loop.serial.test.ts b/test/e2e/skillopt-loop.serial.test.tsindex c473dd2267..7dda2de168 100644--- a/test/e2e/skillopt-loop.serial.test.ts+++ b/test/e2e/skillopt-loop.serial.test.ts@@ -326,7 +326,7 @@ async function runOnce(fixture: Fixture, over: RunOptsOverride = {}) {maxRuntimeMin: over.maxRuntimeMin ?? 1,force: true, // bypass dirty-tree (tempdir isn't a git repo)...(over.heldOutPath ? { heldOutPath: over.heldOutPath } : {}),- ...(over.optimizerMode ? { optimizerMode: over.optimizerMode } : {}),+ optimizerMode: over.optimizerMode ?? 'reflect',...(over.reflectMode ? { reflectMode: over.reflectMode } : {}),...(over.disableValidationGate ? { disableValidationGate: over.disableValidationGate } : {}),});diff --git a/test/e2e/skillopt-models-strict.serial.test.ts b/test/e2e/skillopt-models-strict.serial.test.tsindex fd8998be34..00ea76066d 100644--- a/test/e2e/skillopt-models-strict.serial.test.ts+++ b/test/e2e/skillopt-models-strict.serial.test.ts@@ -116,7 +116,7 @@ async function baseOpts(over: Partial<SkillOptOpts> = {}): Promise<SkillOptOpts>engine, skillName: SKILL, skillsDir, benchmarkPath,epochs: 1, batchSize: 2, lr: 4, lrSchedule: 'constant', split: [4, 1, 5],...skillOptModelOpts(await resolveSkillOptModels(engine)),- mode: 'patch', dryRun: false, noMutate: false, allowMutateBundled: true, bootstrapReviewed: false,+ mode: 'patch', optimizerMode: 'reflect', dryRun: false, noMutate: false, allowMutateBundled: true, bootstrapReviewed: false,json: true, maxCostUsd: 100, maxRuntimeMin: 2, force: true,...over,};@@ -248,7 +248,7 @@ describe('background job', () => {const models = await resolveSkillOptModels(engine);const data = buildSkillOptJobData({skillsDir, skillName: SKILL, benchmarkPath, epochs: 1, batchSize: 2, lr: 4, lrSchedule: 'constant', split: [4, 1, 5],- mode: 'patch', dryRun: true, noMutate: false, allowMutateBundled: true, bootstrapReviewed: false,+ mode: 'patch', optimizerMode: 'reflect', dryRun: true, noMutate: false, allowMutateBundled: true, bootstrapReviewed: false,maxCostUsd: 100, maxRuntimeMin: 2, force: true, models, modelFlags: {}, modelsStrict: false,});expect(data.model_sources).toEqual({ optimizer: 'tier_config', target: 'tier_default', judge: 'tier_default' });@@ -283,7 +283,7 @@ describe('strict propagation through batch and fleet', () => {engine, skillsDir, perSkillMaxCostUsd: 100, brainWideMaxCostUsd: 100,...skillOptModelOpts(models), modelsStrict: true,epochs: 1, batchSize: 2, lr: 4, lrSchedule: 'constant', split: [4, 1, 5],- dryRun: false, noMutate: true, allowMutateBundled: true, force: true,+ dryRun: false, noMutate: true, allowMutateBundled: true, force: true, optimizerMode: 'reflect',});expect(out.per_skill.map((s) => s.outcome)).toEqual(['errored', 'errored']);expect(out.per_skill.every((s) => s.reason?.includes('models strict check'))).toBe(true);@@ -297,7 +297,7 @@ describe('strict propagation through batch and fleet', () => {const out = await runFleet({engine, skillName: SKILL, skillsDir, benchmarkPath, targetModels: [HAIKU, SONNET],optimizerModel: models.optimizer.model, judgeModel: models.judge.model,- models: { optimizer: models.optimizer, judge: models.judge }, modelsStrict: true,+ models: { optimizer: models.optimizer, judge: models.judge }, modelsStrict: true, optimizerMode: 'reflect',epochs: 1, batchSize: 2, lr: 4, lrSchedule: 'constant', split: [4, 1, 5],dryRun: false, noMutate: true, allowMutateBundled: true, bootstrapReviewed: false,maxCostUsd: 100, maxRuntimeMin: 2, force: true,diff --git a/test/e2e/skillopt-models-used.serial.test.ts b/test/e2e/skillopt-models-used.serial.test.tsindex 94e49c74de..cd002cf987 100644--- a/test/e2e/skillopt-models-used.serial.test.ts+++ b/test/e2e/skillopt-models-used.serial.test.ts@@ -99,7 +99,7 @@ async function run(over: Partial<SkillOptOpts> = {}) {optimizerModel: SONNET,targetModel: HAIKU,judgeModel: SONNET,- mode: 'patch',+ mode: 'patch', optimizerMode: 'reflect',dryRun: false,noMutate: false,allowMutateBundled: true,diff --git a/test/e2e/skillopt-outcome.serial.test.ts b/test/e2e/skillopt-outcome.serial.test.tsindex 426a1e1b22..087fb22c73 100644--- a/test/e2e/skillopt-outcome.serial.test.ts+++ b/test/e2e/skillopt-outcome.serial.test.ts@@ -105,7 +105,7 @@ async function run(over: Partial<SkillOptOpts> = {}) {optimizerModel: PLAIN,targetModel: TARGET,judgeModel: TARGET,- mode: 'patch',+ mode: 'patch', optimizerMode: 'reflect',dryRun: false,noMutate: false,allowMutateBundled: true,@@ -271,7 +271,7 @@ describe('--rewrite mode', () => {expect(optimizerCalls.length).toBeGreaterThan(0);expect(optimizerCalls.some((c) => c.mode === 'one-shot')).toBe(false);expect(res.outcome).toBe('accepted');- expect(res.receipt.optimizer_mode).toBeUndefined();+ expect(res.receipt.optimizer_mode).toBe('reflect');});});diff --git a/test/e2e/skillopt-pglite.serial.test.ts b/test/e2e/skillopt-pglite.serial.test.tsindex 33febee85b..6644cb15d6 100644--- a/test/e2e/skillopt-pglite.serial.test.ts+++ b/test/e2e/skillopt-pglite.serial.test.ts@@ -103,7 +103,7 @@ describe('skillopt E2E (PGLite + DI LLM)', () => {optimizerModel: 'anthropic:claude-opus-4-7',targetModel: 'anthropic:claude-sonnet-4-6',judgeModel: 'anthropic:claude-sonnet-4-6',- mode: 'patch',+ mode: 'patch', optimizerMode: 'reflect',dryRun: true, // SHORT-CIRCUITSnoMutate: false,allowMutateBundled: true,@@ -165,7 +165,7 @@ describe('skillopt E2E (PGLite + DI LLM)', () => {optimizerModel: 'anthropic:claude-opus-4-7',targetModel: 'anthropic:claude-sonnet-4-6',judgeModel: 'anthropic:claude-sonnet-4-6',- mode: 'patch',+ mode: 'patch', optimizerMode: 'reflect',dryRun: false,noMutate: false,allowMutateBundled: true,diff --git a/test/e2e/skillopt-wave-e1b1.serial.test.ts b/test/e2e/skillopt-wave-e1b1.serial.test.tsnew file mode 100644index 0000000000..da9d5c6f84--- /dev/null+++ b/test/e2e/skillopt-wave-e1b1.serial.test.ts@@ -0,0 +1,337 @@+/**+ * SkillOpt wave E1 + B1 (GBRA-274): one-shot rewrite by default, the benchmark+ * hooks gbrain-evals needs (explicit task lists, a batch rollout hook, receipt+ * counts, no gold-leaking criteria) and defects 2-4.+ *+ * Hermetic: PGLite in-memory engine, tempdir skill, a stub chat transport for+ * the optimizer and target, rule judges. No API keys, no network.+ */++import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';+import * as fs from 'node:fs';+import * as os from 'node:os';+import * as path from 'node:path';+import { PGLiteEngine } from '../../src/core/pglite-engine.ts';+import { resetPgliteState } from '../helpers/reset-pglite.ts';+import { withEnv } from '../helpers/with-env.ts';+import { __setChatTransportForTests, type ChatOpts, type ChatResult } from '../../src/core/ai/gateway.ts';+import { runSkillOpt } from '../../src/core/skillopt/orchestrator.ts';+import { loadHistory } from '../../src/core/skillopt/version-store.ts';+import { loadRejectedBuffer, rejectedFilePath } from '../../src/core/skillopt/rejected-buffer.ts';+import { runSkillOptJob } from '../../src/core/skillopt/job.ts';+import type { BenchmarkTask, EditOp, RolloutBatchRequest, ScoredRollout, SkillOptOpts } from '../../src/core/skillopt/types.ts';++const SKILL = 'wave-skill';+const OPTIMIZER = 'anthropic:claude-opus-4-7';+const TARGET = 'anthropic:claude-sonnet-4-6';+const BASE = `---+name: wave-skill+version: 0.1.0+description: Test skill for the SkillOpt wave.+triggers:+ - "do the wave task"+brain_first: exempt+---++# Wave++Answer the task.++## People+List people.+`;+const SECTIONS = ['People', 'Citations', 'Summary'];+const BUDGET_TAG = 'skillopt_hook_budget_exhausted';++let engine: PGLiteEngine;+let dir: string;+let skillsDir: string;+let optimizerPrompts: string[];+let targetTasks: string[];++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+});+afterAll(async () => { await engine.disconnect(); });+beforeEach(async () => {+ await resetPgliteState(engine);+ dir = fs.mkdtempSync(path.join(os.tmpdir(), 'skillopt-wave-'));+ skillsDir = dir;+ fs.mkdirSync(path.join(skillsDir, SKILL), { recursive: true });+ fs.writeFileSync(skillFile(), BASE);+ optimizerPrompts = [];+ targetTasks = [];+});+afterEach(() => {+ __setChatTransportForTests(null);+ fs.rmSync(dir, { recursive: true, force: true });+});++const skillFile = () => path.join(skillsDir, SKILL, 'SKILL.md');+const add = (section: string): EditOp => ({ op: 'add', anchor: '## People', content: `## ${section}\nok.`, reason: 'r' });++function task(id: string, text: string, checks: Array<{ op: string; arg: string }>): BenchmarkTask {+ return { task_id: id, task: text, judge: { kind: 'rule', checks } } as BenchmarkTask;+}+/** Sorted ids a-/b-/c- land in train/sel/test under --split 1:1:1. */+function writeBench(checks: (id: string) => Array<{ op: string; arg: string }>, n = 5): string {+ const rows: BenchmarkTask[] = [];+ for (const [prefix, label] of [['a', 'TRAIN'], ['b', 'SEL'], ['c', 'TEST']] as const) {+ for (let i = 1; i <= n; i++) rows.push(task(`${prefix}-${i}`, `${label} task ${i}`, checks(`${prefix}-${i}`)));+ }+ const p = path.join(skillsDir, SKILL, 'skillopt-benchmark.jsonl');+ fs.writeFileSync(p, rows.map((r) => JSON.stringify(r)).join('\n') + '\n');+ return p;+}+function writeHeldOut(): string {+ const p = path.join(skillsDir, SKILL, 'held-out.jsonl');+ const rows = Array.from({ length: 5 }, (_, i) => task(`h-${i + 1}`, `HELDOUT task ${i + 1}`, [{ op: 'contains', arg: 'People' }]));+ fs.writeFileSync(p, rows.map((r) => JSON.stringify(r)).join('\n') + '\n');+ return p;+}++function reply(text: string, model: string): ChatResult {+ return { text, blocks: [{ type: 'text', text }], stopReason: 'end', model, providerId: 'anthropic',+ usage: { input_tokens: 100, output_tokens: 20, cache_read_tokens: 0, cache_creation_tokens: 0 } };+}+/** Target emits the sections its skill has; each optimizer call takes the next queued reply. */+function installStub(optimizer: Array<EditOp[] | string>): void {+ let n = 0;+ __setChatTransportForTests(async (o: ChatOpts) => {+ const sys = o.system ?? '';+ const user = typeof o.messages[0]?.content === 'string' ? o.messages[0].content : '';+ if (sys.startsWith("You are SkillOpt's optimizer")) {+ optimizerPrompts.push(`${sys}\n${user}`);+ const next = optimizer[n++] ?? [];+ return reply(typeof next === 'string' ? next : JSON.stringify({ edits: next }), o.model ?? OPTIMIZER);+ }+ targetTasks.push(user);+ const out = SECTIONS.filter((s) => sys.includes(`## ${s}`)).map((s) => `## ${s}\nok`).join('\n\n');+ return reply(out || 'nothing', o.model ?? TARGET);+ });+}++function opts(over: Partial<SkillOptOpts>): SkillOptOpts {+ return {+ engine, skillName: SKILL, skillsDir,+ epochs: 1, batchSize: 5, lr: 4, lrSchedule: 'constant', split: [1, 1, 1],+ optimizerModel: OPTIMIZER, targetModel: TARGET, judgeModel: TARGET,+ mode: 'patch', optimizerMode: 'reflect', dryRun: false, noMutate: false, allowMutateBundled: true,+ bootstrapReviewed: false, json: true, maxCostUsd: 100, maxRuntimeMin: 2, force: true,+ ...over,+ } as SkillOptOpts;+}+const run = (o: SkillOptOpts) => withEnv({ GBRAIN_AUDIT_DIR: dir }, () => runSkillOpt(o));++describe('defect 4: only applied edits are filed or recorded', () => {+ test('LR budget 1, three proposals, gate rejects: one rejected-buffer row', async () => {+ const benchmarkPath = writeBench(() => [{ op: 'contains', arg: 'Citations' }]);+ installStub([[add('Bogus1'), add('Bogus2'), add('Bogus3')]]);+ const res = await run(opts({ benchmarkPath, lr: 1 }));+ expect(res.outcome).toBe('no_improvement');+ const rows = loadRejectedBuffer(skillsDir, SKILL);+ expect(rows).toHaveLength(1);+ expect(JSON.stringify(rows[0]!.edits)).toContain('Bogus1');+ });++ test('LR budget 1, three proposals, gate accepts: history lists the applied edit only, clipped edits stay unfiled', async () => {+ const benchmarkPath = writeBench(() => [{ op: 'contains', arg: 'Citations' }]);+ installStub([[add('Citations'), add('Bogus2'), add('Bogus3')]]);+ const res = await run(opts({ benchmarkPath, lr: 1 }));+ expect(res.outcome).toBe('accepted');+ const committed = loadHistory(skillsDir, SKILL).filter((r) => r.status === 'committed');+ expect(committed).toHaveLength(1);+ expect(committed[0]!.edits).toHaveLength(1);+ expect(JSON.stringify(committed[0]!.edits)).toContain('Citations');+ expect(loadRejectedBuffer(skillsDir, SKILL)).toHaveLength(0);+ });+});++describe('defect 3 + B1.4: no gold-leaking criteria', () => {+ const checks = (id: string) => [{ op: 'contains', arg: `ANSWER-${id}` }, { op: 'section_present', arg: 'Citations' }];++ for (const optimizerMode of ['reflect', 'one-shot-rewrite'] as const) {+ test(`${optimizerMode}: a shared format rule reaches the optimizer, per-task answers never do`, async () => {+ const benchmarkPath = writeBench(checks);+ installStub([optimizerMode === 'reflect' ? [] : '# Wave\n\n## People\nList people.\n']);+ await run(opts({ benchmarkPath, optimizerMode }));+ expect(optimizerPrompts.length).toBeGreaterThan(0);+ for (const p of optimizerPrompts) {+ expect(p).toContain('"Citations"');+ expect(p).not.toContain('ANSWER-');+ }+ });+ }++ test('criteriaText replaces the generated criteria verbatim', async () => {+ const benchmarkPath = writeBench(checks);+ installStub([[]]);+ await run(opts({ benchmarkPath, criteriaText: 'OPERATOR CRITERIA: cite every claim' }));+ for (const p of optimizerPrompts) {+ expect(p).toContain('OPERATOR CRITERIA: cite every claim');+ expect(p).not.toContain('task-specific');+ expect(p).not.toContain('"Citations"');+ }+ });+});++describe('defect 2: held-out stays out of the proposer and runs once', () => {+ test('a legacy held_out_regression row on disk never reaches an optimizer prompt', async () => {+ const benchmarkPath = writeBench(() => [{ op: 'contains', arg: 'Citations' }]);+ const legacy = { key: 'deadbeef', skill_sha8: 'deadbeef', reason: 'held_out_regression', ts: new Date().toISOString(),+ edits: [{ op: 'add', anchor: '## People', content: 'LEAKED-HELDOUT-VERDICT' }] };+ fs.mkdirSync(path.dirname(rejectedFilePath(skillsDir, SKILL)), { recursive: true });+ fs.writeFileSync(rejectedFilePath(skillsDir, SKILL), JSON.stringify({ schema: 1, entries: [legacy] }));+ installStub([[]]);+ await run(opts({ benchmarkPath }));+ expect(optimizerPrompts.length).toBeGreaterThan(0);+ for (const p of optimizerPrompts) {+ expect(p).not.toContain('LEAKED-HELDOUT-VERDICT');+ expect(p).not.toContain('held_out');+ }+ });++ test('two accepted loop steps: the held-out set is scored once, on the final champion, then written once', async () => {+ const benchmarkPath = writeBench(() => [{ op: 'contains', arg: 'Citations' }, { op: 'contains', arg: 'Summary' }]);+ installStub([[add('Citations')], [add('Summary')]]);+ const res = await run(opts({ benchmarkPath, heldOutPath: writeHeldOut(), batchSize: 2 }));+ expect(res.outcome).toBe('accepted');+ expect(targetTasks.filter((t) => t.startsWith('HELDOUT')).length).toBe(5 * 3 * 2);+ expect(res.receipt.held_out).toEqual({ baseline_score: 1, candidate_score: 1, passed: true });+ expect(res.receipt.rollouts?.held_out).toBe(30);+ expect(loadHistory(skillsDir, SKILL).filter((r) => r.status === 'committed')).toHaveLength(1);+ const text = fs.readFileSync(skillFile(), 'utf8');+ expect(text).toContain('## Citations');+ expect(text).toContain('## Summary');+ });++ test('a champion the held-out check refuses is never written and never filed for the proposer', async () => {+ const benchmarkPath = writeBench(() => [{ op: 'contains', arg: 'Citations' }]);+ installStub([[{ op: 'replace', target: '## People\nList people.', replacement: '## Citations\nCite.', reason: 'r' }]]);+ const res = await run(opts({ benchmarkPath, heldOutPath: writeHeldOut() }));+ expect(res.outcome).toBe('no_improvement');+ expect(res.receipt.held_out?.passed).toBe(false);+ expect(res.mutatedSkillFile).toBe(false);+ expect(fs.readFileSync(skillFile(), 'utf8')).toBe(BASE);+ expect(loadRejectedBuffer(skillsDir, SKILL)).toHaveLength(0);+ const buf = rejectedFilePath(skillsDir, SKILL);+ if (fs.existsSync(buf)) expect(fs.readFileSync(buf, 'utf8')).not.toContain('held_out');+ });+});++describe('E1: one-shot rewrite is a gated step on train data', () => {+ test('the rewrite reflects on train tasks only, never selection tasks', async () => {+ const benchmarkPath = writeBench(() => [{ op: 'contains', arg: 'Citations' }]);+ installStub(['# Wave\n\n## People\nList people.\n\n## Citations\nCite.\n']);+ const res = await run(opts({ benchmarkPath, optimizerMode: 'one-shot-rewrite' }));+ expect(res.receipt.optimizer_mode).toBe('one-shot-rewrite');+ expect(optimizerPrompts).toHaveLength(1);+ expect(optimizerPrompts[0]).toContain('TRAIN task');+ expect(optimizerPrompts[0]).not.toContain('SEL task');+ expect(res.outcome).toBe('accepted');+ expect(fs.readFileSync(skillFile(), 'utf8')).toContain('## Citations');+ });++ test('a rewrite that scores below baseline on selection is not promoted', async () => {+ fs.writeFileSync(skillFile(), BASE + '\n## Citations\nCite.\n');+ const before = fs.readFileSync(skillFile(), 'utf8');+ const benchmarkPath = writeBench(() => [{ op: 'contains', arg: 'Citations' }]);+ installStub(['# Wave\n\n## People\nList people.\n']);+ const res = await run(opts({ benchmarkPath, optimizerMode: 'one-shot-rewrite' }));+ expect(res.outcome).toBe('no_improvement');+ expect(res.mutatedSkillFile).toBe(false);+ expect(fs.readFileSync(skillFile(), 'utf8')).toBe(before);+ });++ test('a queued job without optimizer_mode keeps the loop it was queued for; a new one carries its mode', async () => {+ const benchmarkPath = writeBench(() => [{ op: 'contains', arg: 'Citations' }]);+ const base = { skills_dir: skillsDir, skill_name: SKILL, benchmark_path: benchmarkPath, split: [1, 1, 1], dry_run: true, force: true,+ optimizer_model: OPTIMIZER, target_model: TARGET, judge_model: TARGET, max_cost_usd: 100 };+ const legacy = await runSkillOptJob(engine, base) as { receipt: { optimizer_mode: string } };+ expect(legacy.receipt.optimizer_mode).toBe('reflect');+ const current = await runSkillOptJob(engine, { ...base, optimizer_mode: 'one-shot-rewrite' }) as { receipt: { optimizer_mode: string } };+ expect(current.receipt.optimizer_mode).toBe('one-shot-rewrite');+ });+});++describe('B1.2 / B1.3 / B1.6: explicit tasks, rollout hook, receipt counts', () => {+ const ids = (prefix: string, n: number, label: string) =>+ Array.from({ length: n }, (_, i) => task(`${prefix}-${i + 1}`, `${label} task ${i + 1}`, [{ op: 'contains', arg: 'Citations' }]));+ const lists = () => ({ train: ids('t', 6, 'TRAIN'), sel: ids('s', 5, 'SEL') });++ function countingHook(onCall?: (req: RolloutBatchRequest, nth: number) => void) {+ const calls: RolloutBatchRequest[] = [];+ const hook = async (req: RolloutBatchRequest): Promise<ScoredRollout[][]> => {+ calls.push(req);+ onCall?.(req, calls.length);+ const score = req.skillText.includes('## Citations') ? 1 : 0;+ return req.tasks.map((t) => Array.from({ length: req.runsPerTask }, () => ({+ score,+ trajectory: {+ task_id: t.task_id, task: req.phase === 'train_forward' ? t.task : '', final_text: req.phase === 'train_forward' ? 'out' : '',+ tool_calls: [], turns: 1, stop_reason: 'end', duration_ms: 0,+ usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 },+ },+ } as ScoredRollout)));+ };+ return { hook, calls };+ }++ test('every rollout goes through the hook, on the given lists, and the receipt counts them', async () => {+ const { hook, calls } = countingHook();+ installStub([[add('Citations')]]);+ const t = lists();+ const res = await run(opts({ tasks: t, rolloutHook: hook, epochs: 2, batchSize: 3 }));+ expect(res.outcome).toBe('accepted');+ expect(targetTasks).toHaveLength(0);+ const train = new Set(t.train.map((x) => x.task_id));+ const sel = new Set(t.sel.map((x) => x.task_id));+ for (const c of calls) {+ const want = c.phase === 'train_forward' ? train : sel;+ for (const x of c.tasks) expect(want.has(x.task_id)).toBe(true);+ expect(c.runsPerTask).toBe(c.phase === 'train_forward' ? 1 : 3);+ }+ expect(calls.some((c) => c.phase === 'test')).toBe(false);+ expect(res.receipt.test_score).toBeUndefined();+ const sum = (phase: string) => calls.filter((c) => c.phase === phase).reduce((n, c) => n + c.tasks.length * c.runsPerTask, 0);+ expect(res.receipt.rollouts).toEqual({ baseline_sel: sum('baseline_sel'), train_forward: sum('train_forward'), sel_gate: sum('sel_gate'), held_out: 0, test: 0 });+ expect(res.receipt.candidates_scored_on_sel).toBe(new Set(calls.filter((c) => c.phase === 'sel_gate').map((c) => c.skillSha256)).size);+ expect(res.receipt.optimizer_calls).toBe(optimizerPrompts.length);+ expect(res.receipt.cost_scope).toBe('optimizer_only');+ expect(res.receipt.optimizer_mode).toBe('reflect');+ });++ test('a hook budget refusal mid-step aborts with the previous best on disk and no new buffer rows', async () => {+ let selGates = 0;+ const { hook } = countingHook((req) => {+ if (req.phase === 'sel_gate' && ++selGates === 2) throw new Error(`${BUDGET_TAG}: shared cap reached`);+ });+ installStub([[add('Citations')], [add('Summary')]]);+ const res = await run(opts({ tasks: lists(), rolloutHook: hook, batchSize: 3 }));+ expect(res.outcome).toBe('aborted');+ expect(res.receipt.abort_reason).toBe('budget_exhausted');+ expect(res.receipt.abort_detail).toContain(BUDGET_TAG);+ const text = fs.readFileSync(skillFile(), 'utf8');+ expect(text).toContain('## Citations');+ expect(text).not.toContain('## Summary');+ expect(loadRejectedBuffer(skillsDir, SKILL)).toHaveLength(0);+ });++ test('one-shot under a hook that returns score-only rows outside train_forward completes', async () => {+ const { hook, calls } = countingHook();+ installStub(['# Wave\n\n## People\nList people.\n\n## Citations\nCite.\n']);+ const res = await run(opts({ tasks: lists(), rolloutHook: hook, optimizerMode: 'one-shot-rewrite', batchSize: 3 }));+ expect(res.outcome).toBe('accepted');+ expect(calls.map((c) => c.phase)).toEqual(['baseline_sel', 'train_forward', 'sel_gate']);+ expect(calls[1]!.tasks.map((x) => x.task_id)).toEqual(['t-1', 't-2', 't-3']);+ });++ test('explicit tasks and a benchmark file together are refused before any call', async () => {+ installStub([]);+ await expect(run(opts({ tasks: lists(), benchmarkPath: writeBench(() => [{ op: 'contains', arg: 'x' }]) })))+ .rejects.toMatchObject({ envelope: { code: 'explicit_tasks_conflict' } });+ expect(optimizerPrompts).toHaveLength(0);+ });+});diff --git a/test/fixtures/goldens/cli/tools-json.json b/test/fixtures/goldens/cli/tools-json.jsonindex bdf0a71950..50cb704701 100644--- a/test/fixtures/goldens/cli/tools-json.json+++ b/test/fixtures/goldens/cli/tools-json.json@@ -2,8 +2,8 @@"golden": {"exitCode": 0,"stderr": "",- "stdoutBytes": 265045,- "stdoutSha256": "4358e6505ccac2f00bcbe3a01fb54daf3d13c8a7c526315a8c53aaf223a01dc8",+ "stdoutBytes": 265405,+ "stdoutSha256": "ce85fa67a616e3c90eda5a73943e277734add153923cd5f19dfef2a8fe026f03","toolNames": ["remember","entity",@@ -7791,6 +7791,7 @@"lr": "number?","max_cost_usd": "number?","no_mutate": "boolean?",+ "optimizer_mode": "string?","pack_id": "string?","reflect_max_tokens": "number?","request_id": "string?",@@ -7842,6 +7843,14 @@"description": "Write proposed.md without replacing SKILL.md","type": "boolean"},+ "optimizer_mode": {+ "description": "Default one-shot-rewrite: one gated rewrite. reflect opts into the iterative edit loop, which has not yet beaten one-shot on gbrain benchmarks.",+ "enum": [+ "one-shot-rewrite",+ "reflect"+ ],+ "type": "string"+ },"pack_id": {"description": "Required with shared_skill: exact catalog pack.","type": "string"diff --git a/test/fixtures/goldens/exports/runtime.json b/test/fixtures/goldens/exports/runtime.jsonindex d39f4681f1..736052faed 100644--- a/test/fixtures/goldens/exports/runtime.json+++ b/test/fixtures/goldens/exports/runtime.json@@ -599,9 +599,18 @@"priceAtPromptSize"],"./core/skillopt": [+ "READ_ONLY_BRAIN_TOOLS",+ "SKILLOPT_HOOK_BUDGET_EXHAUSTED",+ "SKILLOPT_HOOK_INFRA_ERROR",+ "describeJudge",+ "describeJudges","loadHeldOut",+ "rolloutToolsForSkill",+ "runRollout","runSkillOpt",- "scoreSkillOnTasks"+ "scoreSkillOnTasks",+ "scoreSkillPerTask",+ "scoreTrajectory"],"./embedding": ["EMBEDDING_COST_PER_1K_TOKENS",diff --git a/test/fixtures/goldens/exports/types.json b/test/fixtures/goldens/exports/types.jsonindex 4b1eff95df..a9a4c2b622 100644--- a/test/fixtures/goldens/exports/types.json+++ b/test/fixtures/goldens/exports/types.json@@ -3129,6 +3129,131 @@],"name": "BenchmarkTask"},+ {+ "kind": "type",+ "name": "Judge",+ "signature": [+ "{ kind: \"rule\"; checks: RuleCheck[]; } | { kind: \"llm\"; rubric: string; model?: string; } | { kind: \"qrels\"; expected_slugs: string[]; k: number; }"+ ]+ },+ {+ "kind": "type",+ "name": "OptimizerMode",+ "signature": [+ "\"reflect\" | \"one-shot-rewrite\""+ ]+ },+ {+ "kind": "interface",+ "members": [+ "perTask: { task_id: string; median: number; runs: number[]; error?: string; }[]",+ "score: number"+ ],+ "name": "PerTaskScore"+ },+ {+ "kind": "variable",+ "name": "READ_ONLY_BRAIN_TOOLS",+ "signature": [+ "ReadonlySet<string>"+ ]+ },+ {+ "kind": "type",+ "name": "RolloutBatchHook",+ "signature": [+ "(req: RolloutBatchRequest, signal?: AbortSignal) => Promise<ScoredRollout[][]>"+ ]+ },+ {+ "kind": "interface",+ "members": [+ "epoch: number",+ "phase: RolloutPhase",+ "runsPerTask: number",+ "skillSha256: string",+ "skillText: string",+ "step: number",+ "tasks: BenchmarkTask[]"+ ],+ "name": "RolloutBatchRequest"+ },+ {+ "kind": "interface",+ "members": [+ "abortSignal?: AbortSignal | undefined",+ "allowedTools?: ReadonlySet<string> | undefined",+ "chatFn?: ((opts: ChatOpts) => Promise<ChatResult>) | undefined",+ "engine: BrainEngine",+ "maxTurns?: number | undefined",+ "operationContext?: OperationContext | undefined",+ "skillText: string",+ "targetModel: string",+ "task: BenchmarkTask",+ "toolLoopFn?: ((opts: ToolLoopOpts) => Promise<ToolLoopResult>) | undefined",+ "writeCapture?: boolean | undefined"+ ],+ "name": "RolloutOpts"+ },+ {+ "kind": "type",+ "name": "RolloutPhase",+ "signature": [+ "\"baseline_sel\" | \"train_forward\" | \"sel_gate\" | \"held_out\" | \"test\""+ ]+ },+ {+ "kind": "interface",+ "members": [+ "abort_detail?: string | undefined",+ "abort_reason?: \"budget_exhausted\" | \"error\" | \"runtime_exceeded\" | \"sigint\" | undefined",+ "baseline_sel_score?: number | undefined",+ "baseline_test_score?: number | undefined",+ "batch_size: number",+ "benchmark_sha8: string",+ "best_sel_score?: number | undefined",+ "candidates_scored_on_sel?: number | undefined",+ "cost_scope?: \"all\" | \"optimizer_only\" | undefined",+ "ended_at?: string | undefined",+ "epochs: number",+ "epochs_completed?: number | undefined",+ "final_cost_usd?: number | undefined",+ "held_out?: { baseline_score: number; candidate_score: number; passed: boolean; } | undefined",+ "judge_model: string",+ "lr: number",+ "lr_schedule: \"cosine\" | \"linear\" | \"constant\"",+ "max_cost_usd: number",+ "models_plan?: ModelsPlanEntry[] | undefined",+ "models_strict?: StrictVerdict | undefined",+ "models_used?: ModelUsageRow[] | undefined",+ "models_used_scope?: \"full_run\" | \"since_resume\" | undefined",+ "no_pricing?: NoPricingGuidance | undefined",+ "optimizer_calls?: number | undefined",+ "optimizer_mode?: OptimizerMode | undefined",+ "optimizer_model: string",+ "outcome?: \"aborted\" | \"accepted\" | \"no_improvement\" | \"errored\" | undefined",+ "prior_segments_cost_usd?: number | undefined",+ "reflect_errors?: string[] | undefined",+ "reflect_invalid_edits_dropped?: number | undefined",+ "reflect_max_tokens?: number | undefined",+ "reflect_max_tokens_source?: \"default\" | \"config\" | \"flag\" | undefined",+ "reflect_mode?: \"both\" | \"failure-only\" | undefined",+ "remediation?: { code: string; fix: string; docs: string; }[] | undefined",+ "resume_command?: string | undefined",+ "rollouts?: Record<RolloutPhase, number> | undefined",+ "run_id: string",+ "skill: string",+ "skill_body_truncated?: { sent_chars: number; total_chars: number; } | undefined",+ "skill_sha8: string",+ "started_at: string",+ "stop_reason?: \"aborted\" | \"completed\" | \"early_stop_unusable_output\" | undefined",+ "target_model: string",+ "test_score?: number | undefined",+ "total_steps?: number | undefined",+ "validation_gate_disabled?: boolean | undefined"+ ],+ "name": "RunReceipt"+ },{"kind": "interface","members": [@@ -3141,6 +3266,20 @@],"name": "RunSkillOptResult"},+ {+ "kind": "variable",+ "name": "SKILLOPT_HOOK_BUDGET_EXHAUSTED",+ "signature": [+ "\"skillopt_hook_budget_exhausted\""+ ]+ },+ {+ "kind": "variable",+ "name": "SKILLOPT_HOOK_INFRA_ERROR",+ "signature": [+ "\"skillopt_hook_infra_error\""+ ]+ },{"kind": "interface","members": [@@ -3149,7 +3288,9 @@"engine: BrainEngine","judgeModel?: string | undefined","operationContext?: OperationContext | undefined",+ "phase?: RolloutPhase | undefined","rolloutFn?: ((opts: RolloutOpts) => Promise<Trajectory>) | undefined",+ "rolloutHook?: RolloutBatchHook | undefined","runsPerTask?: number | undefined","scoreFn?: ((trajectory: Trajectory, judge: Judge, opts?: { judgeModel?: string; chatFn?: typeof chat; now?: () => Date; }) => Promise<ScoredRollout>) | undefined","skillText: string",@@ -3158,13 +3299,24 @@],"name": "ScoreOnTasksOpts"},+ {+ "kind": "interface",+ "members": [+ "judge_error?: string | undefined",+ "rationale?: string | undefined",+ "score: number",+ "trajectory: Trajectory"+ ],+ "name": "ScoredRollout"+ },{"kind": "interface","members": ["allowMutateBundled: boolean","batchSize: number",- "benchmarkPath: string",+ "benchmarkPath?: string | undefined","bootstrapReviewed: boolean",+ "criteriaText?: string | undefined","disableValidationGate?: boolean | undefined","dryRun: boolean","engine: BrainEngine",@@ -3184,20 +3336,50 @@"modelsStrict?: boolean | undefined","noMutate: boolean","operationContext?: OperationContext | undefined",- "optimizerMode?: \"reflect\" | \"one-shot-rewrite\" | undefined",+ "optimizerMode: OptimizerMode","optimizerModel: string","reflectMaxTokens?: number | undefined","reflectMode?: \"both\" | \"failure-only\" | undefined","resumeRunId?: string | undefined",+ "rolloutHook?: RolloutBatchHook | undefined","sharedSkill?: { source_id: string; source_incarnation: string; pack_id: string; expected_revision: string; request_id: string; } | undefined","skillName: string","skillsDir: string",- "split: [number, number, number]",+ "split?: [number, number, number] | undefined","targetModel: string",+ "tasks?: { train: BenchmarkTask[]; sel: BenchmarkTask[]; test?: BenchmarkTask[]; } | undefined","writeCapture?: boolean | undefined"],"name": "SkillOptOpts"},+ {+ "kind": "interface",+ "members": [+ "duration_ms: number",+ "final_text: string",+ "stop_reason: ToolLoopStopReason",+ "task: string",+ "task_id: string",+ "tool_calls: { name: string; input: unknown; output?: unknown; failed?: boolean; }[]",+ "turns: number",+ "usage: { input_tokens: number; output_tokens: number; cache_read_tokens: number; cache_creation_tokens: number; }"+ ],+ "name": "Trajectory"+ },+ {+ "kind": "function",+ "name": "describeJudge",+ "signature": [+ "(judge: Judge) => string"+ ]+ },+ {+ "kind": "function",+ "name": "describeJudges",+ "signature": [+ "(tasks: ReadonlyArray<{ judge: Judge; }>) => string"+ ]+ },{"kind": "function","name": "loadHeldOut",@@ -3205,6 +3387,20 @@"(heldOutPath: string) => BenchmarkTask[]"]},+ {+ "kind": "function",+ "name": "rolloutToolsForSkill",+ "signature": [+ "(skillText: string) => ReadonlySet<string>"+ ]+ },+ {+ "kind": "function",+ "name": "runRollout",+ "signature": [+ "(opts: RolloutOpts) => Promise<Trajectory>"+ ]+ },{"kind": "function","name": "runSkillOpt",@@ -3218,6 +3414,20 @@"signature": ["(opts: ScoreOnTasksOpts) => Promise<number>"]+ },+ {+ "kind": "function",+ "name": "scoreSkillPerTask",+ "signature": [+ "(opts: ScoreOnTasksOpts) => Promise<PerTaskScore>"+ ]+ },+ {+ "kind": "function",+ "name": "scoreTrajectory",+ "signature": [+ "(trajectory: Trajectory, judge: Judge, opts?: { judgeModel?: string; chatFn?: typeof chat; now?: () => Date; }) => Promise<ScoredRollout>"+ ]}],"./embedding": [diff --git a/test/shared-skills-optimizer.serial.test.ts b/test/shared-skills-optimizer.serial.test.tsindex e533f10983..547560ec03 100644--- a/test/shared-skills-optimizer.serial.test.ts+++ b/test/shared-skills-optimizer.serial.test.ts@@ -70,7 +70,7 @@ async function fixture(run: (f: Fixture) => Promise<void>) {pack_id: detail.pack_id, expected_revision: detail.revision, request_id: randomUUID() }, skillName: 'alpha', skillsDir: join(root, 'skills'),benchmarkPath: join(root, 'skills/alpha/skillopt-benchmark.jsonl'), epochs: 1, batchSize: 100, lr: 1, lrSchedule: 'constant', split: [4, 1, 5],optimizerModel: 'anthropic:claude-opus-4-7', targetModel: 'anthropic:claude-sonnet-4-6', judgeModel: 'anthropic:claude-sonnet-4-6',- mode: 'patch', dryRun: false, noMutate: false, allowMutateBundled: false, bootstrapReviewed: true, json: true, maxCostUsd: 1000, maxRuntimeMin: 5, force: false };+ mode: 'patch', optimizerMode: 'reflect', dryRun: false, noMutate: false, allowMutateBundled: false, bootstrapReviewed: true, json: true, maxCostUsd: 1000, maxRuntimeMin: 5, force: false };const reply = (request: ChatOpts, text: string): ChatResult => ({ text, blocks: [{ type: 'text', text }], stopReason: 'end',usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: request.model ?? opts.targetModel, providerId: 'synthetic' });__setChatTransportForTests(async request => {@@ -108,7 +108,7 @@ function proposals(f: Fixture) { return readdirSync(join(f.home, 'skillopt-propotest('explicit remote operation optimizes privately then publishes one complete bundle using original source grants and CAS', () => fixture(async f => {const operation = skilloptOperations.find(op => op.name === 'run_skillopt')!;const result = await operation.handler(f.remote, { skill_name: 'alpha', shared_skill: true, ...f.opts.sharedSkill,- epochs: 1, batch_size: 100, lr: 1, max_cost_usd: 1000 }) as { outcome: string; mutated_skill_file: boolean; shared_optimization: { publication: { state: string } } };+ epochs: 1, batch_size: 100, lr: 1, max_cost_usd: 1000, optimizer_mode: 'reflect' }) as { outcome: string; mutated_skill_file: boolean; shared_optimization: { publication: { state: string } } };expect(result.outcome).toBe('accepted'); expect(result.mutated_skill_file).toBe(true);expect(result.shared_optimization.publication.state).toBe('committed');expect((await getSharedSkill(f.local, { name: 'alpha' })).body).toBe(original.replace('BASELINE', 'IMPROVED'));@@ -270,7 +270,7 @@ test('remote evaluation inputs fail closed on current policy changes and stale rtest('remote evaluation inputs reject preexisting unpublished mutations to both approved assets', () => fixture(async f => {await publishHeldOut(f);let invoked = 0;- for (const [path, originalBytes] of [[f.opts.benchmarkPath, benchmark], [f.opts.heldOutPath!, heldOut]]) {+ for (const [path, originalBytes] of [[f.opts.benchmarkPath!, benchmark], [f.opts.heldOutPath!, heldOut]]) {writeFileSync(path, 'SYNTHETIC_UNPUBLISHED_INPUT');const before = await snapshot(f);await expect(optimizeSharedSkill(f.opts, async () => { invoked++; throw new Error('Unpublished asset reached evaluation'); }))@@ -288,9 +288,9 @@ test('remote evaluation inputs stage exact authorized sealed bytes across a livelet replaced = false, invoked = 0;const reads = spyOn(fs, 'readFileSync').mockImplementation(((...args: Parameters<typeof fs.readFileSync>) => {const bytes = Reflect.apply(read, fs, args);- if (String(args[0]) === f.opts.benchmarkPath && !replaced) {+ if (String(args[0]) === f.opts.benchmarkPath! && !replaced) {replaced = true;- writeFileSync(f.opts.benchmarkPath, 'SYNTHETIC_RACE_BENCHMARK');+ writeFileSync(f.opts.benchmarkPath!, 'SYNTHETIC_RACE_BENCHMARK');writeFileSync(f.opts.heldOutPath!, 'SYNTHETIC_RACE_HELD_OUT');}return bytes;@@ -299,7 +299,7 @@ test('remote evaluation inputs stage exact authorized sealed bytes across a liveconst result = await optimizeSharedSkill({ ...f.opts, noMutate: true, dryRun: true }, async staged => {invoked++;expect(replaced).toBe(true);- expect(readFileSync(staged.benchmarkPath, 'utf8')).toBe(benchmark);+ expect(readFileSync(staged.benchmarkPath!, 'utf8')).toBe(benchmark);expect(readFileSync(staged.heldOutPath!, 'utf8')).toBe(heldOut);expect(readFileSync(join(staged.skillsDir, 'alpha/skillopt-benchmark.jsonl'), 'utf8')).toBe(benchmark);expect(readFileSync(join(staged.skillsDir, 'alpha/held-out.jsonl'), 'utf8')).toBe(heldOut);@@ -308,7 +308,7 @@ test('remote evaluation inputs stage exact authorized sealed bytes across a liveexpect(result.outcome).toBe('aborted'); expect(result.mutatedSkillFile).toBe(false);} finally { reads.mockRestore(); }expect(invoked).toBe(1); expect(f.calls()).toBe(0);- expect(readFileSync(f.opts.benchmarkPath, 'utf8')).toBe('SYNTHETIC_RACE_BENCHMARK');+ expect(readFileSync(f.opts.benchmarkPath!, 'utf8')).toBe('SYNTHETIC_RACE_BENCHMARK');expect(readFileSync(f.opts.heldOutPath!, 'utf8')).toBe('SYNTHETIC_RACE_HELD_OUT');}), 120_000);@@ -319,7 +319,7 @@ test('trusted local evaluation inputs may use bounded explicit files outside thelet invoked = 0;const result = await optimizeSharedSkill({ ...f.opts, operationContext: f.local, benchmarkPath, heldOutPath, noMutate: true, dryRun: true }, async staged => {invoked++;- expect(readFileSync(staged.benchmarkPath, 'utf8')).toBe(benchmark);+ expect(readFileSync(staged.benchmarkPath!, 'utf8')).toBe(benchmark);expect(readFileSync(staged.heldOutPath!, 'utf8')).toBe(heldOut);return runSkillOpt(staged);});diff --git a/test/shared-skills-writer-boundary.test.ts b/test/shared-skills-writer-boundary.test.tsindex 8eb0aedaed..701a21bbcf 100644--- a/test/shared-skills-writer-boundary.test.ts+++ b/test/shared-skills-writer-boundary.test.ts@@ -237,9 +237,9 @@ test('legacy skill writers refuse before partial copies, history, optimizer evalexpect(existsSync(join(skillsDir, 'alpha/skillopt'))).toBe(false);expect(readFileSync(statePath, 'utf8')).toBe(bridgeStateBefore);}- const optimizer: SkillOptOpts = { engine: f.engine, skillsDir, skillName: 'alpha', benchmarkPath: join(f.scratch, 'absent.jsonl'), epochs: 1, batchSize: 1,+ const optimizer: SkillOptOpts & { benchmarkPath: string; split: [number, number, number] } = { engine: f.engine, skillsDir, skillName: 'alpha', benchmarkPath: join(f.scratch, 'absent.jsonl'), epochs: 1, batchSize: 1,lr: 1, lrSchedule: 'constant', split: [4, 1, 5], optimizerModel: 'unconfigured', targetModel: 'unconfigured', judgeModel: 'unconfigured',- mode: 'patch', dryRun: false, noMutate: false, allowMutateBundled: true, bootstrapReviewed: true, json: false, maxCostUsd: 1, maxRuntimeMin: 1, force: true };+ mode: 'patch', optimizerMode: 'reflect', dryRun: false, noMutate: false, allowMutateBundled: true, bootstrapReviewed: true, json: false, maxCostUsd: 1, maxRuntimeMin: 1, force: true };await expect(runSkillOpt(optimizer)).rejects.toMatchObject({ code: 'skill_bundle_required' });await expect(runFleet({ ...optimizer, targetModels: ['unconfigured'] })).rejects.toMatchObject({ code: 'skill_bundle_required' });await expect(runDoctor({ packRoot: f.root, mode: 'quick', fix: true, yes: true })).rejects.toMatchObject({ code: 'skill_bundle_required' });diff --git a/test/skillopt/adversarial/side-effecting-tool.test.ts b/test/skillopt/adversarial/side-effecting-tool.test.tsindex f76e19f699..eece46db8c 100644--- a/test/skillopt/adversarial/side-effecting-tool.test.ts+++ b/test/skillopt/adversarial/side-effecting-tool.test.ts@@ -58,7 +58,8 @@ describe('adversarial: D13 read-only tool sandbox', () => {}});- test('SkillOpt allowlist size = BRAIN_TOOL_ALLOWLIST size minus 1 (put_page)', () => {- expect(READ_ONLY_BRAIN_TOOLS.size).toBe(BRAIN_TOOL_ALLOWLIST.size - 1);+ test('SkillOpt allowlist size = BRAIN_TOOL_ALLOWLIST size minus its 2 mutating ops (put_page, add_timeline_entry)', () => {+ expect(READ_ONLY_BRAIN_TOOLS.has('add_timeline_entry')).toBe(false);+ expect(READ_ONLY_BRAIN_TOOLS.size).toBe(BRAIN_TOOL_ALLOWLIST.size - 2);});});diff --git a/test/skillopt/benchmark-hooks.test.ts b/test/skillopt/benchmark-hooks.test.tsnew file mode 100644index 0000000000..994d78d208--- /dev/null+++ b/test/skillopt/benchmark-hooks.test.ts@@ -0,0 +1,95 @@+/**+ * B1 benchmark hooks (GBRA-274): the batch rollout hook contract, explicit task+ * lists, per-task scores, rollout tool declarations and train-only criteria.+ * Pure: no engine work, no models.+ */++import { describe, expect, test } from 'bun:test';+import type { BrainEngine } from '../../src/core/engine.ts';+import { runValidationGate, scoreSkillPerTask } from '../../src/core/skillopt/validate-gate.ts';+import { isSkilloptMustAbort, SKILLOPT_HOOK_INFRA_ERROR } from '../../src/core/skillopt/must-abort.ts';+import { loadRunTasks } from '../../src/core/skillopt/benchmark.ts';+import { describeJudges } from '../../src/core/skillopt/reflect.ts';+import { READ_ONLY_BRAIN_TOOLS, rolloutToolsForSkill } from '../../src/core/skillopt/rollout.ts';+import type { BenchmarkTask, RolloutBatchHook, RolloutBatchRequest, ScoredRollout } from '../../src/core/skillopt/types.ts';++const engine = {} as BrainEngine;+const task = (id: string, arg = 'x'): BenchmarkTask => ({ task_id: id, task: `do ${id}`, judge: { kind: 'rule', checks: [{ op: 'contains', arg }] } } as BenchmarkTask);+const tasks = (prefix: string, n: number) => Array.from({ length: n }, (_, i) => task(`${prefix}-${i + 1}`));+const row = (score: number): ScoredRollout => ({ score, trajectory: { task_id: '', task: '', final_text: '', tool_calls: [], turns: 1, stop_reason: 'end', duration_ms: 0,+ usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 } } } as ScoredRollout);++describe('rollout hook (B1.3)', () => {+ test('one call for the whole batch, per-task medians and the request labels', async () => {+ const calls: RolloutBatchRequest[] = [];+ const hook: RolloutBatchHook = async (req) => { calls.push(req); return req.tasks.map((_, i) => [row(i % 2), row(1), row(0)]); };+ const res = await scoreSkillPerTask({ engine, skillText: 'SKILL', tasks: tasks('s', 4), targetModel: 'm', rolloutHook: hook });+ expect(calls).toHaveLength(1);+ expect(calls[0]).toMatchObject({ phase: 'test', skillText: 'SKILL', runsPerTask: 3, epoch: 0, step: 0 });+ expect(calls[0]!.skillSha256).toMatch(/^[0-9a-f]{64}$/);+ expect(res.perTask.map((p) => p.median)).toEqual([0, 1, 0, 1]);+ expect(res.score).toBe(0.5);+ });++ test.each([+ ['too few rows', (req: RolloutBatchRequest) => req.tasks.slice(1).map(() => [row(1), row(1), row(1)])],+ ['too few runs', (req: RolloutBatchRequest) => req.tasks.map(() => [row(1)])],+ ['not an array', () => ({}) as unknown as ScoredRollout[][]],+ ])('a malformed batch (%s) is an infra error that aborts, never a score', async (_name, reply) => {+ const hook: RolloutBatchHook = async (req) => reply(req);+ const err = await runValidationGate({ engine, candidateSkillText: 'S', selSet: tasks('s', 5), bestScore: 0, targetModel: 'm', rolloutHook: hook })+ .then(() => null, (e) => e);+ expect(String(err?.message)).toContain(SKILLOPT_HOOK_INFRA_ERROR);+ expect(isSkilloptMustAbort(err)).toBe(true);+ });+});++describe('explicit task lists (B1.2)', () => {+ test('lists are used as given, with a stable sha', () => {+ const t = { train: tasks('t', 2), sel: tasks('s', 5) };+ const a = loadRunTasks({ tasks: t });+ expect(a.split.train.map((x) => x.task_id)).toEqual(['t-1', 't-2']);+ expect(a.split.sel).toHaveLength(5);+ expect(a.split.test).toHaveLength(0);+ expect(a.bench.benchmark_sha8).toBe(loadRunTasks({ tasks: t }).bench.benchmark_sha8);+ expect(a.bench.benchmark_sha8).not.toBe(loadRunTasks({ tasks: { ...t, train: tasks('t', 3) } }).bench.benchmark_sha8);+ });+ test('a task id in two lists, an empty train list or a small sel list is refused', () => {+ const code = (fn: () => unknown) => { try { fn(); return null; } catch (e) { return (e as { envelope?: { code: string } }).envelope?.code; } };+ expect(code(() => loadRunTasks({ tasks: { train: [task('s-1')], sel: tasks('s', 5) } }))).toBe('benchmark_duplicate_task_id');+ expect(code(() => loadRunTasks({ tasks: { train: [], sel: tasks('s', 5) } }))).toBe('benchmark_empty');+ expect(code(() => loadRunTasks({ tasks: { train: tasks('t', 2), sel: tasks('s', 4) } }))).toBe('d_sel_too_small');+ });+});++describe('criteria (defect 3 / B1.4)', () => {+ test('a check every task shares is verbatim; a per-task answer is described, never quoted', () => {+ const ts = ['a', 'b', 'c'].map((id) => ({ judge: { kind: 'rule' as const, checks: [{ op: 'contains', arg: `GOLD-${id}` }, { op: 'section_present', arg: 'Sources' }] } }));+ const text = describeJudges(ts as never);+ expect(text).toContain('"Sources"');+ expect(text).not.toContain('GOLD-');+ expect(text.split('\n')).toHaveLength(2);+ });+ test('a single task never counts as shared', () => {+ expect(describeJudges([{ judge: { kind: 'rule', checks: [{ op: 'contains', arg: 'GOLD-1' }] } }] as never)).not.toContain('GOLD-1');+ });+ test('per-task LLM rubrics are not quoted', () => {+ const text = describeJudges([{ judge: { kind: 'llm', rubric: 'RUBRIC-A' } }, { judge: { kind: 'llm', rubric: 'RUBRIC-B' } }] as never);+ expect(text).not.toContain('RUBRIC-');+ });+});++describe('rollout tools (B1.5)', () => {+ const skill = (tools: string) => `---\nname: s\ntools: [${tools}]\n---\n# S\n`;+ test('the base set never offers a write', () => {+ expect(READ_ONLY_BRAIN_TOOLS.has('put_page')).toBe(false);+ expect(READ_ONLY_BRAIN_TOOLS.has('add_timeline_entry')).toBe(false);+ });+ test('a skill adds reviewed read ops by declaring them; anything else it names is ignored', () => {+ const got = rolloutToolsForSkill(skill('recall, get_timeline, put_page, add_timeline_entry, advisor, context_pack'));+ expect(got.has('recall')).toBe(true);+ expect(got.has('get_timeline')).toBe(true);+ for (const n of ['put_page', 'add_timeline_entry', 'advisor', 'context_pack']) expect(got.has(n)).toBe(false);+ expect(rolloutToolsForSkill(skill('')).size).toBe(READ_ONLY_BRAIN_TOOLS.size);+ });+});diff --git a/test/skillopt/models-four-paths.serial.test.ts b/test/skillopt/models-four-paths.serial.test.tsindex 18720155bc..01ba42d2d2 100644--- a/test/skillopt/models-four-paths.serial.test.ts+++ b/test/skillopt/models-four-paths.serial.test.ts@@ -105,7 +105,7 @@ async function viaJob(flags: SkillOptModelFlags = {}): Promise<Record<string, uncaptured = [];const data = buildSkillOptJobData({skillsDir, skillName: SKILL, benchmarkPath: path.join(skillsDir, SKILL, 'skillopt-benchmark.jsonl'),- epochs: 1, batchSize: 2, lr: 4, lrSchedule: 'constant', split: [4, 1, 5], mode: 'patch', dryRun: false,+ epochs: 1, batchSize: 2, lr: 4, lrSchedule: 'constant', split: [4, 1, 5], mode: 'patch', optimizerMode: 'reflect', dryRun: false,noMutate: false, allowMutateBundled: false, bootstrapReviewed: false, maxCostUsd: 5, maxRuntimeMin: 30, force: false,models: await resolveSkillOptModels(engine, flags), modelFlags: flags, modelsStrict: false,});@@ -224,7 +224,7 @@ describe('#5563 the cap source reaches runSkillOpt on every path', () => {captured = [];const data = buildSkillOptJobData({skillsDir, skillName: SKILL, benchmarkPath: path.join(skillsDir, SKILL, 'skillopt-benchmark.jsonl'),- epochs: 1, batchSize: 2, lr: 4, lrSchedule: 'constant', split: [4, 1, 5], mode: 'patch', dryRun: false,+ epochs: 1, batchSize: 2, lr: 4, lrSchedule: 'constant', split: [4, 1, 5], mode: 'patch', optimizerMode: 'reflect', dryRun: false,noMutate: false, allowMutateBundled: false, bootstrapReviewed: false, maxCostUsd: 5, maxCostSource: 'default', maxRuntimeMin: 30, force: false,models: await resolveSkillOptModels(engine, {}), modelFlags: {}, modelsStrict: false,});@@ -234,3 +234,17 @@ describe('#5563 the cap source reaches runSkillOpt on every path', () => {});});});++describe('E1 optimizer mode (GBRA-274)', () => {+ test('CLI, MCP op and cycle phase run one-shot rewrite by default; --loop opts into the reflect loop', async () => {+ await withEnv(ENV, async () => {+ expect((await viaCli()).optimizerMode).toBe('one-shot-rewrite');+ expect((await viaCli(['--loop'])).optimizerMode).toBe('reflect');+ expect(parseFlags([SKILL, '--loop']).loop).toBe(true);+ expect((await viaMcp()).optimizerMode).toBe('one-shot-rewrite');+ expect((await viaCycle()).optimizerMode).toBe('one-shot-rewrite');+ expect((await viaJob()).optimizerMode).toBe('reflect');+ });+ });+});+