The design language that makes your AI harness better at design.
JavaScript
Raise the skill-behavior timeout that was grading haste over thoroughness
Paul Bakaus65de2d294b5c
diff --git a/scripts/test-suites.mjs b/scripts/test-suites.mjsindex 608cea352..c488973e1 100644--- a/scripts/test-suites.mjs+++ b/scripts/test-suites.mjs@@ -311,7 +311,15 @@ export const SUITES = {commands: [{runner: 'node',- timeoutMs: 300000,+ // 300000 was too low to measure what these scenarios assert. The+ // workflow-contract turns run 20+ steps against a frontier model, and+ // the *correct* path is the slow one: a run that stops to put the+ // concept to the user before building was measured at 579s, while the+ // runs that skipped that checkpoint and failed the assertion finished+ // in 130-200s. At a 300s cap the thorough path is killed and the hasty+ // path is graded, so the cap was selecting for the behavior the suite+ // exists to forbid.+ timeoutMs: 900000,files: ['tests/skill-behavior/scenarios.test.mjs','tests/skill-behavior/workflow-contract.test.mjs',diff --git a/tests/skill-behavior/README.md b/tests/skill-behavior/README.mdindex d3ff2c5c4..01054e417 100644--- a/tests/skill-behavior/README.md+++ b/tests/skill-behavior/README.md@@ -86,15 +86,24 @@ stay here because they are the record of what a weaker model does with this textand that is the useful part. Reproduce with`IMPECCABLE_SKILL_BEHAVIOR_MODELS=gpt-5.6-luna,deepseek-v4-flash`.-Against the current default lineup, three cells are the known floor:-`redesign replaces DESIGN` is flaky on every model, `critique closes` is flaky on-gemini-3.6-flash, and `initialized natural build` fails on claude-sonnet-5.-A regression is a failure beyond those three.+Against the current default lineup, two cells are the known floor:+`redesign replaces DESIGN` is flaky on every model, and `critique closes` is+flaky on gemini-3.6-flash. A regression is a failure beyond those two.++**Read any failure against the clock before calling it behavior.** The suite ran+at a 300s per-test timeout until 2026-08-13, and for the workflow-contract+scenarios that cap was below the runtime of a correct run. `initialized natural+build` on claude-sonnet-5 was measured at 579s when it stopped to put the+concept to the user before building, while the runs that skipped that checkpoint+and failed the assertion finished in 130-200s. The cap was therefore selecting+for the behavior the scenario forbids: thorough runs were killed, hasty ones+were graded. The timeout is now 900s (`scripts/test-suites.mjs`). A duration at+or just past the cap is a timeout, not a verdict.| Scenario | claude-sonnet-5 | gpt-5.6-terra | gemini-3.6-flash | luna / deepseek (dropped) ||---|---|---|---|---|| attended fresh init | pass | pass | pass | not measured |-| initialized natural build | **fail (3 of 3)** | pass | pass | not measured |+| initialized natural build | flaky (1 of 4, and see the clock note) | pass | pass | not measured || redesign replaces DESIGN | flaky (timeout this run) | **fail** | **fail (timeout)** | not measured || bolder refinement | pass | pass | pass (on 3.5) | luna pass, deepseek **fail** || critique closes | pass (4 of 4) | pass (3 of 3) | **flaky (1 of 4)** | luna **fail (1 of 6)**, deepseek flaky |@@ -113,24 +122,36 @@ problem that one model's priors expose rather than as a model floor.| Scenario | claude-sonnet-5 | gpt-5.6-terra | gemini-3.6-flash ||---|---|---|---|-| 1-7, 10, 12-14 | pass | pass | pass |+| 1-7, 10, 12-15 | pass | pass | pass || 8 (SvelteKit exploration) | flaky | pass | pass |-| 9 (update surfaced, never auto-run) | **fail (3 of 3)** | pass | pass || 11 (shape resolves the build gate) | flaky | pass | pass |-| 15 (native audit variant) | **fail (3 of 3)** | pass | pass |--**Scenarios 9 and 15 and `initialized natural build` fail on unmodified `main`.**-Confirmed against a clean worktree at `ddd23b18` with the same model and prompt:-same assertion, same shape. They are open defects in the current text, not-regressions from whatever change you are testing. Scenario 9 fails by auto-running-the skill update instead of surfacing it; scenario 15 loads `audit.md` where the-iOS platform should route it to `audit.native.md`; the natural-build contract-begins implementation before the attended concept checkpoint. Check them against-`main` before attributing any of the three to your branch.Scenarios 8 and 11 pass on re-run, so treat a single failure there as flake andconfirm with a second run before investigating.+Scenarios 9 and 15 both failed on sonnet when this baseline was first measured,+and the two causes are worth keeping because neither was where it looked:++- **9 was a real defect in the directive.** `UPDATE_AVAILABLE` said to ask the+ user, then "If they agree, run `npx impeccable update`", then to continue+ without waiting. With no wait there is no agreement to read, so the command+ was the only concrete instruction left standing and sonnet ran it. Fixed by+ removing the command from the turn entirely rather than by strengthening the+ warning around it.+- **15 was a broken fixture.** The iOS workspace held PRODUCT.md and nothing+ else, so `audit the app in this workspace` named an app that did not exist.+ Sonnet spent its whole step budget looking for it and read no reference file+ at all, which the assertion reported as "loaded `audit.md` instead of the+ variant". The fixture now ships one SwiftUI screen, the same courtesy+ `MINIMAL_LANDING_HTML` already did for the web scenarios. The scenario passes+ on unmodified `main` once the fixture is answerable, which is the proof the+ routing text was never at fault.++The general lesson is worth more than either fix: **an assertion reports the+property it checks, not the reason it failed.** Both of these read as routing+defects and neither was one. Pull the trace before writing the diagnosis, and+prefer `IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1` over inference from the message.+Gemini cells marked `on 3.5` were measured on the superseded `gemini-3.5-flash`and have not been re-run on 3.6. That distinction is not pedantic. `critiquecloses` passed twice on 3.5-flash, then failed three times in a row on 3.6-flash