Skip to content

The design language that makes your AI harness better at design.

JavaScript

Raise the skill-behavior timeout that was grading haste over thoroughness

Paul Bakaus65de2d294b5c
diff --git a/scripts/test-suites.mjs b/scripts/test-suites.mjs
index 608cea352..c488973e1 100644
--- a/scripts/test-suites.mjs
+++ b/scripts/test-suites.mjs
@@ -311,7 +311,15 @@ export const SUITES = {
commands: [
{
runner: 'node',
- timeoutMs: 300000,
+ // 300000 was too low to measure what these scenarios assert. The
+ // workflow-contract turns run 20+ steps against a frontier model, and
+ // the *correct* path is the slow one: a run that stops to put the
+ // concept to the user before building was measured at 579s, while the
+ // runs that skipped that checkpoint and failed the assertion finished
+ // in 130-200s. At a 300s cap the thorough path is killed and the hasty
+ // path is graded, so the cap was selecting for the behavior the suite
+ // exists to forbid.
+ timeoutMs: 900000,
files: [
'tests/skill-behavior/scenarios.test.mjs',
'tests/skill-behavior/workflow-contract.test.mjs',
diff --git a/tests/skill-behavior/README.md b/tests/skill-behavior/README.md
index d3ff2c5c4..01054e417 100644
--- a/tests/skill-behavior/README.md
+++ b/tests/skill-behavior/README.md
@@ -86,15 +86,24 @@ stay here because they are the record of what a weaker model does with this text
and that is the useful part. Reproduce with
`IMPECCABLE_SKILL_BEHAVIOR_MODELS=gpt-5.6-luna,deepseek-v4-flash`.
-Against the current default lineup, three cells are the known floor:
-`redesign replaces DESIGN` is flaky on every model, `critique closes` is flaky on
-gemini-3.6-flash, and `initialized natural build` fails on claude-sonnet-5.
-A regression is a failure beyond those three.
+Against the current default lineup, two cells are the known floor:
+`redesign replaces DESIGN` is flaky on every model, and `critique closes` is
+flaky on gemini-3.6-flash. A regression is a failure beyond those two.
+
+**Read any failure against the clock before calling it behavior.** The suite ran
+at a 300s per-test timeout until 2026-08-13, and for the workflow-contract
+scenarios that cap was below the runtime of a correct run. `initialized natural
+build` on claude-sonnet-5 was measured at 579s when it stopped to put the
+concept to the user before building, while the runs that skipped that checkpoint
+and failed the assertion finished in 130-200s. The cap was therefore selecting
+for the behavior the scenario forbids: thorough runs were killed, hasty ones
+were graded. The timeout is now 900s (`scripts/test-suites.mjs`). A duration at
+or just past the cap is a timeout, not a verdict.
| Scenario | claude-sonnet-5 | gpt-5.6-terra | gemini-3.6-flash | luna / deepseek (dropped) |
|---|---|---|---|---|
| attended fresh init | pass | pass | pass | not measured |
-| initialized natural build | **fail (3 of 3)** | pass | pass | not measured |
+| initialized natural build | flaky (1 of 4, and see the clock note) | pass | pass | not measured |
| redesign replaces DESIGN | flaky (timeout this run) | **fail** | **fail (timeout)** | not measured |
| bolder refinement | pass | pass | pass (on 3.5) | luna pass, deepseek **fail** |
| critique closes | pass (4 of 4) | pass (3 of 3) | **flaky (1 of 4)** | luna **fail (1 of 6)**, deepseek flaky |
@@ -113,24 +122,36 @@ problem that one model's priors expose rather than as a model floor.
| Scenario | claude-sonnet-5 | gpt-5.6-terra | gemini-3.6-flash |
|---|---|---|---|
-| 1-7, 10, 12-14 | pass | pass | pass |
+| 1-7, 10, 12-15 | pass | pass | pass |
| 8 (SvelteKit exploration) | flaky | pass | pass |
-| 9 (update surfaced, never auto-run) | **fail (3 of 3)** | pass | pass |
| 11 (shape resolves the build gate) | flaky | pass | pass |
-| 15 (native audit variant) | **fail (3 of 3)** | pass | pass |
-
-**Scenarios 9 and 15 and `initialized natural build` fail on unmodified `main`.**
-Confirmed against a clean worktree at `ddd23b18` with the same model and prompt:
-same assertion, same shape. They are open defects in the current text, not
-regressions from whatever change you are testing. Scenario 9 fails by auto-running
-the skill update instead of surfacing it; scenario 15 loads `audit.md` where the
-iOS platform should route it to `audit.native.md`; the natural-build contract
-begins implementation before the attended concept checkpoint. Check them against
-`main` before attributing any of the three to your branch.
Scenarios 8 and 11 pass on re-run, so treat a single failure there as flake and
confirm with a second run before investigating.
+Scenarios 9 and 15 both failed on sonnet when this baseline was first measured,
+and the two causes are worth keeping because neither was where it looked:
+
+- **9 was a real defect in the directive.** `UPDATE_AVAILABLE` said to ask the
+ user, then "If they agree, run `npx impeccable update`", then to continue
+ without waiting. With no wait there is no agreement to read, so the command
+ was the only concrete instruction left standing and sonnet ran it. Fixed by
+ removing the command from the turn entirely rather than by strengthening the
+ warning around it.
+- **15 was a broken fixture.** The iOS workspace held PRODUCT.md and nothing
+ else, so `audit the app in this workspace` named an app that did not exist.
+ Sonnet spent its whole step budget looking for it and read no reference file
+ at all, which the assertion reported as "loaded `audit.md` instead of the
+ variant". The fixture now ships one SwiftUI screen, the same courtesy
+ `MINIMAL_LANDING_HTML` already did for the web scenarios. The scenario passes
+ on unmodified `main` once the fixture is answerable, which is the proof the
+ routing text was never at fault.
+
+The general lesson is worth more than either fix: **an assertion reports the
+property it checks, not the reason it failed.** Both of these read as routing
+defects and neither was one. Pull the trace before writing the diagnosis, and
+prefer `IMPECCABLE_SKILL_BEHAVIOR_VERBOSE=1` over inference from the message.
+
Gemini cells marked `on 3.5` were measured on the superseded `gemini-3.5-flash`
and have not been re-run on 3.6. That distinction is not pedantic. `critique
closes` passed twice on 3.5-flash, then failed three times in a row on 3.6-flash