Skip to content

The open alternative to Salesforce, designed for AI.

Make QA Scout failures quiet on PRs and legible in CI (#25060)

Félix Malfait73a451d841b3
diff --git a/.claude/skills/qa-scout/SKILL.md b/.claude/skills/qa-scout/SKILL.md
index c69f6d975df..3ce4c29da77 100644
--- a/.claude/skills/qa-scout/SKILL.md
+++ b/.claude/skills/qa-scout/SKILL.md
@@ -44,7 +44,7 @@ own browser observations unreliable.
its noise is not yours. Only lines after your offsets count as evidence.
2. **Scope from the diff.** Read `pr.json` and `files.json`; Grep and read
- `pr.diff` selectively rather than end to end. Pick 2 to 5 user-visible
+ `pr.diff` selectively rather than end to end. Pick 2 to 4 user-visible
scenarios this change could plausibly break. Bias toward:
- writes over reads;
- cross-object side effects (timeline entries, search, favorites,
@@ -55,13 +55,14 @@ own browser observations unreliable.
types only), write a PASS verdict with an empty scenario list saying why,
and stop. Do not perform browser theater.
-3. **Sanity-check the app, then log in.** If `http://localhost:3000` does not
- respond, write a FAIL verdict with headline "app did not boot" immediately;
- do not burn time. The app redirects to workspace subdomains
- (`http://app.localhost:3000`, then `http://apple.localhost:3000` after
- picking the workspace); those are in scope. Open the base URL, click
- "Continue with Email" if visible, enter the email, Continue, enter the
- password, Sign in, and pick the `Apple` workspace when asked.
+3. **Sanity-check the app, then log in.** Navigate to the app with the
+ browser; you have no `curl`. If it does not load, write a FAIL verdict with
+ headline "app did not boot" immediately; do not burn time. The app
+ redirects to workspace subdomains (`http://app.localhost:3000`, then
+ `http://apple.localhost:3000` once the workspace is picked); those are in
+ scope. Open the base URL, click "Continue with Email" if visible, enter the
+ email, Continue, enter the password, Sign in, and pick the `Apple`
+ workspace when asked.
4. **Execute each scenario.** Use the Playwright tools: snapshot, act, verify
the outcome a user would check (the record exists, the value stuck, no error
@@ -76,20 +77,33 @@ own browser observations unreliable.
looked fine. Do not blame yourself for noise that predates your offsets.
6. **Collect evidence.** Take a screenshot at each scenario's end state and at
- every failure, with descriptive filenames (`03-note-timeline-missing.png`).
+ every failure, giving the browser tool an absolute path under
+ `/tmp/qa-scout/browser/` and a descriptive filename
+ (`03-note-timeline-missing.png`). That directory ships with the report, so
+ never copy or move screenshots afterwards.
## Verdict contract
-Always write both files to the output directory, whatever happens. Write
-first versions right after scoping, before the first scenario (verdict
-INVESTIGATE, headline "run still in progress", scenarios listed as pending),
-then update both after each scenario and finalize at the end: a run that dies
-on turns or time then leaves your last known state instead of silence.
+Always write both files to the output directory, whatever happens, and keep
+them current: write first versions right after scoping (`status`
+`in-progress`, verdict INVESTIGATE, headline "run still in progress",
+scenarios listed as pending), rewrite both immediately after each scenario
+with what you now know, and set `status` to `final` once you stop testing.
+Stopping early still counts as stopping: a verdict you reach without running
+scenarios, such as "app did not boot" or a change with no user-visible
+surface, is `final` the moment you write it.
+
+`status` decides who hears you. A `final` verdict is reported as a result;
+an `in-progress` one means the run died, so it stays out of the PR unless it
+already recorded a failing scenario. Two consequences: never mark `final`
+before you are done, and never leave a real failure sitting in an
+unfinished file, because a checkpoint no scenario has failed in is silence.
`verdict.json`:
```json
{
+ "status": "in-progress | final",
"verdict": "PASS | INVESTIGATE | FAIL",
"headline": "one sentence, user language",
"prNumber": 12345,
@@ -123,9 +137,15 @@ on turns or time then leaves your last known state instead of silence.
the report.
- Never navigate outside `localhost:3000` and its `*.localhost:3000`
workspace subdomains.
-- Budget roughly 12 minutes of browsing. Three scenarios done well beat eight
- done badly. Out of time means INVESTIGATE with what you saw, not silence.
-- Do not modify the repository. Write only inside the output directory.
+- Two to four scenarios, finalized by roughly the 10-minute mark. Three done
+ well beat eight done badly, and a finished verdict on two beats an
+ unfinished one on five.
+- Your shell is a small allowlist, and a denied command costs a turn you
+ needed for testing: `jq` reads JSON (not `python3` or `node`), `tail`,
+ `head`, `grep` and `wc` read logs, `psql` reads the database. There is no
+ `curl`, `cp`, `mv`, `find` or `git`.
+- Do not modify the repository. Write your outputs to the output directory;
+ screenshots belong in the browser directory above.
## Running locally
diff --git a/.github/workflows/ci-e2e-main.yaml b/.github/workflows/ci-e2e-main.yaml
index 968eddea1e5..eb7b18a1fc8 100644
--- a/.github/workflows/ci-e2e-main.yaml
+++ b/.github/workflows/ci-e2e-main.yaml
@@ -292,7 +292,10 @@ jobs:
npm install -g @anthropic-ai/[email protected]
# stream-json teed into the artifact tree, ahead of the scrub,
# keeps per-turn detail and permission denials so allowlist tuning
- # reads real data.
+ # reads real data. stderr and the exit code are kept too: a run that
+ # never reaches a result line (bad flag, exhausted quota) explains
+ # itself only there, and the summary step reads both back.
+ set +e
claude -p "Read .claude/skills/qa-scout/SKILL.md and follow it exactly. This is the CI invocation: every path and credential in the skill's Inputs table applies verbatim." \
--max-turns 150 \
--model opus \
@@ -301,7 +304,12 @@ jobs:
--mcp-config /tmp/qa-scout/mcp.json \
--allowedTools "Read,Grep,Glob,Write,TodoWrite,Bash(cat *),Bash(tail *),Bash(head *),Bash(wc *),Bash(ls *),Bash(grep *),Bash(jq *),Bash(echo *),Bash(diff *),Bash(sort *),Bash(uniq *),Bash(cut *),Bash(tr *),Bash(mkdir *),Bash(sleep *),Bash(date *),Bash(psql *),mcp__playwright" \
--output-format stream-json --verbose \
+ 2> >(tee /tmp/qa-scout/agent-stderr.log >&2) \
| tee /tmp/qa-scout/execution.json
+ AGENT_EXIT=${PIPESTATUS[0]}
+ set -e
+ printf '%s' "$AGENT_EXIT" > /tmp/qa-scout/agent-exit.txt
+ exit "$AGENT_EXIT"
# The agent process necessarily holds the inference credential, and a
# prompt-injected agent could copy it into any file it can write. The
@@ -332,20 +340,67 @@ jobs:
SCOUT_RUN: ${{ steps.qa-scout-pr.outputs.run }}
SCOUT_PREP_OUTCOME: ${{ steps.qa-scout-pr.outcome }}
run: |
+ # Why the run ended, from the evidence the agent leaves behind: the
+ # final stream-json result when it produced one, stderr when it died
+ # before that. Without this the summary can only say "no report".
+ scout_failure_reason() {
+ local result_line exit_code
+ exit_code=$(cat /tmp/qa-scout/agent-exit.txt 2>/dev/null || echo "unknown")
+ echo "**Why it stopped** (exit code \`${exit_code}\`)"
+ echo ""
+ result_line=$(grep -a '"type":"result"' /tmp/qa-scout/execution.json 2>/dev/null | tail -1 || true)
+ if [ -n "$result_line" ]; then
+ printf '%s' "$result_line" | jq -r '
+ "- outcome: `\(.subtype // "unknown")`" +
+ " · \(.num_turns // "?") turns" +
+ " · \(((.duration_ms // 0) / 60000 * 10 | round) / 10) min" +
+ " · $\(((.total_cost_usd // 0) * 100 | round) / 100)",
+ (if ((.errors? // []) | length) > 0 then "- error: " + ((.errors | join("; "))) else empty end)
+ ' 2>/dev/null || echo "- the final result line could not be parsed; see the artifact"
+ else
+ echo "- the agent produced no result line, so it failed before or during startup"
+ fi
+ if [ -s /tmp/qa-scout/agent-stderr.log ]; then
+ echo ""
+ echo "<details><summary>stderr (last 10 lines)</summary>"
+ echo ""
+ echo '```'
+ tail -n 10 /tmp/qa-scout/agent-stderr.log
+ echo '```'
+ echo ""
+ echo "</details>"
+ fi
+ }
{
echo "## QA Scout"
if [ -f /tmp/qa-scout/output/report.md ]; then
+ if [ "$(jq -r '.status // "final"' /tmp/qa-scout/output/verdict.json 2>/dev/null || echo final)" != "final" ]; then
+ echo "> [!WARNING]"
+ echo "> The Scout stopped before finishing; the partial state it left follows."
+ echo ""
+ scout_failure_reason
+ echo ""
+ fi
cat /tmp/qa-scout/output/report.md
elif [ -f /tmp/qa-scout/output/verdict.json ]; then
- jq -r '"**\(.verdict)**: \(.headline // "no headline recorded")\n\n_Full report missing (agent stopped early); details in the qa-scout-report artifact._"' /tmp/qa-scout/output/verdict.json
+ jq -r '"**\(.verdict)**: \(.headline // "no headline recorded")\n\n_Full report missing (agent stopped early)._"' /tmp/qa-scout/output/verdict.json
+ echo ""
+ scout_failure_reason
elif [ "$SCOUT_RUN" = "true" ]; then
- echo "No report produced (agent run failed or timed out); see the qa-scout-report artifact and step logs."
+ echo "The Scout produced no verdict."
+ echo ""
+ scout_failure_reason
elif [ "$SCOUT_PREP_OUTCOME" = "failure" ]; then
echo "Did not run: the prepare step failed; see its logs."
else
echo "Did not run: skipped by the prepare step (no merged PR, [no-qa], or i18n translation PR)."
fi
} >> "$GITHUB_STEP_SUMMARY"
+ # Surfaced on the run page too, so a dead Scout is visible without
+ # opening the summary.
+ if [ "$SCOUT_RUN" = "true" ] && [ ! -f /tmp/qa-scout/output/report.md ]; then
+ echo "::warning title=QA Scout produced no report::$(head -n 1 /tmp/qa-scout/agent-stderr.log 2>/dev/null || echo 'see the job summary for why')"
+ fi
- name: QA Scout / Upload report
if: ${{ !cancelled() && steps.qa-scout-pr.outputs.run == 'true' }}
@@ -360,6 +415,8 @@ jobs:
/tmp/qa-scout/server.log
/tmp/qa-scout/worker.log
/tmp/qa-scout/execution.json
+ /tmp/qa-scout/agent-stderr.log
+ /tmp/qa-scout/agent-exit.txt
if-no-files-found: ignore
retention-days: 14
@@ -449,6 +506,17 @@ jobs:
exit 0
;;
esac
+ # A checkpoint left by a run that died is not a finding: it says
+ # the Scout failed, not that the PR is broken. Only a finished run
+ # speaks, unless the partial state already caught something real: a
+ # failed scenario, or a top-level FAIL (which is how a boot failure
+ # arrives, before any scenario exists to fail).
+ STATUS=$(jq -r '.status // "final"' "$VERDICT_FILE" 2>/dev/null || echo final)
+ FAILED_COUNT=$(jq '[.scenarios[]? | select(.result == "fail")] | length' "$VERDICT_FILE" 2>/dev/null || echo 0)
+ if [ "$STATUS" != "final" ] && [ "${FAILED_COUNT:-0}" -eq 0 ] && [ "$VERDICT" != "FAIL" ]; then
+ echo "Scout did not finish (status=${STATUS}) and recorded no failing scenario; nothing to post. See the job summary and artifact."
+ exit 0
+ fi
ARTIFACT_PR=$(jq -r '.prNumber // empty' "$VERDICT_FILE")
if [ "$ARTIFACT_PR" != "$PR_NUMBER" ]; then
echo "verdict.json prNumber (${ARTIFACT_PR:-none}) differs from the trusted target #${PR_NUMBER}; the trusted target wins."
@@ -476,6 +544,11 @@ jobs:
fi
{
echo "<!-- qa-scout -->"
+ if [ "$STATUS" != "final" ]; then
+ echo "> [!WARNING]"
+ echo "> The QA Scout stopped before finishing; what follows is partial."
+ echo ""
+ fi
# Drops a multi-byte character split by the byte cap so the payload
# stays valid UTF-8 (iconv -c exits 1 on exactly this case, so
# python does the lenient decode).