The open alternative to Salesforce, designed for AI.
TypeScript crmcrm-systemcustomergood-first-issuegraphqlhacktoberfestjavascriptmarketingmonoreponestjsopen-sourcepostgresqlreactreactjssalestypescriptweb
Make QA Scout failures quiet on PRs and legible in CI (#25060)
Félix Malfait73a451d841b3
diff --git a/.claude/skills/qa-scout/SKILL.md b/.claude/skills/qa-scout/SKILL.mdindex c69f6d975df..3ce4c29da77 100644--- a/.claude/skills/qa-scout/SKILL.md+++ b/.claude/skills/qa-scout/SKILL.md@@ -44,7 +44,7 @@ own browser observations unreliable.its noise is not yours. Only lines after your offsets count as evidence.2. **Scope from the diff.** Read `pr.json` and `files.json`; Grep and read- `pr.diff` selectively rather than end to end. Pick 2 to 5 user-visible+ `pr.diff` selectively rather than end to end. Pick 2 to 4 user-visiblescenarios this change could plausibly break. Bias toward:- writes over reads;- cross-object side effects (timeline entries, search, favorites,@@ -55,13 +55,14 @@ own browser observations unreliable.types only), write a PASS verdict with an empty scenario list saying why,and stop. Do not perform browser theater.-3. **Sanity-check the app, then log in.** If `http://localhost:3000` does not- respond, write a FAIL verdict with headline "app did not boot" immediately;- do not burn time. The app redirects to workspace subdomains- (`http://app.localhost:3000`, then `http://apple.localhost:3000` after- picking the workspace); those are in scope. Open the base URL, click- "Continue with Email" if visible, enter the email, Continue, enter the- password, Sign in, and pick the `Apple` workspace when asked.+3. **Sanity-check the app, then log in.** Navigate to the app with the+ browser; you have no `curl`. If it does not load, write a FAIL verdict with+ headline "app did not boot" immediately; do not burn time. The app+ redirects to workspace subdomains (`http://app.localhost:3000`, then+ `http://apple.localhost:3000` once the workspace is picked); those are in+ scope. Open the base URL, click "Continue with Email" if visible, enter the+ email, Continue, enter the password, Sign in, and pick the `Apple`+ workspace when asked.4. **Execute each scenario.** Use the Playwright tools: snapshot, act, verifythe outcome a user would check (the record exists, the value stuck, no error@@ -76,20 +77,33 @@ own browser observations unreliable.looked fine. Do not blame yourself for noise that predates your offsets.6. **Collect evidence.** Take a screenshot at each scenario's end state and at- every failure, with descriptive filenames (`03-note-timeline-missing.png`).+ every failure, giving the browser tool an absolute path under+ `/tmp/qa-scout/browser/` and a descriptive filename+ (`03-note-timeline-missing.png`). That directory ships with the report, so+ never copy or move screenshots afterwards.## Verdict contract-Always write both files to the output directory, whatever happens. Write-first versions right after scoping, before the first scenario (verdict-INVESTIGATE, headline "run still in progress", scenarios listed as pending),-then update both after each scenario and finalize at the end: a run that dies-on turns or time then leaves your last known state instead of silence.+Always write both files to the output directory, whatever happens, and keep+them current: write first versions right after scoping (`status`+`in-progress`, verdict INVESTIGATE, headline "run still in progress",+scenarios listed as pending), rewrite both immediately after each scenario+with what you now know, and set `status` to `final` once you stop testing.+Stopping early still counts as stopping: a verdict you reach without running+scenarios, such as "app did not boot" or a change with no user-visible+surface, is `final` the moment you write it.++`status` decides who hears you. A `final` verdict is reported as a result;+an `in-progress` one means the run died, so it stays out of the PR unless it+already recorded a failing scenario. Two consequences: never mark `final`+before you are done, and never leave a real failure sitting in an+unfinished file, because a checkpoint no scenario has failed in is silence.`verdict.json`:```json{+ "status": "in-progress | final","verdict": "PASS | INVESTIGATE | FAIL","headline": "one sentence, user language","prNumber": 12345,@@ -123,9 +137,15 @@ on turns or time then leaves your last known state instead of silence.the report.- Never navigate outside `localhost:3000` and its `*.localhost:3000`workspace subdomains.-- Budget roughly 12 minutes of browsing. Three scenarios done well beat eight- done badly. Out of time means INVESTIGATE with what you saw, not silence.-- Do not modify the repository. Write only inside the output directory.+- Two to four scenarios, finalized by roughly the 10-minute mark. Three done+ well beat eight done badly, and a finished verdict on two beats an+ unfinished one on five.+- Your shell is a small allowlist, and a denied command costs a turn you+ needed for testing: `jq` reads JSON (not `python3` or `node`), `tail`,+ `head`, `grep` and `wc` read logs, `psql` reads the database. There is no+ `curl`, `cp`, `mv`, `find` or `git`.+- Do not modify the repository. Write your outputs to the output directory;+ screenshots belong in the browser directory above.## Running locallydiff --git a/.github/workflows/ci-e2e-main.yaml b/.github/workflows/ci-e2e-main.yamlindex 968eddea1e5..eb7b18a1fc8 100644--- a/.github/workflows/ci-e2e-main.yaml+++ b/.github/workflows/ci-e2e-main.yaml@@ -292,7 +292,10 @@ jobs:npm install -g @anthropic-ai/[email protected]# stream-json teed into the artifact tree, ahead of the scrub,# keeps per-turn detail and permission denials so allowlist tuning- # reads real data.+ # reads real data. stderr and the exit code are kept too: a run that+ # never reaches a result line (bad flag, exhausted quota) explains+ # itself only there, and the summary step reads both back.+ set +eclaude -p "Read .claude/skills/qa-scout/SKILL.md and follow it exactly. This is the CI invocation: every path and credential in the skill's Inputs table applies verbatim." \--max-turns 150 \--model opus \@@ -301,7 +304,12 @@ jobs:--mcp-config /tmp/qa-scout/mcp.json \--allowedTools "Read,Grep,Glob,Write,TodoWrite,Bash(cat *),Bash(tail *),Bash(head *),Bash(wc *),Bash(ls *),Bash(grep *),Bash(jq *),Bash(echo *),Bash(diff *),Bash(sort *),Bash(uniq *),Bash(cut *),Bash(tr *),Bash(mkdir *),Bash(sleep *),Bash(date *),Bash(psql *),mcp__playwright" \--output-format stream-json --verbose \+ 2> >(tee /tmp/qa-scout/agent-stderr.log >&2) \| tee /tmp/qa-scout/execution.json+ AGENT_EXIT=${PIPESTATUS[0]}+ set -e+ printf '%s' "$AGENT_EXIT" > /tmp/qa-scout/agent-exit.txt+ exit "$AGENT_EXIT"# The agent process necessarily holds the inference credential, and a# prompt-injected agent could copy it into any file it can write. The@@ -332,20 +340,67 @@ jobs:SCOUT_RUN: ${{ steps.qa-scout-pr.outputs.run }}SCOUT_PREP_OUTCOME: ${{ steps.qa-scout-pr.outcome }}run: |+ # Why the run ended, from the evidence the agent leaves behind: the+ # final stream-json result when it produced one, stderr when it died+ # before that. Without this the summary can only say "no report".+ scout_failure_reason() {+ local result_line exit_code+ exit_code=$(cat /tmp/qa-scout/agent-exit.txt 2>/dev/null || echo "unknown")+ echo "**Why it stopped** (exit code \`${exit_code}\`)"+ echo ""+ result_line=$(grep -a '"type":"result"' /tmp/qa-scout/execution.json 2>/dev/null | tail -1 || true)+ if [ -n "$result_line" ]; then+ printf '%s' "$result_line" | jq -r '+ "- outcome: `\(.subtype // "unknown")`" ++ " · \(.num_turns // "?") turns" ++ " · \(((.duration_ms // 0) / 60000 * 10 | round) / 10) min" ++ " · $\(((.total_cost_usd // 0) * 100 | round) / 100)",+ (if ((.errors? // []) | length) > 0 then "- error: " + ((.errors | join("; "))) else empty end)+ ' 2>/dev/null || echo "- the final result line could not be parsed; see the artifact"+ else+ echo "- the agent produced no result line, so it failed before or during startup"+ fi+ if [ -s /tmp/qa-scout/agent-stderr.log ]; then+ echo ""+ echo "<details><summary>stderr (last 10 lines)</summary>"+ echo ""+ echo '```'+ tail -n 10 /tmp/qa-scout/agent-stderr.log+ echo '```'+ echo ""+ echo "</details>"+ fi+ }{echo "## QA Scout"if [ -f /tmp/qa-scout/output/report.md ]; then+ if [ "$(jq -r '.status // "final"' /tmp/qa-scout/output/verdict.json 2>/dev/null || echo final)" != "final" ]; then+ echo "> [!WARNING]"+ echo "> The Scout stopped before finishing; the partial state it left follows."+ echo ""+ scout_failure_reason+ echo ""+ ficat /tmp/qa-scout/output/report.mdelif [ -f /tmp/qa-scout/output/verdict.json ]; then- jq -r '"**\(.verdict)**: \(.headline // "no headline recorded")\n\n_Full report missing (agent stopped early); details in the qa-scout-report artifact._"' /tmp/qa-scout/output/verdict.json+ jq -r '"**\(.verdict)**: \(.headline // "no headline recorded")\n\n_Full report missing (agent stopped early)._"' /tmp/qa-scout/output/verdict.json+ echo ""+ scout_failure_reasonelif [ "$SCOUT_RUN" = "true" ]; then- echo "No report produced (agent run failed or timed out); see the qa-scout-report artifact and step logs."+ echo "The Scout produced no verdict."+ echo ""+ scout_failure_reasonelif [ "$SCOUT_PREP_OUTCOME" = "failure" ]; thenecho "Did not run: the prepare step failed; see its logs."elseecho "Did not run: skipped by the prepare step (no merged PR, [no-qa], or i18n translation PR)."fi} >> "$GITHUB_STEP_SUMMARY"+ # Surfaced on the run page too, so a dead Scout is visible without+ # opening the summary.+ if [ "$SCOUT_RUN" = "true" ] && [ ! -f /tmp/qa-scout/output/report.md ]; then+ echo "::warning title=QA Scout produced no report::$(head -n 1 /tmp/qa-scout/agent-stderr.log 2>/dev/null || echo 'see the job summary for why')"+ fi- name: QA Scout / Upload reportif: ${{ !cancelled() && steps.qa-scout-pr.outputs.run == 'true' }}@@ -360,6 +415,8 @@ jobs:/tmp/qa-scout/server.log/tmp/qa-scout/worker.log/tmp/qa-scout/execution.json+ /tmp/qa-scout/agent-stderr.log+ /tmp/qa-scout/agent-exit.txtif-no-files-found: ignoreretention-days: 14@@ -449,6 +506,17 @@ jobs:exit 0;;esac+ # A checkpoint left by a run that died is not a finding: it says+ # the Scout failed, not that the PR is broken. Only a finished run+ # speaks, unless the partial state already caught something real: a+ # failed scenario, or a top-level FAIL (which is how a boot failure+ # arrives, before any scenario exists to fail).+ STATUS=$(jq -r '.status // "final"' "$VERDICT_FILE" 2>/dev/null || echo final)+ FAILED_COUNT=$(jq '[.scenarios[]? | select(.result == "fail")] | length' "$VERDICT_FILE" 2>/dev/null || echo 0)+ if [ "$STATUS" != "final" ] && [ "${FAILED_COUNT:-0}" -eq 0 ] && [ "$VERDICT" != "FAIL" ]; then+ echo "Scout did not finish (status=${STATUS}) and recorded no failing scenario; nothing to post. See the job summary and artifact."+ exit 0+ fiARTIFACT_PR=$(jq -r '.prNumber // empty' "$VERDICT_FILE")if [ "$ARTIFACT_PR" != "$PR_NUMBER" ]; thenecho "verdict.json prNumber (${ARTIFACT_PR:-none}) differs from the trusted target #${PR_NUMBER}; the trusted target wins."@@ -476,6 +544,11 @@ jobs:fi{echo "<!-- qa-scout -->"+ if [ "$STATUS" != "final" ]; then+ echo "> [!WARNING]"+ echo "> The QA Scout stopped before finishing; what follows is partial."+ echo ""+ fi# Drops a multi-byte character split by the byte cap so the payload# stays valid UTF-8 (iconv -c exits 1 on exactly this case, so# python does the lenient decode).