Skip to content

The open alternative to Salesforce, designed for AI.

Fix QA Scout turn exhaustion found by the first live audit (#24934)

Félix Malfait94ca5a7c64ea
diff --git a/.claude/skills/qa-scout/SKILL.md b/.claude/skills/qa-scout/SKILL.md
index 42a835dcfbe..c69f6d975df 100644
--- a/.claude/skills/qa-scout/SKILL.md
+++ b/.claude/skills/qa-scout/SKILL.md
@@ -27,7 +27,7 @@ The invoking prompt gives you concrete paths. In CI they are:
| Server log (live) | `/tmp/qa-scout/server.log` |
| Worker log (live) | `/tmp/qa-scout/worker.log` |
| Run mode | `/tmp/qa-scout/context/mode`: `post-merge` (the change is on main) or `pre-merge` (label-triggered validation of the PR head before merge) |
-| Database (disposable, full access) | `postgres://postgres:postgres@localhost:5432/default` via `psql` |
+| Database (disposable, full access) | `psql -h localhost -U postgres -d default` (PGPASSWORD is set; the URI `postgres://postgres:postgres@localhost:5432/default` also works) |
This environment is ephemeral, so unlike a shared instance you have full
power here: use `psql` to attest what the UI cannot show. After a write flow,
@@ -80,7 +80,11 @@ own browser observations unreliable.
## Verdict contract
-Always write both files to the output directory, whatever happens:
+Always write both files to the output directory, whatever happens. Write
+first versions right after scoping, before the first scenario (verdict
+INVESTIGATE, headline "run still in progress", scenarios listed as pending),
+then update both after each scenario and finalize at the end: a run that dies
+on turns or time then leaves your last known state instead of silence.
`verdict.json`:
diff --git a/.github/workflows/ci-e2e-main.yaml b/.github/workflows/ci-e2e-main.yaml
index 7633d008422..968eddea1e5 100644
--- a/.github/workflows/ci-e2e-main.yaml
+++ b/.github/workflows/ci-e2e-main.yaml
@@ -286,6 +286,7 @@ jobs:
timeout-minutes: 15
env:
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+ PGPASSWORD: postgres
run: |
set -euo pipefail
npm install -g @anthropic-ai/[email protected]
@@ -293,8 +294,10 @@ jobs:
# keeps per-turn detail and permission denials so allowlist tuning
# reads real data.
claude -p "Read .claude/skills/qa-scout/SKILL.md and follow it exactly. This is the CI invocation: every path and credential in the skill's Inputs table applies verbatim." \
- --max-turns 120 \
+ --max-turns 150 \
--model opus \
+ --add-dir /tmp/qa-scout \
+ --strict-mcp-config \
--mcp-config /tmp/qa-scout/mcp.json \
--allowedTools "Read,Grep,Glob,Write,TodoWrite,Bash(cat *),Bash(tail *),Bash(head *),Bash(wc *),Bash(ls *),Bash(grep *),Bash(jq *),Bash(echo *),Bash(diff *),Bash(sort *),Bash(uniq *),Bash(cut *),Bash(tr *),Bash(mkdir *),Bash(sleep *),Bash(date *),Bash(psql *),mcp__playwright" \
--output-format stream-json --verbose \
@@ -333,6 +336,8 @@ jobs:
echo "## QA Scout"
if [ -f /tmp/qa-scout/output/report.md ]; then
cat /tmp/qa-scout/output/report.md
+ elif [ -f /tmp/qa-scout/output/verdict.json ]; then
+ jq -r '"**\(.verdict)**: \(.headline // "no headline recorded")\n\n_Full report missing (agent stopped early); details in the qa-scout-report artifact._"' /tmp/qa-scout/output/verdict.json
elif [ "$SCOUT_RUN" = "true" ]; then
echo "No report produced (agent run failed or timed out); see the qa-scout-report artifact and step logs."
elif [ "$SCOUT_PREP_OUTCOME" = "failure" ]; then
@@ -425,10 +430,17 @@ jobs:
fi
VERDICT_FILE=$(find qa-scout-report -name verdict.json | head -1 || true)
REPORT_FILE=$(find qa-scout-report -name report.md | head -1 || true)
- if [ -z "$VERDICT_FILE" ] || [ -z "$REPORT_FILE" ]; then
- echo "No verdict or report in the artifact; nothing to post."
+ if [ -z "$VERDICT_FILE" ]; then
+ echo "No verdict in the artifact; nothing to post."
exit 0
fi
+ # A run killed on turns or time can leave the verdict without the
+ # report; a synthesized body beats silence on a FAIL.
+ if [ -z "$REPORT_FILE" ]; then
+ REPORT_FILE=report-fallback.md
+ jq -r '"**\(.verdict)**: \(.headline // "no headline recorded")\n\n" + ([.scenarios[]? | "- \(.name): \(.result)"] | join("\n")) + "\n\n_Full report missing (agent stopped early); details in the qa-scout-report artifact._"' \
+ "$VERDICT_FILE" > "$REPORT_FILE"
+ fi
VERDICT=$(jq -r '.verdict // empty' "$VERDICT_FILE")
case "$VERDICT" in
PASS | INVESTIGATE | FAIL) ;;