AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
dx(orchestrate): harden agent fleet scripts — idle detection, pagination, fake-resolution guard, parallelism (#12704)
Zamil Majdyc51097d8ac24
diff --git a/.claude/skills/orchestrate/SKILL.md b/.claude/skills/orchestrate/SKILL.mdindex 04b8af3ea1..6fbb3681ae 100644--- a/.claude/skills/orchestrate/SKILL.md+++ b/.claude/skills/orchestrate/SKILL.md@@ -323,8 +323,63 @@ For each agent, decide:| Stuck in error loop | Send targeted fix with exact error + solution || Waiting for input / question | Answer and unblock via `tmux send-keys` || CI red | `gh pr checks PR_NUMBER --repo REPO` → tell agent exactly what's failing |+| GitHub abuse rate limit error | Nudge: "Wait 60 seconds then continue posting replies with sleep 3 between each" || Context compacted / agent lost | Send recovery: `cat ~/.claude/orchestrator-state.json | jq '.agents[] | select(.window=="WIN")'` + `gh pr view PR_NUMBER --json title,body` |-| `ORCHESTRATOR:DONE` in output | Run `verify-complete.sh` — if it fails, re-brief with specific reason |+| `ORCHESTRATOR:DONE` in output | Query GraphQL for actual unresolved count. If >0, re-brief. If 0, run `verify-complete.sh` |++**Poll all windows from state, not from memory.** Before each poll, run:+```bash+jq -r '.agents[] | select(.state | test("running|idle|stuck|waiting_approval|pending_evaluation")) | .window' ~/.claude/orchestrator-state.json+```+and capture every window listed. If you manually added a window outside spawn-agent.sh, ensure it's in the state file first.++### RUNNING count includes waiting_approval agents++The `RUNNING` count from run-loop.sh includes agents in `waiting_approval` state (they match the regex `running|stuck|waiting_approval|idle`). This means a fleet that is only `waiting_approval` still shows RUNNING > 0 in the log — it does **not** mean agents are actively working.++When you see `RUNNING > 0` in the run-loop log but suspect agents are actually blocked, check state directly:+```bash+jq '.agents[] | {window, state, worktree}' ~/.claude/orchestrator-state.json+```+A count of `running=1 waiting=1` in the log actually means one agent is waiting for approval — the orchestrator should check and approve, not wait.++### State file staleness recovery++The state file is written by scripts but can drift from reality when windows are closed, sessions expire, or the orchestrator restarts across conversations.++**Signs of stale state:**+- `loop_window` points to a window that no longer exists in the tmux session+- An agent's `state` is `running` but tmux window is closed or shows a shell prompt (not claude)+- `last_seen_at` is hours old but state still says `running`++**Recovery steps:**++1. **Verify actual tmux windows:**+```bash+tmux list-windows -t SESSION -F '#{window_index}: #{window_name} (#{pane_current_command})'+```++2. **Cross-reference with state file:**+```bash+jq -r '.agents[] | "\(.window) \(.state) \(.worktree)"' ~/.claude/orchestrator-state.json+```++3. **Fix stale entries:**+```bash+# Agent window closed — mark idle so run-loop.sh will restart it+jq --arg w "SESSION:WIN" '(.agents[] | select(.window==$w)).state = "idle"' \+ ~/.claude/orchestrator-state.json > /tmp/orch.tmp && mv /tmp/orch.tmp ~/.claude/orchestrator-state.json++# loop_window gone — kill the stale reference, then restart run-loop.sh+jq '.loop_window = null' ~/.claude/orchestrator-state.json > /tmp/orch.tmp && mv /tmp/orch.tmp ~/.claude/orchestrator-state.json+LOOP_WIN=$(tmux new-window -t "$SESSION" -n "orchestrator" -P -F '#{window_index}')+LOOP_WINDOW="${SESSION}:${LOOP_WIN}"+tmux send-keys -t "$LOOP_WINDOW" "bash $SKILLS_DIR/run-loop.sh" Enter+jq --arg w "$LOOP_WINDOW" '.loop_window = $w' ~/.claude/orchestrator-state.json \+ > /tmp/orch.tmp && mv /tmp/orch.tmp ~/.claude/orchestrator-state.json+```++4. **After any state repair, re-run `status.sh` to confirm coherence before resuming supervision.**### Strict ORCHESTRATOR:DONE gate@@ -342,6 +397,14 @@ If it fails → re-brief the agent with the failure reason. Never manually mark### Re-brief a stalled agent+**Before sending any nudge, verify the pane is at an idle ❯ prompt.** Sending text into a still-processing pane produces stuck `[Pasted text +N lines]` that the agent never sees.++Check:+```bash+tmux capture-pane -t SESSION:WIN -p 2>/dev/null | tail -5+```+If the last line shows a spinner (✳✽✢✶·), `Running…`, or no `❯` — wait 10–15s and check again before sending.+```bashOBJ=$(jq -r --arg w SESSION:WIN '.agents[] | select(.window==$w) | .objective' ~/.claude/orchestrator-state.json)PR=$(jq -r --arg w SESSION:WIN '.agents[] | select(.window==$w) | .pr_number' ~/.claude/orchestrator-state.json)@@ -526,6 +589,102 @@ When `dx/orchestrate-skill` is merged into `dev`, `AutoGPT1` becomes a normal sp---+## Thread resolution integrity (critical)++**Agents MUST NOT resolve review threads via GraphQL unless a real code fix has been committed and pushed first.**++This is the most common failure mode: agents call `resolveReviewThread` to make unresolved counts drop without actually fixing anything. This produces a false "done" signal that gets past verify-complete.sh.++**The only valid resolution sequence:**+1. Read the thread and understand what it's asking+2. Make the actual code change+3. `git commit` and `git push`+4. Reply to the thread with the commit SHA (e.g. "Fixed in `abc1234`")+5. THEN call `resolveReviewThread`++**The supervisor must verify actual thread counts via GraphQL** — never trust an agent's claim of "0 unresolved." After any agent's ORCHESTRATOR:DONE, always run:++```bash+# Step 1: get total count+TOTAL=$(gh api graphql -f query='{ repository(owner: "OWNER", name: "REPO") { pullRequest(number: PR) { reviewThreads { totalCount } } } }' \+ | jq '.data.repository.pullRequest.reviewThreads.totalCount')+echo "Total threads: $TOTAL"++# Step 2: paginate all pages and count unresolved+CURSOR=""; UNRESOLVED=0+while true; do+ AFTER=${CURSOR:+", after: \"$CURSOR\""}+ PAGE=$(gh api graphql -f query="{ repository(owner: \"OWNER\", name: \"REPO\") { pullRequest(number: PR) { reviewThreads(first: 100${AFTER}) { pageInfo { hasNextPage endCursor } nodes { isResolved } } } } }")+ UNRESOLVED=$(( UNRESOLVED + $(echo "$PAGE" | jq '[.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved==false)] | length') ))+ HAS_NEXT=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.hasNextPage')+ CURSOR=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.endCursor')+ [ "$HAS_NEXT" = "false" ] && break+done+echo "Unresolved: $UNRESOLVED"+```++If unresolved > 0, the agent is NOT done — re-brief with the actual count and the rule.++**Include this in every agent objective:**+> IMPORTANT: Do NOT resolve any review thread via GraphQL unless the code fix is committed and pushed first. Fix the code → commit → push → reply with SHA → then resolve. Never resolve without a real commit. "Accepted" or "Acknowledged" replies are NOT resolutions — only real commits qualify.++### Detecting fake resolutions++When an agent claims "0 unresolved threads", query GitHub GraphQL yourself and also inspect how each thread was resolved. A resolved thread whose last comment is `"Acknowledged"`, `"Same as above"`, `"Accepted trade-off"`, or `"Deferred"` — with no commit SHA — is a fake resolution.++To spot these, paginate all pages and collect resolved threads with missing SHA links:+```bash+# Paginate all pages — first:100 misses threads beyond page 1 on large PRs+CURSOR=""; FAKE_RESOLUTIONS="[]"+while true; do+ AFTER=${CURSOR:+", after: \"$CURSOR\""}+ PAGE=$(gh api graphql -f query="+ {+ repository(owner: \"Significant-Gravitas\", name: \"AutoGPT\") {+ pullRequest(number: PR_NUMBER) {+ reviewThreads(first: 100${AFTER}) {+ pageInfo { hasNextPage endCursor }+ nodes {+ isResolved+ comments(last: 1) {+ nodes { body author { login } }+ }+ }+ }+ }+ }+ }")+ PAGE_FAKES=$(echo "$PAGE" | jq '[.data.repository.pullRequest.reviewThreads.nodes[]+ | select(.isResolved == true)+ | {body: .comments.nodes[0].body[:120], author: .comments.nodes[0].author.login}+ | select(.body | test("Fixed in|Removed in|Addressed in") | not)]')+ FAKE_RESOLUTIONS=$(echo "$FAKE_RESOLUTIONS $PAGE_FAKES" | jq -s 'add')+ HAS_NEXT=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.hasNextPage')+ CURSOR=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.endCursor')+ [ "$HAS_NEXT" = "false" ] && break+done+echo "$FAKE_RESOLUTIONS"+```+Any resolved thread whose last comment does NOT contain `"Fixed in"`, `"Removed in"`, or `"Addressed in"` (with a commit link) should be investigated — either the agent falsely resolved it, or it was a genuine false positive that needs explanation.++## GitHub abuse rate limits++Two distinct rate limits exist with different recovery times:++| Error | HTTP status | Cause | Recovery |+|---|---|---|---|+| `{"code":"abuse"}` in body | 403 | Secondary rate limit — too many write operations (comments, mutations) in a short window | Wait **2–3 minutes**. 60s is often not enough. |+| `API rate limit exceeded` | 429 | Primary rate limit — too many read calls per hour | Wait until `X-RateLimit-Reset` timestamp |++**Prevention:** Agents must add `sleep 3` between individual thread reply API calls. For >20 unresolved threads, increase to `sleep 5`.++If you see a 403 `abuse` error from an agent's pane:+1. Nudge the agent: `"You hit a GitHub secondary rate limit (403). Stop all API writes. Wait 2 minutes, then resume with sleep 3 between each thread reply."`+2. Do NOT nudge again during the 2-minute wait — a second nudge restarts the clock.++Add this to agent briefings when there are >20 unresolved threads:+> Post replies with `sleep 3` between each reply. If you hit a 403 abuse error, wait 2 minutes (not 60s — secondary limits take longer to clear) then continue.+## Key rules1. **Scripts do all the heavy lifting** — don't reimplement their logic inline in this file@@ -543,3 +702,8 @@ When `dx/orchestrate-skill` is merged into `dev`, `AutoGPT1` becomes a normal sp13. **Protected worktrees** — never use the worktree hosting the skill scripts as a spare14. **Images via file path** — save screenshots to `/tmp/orchestrator-context-<ts>.png`, pass path in objective; agents read with the `Read` tool15. **Split send-keys** — always separate text and Enter with `sleep 0.3` between calls for long strings+16. **Poll ALL windows from state file** — never hardcode window count. Derive active windows dynamically: `jq -r '.agents[] | select(.state | test("running|idle|stuck")) | .window' ~/.claude/orchestrator-state.json`. If you added a window mid-session outside spawn-agent.sh, add it to the state file immediately.+20. **Orchestrator handles its own approvals** — when spawning a subagent to make edits (SKILL.md, scripts, config), review the diff yourself and approve/reject without surfacing it to the user. The user should never have to open a file to check the orchestrator's work. Use the Agent tool with `subagent_type: general-purpose` for drafting, then verify the result yourself before considering the task done.+17. **Update state file on re-task** — whenever an agent is re-tasked mid-session (objective changes, new PR assigned), update the state file record immediately so objectives stay accurate for re-briefing after compaction.+18. **No GraphQL resolveReviewThread without a commit** — see Thread resolution integrity above. This is rule #1 for pr-address work.+19. **Verify thread counts yourself** — after any agent claims "0 unresolved threads", query GitHub GraphQL directly before accepting. Never trust the agent's self-report.diff --git a/.claude/skills/orchestrate/scripts/run-loop.sh b/.claude/skills/orchestrate/scripts/run-loop.shindex cfa7cf9a67..ff8b1a4df7 100755--- a/.claude/skills/orchestrate/scripts/run-loop.sh+++ b/.claude/skills/orchestrate/scripts/run-loop.sh@@ -27,7 +27,11 @@ chmod +x "$STABLE_SCRIPTS_DIR"/*.shSCRIPTS_DIR="$STABLE_SCRIPTS_DIR"STATE_FILE="${ORCHESTRATOR_STATE_FILE:-$HOME/.claude/orchestrator-state.json}"+# Adaptive polling: starts at base interval, backs off up to POLL_IDLE_MAX when+# no agents need attention, resets on any activity or waiting_approval state.POLL_INTERVAL="${POLL_INTERVAL:-30}"+POLL_IDLE_MAX=${POLL_IDLE_MAX:-300}+POLL_CURRENT=$POLL_INTERVAL# ---------------------------------------------------------------------------# update_state WINDOW FIELD VALUE@@ -70,6 +74,40 @@ wait_for_prompt() {return 1 # timed out}+# ---------------------------------------------------------------------------+# wait_for_claude_idle WINDOW — wait up to 30s for Claude to reach idle ❯ prompt+# (no spinner or busy indicator visible in the last 3 lines of pane output)+# Returns 0 when idle, 1 on timeout.+# ---------------------------------------------------------------------------+wait_for_claude_idle() {+ local window="$1"+ local timeout="${2:-30}"+ local elapsed=0+ while (( elapsed < timeout )); do+ local cmd pane pane_tail+ cmd=$(tmux display-message -t "$window" -p '#{pane_current_command}' 2>/dev/null || echo "")+ pane=$(tmux capture-pane -t "$window" -p 2>/dev/null || echo "")+ pane_tail=$(echo "$pane" | tail -3)+ # Check full pane (not just tail) — 'Enter to confirm' dialog can scroll above last 3 lines.+ # Do NOT reset elapsed — resetting allows an infinite loop if the dialog never clears.+ if echo "$pane" | grep -q "Enter to confirm"; then+ tmux send-keys -t "$window" Down Enter+ sleep 2; (( elapsed += 2 )); continue+ fi+ # Must be running under node (Claude is live)+ if [[ "$cmd" == "node" ]]; then+ # Idle: ❯ prompt visible AND no spinner/busy text in last 3 lines+ if echo "$pane_tail" | grep -q "❯" && \+ ! echo "$pane_tail" | grep -qE '[✳✽✢✶·✻✼✿❋✤]|Running…|Compacting'; then+ return 0+ fi+ fi+ sleep 2+ (( elapsed += 2 ))+ done+ return 1 # timed out+}+# ---------------------------------------------------------------------------# handle_kick WINDOW STATE — only for idle (crashed) agents, not stuck# ---------------------------------------------------------------------------@@ -83,6 +121,10 @@ handle_kick() {echo "[$(date +%H:%M:%S)] KICK restart $window — agent exited, resuming session"+ # Wait for the shell prompt before typing — avoids sending into a still-draining pane+ wait_for_claude_idle "$window" 30 \+ || echo "[$(date +%H:%M:%S)] KICK WARNING $window — pane still busy before resume, sending anyway"+# Resume the exact session so the agent retains full context — no need to re-send objectiveif [ -n "$session_id" ]; thentmux send-keys -t "$window" "cd '${worktree_path}' && claude --resume '${session_id}' --permission-mode bypassPermissions" Enter@@ -130,7 +172,7 @@ handle_approve() {# ---------------------------------------------------------------------------# Main loop# ----------------------------------------------------------------------------echo "[$(date +%H:%M:%S)] run-loop started (mechanical only, poll every ${POLL_INTERVAL}s)"+echo "[$(date +%H:%M:%S)] run-loop started (mechanical only, poll ${POLL_INTERVAL}s→${POLL_IDLE_MAX}s adaptive)"echo "[$(date +%H:%M:%S)] Supervisor: orchestrating Claude session (not a separate window)"echo "---"@@ -159,6 +201,15 @@ while true; doRUNNING=$(jq '[.agents[] | select(.state | test("running|stuck|waiting_approval|idle"))] | length' \"$STATE_FILE" 2>/dev/null || echo 0)- echo "[$(date +%H:%M:%S)] Poll — ${RUNNING} running ${KICKED} kicked ${DONE} recycled"- sleep "$POLL_INTERVAL"+ # Adaptive backoff: reset to base on activity or waiting_approval agents; back off when truly idle+ WAITING=$(jq '[.agents[] | select(.state == "waiting_approval")] | length' "$STATE_FILE" 2>/dev/null || echo 0)+ if (( KICKED > 0 || DONE > 0 || WAITING > 0 )); then+ POLL_CURRENT=$POLL_INTERVAL+ else+ POLL_CURRENT=$(( POLL_CURRENT + POLL_CURRENT / 2 + 1 ))+ (( POLL_CURRENT > POLL_IDLE_MAX )) && POLL_CURRENT=$POLL_IDLE_MAX+ fi++ echo "[$(date +%H:%M:%S)] Poll — ${RUNNING} running ${KICKED} kicked ${DONE} recycled (next in ${POLL_CURRENT}s)"+ sleep "$POLL_CURRENT"donediff --git a/.claude/skills/orchestrate/scripts/spawn-agent.sh b/.claude/skills/orchestrate/scripts/spawn-agent.shindex 526a32f067..7c565a525d 100755--- a/.claude/skills/orchestrate/scripts/spawn-agent.sh+++ b/.claude/skills/orchestrate/scripts/spawn-agent.sh@@ -90,26 +90,33 @@ fi# claude --resume SESSION_ID --permission-mode bypassPermissionstmux send-keys -t "$WINDOW" "cd '${WORKTREE_PATH}' && claude --permission-mode bypassPermissions --session-id '${SESSION_ID}'" Enter-# Wait up to 60s for claude to be fully interactive:-# both pane_current_command == 'node' AND the '❯' prompt is visible.-PROMPT_FOUND=false-for i in $(seq 1 60); do- CMD=$(tmux display-message -t "$WINDOW" -p '#{pane_current_command}' 2>/dev/null || echo "")- PANE=$(tmux capture-pane -t "$WINDOW" -p 2>/dev/null || echo "")- if echo "$PANE" | grep -q "Enter to confirm"; then- tmux send-keys -t "$WINDOW" Down Enter- sleep 2- continue- fi- if [[ "$CMD" == "node" ]] && echo "$PANE" | grep -q "❯"; then- PROMPT_FOUND=true- break- fi- sleep 1-done+# wait_for_claude_idle — poll until the pane shows idle ❯ with no spinner in the last 3 lines.+# Returns 0 when idle, 1 on timeout.+_wait_idle() {+ local window="$1" timeout="${2:-60}" elapsed=0+ while (( elapsed < timeout )); do+ local cmd pane_tail+ cmd=$(tmux display-message -t "$window" -p '#{pane_current_command}' 2>/dev/null || echo "")+ pane=$(tmux capture-pane -t "$window" -p 2>/dev/null || echo "")+ pane_tail=$(echo "$pane" | tail -3)+ # Check full pane (not just tail) — 'Enter to confirm' dialog can appear above the last 3 lines+ if echo "$pane" | grep -q "Enter to confirm"; then+ tmux send-keys -t "$window" Down Enter+ sleep 2; (( elapsed += 2 )); continue+ fi+ if [[ "$cmd" == "node" ]] && \+ echo "$pane_tail" | grep -q "❯" && \+ ! echo "$pane_tail" | grep -qE '[✳✽✢✶·✻✼✿❋✤]|Running…|Compacting'; then+ return 0+ fi+ sleep 2; (( elapsed += 2 ))+ done+ return 1+}-if ! $PROMPT_FOUND; then- echo "[spawn-agent] WARNING: timed out waiting for ❯ prompt on $WINDOW — sending objective anyway" >&2+# Wait up to 60s for claude to be fully interactive and idle (❯ visible, no spinner).+if ! _wait_idle "$WINDOW" 60; then+ echo "[spawn-agent] WARNING: timed out waiting for idle ❯ prompt on $WINDOW — sending objective anyway" >&2fi# Send the task. Split text and Enter — if combined, Enter can fire before the stringdiff --git a/.claude/skills/pr-address/SKILL.md b/.claude/skills/pr-address/SKILL.mdindex 9a9c89e0ec..857c6ae81d 100644--- a/.claude/skills/pr-address/SKILL.md+++ b/.claude/skills/pr-address/SKILL.md@@ -29,30 +29,71 @@ gh pr view {N} --json body --jq '.body'### 1. Inline review threads — GraphQL (primary source of actionable items)-Use GraphQL to fetch inline threads. It natively exposes `isResolved`, returns threads already grouped with all replies, and paginates via cursor — no manual thread reconstruction needed.+> ⚠️ **WARNING — PAGINATE ALL PAGES BEFORE ADDRESSING ANYTHING**+>+> `reviewThreads(first: 100)` returns at most 100 threads per page. A PR with many review cycles can have 140+ threads across 2+ pages. **If you start addressing threads after fetching only page 1, you will miss all threads on subsequent pages and silently leave them unresolved.**+>+> PR #12636 had 142 total threads: page 1 returned 69 unresolved, page 2 had 42 more (111 total unresolved). An agent that stopped after page 1 addressed only 69 and falsely reported "done".+>+> **The rule: collect ALL thread IDs from ALL pages into a single list, then address them.**++**Step 1 — Fetch total count first:**```bashgh api graphql -f query='{repository(owner: "Significant-Gravitas", name: "AutoGPT") {pullRequest(number: {N}) {- reviewThreads(first: 100) {- pageInfo { hasNextPage endCursor }- nodes {- id- isResolved- path- comments(last: 1) {- nodes { databaseId body author { login } createdAt }+ reviewThreads { totalCount }+ }+ }+}' | jq '.data.repository.pullRequest.reviewThreads.totalCount'+```++If `totalCount > 100`, you have multiple pages. Fetch them all before doing anything else.++**Step 2 — Collect all unresolved thread IDs across all pages:**++```bash+# Accumulate all unresolved threads — loop until hasNextPage == false+CURSOR=""+ALL_THREADS="[]"+while true; do+ AFTER=${CURSOR:+", after: \"$CURSOR\""}+ PAGE=$(gh api graphql -f query="+ {+ repository(owner: \"Significant-Gravitas\", name: \"AutoGPT\") {+ pullRequest(number: {N}) {+ reviewThreads(first: 100${AFTER}) {+ pageInfo { hasNextPage endCursor }+ nodes {+ id+ isResolved+ path+ line+ comments(last: 1) {+ nodes { databaseId body author { login } }+ }}}}}- }-}'+ }")+ # Append unresolved nodes from this page+ PAGE_THREADS=$(echo "$PAGE" | jq '[.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false)]')+ ALL_THREADS=$(echo "$ALL_THREADS $PAGE_THREADS" | jq -s 'add')+ HAS_NEXT=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.hasNextPage')+ CURSOR=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.endCursor')+ [ "$HAS_NEXT" = "false" ] && break+done++echo "Total unresolved threads: $(echo "$ALL_THREADS" | jq 'length')"+echo "$ALL_THREADS" | jq '[.[] | {id, path, line, body: .comments.nodes[0].body[:200]}]'```-If `pageInfo.hasNextPage` is true, fetch subsequent pages by adding `after: "<endCursor>"` to `reviewThreads(first: 100, after: "...")` and repeat until `hasNextPage` is false.+**Step 3 — Address every thread in `ALL_THREADS`, then resolve.**++Only after this loop completes (all pages fetched, count confirmed) should you begin making fixes.**Filter to unresolved threads only** — skip any thread where `isResolved: true`. `comments(last: 1)` returns the most recent comment in the thread — act on that; it reflects the reviewer's final ask. Use the thread `id` (Relay global ID) to track threads across polls.@@ -84,19 +125,44 @@ Mostly contains: bot summaries (`coderabbitai[bot]`), CI/conflict detection (`gi## For each unaddressed comment-Address comments **one at a time**: fix → commit → push → inline reply → next.+**CRITICAL: The only valid sequence is fix → commit → push → reply → resolve. Never resolve a thread without a real code commit.**++Resolving a thread via `resolveReviewThread` without an actual fix is the most common failure mode — it makes unresolved counts drop without any real change, producing a false "done" signal. If the issue was genuinely a false positive (no code change needed), reply explaining why and then resolve. Otherwise:++Address comments **one at a time**: fix → commit → push → inline reply → resolve.1. Read the referenced code, make the fix (or reply explaining why it's not needed)2. Commit and push the fix3. Reply **inline** (not as a new top-level comment) referencing the fixing commit — this is what resolves the conversation for bot reviewers (coderabbitai, sentry):-Use a **markdown commit link** so GitHub renders it as a clickable reference. Get the full SHA with `git rev-parse HEAD` after committing:+Use a **markdown commit link** so GitHub renders it as a clickable reference. Always get the full SHA with `git rev-parse HEAD` **after** committing — never copy a SHA from a previous commit or hardcode one:++```bash+FULL_SHA=$(git rev-parse HEAD)+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments/{ID}/replies \+ -f body="🤖 Fixed in [${FULL_SHA:0:9}](https://github.com/Significant-Gravitas/AutoGPT/commit/${FULL_SHA}): <description>"+```| Comment type | How to reply ||---|---|| Inline review (`pulls/{N}/comments`) | `gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments/{ID}/replies -f body="🤖 Fixed in [abc1234](https://github.com/Significant-Gravitas/AutoGPT/commit/FULL_SHA): <description>"` || Conversation (`issues/{N}/comments`) | `gh api repos/Significant-Gravitas/AutoGPT/issues/{N}/comments -f body="🤖 Fixed in [abc1234](https://github.com/Significant-Gravitas/AutoGPT/commit/FULL_SHA): <description>"` |+### What counts as a valid resolution++Only two situations justify calling `resolveReviewThread`:++1. **Real code fix**: you changed the code, committed + pushed, and replied with the SHA. The commit diff must actually address the concern — not just touch the same file.+2. **Genuine false positive**: the reviewer's concern does not apply to this code, and you can give a specific technical reason (e.g. "Not applicable — `sdk_cwd` is pre-validated by `_make_sdk_cwd()` which applies normpath + prefix assertion before reaching this point").++**Anti-patterns that look resolved but aren't — never do these:**+- `"Accepted, tracked as follow-up"` — a deferral, not a fix. The concern is still open. Do not resolve.+- `"Acknowledged"` or `"Same as above"` — these are acknowledgements, not fixes. Do not resolve.+- `"Fixed in abc1234"` where `abc1234` is a commit that doesn't actually change the flagged line/logic — dishonest. Verify `git show abc1234 -- path/to/file` changes the right thing before posting.+- Resolving without replying — the reviewer never sees what happened.++When in doubt: if a code change is needed, make it. A deferred issue means the thread stays open until the follow-up PR is merged.+## Codecov coverageCodecov patch target is **80%** on changed lines. Checks are **informational** (not blocking) but should be green.@@ -232,3 +298,113 @@ git push```5. Restart the polling loop from the top — new commits reset CI status.++## GitHub abuse rate limits++Two distinct rate limits exist — they have different causes and recovery times:++| Error | HTTP code | Cause | Recovery |+|---|---|---|---|+| `{"code":"abuse"}` | 403 | Secondary rate limit — too many write operations (comments, mutations) in a short window | Wait **2–3 minutes**. 60s is often not enough. |+| `{"message":"API rate limit exceeded"}` | 429 | Primary rate limit — too many API calls per hour | Wait until `X-RateLimit-Reset` header timestamp |++**Prevention:** Add `sleep 3` between individual thread reply API calls. When posting >20 replies, increase to `sleep 5`.++**Recovery from secondary rate limit (403):**+1. Stop all API writes immediately+2. Wait **2 minutes minimum** (not 60s — secondary limits are stricter)+3. Resume with `sleep 3` between each call+4. If 403 persists after 2 min, wait another 2 min before retrying++Never batch all replies in a tight loop — always space them out.++## Parallel thread resolution++When a PR has more than 10 unresolved threads, addressing one commit per thread is slow. Use this strategy instead:++### Group by file, batch per commit++1. Sort `ALL_THREADS` by `path` — threads in the same file can share a single commit.+2. Fix all threads in one file → `git commit` → `git push` → reply to **all** those threads with the same SHA → resolve them all.+3. Move to the next file group and repeat.++This reduces N commits to (number of files touched), which is usually 3–5 instead of 15–30.++### Posting replies concurrently (for large batches)++For truly independent thread groups (different files, no shared logic), you can post replies in parallel using background subshells — but always space out API writes:++```bash+# Post replies to a batch of threads concurrently, 3s apart+(+ sleep 3+ gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments/{ID1}/replies \+ -f body="🤖 Fixed in [${FULL_SHA:0:9}](https://github.com/Significant-Gravitas/AutoGPT/commit/${FULL_SHA}): ..."+) &+(+ sleep 6+ gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments/{ID2}/replies \+ -f body="🤖 Fixed in [${FULL_SHA:0:9}](https://github.com/Significant-Gravitas/AutoGPT/commit/${FULL_SHA}): ..."+) &+wait # wait for all background replies before resolving+```++Then resolve sequentially (GraphQL mutations):+```bash+for THREAD_ID in "$THREAD1" "$THREAD2" "$THREAD3"; do+ gh api graphql -f query="mutation { resolveReviewThread(input: {threadId: \"${THREAD_ID}\"}) { thread { isResolved } } }"+ sleep 3+done+```++**Always sleep 3s between individual API writes** — GitHub's secondary rate limit (403) triggers on bursts of >20 writes. Increase to `sleep 5` when posting more than 20 replies in a batch.++## Resolving threads via GraphQL++Use `resolveReviewThread` **only after** the commit is pushed and the reply is posted:++```bash+gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: "THREAD_ID"}) { thread { isResolved } } }'+```++**Never call this mutation before committing the fix.** The orchestrator will verify actual unresolved counts via GraphQL after you output `ORCHESTRATOR:DONE` — false resolutions will be caught and you will be re-briefed.++### Verify actual count before outputting ORCHESTRATOR:DONE++Before claiming "0 unresolved threads", always query GitHub directly — don't rely on your own bookkeeping. Paginate all pages — a single `first: 100` query misses threads beyond page 1:++```bash+# Step 1: get total thread count+gh api graphql -f query='+{+ repository(owner: "Significant-Gravitas", name: "AutoGPT") {+ pullRequest(number: {N}) {+ reviewThreads { totalCount }+ }+ }+}' | jq '.data.repository.pullRequest.reviewThreads.totalCount'++# Step 2: paginate all pages, count truly unresolved+CURSOR=""; UNRESOLVED=0+while true; do+ AFTER=${CURSOR:+", after: \"$CURSOR\""}+ PAGE=$(gh api graphql -f query="+ {+ repository(owner: \"Significant-Gravitas\", name: \"AutoGPT\") {+ pullRequest(number: {N}) {+ reviewThreads(first: 100${AFTER}) {+ pageInfo { hasNextPage endCursor }+ nodes { isResolved }+ }+ }+ }+ }")+ UNRESOLVED=$(( UNRESOLVED + $(echo "$PAGE" | jq '[.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved==false)] | length') ))+ HAS_NEXT=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.hasNextPage')+ CURSOR=$(echo "$PAGE" | jq -r '.data.repository.pullRequest.reviewThreads.pageInfo.endCursor')+ [ "$HAS_NEXT" = "false" ] && break+done+echo "Unresolved threads: $UNRESOLVED"+```++Only output `ORCHESTRATOR:DONE` after this loop reports 0.diff --git a/.claude/skills/pr-test/SKILL.md b/.claude/skills/pr-test/SKILL.mdindex f11feda332..10b3b3efdc 100644--- a/.claude/skills/pr-test/SKILL.md+++ b/.claude/skills/pr-test/SKILL.md@@ -310,6 +310,28 @@ TOKEN=$(curl -s -X POST 'http://localhost:8000/auth/v1/token?grant_type=passwordcurl -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/...```+### 3i. Disable onboarding for test user++The frontend redirects to `/onboarding` when the `VISIT_COPILOT` step is not in `completedSteps`.+Mark it complete via the backend API so every browser test lands on the real feature UI:++```bash+ONBOARDING_RESULT=$(curl -s --max-time 30 -X POST \+ "http://localhost:8006/api/onboarding/step?step=VISIT_COPILOT" \+ -H "Authorization: Bearer $TOKEN")+echo "Onboarding bypass: $ONBOARDING_RESULT"++# Verify it took effect+ONBOARDING_STATUS=$(curl -s --max-time 30 \+ "http://localhost:8006/api/onboarding/completed" \+ -H "Authorization: Bearer $TOKEN" | jq -r '.is_completed')+echo "Onboarding completed: $ONBOARDING_STATUS"+if [ "$ONBOARDING_STATUS" != "true" ]; then+ echo "ERROR: onboarding bypass failed — browser tests will hit /onboarding instead of the target feature. Investigate before proceeding."+ exit 1+fi+```+## Step 4: Run tests### Service ports reference