AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
ci: apply E2E CI optimizations to Claude workflows (#12097)
Otto86af8fc856d9
diff --git a/.github/workflows/claude-ci-failure-auto-fix.yml b/.github/workflows/claude-ci-failure-auto-fix.ymlindex ab07c8ae10..dbca6dc3f3 100644--- a/.github/workflows/claude-ci-failure-auto-fix.yml+++ b/.github/workflows/claude-ci-failure-auto-fix.yml@@ -40,6 +40,48 @@ jobs:git checkout -b "$BRANCH_NAME"echo "branch_name=$BRANCH_NAME" >> $GITHUB_OUTPUT+ # Backend Python/Poetry setup (so Claude can run linting/tests)+ - name: Set up Python+ uses: actions/setup-python@v5+ with:+ python-version: "3.11"++ - name: Set up Python dependency cache+ uses: actions/cache@v5+ with:+ path: ~/.cache/pypoetry+ key: poetry-${{ runner.os }}-${{ hashFiles('autogpt_platform/backend/poetry.lock') }}++ - name: Install Poetry+ run: |+ cd autogpt_platform/backend+ HEAD_POETRY_VERSION=$(python3 ../../.github/workflows/scripts/get_package_version_from_lockfile.py poetry)+ curl -sSL https://install.python-poetry.org | POETRY_VERSION=$HEAD_POETRY_VERSION python3 -+ echo "$HOME/.local/bin" >> $GITHUB_PATH++ - name: Install Python dependencies+ working-directory: autogpt_platform/backend+ run: poetry install++ - name: Generate Prisma Client+ working-directory: autogpt_platform/backend+ run: poetry run prisma generate && poetry run gen-prisma-stub++ # Frontend Node.js/pnpm setup (so Claude can run linting/tests)+ - name: Enable corepack+ run: corepack enable++ - name: Set up Node.js+ uses: actions/setup-node@v6+ with:+ node-version: "22"+ cache: "pnpm"+ cache-dependency-path: autogpt_platform/frontend/pnpm-lock.yaml++ - name: Install JavaScript dependencies+ working-directory: autogpt_platform/frontend+ run: pnpm install --frozen-lockfile+- name: Get CI failure detailsid: failure_detailsuses: actions/github-script@v8diff --git a/.github/workflows/claude-dependabot.yml b/.github/workflows/claude-dependabot.ymlindex da37df6de7..274c6d2cab 100644--- a/.github/workflows/claude-dependabot.yml+++ b/.github/workflows/claude-dependabot.yml@@ -77,27 +77,15 @@ jobs:run: poetry run prisma generate && poetry run gen-prisma-stub# Frontend Node.js/pnpm setup (mirrors platform-frontend-ci.yml)- - name: Set up Node.js- uses: actions/setup-node@v6- with:- node-version: "22"-- name: Enable corepackrun: corepack enable- - name: Set pnpm store directory- run: |- pnpm config set store-dir ~/.pnpm-store- echo "PNPM_HOME=$HOME/.pnpm-store" >> $GITHUB_ENV-- - name: Cache frontend dependencies- uses: actions/cache@v5+ - name: Set up Node.js+ uses: actions/setup-node@v6with:- path: ~/.pnpm-store- key: ${{ runner.os }}-pnpm-${{ hashFiles('autogpt_platform/frontend/pnpm-lock.yaml', 'autogpt_platform/frontend/package.json') }}- restore-keys: |- ${{ runner.os }}-pnpm-${{ hashFiles('autogpt_platform/frontend/pnpm-lock.yaml') }}- ${{ runner.os }}-pnpm-+ node-version: "22"+ cache: "pnpm"+ cache-dependency-path: autogpt_platform/frontend/pnpm-lock.yaml- name: Install JavaScript dependenciesworking-directory: autogpt_platform/frontenddiff --git a/.github/workflows/claude.yml b/.github/workflows/claude.ymlindex ee901fe5d4..8b8260af6b 100644--- a/.github/workflows/claude.yml+++ b/.github/workflows/claude.yml@@ -93,27 +93,15 @@ jobs:run: poetry run prisma generate && poetry run gen-prisma-stub# Frontend Node.js/pnpm setup (mirrors platform-frontend-ci.yml)- - name: Set up Node.js- uses: actions/setup-node@v6- with:- node-version: "22"-- name: Enable corepackrun: corepack enable- - name: Set pnpm store directory- run: |- pnpm config set store-dir ~/.pnpm-store- echo "PNPM_HOME=$HOME/.pnpm-store" >> $GITHUB_ENV-- - name: Cache frontend dependencies- uses: actions/cache@v5+ - name: Set up Node.js+ uses: actions/setup-node@v6with:- path: ~/.pnpm-store- key: ${{ runner.os }}-pnpm-${{ hashFiles('autogpt_platform/frontend/pnpm-lock.yaml', 'autogpt_platform/frontend/package.json') }}- restore-keys: |- ${{ runner.os }}-pnpm-${{ hashFiles('autogpt_platform/frontend/pnpm-lock.yaml') }}- ${{ runner.os }}-pnpm-+ node-version: "22"+ cache: "pnpm"+ cache-dependency-path: autogpt_platform/frontend/pnpm-lock.yaml- name: Install JavaScript dependenciesworking-directory: autogpt_platform/frontenddiff --git a/plans/SECRT-1950-claude-ci-optimizations.md b/plans/SECRT-1950-claude-ci-optimizations.mdnew file mode 100644index 0000000000..15d1419b0e--- /dev/null+++ b/plans/SECRT-1950-claude-ci-optimizations.md@@ -0,0 +1,165 @@+# Implementation Plan: SECRT-1950 - Apply E2E CI Optimizations to Claude Code Workflows++## Ticket+[SECRT-1950](https://linear.app/autogpt/issue/SECRT-1950)++## Summary+Apply Pwuts's CI performance optimizations from PR #12090 to Claude Code workflows.++## Reference PR+https://github.com/Significant-Gravitas/AutoGPT/pull/12090++---++## Analysis++### Current State (claude.yml)++**pnpm caching (lines 104-118):**+```yaml+- name: Set up Node.js+ uses: actions/setup-node@v6+ with:+ node-version: "22"++- name: Enable corepack+ run: corepack enable++- name: Set pnpm store directory+ run: |+ pnpm config set store-dir ~/.pnpm-store+ echo "PNPM_HOME=$HOME/.pnpm-store" >> $GITHUB_ENV++- name: Cache frontend dependencies+ uses: actions/cache@v5+ with:+ path: ~/.pnpm-store+ key: ${{ runner.os }}-pnpm-${{ hashFiles('autogpt_platform/frontend/pnpm-lock.yaml', 'autogpt_platform/frontend/package.json') }}+ restore-keys: |+ ${{ runner.os }}-pnpm-${{ hashFiles('autogpt_platform/frontend/pnpm-lock.yaml') }}+ ${{ runner.os }}-pnpm-+```++**Docker setup (lines 134-165):**+- Uses `docker-buildx-action@v3`+- Has manual Docker image caching via `actions/cache`+- Runs `docker compose up` without buildx bake optimization++### Pwuts's Optimizations (PR #12090)++1. **Simplified pnpm caching** - Use `setup-node` built-in cache:+```yaml+- name: Enable corepack+ run: corepack enable++- name: Set up Node+ uses: actions/setup-node@v6+ with:+ node-version: "22.18.0"+ cache: "pnpm"+ cache-dependency-path: autogpt_platform/frontend/pnpm-lock.yaml+```++2. **Docker build caching via buildx bake**:+```yaml+- name: Set up Docker Buildx+ uses: docker/setup-buildx-action@v3+ with:+ driver: docker-container+ driver-opts: network=host++- name: Expose GHA cache to docker buildx CLI+ uses: crazy-max/ghaction-github-runtime@v3++- name: Build Docker images (with cache)+ run: |+ pip install pyyaml+ docker compose -f docker-compose.yml config > docker-compose.resolved.yml+ python ../.github/workflows/scripts/docker-ci-fix-compose-build-cache.py \+ --source docker-compose.resolved.yml \+ --cache-from "type=gha" \+ --cache-to "type=gha,mode=max" \+ ...+ docker buildx bake --allow=fs.read=.. -f docker-compose.resolved.yml --load+```++---++## Proposed Changes++### 1. Update pnpm caching in `claude.yml`++**Before:**+- Manual cache key generation+- Separate `actions/cache` step+- Manual pnpm store directory config++**After:**+- Use `setup-node` built-in `cache: "pnpm"` option+- Remove manual cache step+- Keep `corepack enable` before `setup-node`++### 2. Update Docker build in `claude.yml`++**Before:**+- Manual Docker layer caching via `actions/cache` with `/tmp/.buildx-cache`+- Simple `docker compose build`++**After:**+- Use `crazy-max/ghaction-github-runtime@v3` to expose GHA cache+- Use `docker-ci-fix-compose-build-cache.py` script+- Build with `docker buildx bake`++### 3. Apply same changes to other Claude workflows++- `claude-dependabot.yml` - Check if it has similar patterns+- `claude-ci-failure-auto-fix.yml` - Check if it has similar patterns+- `copilot-setup-steps.yml` - Reusable workflow, may be the source of truth++---++## Files to Modify++1. `.github/workflows/claude.yml`+2. `.github/workflows/claude-dependabot.yml` (if applicable)+3. `.github/workflows/claude-ci-failure-auto-fix.yml` (if applicable)++## Dependencies++- PR #12090 must be merged first (provides the `docker-ci-fix-compose-build-cache.py` script)+- Backend Dockerfile optimizations (already in PR #12090)++---++## Test Plan++1. Create PR with changes+2. Trigger Claude workflow manually or via `@claude` mention on a test issue+3. Compare CI runtime before/after+4. Verify Claude agent still works correctly (can checkout, build, run tests)++---++## Risk Assessment++**Low risk:**+- These are CI infrastructure changes, not code changes+- If caching fails, builds fall back to uncached (slower but works)+- Changes mirror proven patterns from PR #12090++---++## Questions for Reviewer++1. Should we wait for PR #12090 to merge before creating this PR?+2. Does `copilot-setup-steps.yml` need updating, or is it a separate concern?+3. Any concerns about cache key collisions between frontend E2E and Claude workflows?++---++## Verified++- ✅ **`claude-dependabot.yml`**: Has same pnpm caching pattern as `claude.yml` (manual `actions/cache`) — NEEDS UPDATE+- ✅ **`claude-ci-failure-auto-fix.yml`**: Simple workflow with no pnpm or Docker caching — NO CHANGES NEEDED+- ✅ **Script path**: `docker-ci-fix-compose-build-cache.py` will be at `.github/workflows/scripts/` after PR #12090 merges+- ✅ **Test seed caching**: NOT APPLICABLE — Claude workflows spin up a dev environment but don't run E2E tests with pre-seeded data. The seed caching in PR #12090 is specific to the frontend E2E test suite which needs consistent test data. Claude just needs the services running.