Skip to content

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

feat(platform): Enhance GitHub Copilot agent setup with CI-aligned environment and comprehensive platform documentation integration (#10708)

Copilot68cd1cb398cf
diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
new file mode 100644
index 0000000000..a0834a6913
--- /dev/null
+++ b/.github/copilot-instructions.md
@@ -0,0 +1,244 @@
+# GitHub Copilot Instructions for AutoGPT
+
+This file provides comprehensive onboarding information for GitHub Copilot coding agent to work efficiently with the AutoGPT repository.
+
+## Repository Overview
+
+**AutoGPT** is a powerful platform for creating, deploying, and managing continuous AI agents that automate complex workflows. This is a large monorepo (~150MB) containing multiple components:
+
+- **AutoGPT Platform** (`autogpt_platform/`) - Main focus: Modern AI agent platform (Polyform Shield License)
+- **Classic AutoGPT** (`classic/`) - Legacy agent system (MIT License)
+- **Documentation** (`docs/`) - MkDocs-based documentation site
+- **Infrastructure** - Docker configurations, CI/CD, and development tools
+
+**Primary Languages & Frameworks:**
+- **Backend**: Python 3.10-3.13, FastAPI, Prisma ORM, PostgreSQL, RabbitMQ
+- **Frontend**: TypeScript, Next.js 15, React, Tailwind CSS, Radix UI
+- **Development**: Docker, Poetry, pnpm, Playwright, Storybook
+
+## Build and Validation Instructions
+
+### Essential Setup Commands
+
+**Always run these commands in the correct directory and in this order:**
+
+1. **Initial Setup** (required once):
+ ```bash
+ # Clone and enter repository
+ git clone <repo> && cd AutoGPT
+
+ # Start all services (database, redis, rabbitmq, clamav)
+ cd autogpt_platform && docker compose --profile local up deps --build --detach
+ ```
+
+2. **Backend Setup** (always run before backend development):
+ ```bash
+ cd autogpt_platform/backend
+ poetry install # Install dependencies
+ poetry run prisma migrate dev # Run database migrations
+ poetry run prisma generate # Generate Prisma client
+ ```
+
+3. **Frontend Setup** (always run before frontend development):
+ ```bash
+ cd autogpt_platform/frontend
+ pnpm install # Install dependencies
+ ```
+
+### Runtime Requirements
+
+**Critical:** Always ensure Docker services are running before starting development:
+```bash
+cd autogpt_platform && docker compose --profile local up deps --build --detach
+```
+
+**Python Version:** Use Python 3.11 (required; managed by Poetry via pyproject.toml)
+**Node.js Version:** Use Node.js 21+ with pnpm package manager
+
+### Development Commands
+
+**Backend Development:**
+```bash
+cd autogpt_platform/backend
+poetry run serve # Start development server (port 8000)
+poetry run test # Run all tests (requires ~5 minutes)
+poetry run pytest path/to/test.py # Run specific test
+poetry run format # Format code (Black + isort) - always run first
+poetry run lint # Lint code (ruff) - run after format
+```
+
+**Frontend Development:**
+```bash
+cd autogpt_platform/frontend
+pnpm dev # Start development server (port 3000) - use for active development
+pnpm build # Build for production (only needed for E2E tests or deployment)
+pnpm test # Run Playwright E2E tests (requires build first)
+pnpm test-ui # Run tests with UI
+pnpm format # Format and lint code
+pnpm storybook # Start component development server
+```
+
+### Testing Strategy
+
+**Backend Tests:**
+- **Block Tests**: `poetry run pytest backend/blocks/test/test_block.py -xvs` (validates all blocks)
+- **Specific Block**: `poetry run pytest 'backend/blocks/test/test_block.py::test_available_blocks[BlockName]' -xvs`
+- **Snapshot Tests**: Use `--snapshot-update` when output changes, always review with `git diff`
+
+**Frontend Tests:**
+- **E2E Tests**: Always run `pnpm dev` before `pnpm test` (Playwright requires running instance)
+- **Component Tests**: Use Storybook for isolated component development
+
+### Critical Validation Steps
+
+**Before committing changes:**
+1. Run `poetry run format` (backend) and `pnpm format` (frontend)
+2. Ensure all tests pass in modified areas
+3. Verify Docker services are still running
+4. Check that database migrations apply cleanly
+
+**Common Issues & Workarounds:**
+- **Prisma issues**: Run `poetry run prisma generate` after schema changes
+- **Permission errors**: Ensure Docker has proper permissions
+- **Port conflicts**: Check the `docker-compose.yml` file for the current list of exposed ports. You can list all mapped ports with:
+- **Test timeouts**: Backend tests can take 5+ minutes, use `-x` flag to stop on first failure
+
+## Project Layout & Architecture
+
+### Core Architecture
+
+**AutoGPT Platform** (`autogpt_platform/`):
+- `backend/` - FastAPI server with async support
+ - `backend/backend/` - Core API logic
+ - `backend/blocks/` - Agent execution blocks
+ - `backend/data/` - Database models and schemas
+ - `schema.prisma` - Database schema definition
+- `frontend/` - Next.js application
+ - `src/app/` - App Router pages and layouts
+ - `src/components/` - Reusable React components
+ - `src/lib/` - Utilities and configurations
+- `autogpt_libs/` - Shared Python utilities
+- `docker-compose.yml` - Development stack orchestration
+
+**Key Configuration Files:**
+- `pyproject.toml` - Python dependencies and tooling
+- `package.json` - Node.js dependencies and scripts
+- `schema.prisma` - Database schema and migrations
+- `next.config.mjs` - Next.js configuration
+- `tailwind.config.ts` - Styling configuration
+
+### Security & Middleware
+
+**Cache Protection**: Backend includes middleware preventing sensitive data caching in browsers/proxies
+**Authentication**: JWT-based with Supabase integration
+**User ID Validation**: All data access requires user ID checks - verify this for any `data/*.py` changes
+
+### Development Workflow
+
+**GitHub Actions**: Multiple CI/CD workflows in `.github/workflows/`
+- `platform-backend-ci.yml` - Backend testing and validation
+- `platform-frontend-ci.yml` - Frontend testing and validation
+- `platform-fullstack-ci.yml` - End-to-end integration tests
+
+**Pre-commit Hooks**: Run linting and formatting checks
+**Conventional Commits**: Use format `type(scope): description` (e.g., `feat(backend): add API`)
+
+### Key Source Files
+
+**Backend Entry Points:**
+- `backend/backend/server/server.py` - FastAPI application setup
+- `backend/backend/data/` - Database models and user management
+- `backend/blocks/` - Agent execution blocks and logic
+
+**Frontend Entry Points:**
+- `frontend/src/app/layout.tsx` - Root application layout
+- `frontend/src/app/page.tsx` - Home page
+- `frontend/src/lib/supabase/` - Authentication and database client
+
+**Protected Routes**: Update `frontend/lib/supabase/middleware.ts` when adding protected routes
+
+### Agent Block System
+
+Agents are built using a visual block-based system where each block performs a single action. Blocks are defined in `backend/blocks/` and must include:
+- Block definition with input/output schemas
+- Execution logic with proper error handling
+- Tests validating functionality
+
+### Database & ORM
+
+**Prisma ORM** with PostgreSQL backend including pgvector for embeddings:
+- Schema in `schema.prisma`
+- Migrations in `backend/migrations/`
+- Always run `prisma migrate dev` and `prisma generate` after schema changes
+
+## Environment Configuration
+
+### Configuration Files Priority Order
+1. **Backend**: `/backend/.env.default` → `/backend/.env` (user overrides)
+2. **Frontend**: `/frontend/.env.default` → `/frontend/.env` (user overrides)
+3. **Platform**: `/.env.default` (Supabase/shared) → `/.env` (user overrides)
+4. Docker Compose `environment:` sections override file-based config
+5. Shell environment variables have highest precedence
+
+### Docker Environment Setup
+- All services use hardcoded defaults (no `${VARIABLE}` substitutions)
+- The `env_file` directive loads variables INTO containers at runtime
+- Backend/Frontend services use YAML anchors for consistent configuration
+- Copy `.env.default` files to `.env` for local development customization
+
+## Advanced Development Patterns
+
+### Adding New Blocks
+1. Create file in `/backend/backend/blocks/`
+2. Inherit from `Block` base class with input/output schemas
+3. Implement `run` method with proper error handling
+4. Generate block UUID using `uuid.uuid4()`
+5. Register in block registry
+6. Write tests alongside block implementation
+7. Consider how inputs/outputs connect with other blocks in graph editor
+
+### API Development
+1. Update routes in `/backend/backend/server/routers/`
+2. Add/update Pydantic models in same directory
+3. Write tests alongside route files
+4. For `data/*.py` changes, validate user ID checks
+5. Run `poetry run test` to verify changes
+
+### Frontend Development
+1. Components in `/frontend/src/components/`
+2. Use existing UI components from `/frontend/src/components/ui/`
+3. Add Storybook stories for component development
+4. Test user-facing features with Playwright E2E tests
+5. Update protected routes in middleware when needed
+
+### Security Guidelines
+**Cache Protection Middleware** (`/backend/backend/server/middleware/security.py`):
+- Default: Disables caching for ALL endpoints with `Cache-Control: no-store, no-cache, must-revalidate, private`
+- Uses allow list approach for cacheable paths (static assets, health checks, public pages)
+- Prevents sensitive data caching in browsers/proxies
+- Add new cacheable endpoints to `CACHEABLE_PATHS`
+
+### CI/CD Alignment
+The repository has comprehensive CI workflows that test:
+- **Backend**: Python 3.11-3.13, services (Redis/RabbitMQ/ClamAV), Prisma migrations, Poetry lock validation
+- **Frontend**: Node.js 21, pnpm, Playwright with Docker Compose stack, API schema validation
+- **Integration**: Full-stack type checking and E2E testing
+
+Match these patterns when developing locally - the copilot setup environment mirrors these CI configurations.
+
+## Collaboration with Other AI Assistants
+
+This repository is actively developed with assistance from Claude (via CLAUDE.md files). When working on this codebase:
+- Check for existing CLAUDE.md files that provide additional context
+- Follow established patterns and conventions already in the codebase
+- Maintain consistency with existing code style and architecture
+- Consider that changes may be reviewed and extended by both human developers and AI assistants
+
+## Trust These Instructions
+
+These instructions are comprehensive and tested. Only perform additional searches if:
+1. Information here is incomplete for your specific task
+2. You encounter errors not covered by the workarounds
+3. You need to understand implementation details not covered above
+
+For detailed platform development patterns, refer to `autogpt_platform/CLAUDE.md` and `AGENTS.md` in the repository root.
\ No newline at end of file
diff --git a/.github/workflows/copilot-setup-steps.yml b/.github/workflows/copilot-setup-steps.yml
new file mode 100644
index 0000000000..7af1ec4365
--- /dev/null
+++ b/.github/workflows/copilot-setup-steps.yml
@@ -0,0 +1,302 @@
+name: "Copilot Setup Steps"
+
+# Automatically run the setup steps when they are changed to allow for easy validation, and
+# allow manual testing through the repository's "Actions" tab
+on:
+ workflow_dispatch:
+ push:
+ paths:
+ - .github/workflows/copilot-setup-steps.yml
+ pull_request:
+ paths:
+ - .github/workflows/copilot-setup-steps.yml
+
+jobs:
+ # The job MUST be called `copilot-setup-steps` or it will not be picked up by Copilot.
+ copilot-setup-steps:
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+
+ # Set the permissions to the lowest permissions possible needed for your steps.
+ # Copilot will be given its own token for its operations.
+ permissions:
+ # If you want to clone the repository as part of your setup steps, for example to install dependencies, you'll need the `contents: read` permission. If you don't clone the repository in your setup steps, Copilot will do this for you automatically after the steps complete.
+ contents: read
+
+ # You can define any steps you want, and they will run before the agent starts.
+ # If you do not check out your code, Copilot will do this for you.
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ submodules: true
+
+ # Backend Python/Poetry setup (mirrors platform-backend-ci.yml)
+ - name: Set up Python
+ uses: actions/setup-python@v5
+ with:
+ python-version: "3.11" # Use standard version matching CI
+
+ - name: Set up Python dependency cache
+ uses: actions/cache@v4
+ with:
+ path: ~/.cache/pypoetry
+ key: poetry-${{ runner.os }}-${{ hashFiles('autogpt_platform/backend/poetry.lock') }}
+
+ - name: Install Poetry
+ run: |
+ # Extract Poetry version from backend/poetry.lock (matches CI)
+ cd autogpt_platform/backend
+ HEAD_POETRY_VERSION=$(python3 ../../.github/workflows/scripts/get_package_version_from_lockfile.py poetry)
+ echo "Found Poetry version ${HEAD_POETRY_VERSION} in backend/poetry.lock"
+
+ # Install Poetry
+ curl -sSL https://install.python-poetry.org | POETRY_VERSION=$HEAD_POETRY_VERSION python3 -
+
+ # Add Poetry to PATH
+ echo "$HOME/.local/bin" >> $GITHUB_PATH
+
+ - name: Check poetry.lock
+ working-directory: autogpt_platform/backend
+ run: |
+ poetry lock
+ if ! git diff --quiet --ignore-matching-lines="^# " poetry.lock; then
+ echo "Warning: poetry.lock not up to date, but continuing for setup"
+ git checkout poetry.lock # Reset for clean setup
+ fi
+
+ - name: Install Python dependencies
+ working-directory: autogpt_platform/backend
+ run: poetry install
+
+ - name: Generate Prisma Client
+ working-directory: autogpt_platform/backend
+ run: poetry run prisma generate
+
+ # Frontend Node.js/pnpm setup (mirrors platform-frontend-ci.yml)
+ - name: Set up Node.js
+ uses: actions/setup-node@v4
+ with:
+ node-version: "21"
+
+ - name: Enable corepack
+ run: corepack enable
+
+ - name: Set pnpm store directory
+ run: |
+ pnpm config set store-dir ~/.pnpm-store
+ echo "PNPM_HOME=$HOME/.pnpm-store" >> $GITHUB_ENV
+
+ - name: Cache frontend dependencies
+ uses: actions/cache@v4
+ with:
+ path: ~/.pnpm-store
+ key: ${{ runner.os }}-pnpm-${{ hashFiles('autogpt_platform/frontend/pnpm-lock.yaml', 'autogpt_platform/frontend/package.json') }}
+ restore-keys: |
+ ${{ runner.os }}-pnpm-${{ hashFiles('autogpt_platform/frontend/pnpm-lock.yaml') }}
+ ${{ runner.os }}-pnpm-
+
+ - name: Install JavaScript dependencies
+ working-directory: autogpt_platform/frontend
+ run: pnpm install --frozen-lockfile
+
+ # Install Playwright browsers for frontend testing
+ # NOTE: Disabled to save ~1 minute of setup time. Re-enable if Copilot needs browser automation (e.g., for MCP)
+ # - name: Install Playwright browsers
+ # working-directory: autogpt_platform/frontend
+ # run: pnpm playwright install --with-deps chromium
+
+ # Docker setup for development environment
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v3
+
+ - name: Copy default environment files
+ working-directory: autogpt_platform
+ run: |
+ # Copy default environment files for development
+ cp .env.default .env
+ cp backend/.env.default backend/.env
+ cp frontend/.env.default frontend/.env
+
+ # Phase 1: Cache and load Docker images for faster setup
+ - name: Set up Docker image cache
+ id: docker-cache
+ uses: actions/cache@v4
+ with:
+ path: ~/docker-cache
+ # Use a versioned key for cache invalidation when image list changes
+ key: docker-images-v2-${{ runner.os }}-${{ hashFiles('.github/workflows/copilot-setup-steps.yml') }}
+ restore-keys: |
+ docker-images-v2-${{ runner.os }}-
+ docker-images-v1-${{ runner.os }}-
+
+ - name: Load or pull Docker images
+ working-directory: autogpt_platform
+ run: |
+ mkdir -p ~/docker-cache
+
+ # Define image list for easy maintenance
+ IMAGES=(
+ "redis:latest"
+ "rabbitmq:management"
+ "clamav/clamav-debian:latest"
+ "busybox:latest"
+ "kong:2.8.1"
+ "supabase/gotrue:v2.170.0"
+ "supabase/postgres:15.8.1.049"
+ "supabase/postgres-meta:v0.86.1"
+ "supabase/studio:20250224-d10db0f"
+ )
+
+ # Check if any cached tar files exist (more reliable than cache-hit)
+ if ls ~/docker-cache/*.tar 1> /dev/null 2>&1; then
+ echo "Docker cache found, loading images in parallel..."
+ for image in "${IMAGES[@]}"; do
+ # Convert image name to filename (replace : and / with -)
+ filename=$(echo "$image" | tr ':/' '--')
+ if [ -f ~/docker-cache/${filename}.tar ]; then
+ echo "Loading $image..."
+ docker load -i ~/docker-cache/${filename}.tar || echo "Warning: Failed to load $image from cache" &
+ fi
+ done
+ wait
+ echo "All cached images loaded"
+ else
+ echo "No Docker cache found, pulling images in parallel..."
+ # Pull all images in parallel
+ for image in "${IMAGES[@]}"; do
+ docker pull "$image" &
+ done
+ wait
+
+ # Only save cache on main branches (not PRs) to avoid cache pollution
+ if [[ "${{ github.ref }}" == "refs/heads/master" ]] || [[ "${{ github.ref }}" == "refs/heads/dev" ]]; then
+ echo "Saving Docker images to cache in parallel..."
+ for image in "${IMAGES[@]}"; do
+ # Convert image name to filename (replace : and / with -)
+ filename=$(echo "$image" | tr ':/' '--')
+ echo "Saving $image..."
+ docker save -o ~/docker-cache/${filename}.tar "$image" || echo "Warning: Failed to save $image" &
+ done
+ wait
+ echo "Docker image cache saved"
+ else
+ echo "Skipping cache save for PR/feature branch"
+ fi
+ fi
+
+ echo "Docker images ready for use"
+
+ # Phase 2: Build migrate service with GitHub Actions cache
+ - name: Build migrate Docker image with cache
+ working-directory: autogpt_platform
+ run: |
+ # Build the migrate image with buildx for GHA caching
+ docker buildx build \
+ --cache-from type=gha \
+ --cache-to type=gha,mode=max \
+ --target migrate \
+ --tag autogpt_platform-migrate:latest \
+ --load \
+ -f backend/Dockerfile \
+ ..
+
+ # Start services using pre-built images
+ - name: Start Docker services for development
+ working-directory: autogpt_platform
+ run: |
+ # Start essential services (migrate image already built with correct tag)
+ docker compose --profile local up deps --no-build --detach
+ echo "Waiting for services to be ready..."
+
+ # Wait for database to be ready
+ echo "Checking database readiness..."
+ timeout 30 sh -c 'until docker compose exec -T db pg_isready -U postgres 2>/dev/null; do
+ echo " Waiting for database..."
+ sleep 2
+ done' && echo "✅ Database is ready" || echo "⚠️ Database ready check timeout after 30s, continuing..."
+
+ # Check migrate service status
+ echo "Checking migration status..."
+ docker compose ps migrate || echo " Migrate service not visible in ps output"
+
+ # Wait for migrate service to complete
+ echo "Waiting for migrations to complete..."
+ timeout 30 bash -c '
+ ATTEMPTS=0
+ while [ $ATTEMPTS -lt 15 ]; do
+ ATTEMPTS=$((ATTEMPTS + 1))
+
+ # Check using docker directly (more reliable than docker compose ps)
+ CONTAINER_STATUS=$(docker ps -a --filter "label=com.docker.compose.service=migrate" --format "{{.Status}}" | head -1)
+
+ if [ -z "$CONTAINER_STATUS" ]; then
+ echo " Attempt $ATTEMPTS: Migrate container not found yet..."
+ elif echo "$CONTAINER_STATUS" | grep -q "Exited (0)"; then
+ echo "✅ Migrations completed successfully"
+ docker compose logs migrate --tail=5 2>/dev/null || true
+ exit 0
+ elif echo "$CONTAINER_STATUS" | grep -q "Exited ([1-9]"; then
+ EXIT_CODE=$(echo "$CONTAINER_STATUS" | grep -oE "Exited \([0-9]+\)" | grep -oE "[0-9]+")
+ echo "❌ Migrations failed with exit code: $EXIT_CODE"
+ echo "Migration logs:"
+ docker compose logs migrate --tail=20 2>/dev/null || true
+ exit 1
+ elif echo "$CONTAINER_STATUS" | grep -q "Up"; then
+ echo " Attempt $ATTEMPTS: Migrate container is running... ($CONTAINER_STATUS)"
+ else
+ echo " Attempt $ATTEMPTS: Migrate container status: $CONTAINER_STATUS"
+ fi
+
+ sleep 2
+ done
+
+ echo "⚠️ Timeout: Could not determine migration status after 30 seconds"
+ echo "Final container check:"
+ docker ps -a --filter "label=com.docker.compose.service=migrate" || true
+ echo "Migration logs (if available):"
+ docker compose logs migrate --tail=10 2>/dev/null || echo " No logs available"
+ ' || echo "⚠️ Migration check completed with warnings, continuing..."
+
+ # Brief wait for other services to stabilize
+ echo "Waiting 5 seconds for other services to stabilize..."
+ sleep 5
+
+ # Verify installations and provide environment info
+ - name: Verify setup and show environment info
+ run: |
+ echo "=== Python Setup ==="
+ python --version
+ poetry --version
+
+ echo "=== Node.js Setup ==="
+ node --version
+ pnpm --version
+
+ echo "=== Additional Tools ==="
+ docker --version
+ docker compose version
+ gh --version || true
+
+ echo "=== Services Status ==="
+ cd autogpt_platform
+ docker compose ps || true
+
+ echo "=== Backend Dependencies ==="
+ cd backend
+ poetry show | head -10 || true
+
+ echo "=== Frontend Dependencies ==="
+ cd ../frontend
+ pnpm list --depth=0 | head -10 || true
+
+ echo "=== Environment Files ==="
+ ls -la ../.env* || true
+ ls -la .env* || true
+ ls -la ../backend/.env* || true
+
+ echo "✅ AutoGPT Platform development environment setup complete!"
+ echo "🚀 Ready for development with Docker services running"
+ echo "📝 Backend server: poetry run serve (port 8000)"
+ echo "🌐 Frontend server: pnpm dev (port 3000)"
\ No newline at end of file