Skip to content

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

fix(backend): make the load tests authenticate through Better Auth (#15482)

Nicholas Tindle784e91b336a9
diff --git a/.claude/skills/perf-compare/SKILL.md b/.claude/skills/perf-compare/SKILL.md
index 42f41e6036..9c68d17479 100644
--- a/.claude/skills/perf-compare/SKILL.md
+++ b/.claude/skills/perf-compare/SKILL.md
@@ -85,32 +85,17 @@ Measuring the candidate preview against `app:dev-behave:cloud` at dev HEAD in th
A preview's backend is `https://pr-<N>-server.agpt.co` (REST under `/api`, health at `/health`), and its frontend is `https://autogpt-pr-<N>.vercel.app`.
-The k6 tests are in `autogpt_platform/backend/load-tests/`. `K6_ENVIRONMENT=PREVIEW` with `PR_NUMBER=<N>` targets a preview.
-
-- **No token needed:** `tests/marketplace/public-access-test.js` (store agents, search, creators).
-
- ```bash
- cd autogpt_platform/backend/load-tests
- k6 run --env K6_ENVIRONMENT=PREVIEW --env PR_NUMBER=<N> \
- --env VUS=5 --env DURATION=10m --env RAMP_UP=30s --env RAMP_DOWN=30s \
- tests/marketplace/public-access-test.js
- ```
-
-- **Token needed:** `tests/api/core-api-test.js`, `tests/api/graph-execution-test.js`, `tests/marketplace/library-access-test.js`, `tests/comprehensive/platform-journey-test.js`. These read `configs/pre-authenticated-tokens.js`, which is gitignored; copy it from `configs/pre-authenticated-tokens.example.js`.
- - The backend verifies Better Auth tokens (ES256, audience `authenticated`) against `<frontend>/api/auth/jwks` and rejects HS256. `generate-tokens.js` signs in against dev's Supabase, so its tokens do not work.
- - Mint a token for a preview's seeded account instead. The `preview-*@previews.agpt.co` accounts use the `PREVIEW_ACCOUNTS_PASSWORD` secret, which is in the team Bitwarden. Tokens last one hour.
- - Export the password first. The recipe below reads it from the environment and sends it on stdin, so it never appears on a command line other users can see. The cookie jar is a private temporary file.
-
- ```bash
- F=https://autogpt-pr-<N>.vercel.app
- jar=$(mktemp)
- python3 -c 'import json, os; print(json.dumps({"email": "[email protected]", "password": os.environ["PREVIEW_ACCOUNTS_PASSWORD"]}))' |
- curl -sS -c "$jar" -H "Origin: $F" -H 'Content-Type: application/json' \
- --data-binary @- "$F/api/auth/sign-in/email" > /dev/null
- curl -sS -b "$jar" "$F/api/auth/token" # {"token": "..."}
- rm -f "$jar"
- ```
+The k6 tests are in `autogpt_platform/backend/load-tests/`; its `README.md` covers every test and setting. Run them on your own machine: `./k6.sh` runs k6 in Docker (`grafana/k6`), so nothing else needs installing, and a local run costs nothing beyond the load on the target. Do not run them on Grafana Cloud k6. It bills every virtual-user hour, so `./k6.sh` refuses cloud runs unless they are explicitly opted into, and the README reserves them for rare runs the team has agreed to. `K6_ENVIRONMENT=PREVIEW` with `PR_NUMBER=<N>` targets a preview:
+```bash
+cd autogpt_platform/backend/load-tests
+export K6_ENVIRONMENT=PREVIEW PR_NUMBER=<N>
+VUS=5 DURATION=10m RAMP_UP=30s RAMP_DOWN=30s ./k6.sh run tests/api/core-api-test.js
+```
+
+- **Authentication needs no setup on a preview.** The authenticated tests (`tests/api/*`, `tests/basic/*`, `tests/marketplace/library-access-test.js`, `tests/comprehensive/platform-journey-test.js`) sign up throwaway `loadtest-*@previews.agpt.co` accounts in `setup()`, once per run, and use the Better Auth tokens the preview's frontend issues for them. `LOAD_TEST_ACCOUNT_COUNT` sets how many (default 5). `tests/marketplace/public-access-test.js` needs no account at all.
+- **To run as specific accounts**, such as the seeded `preview-*@previews.agpt.co` personas, set `LOAD_TEST_ACCOUNTS` to a JSON array of `{"email", "password"}`. The tests sign each one in once per run. Dev has no throwaway sign-up, so it always needs `LOAD_TEST_ACCOUNTS`.
+- **Note the run's UTC window** (`date -u +%FT%TZ` before and after) for the baseline and candidate windows above, and drive the same test, VUs and duration at both sides.
- **Previews start with an empty database.** Store and library lists return nothing, so costs that grow with data (N+1 queries, scans) do not show. Seed data through the API first, or confirm on dev and prod after merge.
- None of the k6 tests open copilot chats. To measure copilot turns, run chats against the preview yourself, or measure on dev and prod traffic.
- The repository's evals (`backend/copilot/eval`, `backend/copilot/capabilities/eval`, `poetry run expert-style-eval`, `poetry run brain-dump-eval`) are offline quality and cost harnesses. They send no traffic to a deploy.
diff --git a/.gitleaks.toml b/.gitleaks.toml
index 641c3034e5..e219261238 100644
--- a/.gitleaks.toml
+++ b/.gitleaks.toml
@@ -16,8 +16,6 @@ paths = [
'''classic/frontend/build/''',
# Docker dev setup (local dev JWTs/keys only)
'''autogpt_platform/db/docker/''',
- # Load test configs (dev JWTs)
- '''load-tests/configs/''',
# Test files with fake/fixture keys (_test.py, test_*.py, conftest.py)
'''(_test|test_.*|conftest)\.py$''',
# Documentation (only contains placeholder keys in curl/API examples)
diff --git a/autogpt_platform/backend/load-tests/.gitignore b/autogpt_platform/backend/load-tests/.gitignore
index 127927159a..2acba26c24 100644
--- a/autogpt_platform/backend/load-tests/.gitignore
+++ b/autogpt_platform/backend/load-tests/.gitignore
@@ -1,8 +1,12 @@
-# Load testing credentials and sensitive data
+# Files that hold real credentials, so they are never committed:
+# configs/accounts.json keeps LOAD_TEST_ACCOUNTS between runs, and the other
+# two hold access tokens.
+configs/accounts.json
configs/pre-authenticated-tokens.js
configs/k6-credentials.env
+
+# Results
results/
-k6-cloud-results.txt
# Node.js
node_modules/
@@ -15,4 +19,4 @@ yarn-error.log*
.env.local
.env.development.local
.env.test.local
-.env.production.local
\ No newline at end of file
+.env.production.local
diff --git a/autogpt_platform/backend/load-tests/README.md b/autogpt_platform/backend/load-tests/README.md
index 5f989a86ed..4919819f9f 100644
--- a/autogpt_platform/backend/load-tests/README.md
+++ b/autogpt_platform/backend/load-tests/README.md
@@ -1,297 +1,202 @@
# AutoGPT Platform Load Tests
-Clean, streamlined load testing infrastructure for the AutoGPT Platform using k6.
+[k6](https://grafana.com/docs/k6/latest/) tests that drive traffic at a deployed platform: a `!deploy` preview, dev, or a local stack. The backend reports every request to Sentry, and the [perf-compare skill](../../../.claude/skills/perf-compare/SKILL.md) compares runs there.
-## πŸš€ Quick Start
+Run them on your own machine with Docker. Don't run them on Grafana Cloud k6: it bills every virtual-user hour (see [What it costs](#what-it-costs)), and it is only for the rare run that needs more load than one machine can generate, agreed with the team first.
-```bash
-# 1. Set up Supabase service key (required for token generation)
-export SUPABASE_SERVICE_KEY="your-supabase-service-key"
-
-# 2. Generate pre-authenticated tokens (first time setup - creates 160+ tokens with 24-hour expiry)
-node generate-tokens.js --count=160
-
-# 3. Set up k6 cloud credentials (for cloud testing - see Credential Setup section below)
-export K6_CLOUD_TOKEN="your-k6-cloud-token"
-export K6_CLOUD_PROJECT_ID="4254406"
-
-# 4. Run orchestrated load tests locally
-node orchestrator/orchestrator.js DEV local
-
-# 5. Run orchestrated load tests in k6 cloud (recommended)
-node orchestrator/orchestrator.js DEV cloud
-```
-
-## πŸ“‹ Load Test Orchestrator
-
-The AutoGPT Platform uses a comprehensive load test orchestrator (`orchestrator/orchestrator.js`) that runs 12 optimized tests with maximum VU counts:
-
-### Available Tests
-
-#### Basic Tests (Simple validation)
-
-- **connectivity-test**: Basic connectivity and authentication validation
-- **single-endpoint-test**: Individual API endpoint testing with high concurrency
-
-#### API Tests (Core functionality)
+## Run it on your machine
-- **core-api-test**: Core API endpoints (`/api/credits`, `/api/graphs`, `/api/blocks`, `/api/executions`)
-- **graph-execution-test**: Complete graph creation and execution pipeline
+Docker is the only requirement. `./k6.sh` runs k6 from the pinned `grafana/k6` image with this directory mounted, and takes the same arguments as `k6`.
-#### Marketplace Tests (User-facing features)
+### A preview
-- **marketplace-public-test**: Public marketplace browsing and search
-- **marketplace-library-test**: Authenticated marketplace and user library operations
+1. Comment `!deploy` on the pull request. A few minutes later its backend answers at `https://pr-<N>-server.agpt.co/health`, and its frontend is `https://autogpt-pr-<N>.vercel.app`.
+2. Run a test against it:
-#### Comprehensive Tests (End-to-end scenarios)
+ ```bash
+ cd autogpt_platform/backend/load-tests
+ export K6_ENVIRONMENT=PREVIEW PR_NUMBER=<N>
+ VUS=2 DURATION=1m RAMP_UP=10s RAMP_DOWN=10s ./k6.sh run tests/api/core-api-test.js
+ ```
-- **comprehensive-test**: Complete user journey simulation with multiple operations
+Nothing else is needed: on a preview, the tests sign up their own throwaway accounts (see [Accounts](#accounts)).
-### Test Modes
+### Dev
-- **Local Mode**: 5 VUs Γ— 30s - Quick validation and debugging
-- **Cloud Mode**: 80-160 VUs Γ— 3-6m - Real performance testing
-
-## πŸ› οΈ Usage
-
-### Basic Commands
+The whole team shares dev, so keep the load small and tell the team first. The tests cannot create accounts on dev; they sign in with accounts you supply (see [Accounts on dev](#accounts-on-dev)):
```bash
-# Run 12 optimized tests locally (for debugging)
-node orchestrator/orchestrator.js DEV local
-
-# Run 12 optimized tests in k6 cloud (recommended for performance testing)
-node orchestrator/orchestrator.js DEV cloud
-
-# Run against production (coordinate with team!)
-node orchestrator/orchestrator.js PROD cloud
-
-# Run individual test directly with k6
-K6_ENVIRONMENT=DEV VUS=100 DURATION=3m k6 run tests/api/core-api-test.js
+cd autogpt_platform/backend/load-tests
+export K6_ENVIRONMENT=DEV
+export LOAD_TEST_ACCOUNTS="$(cat configs/accounts.json)" # gitignored
+VUS=2 DURATION=1m RAMP_UP=10s RAMP_DOWN=10s ./k6.sh run tests/api/core-api-test.js
```
-### NPM Scripts
+### The variables a run needs
-```bash
-# Run orchestrator locally
-npm run local
+- `K6_ENVIRONMENT`: `PREVIEW`, `DEV` or `LOCAL`.
+- `PR_NUMBER`: the pull request, for `PREVIEW`.
+- `LOAD_TEST_ACCOUNTS`: accounts to sign in with; required on `DEV` (see [`LOAD_TEST_ACCOUNTS`](#load_test_accounts)).
+- `VUS`, `DURATION`, `RAMP_UP`, `RAMP_DOWN`: the load shape. Each test has its own defaults.
-# Run orchestrator in k6 cloud
-npm run cloud
-```
-
-## πŸ”§ Test Configuration
+[Environment variables](#environment-variables) lists the rest.
-### Pre-Authenticated Tokens
+### The Docker command
-- **Generation**: Run `node generate-tokens.js --count=160` to create tokens
-- **File**: `configs/pre-authenticated-tokens.js` (gitignored for security)
-- **Capacity**: 160+ tokens supporting high-concurrency testing
-- **Expiry**: 24 hours (86400 seconds) - extended for long-duration testing
-- **Benefit**: Eliminates Supabase auth rate limiting at scale
-- **Regeneration**: Run `node generate-tokens.js --count=160` when tokens expire after 24 hours
+`./k6.sh` passes each setting into the container by name, so values such as `LOAD_TEST_ACCOUNTS` never appear on a command line. Without the wrapper, the two runs above are:
-### Environment Configuration
+```bash
+cd autogpt_platform/backend/load-tests
-- **LOCAL**: `http://localhost:8006` (local development)
-- **DEV**: `https://dev-server.agpt.co` (development environment)
-- **PROD**: `https://api.agpt.co` (production environment - coordinate with team!)
-- **PREVIEW**: `https://pr-<N>-server.agpt.co`, the `!deploy` preview of PR `<N>`. Set `PR_NUMBER=<N>` next to `K6_ENVIRONMENT=PREVIEW`. A preview starts with an empty database and has its own sign-in, so `generate-tokens.js` tokens are not valid there; `tests/marketplace/public-access-test.js` needs no token. `tests/basic/connectivity-test.js` also calls the preview's own Supabase project, so it needs `SUPABASE_URL` and `SUPABASE_ANON_KEY` for that project. Measuring a preview against a baseline is covered in `.claude/skills/perf-compare/SKILL.md`.
+# A preview
+docker run --rm -i --user "$(id -u):$(id -g)" -v "$PWD:/load-tests" -w /load-tests \
+ -e K6_ENVIRONMENT=PREVIEW -e PR_NUMBER=<N> \
+ -e VUS=2 -e DURATION=1m -e RAMP_UP=10s -e RAMP_DOWN=10s \
+ grafana/k6:2.3.0 run tests/api/core-api-test.js
-## πŸ“Š Performance Testing Features
+# Dev, with LOAD_TEST_ACCOUNTS exported in your shell
+docker run --rm -i --user "$(id -u):$(id -g)" -v "$PWD:/load-tests" -w /load-tests \
+ -e K6_ENVIRONMENT=DEV -e LOAD_TEST_ACCOUNTS \
+ -e VUS=2 -e DURATION=1m -e RAMP_UP=10s -e RAMP_DOWN=10s \
+ grafana/k6:2.3.0 run tests/api/core-api-test.js
+```
-### Real-Time Monitoring
+### What it costs
-- **k6 Cloud Dashboard**: Live performance metrics during cloud test execution
-- **URL Tracking**: Test URLs automatically saved to `k6-cloud-results.txt`
-- **Error Tracking**: Detailed failure analysis and HTTP status monitoring
-- **Custom Metrics**: Request success/failure rates, response times, user journey tracking
-- **Authentication Monitoring**: Tracks auth success/failure rates separately from HTTP errors
+A run on your machine costs nothing beyond the load it puts on the target: the preview's or dev's own pods and database.
-### Load Testing Capabilities
+Grafana Cloud k6 bills each test as its peak VUs times its whole duration, in virtual-user hours (VUh), whether it runs in the cloud (`k6 cloud run`) or streams results there (`--out cloud`). The Grafana Cloud org is on the Free plan, which allows 500 VUh a month; beyond that the paid plan costs about $0.15 per VUh on top of a $19 monthly platform fee. A 5-VU, 11-minute comparison run is about 1 VUh, and the orchestrator's full suite is about 175 VUh. So run locally; see [Grafana Cloud k6](#grafana-cloud-k6-rare-opt-in) for the exception.
-- **High Concurrency**: Up to 160+ virtual users per test
-- **Authentication Scaling**: Pre-auth tokens support 160+ concurrent users
-- **Sequential Execution**: Multiple tests run one after another with proper delays
-- **Cloud Infrastructure**: Tests run on k6 cloud servers for consistent results
-- **ES Module Support**: Full ES module compatibility with modern JavaScript features
+## Tests
-## πŸ“ˆ Performance Expectations
+| Test | Auth | What it does |
+| ---------------------------------------------- | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `tests/basic/connectivity-test.js` | yes | Backend `/health`, the frontend's JWKS, and one authenticated request |
+| `tests/basic/single-endpoint-test.js` | yes | One endpoint (`ENDPOINT=credits\|graphs\|blocks\|executions`) |
+| `tests/api/core-api-test.js` | yes | `/api/credits`, `/api/graphs` and `/api/blocks` together |
+| `tests/api/graph-execution-test.js` | yes | Creates small graphs and executes them. An account without credits, or without a subscription where the paywall applies, gets a 402; k6 counts those in `graph_executions_refused_no_credits_total` and `graph_executions_refused_no_subscription_total` rather than as runs |
+| `tests/marketplace/public-access-test.js` | no | Store browsing: agents, search, categories, creators |
+| `tests/marketplace/library-access-test.js` | yes | Library listing and search; adds a store agent to the library when the store has one |
+| `tests/comprehensive/platform-journey-test.js` | yes | Credits, graphs, blocks and executions with think time |
-### Validated Performance Limits
+A preview starts with an empty database: store and library lists come back empty, so costs that grow with data do not show there.
-- **Core API**: 100+ VUs successfully handling `/api/credits`, `/api/graphs`, `/api/blocks`, `/api/executions`
-- **Graph Execution**: 80+ VUs for complete workflow pipeline
-- **Marketplace Browsing**: 160 VUs for public marketplace access (verified)
-- **Marketplace Library**: 160 VUs for authenticated library operations (verified)
-- **Authentication**: 160+ concurrent users with pre-authenticated tokens
+## How authentication works
-### Target Metrics
+The backend accepts only the ES256 JWTs that the frontend's Better Auth issues (audience `authenticated`), and verifies them against `<frontend>/api/auth/jwks`. `lib/auth.js` gets them the way the browser does:
-- **P95 Latency**: Target < 5 seconds (marketplace), < 2 seconds (core API)
-- **P99 Latency**: Target < 10 seconds (marketplace), < 5 seconds (core API)
-- **Success Rate**: Target > 95% under normal load
-- **Error Rate**: Target < 5% for all endpoints
+- `setup()` signs each account in once per run (`POST <frontend>/api/auth/sign-in/email`, or `sign-up/email` for throwaway accounts) and exchanges the session for a token (`GET <frontend>/api/auth/token`). Every VU then uses one account's token, so a run signs in once per account however many VUs it has.
+- Better Auth lets one IP make 3 sign-in or sign-up requests, then refuses more until 10 seconds pass without one. `setup()` sends them in bursts of 3 with an 11-second pause, and waits out any `429`. Five accounts take about 15 seconds.
+- Tokens last an hour. Each VU fetches a new one with its account's session 10 to 20 minutes before the old one expires, so runs can last longer than an hour.
+- Before the load starts, `setup()` sends one authenticated request and stops the run unless it succeeds.
-### Recent Performance Results (160 VU Test - Verified)
+Passwords, cookies and tokens stay in memory and are never logged.
-- **Marketplace Library Operations**: 500-1000ms response times at 160 VUs
-- **Authentication**: 100% success rate with pre-authenticated tokens
-- **Library Journeys**: 5 operations per journey completing successfully
-- **Test Duration**: 6+ minutes sustained load without degradation
-- **k6 Cloud Execution**: Stable performance on Amazon US Columbus infrastructure
+## Accounts
-## πŸ” Troubleshooting
+### Accounts on previews
-### Common Issues
+With `LOAD_TEST_ACCOUNTS` unset, `setup()` signs up `LOAD_TEST_ACCOUNT_COUNT` (default 5) throwaway accounts named `loadtest-<run>-<n>@previews.agpt.co`, with random passwords that never leave the run. This works because previews allow sign-up for `@previews.agpt.co` and do not require email verification, so no email is sent and no shared secret is needed. The accounts stay in the preview's database until the preview is torn down.
-**1. Authentication Failures**
+To run as the seeded personas instead (`[email protected]` and the others), set `LOAD_TEST_ACCOUNTS`. Their password is the `PREVIEW_ACCOUNTS_PASSWORD` secret in the team Bitwarden:
-```
-❌ No valid authentication token available
-❌ Token has expired
+```bash
+export LOAD_TEST_ACCOUNTS="$(jq -n --arg p "$PREVIEW_ACCOUNTS_PASSWORD" \
+ '[{email: "[email protected]", password: $p}]')"
```
-- **Solution**: Run `node generate-tokens.js --count=160` to create fresh 24-hour tokens
-- **Note**: Use `--count` parameter to generate appropriate number of tokens for your test scale
+### Accounts on dev
-**2. Cloud Credentials Missing**
+Dev's sign-up allowlist does not include `@previews.agpt.co`, and `@agpt.co` addresses can only sign up through Google, so the tests cannot sign up on dev. Running the authenticated tests there needs:
-```
-❌ Missing k6 cloud credentials
-```
+- A few dedicated email-and-password accounts. An admin can create them with Better Auth's admin API (`POST https://dev-builder.agpt.co/api/auth/admin/create-user`, signed in as an admin), or a domain for them can be added to dev's `AUTH_SIGNUP_ALLOWLIST` (`DEV_AUTH_SIGNUP_ALLOWLIST` in the infra repo's dev CD workflow) so they can sign up.
+- Verified email addresses, if dev requires email verification.
+- Their passwords in the team secret store, handed to the tests as `LOAD_TEST_ACCOUNTS`.
-- **Solution**: Set `K6_CLOUD_TOKEN` and `K6_CLOUD_PROJECT_ID=4254406`
+### `LOAD_TEST_ACCOUNTS`
-**3. k6 Cloud VU Scaling Issue**
+A JSON array of accounts that already exist on the target:
-```
-❌ Test shows only 5 VUs instead of requested 100+ VUs
+```json
+[{ "email": "[email protected]", "password": "..." }]
```
-- **Problem**: Using `K6_ENVIRONMENT=DEV VUS=160 k6 cloud run test.js` (incorrect)
-- **Solution**: Use `k6 cloud run --env K6_ENVIRONMENT=DEV --env VUS=160 test.js` (correct)
-- **Note**: The unified test runner (`run-tests.js`) already uses the correct syntax
+The tests read it only from the environment. To keep it between runs, save it as `configs/accounts.json` (gitignored) and `export LOAD_TEST_ACCOUNTS="$(cat configs/accounts.json)"`. The tests sign in each account in the list once; VUs share them round-robin.
-**4. Setup Verification Failed**
+### Local stack
-```
-❌ Verification failed
-```
+`K6_ENVIRONMENT=LOCAL` targets `http://localhost:8006` and `http://localhost:3000`; inside Docker, `./k6.sh` points it at `host.docker.internal`. The tests sign up throwaway accounts unless the local frontend restricts sign-up.
-- **Solution**: Check tokens exist and local API is accessible
+## Environment variables
-### Required Setup
+`./k6.sh` passes these into the container by name when they are set in your shell, so values never appear on a command line.
-**1. Supabase Service Key (Required for all testing):**
+| Variable | Used for |
+| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------- |
+| `K6_ENVIRONMENT` | `PREVIEW`, `DEV`, `LOCAL` or `PROD` (default `DEV`) |
+| `PR_NUMBER` | The pull request whose preview `PREVIEW` targets |
+| `LOAD_TEST_ACCOUNTS` | Accounts to sign in with; required on `DEV` and `PROD` |
+| `LOAD_TEST_ACCOUNT_COUNT` | Throwaway accounts to sign up when `LOAD_TEST_ACCOUNTS` is unset (default 5, at most 100) |
+| `VUS`, `DURATION`, `RAMP_UP`, `RAMP_DOWN` | Load shape |
+| `REQUESTS_PER_VU` | Requests per VU per iteration |
+| `ENDPOINT`, `CONCURRENT_REQUESTS` | `single-endpoint-test.js` only |
+| `THRESHOLD_*` | Threshold overrides in the marketplace tests |
+| `K6_ALLOW_PROD` | Must be `true` for `PROD`; agree a production run with the team first |
+| `K6_CLOUD_TOKEN`, `K6_CLOUD_STACK_ID`, `K6_CLOUD_PROJECT_ID` | Grafana Cloud k6 only, and passed in only with `--cloud` |
+| `K6_IMAGE` | Another k6 image for `./k6.sh` (default `grafana/k6:2.3.0`) |
+| `K6_BIN` | A k6 binary for the orchestrator to use instead of `./k6.sh` |
-```bash
-# Option 1: From your local environment (if available)
-export SUPABASE_SERVICE_KEY="your-supabase-service-key"
+## Results in Sentry and perf-compare
-# Option 2: From Kubernetes secret (for platform developers)
-kubectl get secret supabase-service-key -o jsonpath='{.data.service-key}' | base64 -d
+k6 prints its own summary at the end of a run. For comparisons, use the backend's traces in Sentry:
-# Option 3: From Supabase dashboard
-# Go to Project Settings > API > service_role key (never commit this!)
-```
+| Target | Sentry project | Environment |
+| ------- | ------------------------------------------------------------------ | -------------------------------- |
+| Preview | `autogpt-server-previews` (backend), `builder-previews` (frontend) | `pr-<N>` |
+| Dev | `autogpt-server` (backend), `builder` (frontend) | `app:dev-behave:cloud` (backend) |
-**2. Generate Pre-Authenticated Tokens (Required):**
+Each route is a transaction named by its path, such as `/api/credits` or `/api/graphs/{graph_id}/execute/{graph_version}`. `/health` is not traced, and the sign-in requests from `setup()` go to the frontend project. Sentry's `failure_rate()` counts the 402 an account without credits gets from graph execution, although k6 does not. Note the UTC start and end of each run (`date -u +%FT%TZ`), wait a few minutes for Sentry to ingest the last events, then compare windows with perf-compare. For example, a candidate preview against a control preview under the same load:
```bash
-# Creates 160 tokens with 24-hour expiry - prevents auth rate limiting
-node generate-tokens.js --count=160
-
-# Generate fewer tokens for smaller tests (minimum 10)
-node generate-tokens.js --count=50
-
-# Regenerate when tokens expire (every 24 hours)
-node generate-tokens.js --count=160
+S=$(git rev-parse --show-toplevel)/.claude/skills/perf-compare/scripts/perf_compare.py
+python3 "$S" --project autogpt-server-previews --transaction '/api/credits' \
+ --baseline env=pr-<CONTROL> window=<START>/<END> \
+ --candidate env=pr-<N> window=<START>/<END>
```
-**3. k6 Cloud Credentials (Required for cloud testing):**
+perf-compare wants at least 200 samples from 30 traces per side, and recommends 1000 requests per transaction. The skill covers choosing baselines, interleaving runs and reading the verdict.
-```bash
-# Get from k6 cloud dashboard: https://app.k6.io/account/api-token
-export K6_CLOUD_TOKEN="your-k6-cloud-token"
-export K6_CLOUD_PROJECT_ID="4254406" # AutoGPT Platform project ID
+## Orchestrator
-# Verify credentials work by running orchestrator
-node orchestrator/orchestrator.js DEV cloud
-```
+`orchestrator/orchestrator.js` runs all twelve scenarios in turn, at 100 to 500 VUs each, for about an hour: a stress suite rather than a comparison run. It runs them on your machine, needs Node and Docker, and checks its settings before the first test:
-## πŸ“‚ File Structure
-
-```
-load-tests/
-β”œβ”€β”€ README.md # This documentation
-β”œβ”€β”€ generate-tokens.js # Generate pre-auth tokens (MAIN TOKEN SETUP)
-β”œβ”€β”€ package.json # Node.js dependencies and scripts
-β”œβ”€β”€ orchestrator/
-β”‚ └── orchestrator.js # Main test orchestrator (MAIN ENTRY POINT)
-β”œβ”€β”€ configs/
-β”‚ β”œβ”€β”€ environment.js # Environment URLs and configuration
-β”‚ └── pre-authenticated-tokens.js # Generated tokens (gitignored)
-β”œβ”€β”€ tests/
-β”‚ β”œβ”€β”€ basic/
-β”‚ β”‚ β”œβ”€β”€ connectivity-test.js # Basic connectivity validation
-β”‚ β”‚ └── single-endpoint-test.js # Individual API endpoint testing
-β”‚ β”œβ”€β”€ api/
-β”‚ β”‚ β”œβ”€β”€ core-api-test.js # Core authenticated API endpoints
-β”‚ β”‚ └── graph-execution-test.js # Graph workflow pipeline testing
-β”‚ β”œβ”€β”€ marketplace/
-β”‚ β”‚ β”œβ”€β”€ public-access-test.js # Public marketplace browsing
-β”‚ β”‚ └── library-access-test.js # Authenticated marketplace/library
-β”‚ └── comprehensive/
-β”‚ └── platform-journey-test.js # Complete user journey simulation
-β”œβ”€β”€ results/ # Local test results (auto-created)
-β”œβ”€β”€ unified-results-*.json # Orchestrator results (auto-created)
-└── *.log # Test execution logs (auto-created)
+```bash
+PR_NUMBER=<N> node orchestrator/orchestrator.js PREVIEW
+LOAD_TEST_ACCOUNTS="$(cat configs/accounts.json)" node orchestrator/orchestrator.js DEV
```
-## 🎯 Best Practices
+It writes a summary to `results/`.
-1. **Generate Tokens First**: Always run `node generate-tokens.js --count=160` before testing
-2. **Local for Development**: Use `DEV local` for debugging and development
-3. **Cloud for Performance**: Use `DEV cloud` for actual performance testing
-4. **Monitor Real-Time**: Check k6 cloud dashboards during test execution
-5. **Regenerate Tokens**: Refresh tokens every 24 hours when they expire
-6. **Unified Testing**: Orchestrator runs 12 optimized tests automatically
+## Grafana Cloud k6 (rare, opt-in)
-## πŸš€ Advanced Usage
+Don't run these tests on Grafana Cloud k6; it costs money (see [What it costs](#what-it-costs)). It is only for the rare run that needs more load than one machine can generate, and only after the team agrees to it.
-### Direct k6 Execution
-
-For granular control over individual test scripts:
+It is never a default. `./k6.sh` refuses any command that would use Grafana Cloud (`k6 cloud ...`, `--out cloud` or `K6_OUT=cloud`) unless `--cloud` is its first argument and `K6_CLOUD_TOKEN` is set. Without `--cloud`, it passes no Grafana Cloud credentials into the container. The orchestrator uses Grafana Cloud only with its explicit `cloud` argument. Both print a one-line cost reminder when they do. An agreed cloud run needs a Grafana Cloud k6 API token and the stack's numeric ID; the project is `4254406`:
```bash
-# k6 Cloud execution (recommended for performance testing)
-# IMPORTANT: Use --env syntax for k6 cloud to ensure proper VU scaling
-k6 cloud run --env K6_ENVIRONMENT=DEV --env VUS=160 --env DURATION=5m --env RAMP_UP=30s --env RAMP_DOWN=30s tests/marketplace/library-access-test.js
-
-# Local execution with cloud output (debugging)
-K6_ENVIRONMENT=DEV VUS=10 DURATION=1m \
-k6 run tests/api/core-api-test.js --out cloud
-
-# Local execution with JSON output (offline testing)
-K6_ENVIRONMENT=DEV VUS=10 DURATION=1m \
-k6 run tests/api/core-api-test.js --out json=results.json
-```
-
-### Custom Token Generation
-
-```bash
-# Generate specific number of tokens
-node generate-tokens.js --count=200
-
-# Generate tokens with custom timeout
-node generate-tokens.js --count=100 --timeout=60
+export K6_CLOUD_TOKEN=... K6_CLOUD_STACK_ID=... K6_CLOUD_PROJECT_ID=4254406
+./k6.sh --cloud cloud run --env K6_ENVIRONMENT=PREVIEW --env PR_NUMBER=<N> --env VUS=50 tests/api/core-api-test.js
+PR_NUMBER=<N> node orchestrator/orchestrator.js PREVIEW cloud
```
-## πŸ”— Related Documentation
+A remote cloud run sees only the variables passed with `--env`, and Grafana Cloud keeps those with the run, so the orchestrator refuses to send `LOAD_TEST_ACCOUNTS` there. On a preview the tests sign up throwaway accounts instead. Results appear at <https://significantgravitas.grafana.net/a/k6-app/>.
-- [k6 Documentation](https://k6.io/docs/)
-- [AutoGPT Platform API Documentation](https://agpt.co/docs)
-- [k6 Cloud Dashboard](https://significantgravitas.grafana.net/a/k6-app/)
+## Troubleshooting
-For questions or issues, please refer to the [AutoGPT Platform issues](https://github.com/Significant-Gravitas/AutoGPT/issues).
+- **`DEV needs LOAD_TEST_ACCOUNTS`**: dev and prod have no throwaway sign-up; see [Accounts on dev](#accounts-on-dev).
+- **`Signing up throwaway account ... failed: HTTP 403`**: the target does not allow sign-up for `@previews.agpt.co`. Use `LOAD_TEST_ACCOUNTS`.
+- **`Getting a token ... failed: HTTP 401` right after sign-up**: the target requires email verification, so a sign-up gets no session. Use verified accounts in `LOAD_TEST_ACCOUNTS`.
+- **`... rejected the token from ...`**: the backend verifies tokens against a different frontend's JWKS than the one the test signed in at.
+- **`The first authenticated request ... failed`**: the backend is down, still starting, or not where `K6_ENVIRONMENT` and `PR_NUMBER` point.
+- **`429` during setup**: another sign-in from the same IP used up the burst. `setup()` waits and retries; it fails only after five refusals in a row.
+- **`k6.sh: this command would use Grafana Cloud k6`**: run it on your machine instead; see [Grafana Cloud k6](#grafana-cloud-k6-rare-opt-in) for the agreed exception.
diff --git a/autogpt_platform/backend/load-tests/configs/environment.js b/autogpt_platform/backend/load-tests/configs/environment.js
index aff0bdba50..caa9478057 100644
--- a/autogpt_platform/backend/load-tests/configs/environment.js
+++ b/autogpt_platform/backend/load-tests/configs/environment.js
@@ -1,83 +1,78 @@
-// Environment configuration for AutoGPT Platform load tests
-export const ENV_CONFIG = {
- DEV: {
- API_BASE_URL: "https://dev-server.agpt.co",
- BUILDER_BASE_URL: "https://dev-builder.agpt.co",
- WS_BASE_URL: "wss://dev-ws-server.agpt.co",
- SUPABASE_URL: "https://adfjtextkuilwuhzdjpf.supabase.co",
- SUPABASE_ANON_KEY:
- "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6ImFkZmp0ZXh0a3VpbHd1aHpkanBmIiwicm9sZSI6ImFub24iLCJpYXQiOjE3MzAyNTE3MDIsImV4cCI6MjA0NTgyNzcwMn0.IuQNXsHEKJNxtS9nyFeqO0BGMYN8sPiObQhuJLSK9xk",
- },
- LOCAL: {
- API_BASE_URL: "http://localhost:8006",
- BUILDER_BASE_URL: "http://localhost:3000",
- WS_BASE_URL: "ws://localhost:8001",
- SUPABASE_URL: "http://localhost:8000",
- SUPABASE_ANON_KEY:
- "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyAgCiAgICAicm9sZSI6ICJhbm9uIiwKICAgICJpc3MiOiAic3VwYWJhc2UtZGVtbyIsCiAgICAiaWF0IjogMTY0MTc2OTIwMCwKICAgICJleHAiOiAxNzk5NTM1NjAwCn0.dc_X5iR_VP_qT0zsiyj_I_OZ2T9FtRU2BBNWN8Bu4GE",
- },
- PROD: {
- API_BASE_URL: "https://api.agpt.co",
- BUILDER_BASE_URL: "https://builder.agpt.co",
- WS_BASE_URL: "wss://ws-server.agpt.co",
- SUPABASE_URL: "https://supabase.agpt.co",
- SUPABASE_ANON_KEY:
- "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6ImJnd3B3ZHN4YmxyeWloaW51dGJ4Iiwicm9sZSI6ImFub24iLCJpYXQiOjE3MzAyODYzMDUsImV4cCI6MjA0NTg2MjMwNX0.ISa2IofTdQIJmmX5JwKGGNajqjsD8bjaGBzK90SubE0",
- },
-};
+// Targets for the AutoGPT Platform load tests, picked by K6_ENVIRONMENT
+// (default DEV). PREVIEW also needs PR_NUMBER.
+//
+// - BACKEND_URL: the backend's origin; /health lives here.
+// - API_BASE_URL: the backend's REST root (BACKEND_URL + /api).
+// - FRONTEND_URL: the Next.js app. Better Auth runs under /api/auth there,
+// and the backend verifies tokens against its /api/auth/jwks.
+// - SIGNUP_ALLOWED: whether the authenticated tests may sign up throwaway
+// accounts when LOAD_TEST_ACCOUNTS is unset (see lib/auth.js).
+function target(name, backendUrl, frontendUrl, signupAllowed) {
+ return {
+ NAME: name,
+ BACKEND_URL: backendUrl,
+ API_BASE_URL: `${backendUrl}/api`,
+ FRONTEND_URL: frontendUrl,
+ SIGNUP_ALLOWED: signupAllowed,
+ };
+}
-// Get environment config based on K6_ENVIRONMENT variable (default: DEV)
export function getEnvironmentConfig() {
- const env = __ENV.K6_ENVIRONMENT || "DEV";
- if (env === "PREVIEW") {
- return getPreviewConfig(__ENV.PR_NUMBER);
+ const env = (__ENV.K6_ENVIRONMENT || "DEV").toUpperCase();
+ switch (env) {
+ case "LOCAL": {
+ // k6.sh sets LOCAL_HOST so k6 inside Docker reaches the host's stack.
+ const host = __ENV.LOCAL_HOST || "localhost";
+ return target(
+ "LOCAL",
+ `http://${host}:8006`,
+ `http://${host}:3000`,
+ true,
+ );
+ }
+ case "DEV":
+ return target(
+ "DEV",
+ "https://dev-server.agpt.co",
+ "https://dev-builder.agpt.co",
+ false,
+ );
+ case "PREVIEW":
+ return getPreviewConfig(__ENV.PR_NUMBER);
+ case "PROD":
+ if (__ENV.K6_ALLOW_PROD !== "true") {
+ throw new Error(
+ "K6_ENVIRONMENT=PROD sends load at production; agree it with the team, then set K6_ALLOW_PROD=true",
+ );
+ }
+ return target(
+ "PROD",
+ "https://backend.agpt.co",
+ "https://platform.agpt.co",
+ false,
+ );
+ default:
+ throw new Error(
+ `Unknown K6_ENVIRONMENT "${env}": use PREVIEW, DEV, LOCAL or PROD`,
+ );
}
- return ENV_CONFIG[env];
}
// K6_ENVIRONMENT=PREVIEW PR_NUMBER=<n> targets the `!deploy` preview of PR <n>.
-// Each preview has its own Supabase project that nothing derives from <n>, so
-// tests that call Supabase directly need SUPABASE_URL and SUPABASE_ANON_KEY.
-export function getPreviewConfig(prNumber) {
+function getPreviewConfig(prNumber) {
if (!/^[1-9]\d{0,6}$/.test(prNumber || "")) {
throw new Error(
"K6_ENVIRONMENT=PREVIEW needs PR_NUMBER=<pull request number>",
);
}
- return {
- API_BASE_URL: `https://pr-${prNumber}-server.agpt.co`,
- BUILDER_BASE_URL: `https://autogpt-pr-${prNumber}.vercel.app`,
- WS_BASE_URL: `wss://pr-${prNumber}-ws.agpt.co`,
- SUPABASE_URL: __ENV.SUPABASE_URL,
- SUPABASE_ANON_KEY: __ENV.SUPABASE_ANON_KEY,
- };
+ return target(
+ `PREVIEW pr-${prNumber}`,
+ `https://pr-${prNumber}-server.agpt.co`,
+ `https://autogpt-pr-${prNumber}.vercel.app`,
+ true,
+ );
}
-// Authentication configuration
-export const AUTH_CONFIG = {
- // Test user credentials - REPLACE WITH ACTUAL TEST ACCOUNTS
- TEST_USERS: [
- {
- email: "[email protected]",
- password: "LoadTest123!",
- user_id: "test-user-1",
- },
- {
- email: "[email protected]",
- password: "LoadTest123!",
- user_id: "test-user-2",
- },
- {
- email: "[email protected]",
- password: "LoadTest123!",
- user_id: "test-user-3",
- },
- ],
-
- // JWT token for API access (will be set during test execution)
- JWT_TOKEN: null,
-};
-
// Performance test configurations - Environment variable overrides supported
export const PERFORMANCE_CONFIG = {
// Default load test parameters (override with env vars: VUS, DURATION, RAMP_UP, RAMP_DOWN)
@@ -147,16 +142,3 @@ export function getLoadTestConfig(testType = "default") {
return configs[testType] || configs.default;
}
-
-// Grafana Cloud K6 configuration
-export const GRAFANA_CONFIG = {
- PROJECT_ID: __ENV.K6_CLOUD_PROJECT_ID || "",
- TOKEN: __ENV.K6_CLOUD_TOKEN || "",
- // Tags for organizing test results
- TEST_TAGS: {
- team: "platform",
- service: "autogpt-platform",
- environment: __ENV.K6_ENVIRONMENT || "dev",
- version: __ENV.GIT_COMMIT || "unknown",
- },
-};
diff --git a/autogpt_platform/backend/load-tests/configs/k6-credentials.env.example b/autogpt_platform/backend/load-tests/configs/k6-credentials.env.example
deleted file mode 100644
index 99201d5376..0000000000
--- a/autogpt_platform/backend/load-tests/configs/k6-credentials.env.example
+++ /dev/null
@@ -1,9 +0,0 @@
-# k6 Cloud Credentials (EXAMPLE FILE)
-# Copy this to k6-credentials.env and fill in your actual credentials
-#
-# Get these from: https://app.k6.io/
-# - K6_CLOUD_TOKEN: Your k6 cloud API token
-# - K6_CLOUD_PROJECT_ID: Your project ID
-
-K6_CLOUD_TOKEN=your-k6-cloud-token-here
-K6_CLOUD_PROJECT_ID=your-project-id-here
\ No newline at end of file
diff --git a/autogpt_platform/backend/load-tests/configs/pre-authenticated-tokens.example.js b/autogpt_platform/backend/load-tests/configs/pre-authenticated-tokens.example.js
deleted file mode 100644
index 0328225c0b..0000000000
--- a/autogpt_platform/backend/load-tests/configs/pre-authenticated-tokens.example.js
+++ /dev/null
@@ -1,51 +0,0 @@
-// Pre-authenticated tokens for load testing (EXAMPLE FILE)
-// Copy this to pre-authenticated-tokens.js and run generate-tokens.js to populate
-//
-// ⚠️ SECURITY: The real file contains authentication tokens
-// ⚠️ DO NOT COMMIT TO GIT - Real file is gitignored
-
-export const PRE_AUTHENTICATED_TOKENS = [
- // Will be populated by generate-tokens.js with 350+ real tokens
- // Example structure:
- // {
- // token: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
- // user: "[email protected]",
- // generated: "2025-01-24T10:08:04.123Z",
- // round: 1
- // }
-];
-
-export function getPreAuthenticatedToken(vuId = 1) {
- if (PRE_AUTHENTICATED_TOKENS.length === 0) {
- throw new Error(
- "No pre-authenticated tokens available. Run: node generate-tokens.js",
- );
- }
-
- const tokenIndex = (vuId - 1) % PRE_AUTHENTICATED_TOKENS.length;
- const tokenData = PRE_AUTHENTICATED_TOKENS[tokenIndex];
-
- return {
- access_token: tokenData.token,
- user: { email: tokenData.user },
- generated: tokenData.generated,
- };
-}
-
-export function getPreAuthenticatedHeaders(vuId = 1) {
- const authData = getPreAuthenticatedToken(vuId);
- return {
- "Content-Type": "application/json",
- Authorization: `Bearer ${authData.access_token}`,
- };
-}
-
-export const TOKEN_STATS = {
- total: PRE_AUTHENTICATED_TOKENS.length,
- users: [...new Set(PRE_AUTHENTICATED_TOKENS.map((t) => t.user))].length,
- generated: PRE_AUTHENTICATED_TOKENS[0]?.generated || "unknown",
-};
-
-console.log(
- `πŸ” Loaded ${TOKEN_STATS.total} pre-authenticated tokens from ${TOKEN_STATS.users} users`,
-);
diff --git a/autogpt_platform/backend/load-tests/generate-tokens.js b/autogpt_platform/backend/load-tests/generate-tokens.js
deleted file mode 100644
index c5e503fd1a..0000000000
--- a/autogpt_platform/backend/load-tests/generate-tokens.js
+++ /dev/null
@@ -1,236 +0,0 @@
-#!/usr/bin/env node
-
-/**
- * Generate Pre-Authenticated Tokens for Load Testing
- * Creates configs/pre-authenticated-tokens.js with 350+ tokens
- *
- * This replaces the old token generation scripts with a clean, single script
- */
-
-import https from "https";
-import fs from "fs";
-import path from "path";
-
-// Get Supabase service key from environment (REQUIRED for token generation)
-const SUPABASE_SERVICE_KEY = process.env.SUPABASE_SERVICE_KEY;
-
-if (!SUPABASE_SERVICE_KEY) {
- console.error("❌ SUPABASE_SERVICE_KEY environment variable is required");
- console.error("Get service key from kubectl or environment:");
- console.error('export SUPABASE_SERVICE_KEY="your-service-key"');
- process.exit(1);
-}
-
-// Generate test users (loadtest4-50 are known to work)
-const TEST_USERS = [];
-for (let i = 4; i <= 50; i++) {
- TEST_USERS.push({
- email: `loadtest${i}@example.com`,
- password: "password123",
- });
-}
-
-console.log(
- `πŸ” Generating pre-authenticated tokens from ${TEST_USERS.length} users...`,
-);
-
-async function authenticateUser(user, attempt = 1) {
- return new Promise((resolve) => {
- const postData = JSON.stringify({
- email: user.email,
- password: user.password,
- expires_in: 86400, // 24 hours in seconds (24 * 60 * 60)
- });
-
- const options = {
- hostname: "adfjtextkuilwuhzdjpf.supabase.co",
- path: "/auth/v1/token?grant_type=password",
- method: "POST",
- headers: {
- Authorization: `Bearer ${SUPABASE_SERVICE_KEY}`,
- apikey: SUPABASE_SERVICE_KEY,
- "Content-Type": "application/json",
- "Content-Length": postData.length,
- },
- };
-
- const req = https.request(options, (res) => {
- let data = "";
- res.on("data", (chunk) => (data += chunk));
- res.on("end", () => {
- try {
- if (res.statusCode === 200) {
- const authData = JSON.parse(data);
- resolve(authData.access_token);
- } else if (res.statusCode === 429) {
- // Rate limited - wait and retry
- console.log(
- `⏳ Rate limited for ${user.email}, waiting 5s (attempt ${attempt}/3)...`,
- );
- setTimeout(() => {
- if (attempt < 3) {
- authenticateUser(user, attempt + 1).then(resolve);
- } else {
- console.log(`❌ Max retries exceeded for ${user.email}`);
- resolve(null);
- }
- }, 5000);
- } else {
- console.log(`❌ Auth failed for ${user.email}: ${res.statusCode}`);
- resolve(null);
- }
- } catch (e) {
- console.log(`❌ Parse error for ${user.email}:`, e.message);
- resolve(null);
- }
- });
- });
-
- req.on("error", (err) => {
- console.log(`❌ Request error for ${user.email}:`, err.message);
- resolve(null);
- });
-
- req.write(postData);
- req.end();
- });
-}
-
-async function generateTokens() {
- console.log("πŸš€ Starting token generation...");
- console.log("Rate limit aware - this will take ~10-15 minutes");
- console.log("===========================================\n");
-
- const tokens = [];
- const startTime = Date.now();
-
- // Generate tokens - configurable via --count argument or default to 150
- const targetTokens =
- parseInt(
- process.argv.find((arg) => arg.startsWith("--count="))?.split("=")[1],
- ) ||
- parseInt(process.env.TOKEN_COUNT) ||
- 150;
- const tokensPerUser = Math.ceil(targetTokens / TEST_USERS.length);
- console.log(
- `πŸ“Š Generating ${tokensPerUser} tokens per user (${TEST_USERS.length} users) - Target: ${targetTokens}\n`,
- );
-
- for (let round = 1; round <= tokensPerUser; round++) {
- console.log(`πŸ”„ Round ${round}/${tokensPerUser}:`);
-
- for (
- let i = 0;
- i < TEST_USERS.length && tokens.length < targetTokens;
- i++
- ) {
- const user = TEST_USERS[i];
-
- process.stdout.write(` ${user.email.padEnd(25)} ... `);
-
- const token = await authenticateUser(user);
-
- if (token) {
- tokens.push({
- token,
- user: user.email,
- generated: new Date().toISOString(),
- round: round,
- });
- console.log(`βœ… (${tokens.length}/${targetTokens})`);
- } else {
- console.log(`❌`);
- }
-
- // Respect rate limits - wait 500ms between requests
- if (tokens.length < targetTokens) {
- await new Promise((resolve) => setTimeout(resolve, 500));
- }
- }
-
- if (tokens.length >= targetTokens) break;
-
- // Wait longer between rounds
- if (round < tokensPerUser) {
- console.log(` ⏸️ Waiting 3s before next round...\n`);
- await new Promise((resolve) => setTimeout(resolve, 3000));
- }
- }
-
- const duration = Math.round((Date.now() - startTime) / 1000);
- console.log(`\nβœ… Generated ${tokens.length} tokens in ${duration}s`);
-
- // Create configs directory if it doesn't exist
- const configsDir = path.join(process.cwd(), "configs");
- if (!fs.existsSync(configsDir)) {
- fs.mkdirSync(configsDir, { recursive: true });
- }
-
- // Write tokens to secure file
- const jsContent = `// Pre-authenticated tokens for load testing
-// Generated: ${new Date().toISOString()}
-// Total tokens: ${tokens.length}
-// Generation time: ${duration} seconds
-//
-// ⚠️ SECURITY: This file contains real authentication tokens
-// ⚠️ DO NOT COMMIT TO GIT - File is gitignored
-
-export const PRE_AUTHENTICATED_TOKENS = ${JSON.stringify(tokens, null, 2)};
-
-export function getPreAuthenticatedToken(vuId = 1) {
- if (PRE_AUTHENTICATED_TOKENS.length === 0) {
- throw new Error('No pre-authenticated tokens available');
- }
-
- const tokenIndex = (vuId - 1) % PRE_AUTHENTICATED_TOKENS.length;
- const tokenData = PRE_AUTHENTICATED_TOKENS[tokenIndex];
-
- return {
- access_token: tokenData.token,
- user: { email: tokenData.user },
- generated: tokenData.generated
- };
-}
-
-// Generate single session ID for this test run
-const LOAD_TEST_SESSION_ID = '${new Date().toISOString().slice(0, 16).replace(/:/g, "-")}-' + Math.random().toString(36).substr(2, 8);
-
-export function getPreAuthenticatedHeaders(vuId = 1) {
- const authData = getPreAuthenticatedToken(vuId);
-
- return {
- 'Content-Type': 'application/json',
- 'Authorization': \`Bearer \${authData.access_token}\`,
- 'X-Load-Test-Session': LOAD_TEST_SESSION_ID,
- 'X-Load-Test-VU': vuId.toString(),
- 'X-Load-Test-User': authData.user.email,
- };
-}
-
-export const TOKEN_STATS = {
- total: PRE_AUTHENTICATED_TOKENS.length,
- users: [...new Set(PRE_AUTHENTICATED_TOKENS.map(t => t.user))].length,
- generated: PRE_AUTHENTICATED_TOKENS[0]?.generated || 'unknown'
-};
-
-console.log(\`πŸ” Loaded \${TOKEN_STATS.total} pre-authenticated tokens from \${TOKEN_STATS.users} users\`);
-`;
-
- const tokenFile = path.join(configsDir, "pre-authenticated-tokens.js");
- fs.writeFileSync(tokenFile, jsContent);
-
- console.log(`πŸ’Ύ Saved to configs/pre-authenticated-tokens.js`);
- console.log(`πŸš€ Ready for ${tokens.length} concurrent VU load testing!`);
- console.log(
- `\nπŸ”’ Security Note: Token file is gitignored and will not be committed`,
- );
-
- return tokens.length;
-}
-
-// Run if called directly
-if (process.argv[1] === new URL(import.meta.url).pathname) {
- generateTokens().catch(console.error);
-}
-
-export { generateTokens };
diff --git a/autogpt_platform/backend/load-tests/k6.sh b/autogpt_platform/backend/load-tests/k6.sh
new file mode 100755
index 0000000000..a4f549be5c
--- /dev/null
+++ b/autogpt_platform/backend/load-tests/k6.sh
@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# Runs k6 on this machine from the grafana/k6 image, with this directory
+# mounted, so Docker is the only thing to install. Arguments go to k6
+# unchanged:
+#
+# ./k6.sh run tests/api/core-api-test.js
+#
+# Settings and secrets travel by name: each K6_*, LOAD_TEST_* or test setting
+# (PR_NUMBER, VUS, DURATION, ...) exported in the calling shell is passed into
+# the container without its value appearing on a command line.
+#
+# Grafana Cloud k6 bills every virtual-user hour, so these tests run locally.
+# A command that would use it (k6 cloud ..., --out cloud, K6_OUT=cloud) is
+# refused unless --cloud comes first and K6_CLOUD_TOKEN is set:
+#
+# ./k6.sh --cloud cloud run tests/api/core-api-test.js
+set -euo pipefail
+
+cd "$(dirname "$0")"
+
+cloud_opt_in=false
+if [ "${1:-}" = "--cloud" ]; then
+ cloud_opt_in=true
+ shift
+fi
+
+uses_cloud=false
+if [ "${1:-}" = "cloud" ]; then
+ uses_cloud=true
+fi
+case "${K6_OUT:-}" in
+ *cloud*) uses_cloud=true ;;
+esac
+previous=""
+for arg in "$@"; do
+ case "$arg" in
+ --out=*cloud* | -o=*cloud*) uses_cloud=true ;;
+ esac
+ if { [ "$previous" = "--out" ] || [ "$previous" = "-o" ]; } &&
+ [[ "$arg" == *cloud* ]]; then
+ uses_cloud=true
+ fi
+ previous="$arg"
+done
+
+if [ "$uses_cloud" = true ]; then
+ if [ "$cloud_opt_in" != true ] || [ -z "${K6_CLOUD_TOKEN:-}" ]; then
+ {
+ echo "k6.sh: this command would use Grafana Cloud k6, which bills every virtual-user hour."
+ echo "k6.sh: run it on this machine instead. Grafana Cloud is only for load one machine"
+ echo "k6.sh: cannot generate, agreed with the team first: then put --cloud first and set"
+ echo "k6.sh: K6_CLOUD_TOKEN (see README.md)."
+ } >&2
+ exit 2
+ fi
+ echo "k6.sh: using Grafana Cloud k6, billed per virtual-user hour (peak VUs x duration); local runs are the default." >&2
+fi
+
+image="${K6_IMAGE:-grafana/k6:2.3.0}"
+docker_args=(
+ run --rm -i
+ --user "$(id -u):$(id -g)"
+ -v "$PWD:/load-tests" -w /load-tests
+ # K6_ENVIRONMENT=LOCAL reaches the stack on the host through this name.
+ --add-host=host.docker.internal:host-gateway
+ -e "LOCAL_HOST=${LOCAL_HOST:-host.docker.internal}"
+)
+if [ -t 0 ] && [ -t 1 ]; then
+ docker_args+=(-t)
+fi
+
+for name in $(compgen -e); do
+ case "$name" in
+ # Without --cloud, k6 in the container gets no Grafana Cloud credentials.
+ K6_CLOUD_*)
+ if [ "$cloud_opt_in" = true ]; then
+ docker_args+=(-e "$name")
+ fi
+ ;;
+ K6_* | LOAD_TEST_* | PR_NUMBER | VUS | DURATION | RAMP_UP | RAMP_DOWN | \
+ REQUESTS_PER_VU | CONCURRENT_REQUESTS | ENDPOINT | THRESHOLD_*)
+ docker_args+=(-e "$name")
+ ;;
+ esac
+done
+
+exec docker "${docker_args[@]}" "$image" "$@"
diff --git a/autogpt_platform/backend/load-tests/lib/auth.js b/autogpt_platform/backend/load-tests/lib/auth.js
new file mode 100644
index 0000000000..dd837b964d
--- /dev/null
+++ b/autogpt_platform/backend/load-tests/lib/auth.js
@@ -0,0 +1,337 @@
+// Better Auth sign-in for the load tests.
+//
+// The backend only accepts the ES256 JWTs that the frontend's Better Auth
+// issues at GET /api/auth/token (audience "authenticated", valid for an hour).
+// setup() signs each account in once per run and gets its token; every VU
+// then uses one account's token. A run therefore signs in once per account,
+// however many VUs and iterations it has.
+//
+// Accounts come from LOAD_TEST_ACCOUNTS, a JSON array of
+// {"email": "...", "password": "..."}. When that is unset and the target
+// allows it (PREVIEW and LOCAL), setup() signs up LOAD_TEST_ACCOUNT_COUNT
+// throwaway accounts instead, with random passwords that never leave the run.
+//
+// Passwords, cookies and tokens are never logged.
+import { sleep } from "k6";
+import encoding from "k6/encoding";
+import http from "k6/http";
+
+import { getEnvironmentConfig } from "../configs/environment.js";
+
+const config = getEnvironmentConfig();
+const AUTH_URL = `${config.FRONTEND_URL}/api/auth`;
+
+// Better Auth lets an IP make 3 sign-in (or sign-up) requests, then refuses
+// more until 10 s pass without one, so setup() sends them in bursts of 3.
+const AUTH_BURST = 3;
+const AUTH_PAUSE_SECONDS = 11;
+const MAX_ATTEMPTS = 5;
+
+const DEFAULT_THROWAWAY_ACCOUNTS = 5;
+const MAX_THROWAWAY_ACCOUNTS = 100;
+// Previews allow sign-up for this domain, and nothing is mailed to it.
+const THROWAWAY_EMAIL_DOMAIN = "previews.agpt.co";
+
+// A VU refreshes its token 10 to 20 minutes before the token expires, at a
+// random point so the VUs sharing an account don't all refresh at once.
+const REFRESH_MARGIN_MS = 10 * 60 * 1000;
+const REFRESH_SPREAD_MS = 10 * 60 * 1000;
+const REFRESH_RETRY_MS = 60 * 1000;
+
+const ACCOUNTS_FORMAT = 'a JSON array of {"email": "...", "password": "..."}';
+
+// Each VU runs this module in its own runtime, so this is per-VU state.
+let vuSession = null;
+
+// For options.setupTimeout: setup() pauses between bursts of sign-ins.
+export function authSetupTimeout() {
+ const listed = readAccounts();
+ const count = listed ? listed.length : readAccountCount();
+ const pauses = Math.ceil(count / AUTH_BURST) - 1;
+ return `${60 + pauses * AUTH_PAUSE_SECONDS + count * 10}s`;
+}
+
+// Call from setup() and return the result (merged with any other setup data).
+export function setupAuth() {
+ const listed = readAccounts();
+ if (!listed && !config.SIGNUP_ALLOWED) {
+ throw new Error(
+ `${config.NAME} needs LOAD_TEST_ACCOUNTS, ${ACCOUNTS_FORMAT}. See README.md.`,
+ );
+ }
+
+ const accounts = listed || throwawayAccounts(readAccountCount());
+ const sessions = accounts.map((account, index) => {
+ if (index > 0 && index % AUTH_BURST === 0) sleep(AUTH_PAUSE_SECONDS);
+ const label = `account ${index + 1} of ${accounts.length}`;
+ return listed ? signIn(account, label) : signUp(account, index, label);
+ });
+
+ checkBackendAcceptsToken(sessions[0].token);
+ console.log(
+ `${listed ? "Signed in" : "Signed up"} ${sessions.length} account(s) at ${config.FRONTEND_URL}`,
+ );
+ return { auth: sessions };
+}
+
+// Headers for the current VU's account, from setupAuth()'s data.
+export function authHeaders(data) {
+ if (vuSession === null) {
+ const sessions = data && data.auth;
+ if (!sessions || sessions.length === 0) {
+ throw new Error(
+ "No sessions: the test's setup() must return setupAuth()'s result",
+ );
+ }
+ const session = sessions[(__VU - 1) % sessions.length];
+ vuSession = {
+ cookies: session.cookies,
+ token: session.token,
+ refreshAt: refreshTime(session.expiresAt),
+ };
+ }
+ if (Date.now() >= vuSession.refreshAt) refreshToken();
+ return {
+ "Content-Type": "application/json",
+ Authorization: `Bearer ${vuSession.token}`,
+ };
+}
+
+function readAccounts() {
+ const raw = __ENV.LOAD_TEST_ACCOUNTS;
+ if (!raw) return null;
+
+ let parsed;
+ try {
+ parsed = JSON.parse(raw);
+ } catch (e) {
+ // The parser's own message can quote the input, passwords included.
+ throw new Error(
+ `LOAD_TEST_ACCOUNTS is not valid JSON: use ${ACCOUNTS_FORMAT}`,
+ );
+ }
+ const valid =
+ Array.isArray(parsed) &&
+ parsed.length > 0 &&
+ parsed.every(
+ (account) =>
+ account &&
+ isNonEmptyString(account.email) &&
+ isNonEmptyString(account.password),
+ );
+ if (!valid) {
+ throw new Error(`LOAD_TEST_ACCOUNTS must be ${ACCOUNTS_FORMAT}`);
+ }
+ return parsed.map(({ email, password }) => ({ email, password }));
+}
+
+function isNonEmptyString(value) {
+ return typeof value === "string" && value.length > 0;
+}
+
+function readAccountCount() {
+ const raw = __ENV.LOAD_TEST_ACCOUNT_COUNT;
+ if (raw === undefined || raw === "") return DEFAULT_THROWAWAY_ACCOUNTS;
+ const count = Number(raw);
+ if (!Number.isInteger(count) || count < 1 || count > MAX_THROWAWAY_ACCOUNTS) {
+ throw new Error(
+ `LOAD_TEST_ACCOUNT_COUNT must be a whole number from 1 to ${MAX_THROWAWAY_ACCOUNTS}`,
+ );
+ }
+ return count;
+}
+
+function throwawayAccounts(count) {
+ const runId = randomHex(4);
+ const accounts = [];
+ for (let i = 1; i <= count; i++) {
+ accounts.push({
+ email: `loadtest-${runId}-${i}@${THROWAWAY_EMAIL_DOMAIN}`,
+ password: randomHex(24),
+ });
+ }
+ return accounts;
+}
+
+function randomHex(bytes) {
+ const buffer = new Uint8Array(bytes);
+ crypto.getRandomValues(buffer);
+ return Array.from(buffer, (b) => b.toString(16).padStart(2, "0")).join("");
+}
+
+function signIn(account, label) {
+ const jar = new http.CookieJar();
+ const res = authPost(
+ "/sign-in/email",
+ { email: account.email, password: account.password },
+ jar,
+ );
+ if (res.status !== 200) {
+ throw new Error(
+ `Signing in ${label} from LOAD_TEST_ACCOUNTS failed: ${describe(res)}`,
+ );
+ }
+ return startSession(jar, label, "");
+}
+
+function signUp(account, index, label) {
+ const jar = new http.CookieJar();
+ const res = authPost(
+ "/sign-up/email",
+ {
+ name: `Load test ${index + 1}`,
+ email: account.email,
+ password: account.password,
+ },
+ jar,
+ );
+ if (res.status !== 200) {
+ throw new Error(
+ `Signing up throwaway ${label} at ${config.FRONTEND_URL} failed: ${describe(res)}. ` +
+ "Set LOAD_TEST_ACCOUNTS to use existing accounts instead.",
+ );
+ }
+ return startSession(
+ jar,
+ label,
+ " A sign-up gets no session when the target requires email verification;" +
+ " set LOAD_TEST_ACCOUNTS to verified accounts instead.",
+ );
+}
+
+function startSession(jar, label, hint) {
+ const cookies = {};
+ const byName = jar.cookiesForURL(`${AUTH_URL}/token`);
+ for (const name of Object.keys(byName)) cookies[name] = byName[name][0];
+
+ const res = requestToken(cookies);
+ if (res.status !== 200) {
+ throw new Error(
+ `Getting a token for ${label} failed: ${describe(res)}.${hint}`,
+ );
+ }
+ const token = readToken(res);
+ return { cookies, token, expiresAt: tokenExpiry(token) };
+}
+
+// If the backend rejects the token, or is down or elsewhere, every request
+// in the run fails fast, which would otherwise look like a very fast run.
+function checkBackendAcceptsToken(token) {
+ const res = http.get(`${config.API_BASE_URL}/credits`, {
+ headers: { Authorization: `Bearer ${token}` },
+ });
+ if (res.status === 401 || res.status === 403) {
+ throw new Error(
+ `${config.API_BASE_URL} rejected the token from ${config.FRONTEND_URL} ` +
+ `(${describe(res)}): the backend must verify tokens against that frontend's /api/auth/jwks`,
+ );
+ }
+ if (res.status !== 200) {
+ throw new Error(
+ `The first authenticated request, to ${config.API_BASE_URL}/credits, failed ` +
+ `(${describe(res)}): check that the backend is up and that K6_ENVIRONMENT points at it`,
+ );
+ }
+}
+
+function refreshToken() {
+ const res = requestToken(vuSession.cookies);
+ if (res.status === 200) {
+ try {
+ const token = readToken(res);
+ vuSession.token = token;
+ vuSession.refreshAt = refreshTime(tokenExpiry(token));
+ return;
+ } catch (e) {
+ // Fall through and retry later, like any other failed refresh.
+ }
+ }
+ console.warn(
+ `VU ${__VU} could not refresh its token (HTTP ${res.status}); retrying in a minute`,
+ );
+ vuSession.refreshAt = Date.now() + REFRESH_RETRY_MS;
+}
+
+function refreshTime(expiresAt) {
+ return expiresAt - REFRESH_MARGIN_MS - Math.random() * REFRESH_SPREAD_MS;
+}
+
+function authPost(path, body, jar) {
+ return withRetries(() =>
+ http.post(`${AUTH_URL}${path}`, JSON.stringify(body), {
+ headers: { "Content-Type": "application/json" },
+ jar,
+ redirects: 0,
+ }),
+ );
+}
+
+// The session cookies go in a fresh jar each time, so a VU's own jar never
+// collects them and they reach only the frontend.
+function requestToken(cookies) {
+ return withRetries(() =>
+ http.get(`${AUTH_URL}/token`, {
+ cookies,
+ jar: new http.CookieJar(),
+ redirects: 0,
+ }),
+ );
+}
+
+function withRetries(send) {
+ let res = send();
+ for (
+ let attempt = 1;
+ res.status === 429 && attempt < MAX_ATTEMPTS;
+ attempt++
+ ) {
+ sleep(retryAfterSeconds(res) + 1);
+ res = send();
+ }
+ return res;
+}
+
+function retryAfterSeconds(res) {
+ const seconds = parseInt(res.headers["X-Retry-After"], 10);
+ return Number.isFinite(seconds) && seconds > 0
+ ? Math.min(seconds, 60)
+ : AUTH_PAUSE_SECONDS;
+}
+
+function readToken(res) {
+ let token;
+ try {
+ token = res.json("token");
+ } catch (e) {
+ token = undefined;
+ }
+ if (typeof token !== "string" || token.split(".").length !== 3) {
+ throw new Error(`${AUTH_URL}/token answered without a JWT`);
+ }
+ return token;
+}
+
+function tokenExpiry(token) {
+ const payload = JSON.parse(
+ encoding.b64decode(token.split(".")[1], "rawurl", "s"),
+ );
+ if (typeof payload.exp !== "number") {
+ throw new Error(`The JWT from ${AUTH_URL}/token has no exp claim`);
+ }
+ return payload.exp * 1000;
+}
+
+// Error bodies carry a code and a message (Better Auth) or a detail (the
+// backend). None of them repeats the credentials that were sent.
+function describe(res) {
+ if (res.status === 0) return res.error || "no response";
+ let detail = "";
+ try {
+ const body = res.json();
+ detail = (body && (body.code || body.message || body.detail)) || "";
+ } catch (e) {
+ detail = "";
+ }
+ if (typeof detail !== "string") detail = "";
+ return `HTTP ${res.status}${detail ? ` ${String(detail).slice(0, 160)}` : ""}`;
+}
diff --git a/autogpt_platform/backend/load-tests/orchestrator/orchestrator.js b/autogpt_platform/backend/load-tests/orchestrator/orchestrator.js
index e7a12ddd66..52320958be 100644
--- a/autogpt_platform/backend/load-tests/orchestrator/orchestrator.js
+++ b/autogpt_platform/backend/load-tests/orchestrator/orchestrator.js
@@ -1,25 +1,41 @@
#!/usr/bin/env node
/**
- * AutoGPT Platform Load Test Orchestrator
- *
- * Optimized test suite with only the highest VU count for each unique test type.
- * Eliminates duplicate tests and focuses on maximum load testing.
+ * AutoGPT Platform load test orchestrator.
+ *
+ * Runs every scenario below in turn, each at its highest VU count: the full
+ * stress suite, about an hour end to end.
+ *
+ * node orchestrator/orchestrator.js [PREVIEW|DEV|LOCAL|PROD] [local|cloud]
+ *
+ * local (the default) runs k6 on this machine through ../k6.sh, which uses
+ * Docker, or through the binary named by K6_BIN. cloud runs each test on
+ * Grafana Cloud k6, which bills every virtual-user hour: it is only for load
+ * one machine cannot generate, agreed with the team first, and it needs the
+ * explicit `cloud` argument plus K6_CLOUD_TOKEN and K6_CLOUD_STACK_ID.
*/
-import { spawn } from 'child_process';
-import fs from 'fs';
-
-console.log("🎯 AUTOGPT PLATFORM LOAD TEST ORCHESTRATOR\n");
-console.log("===========================================\n");
+import { spawn } from "child_process";
+import fs from "fs";
+import path from "path";
+import { fileURLToPath } from "url";
+
+const LOAD_TESTS_DIR = path.resolve(
+ path.dirname(fileURLToPath(import.meta.url)),
+ "..",
+);
+const K6_BIN = process.env.K6_BIN || path.join(LOAD_TESTS_DIR, "k6.sh");
+// Not a secret: the AutoGPT Platform project in Grafana Cloud k6.
+const K6_CLOUD_PROJECT_ID = process.env.K6_CLOUD_PROJECT_ID || "4254406";
+const PAUSE_BETWEEN_TESTS = 30; // seconds
-// Parse command line arguments
const args = process.argv.slice(2);
-const environment = args[0] || "DEV"; // LOCAL, DEV, PROD
-const executionMode = args[1] || "cloud"; // local, cloud
-
-console.log(`🌍 Target Environment: ${environment}`);
-console.log(`πŸš€ Execution Mode: ${executionMode}`);
+const environment = (
+ args[0] ||
+ process.env.K6_ENVIRONMENT ||
+ "DEV"
+).toUpperCase();
+const executionMode = (args[1] || "local").toLowerCase();
// Unified test scenarios - only highest VUs for each unique test
const unifiedTestScenarios = [
@@ -31,18 +47,18 @@ const unifiedTestScenarios = [
duration: "3m",
rampUp: "30s",
rampDown: "30s",
- description: "Public marketplace browsing at maximum load"
+ description: "Public marketplace browsing at maximum load",
},
- // 2. Marketplace Authenticated Access (highest VUs: 157)
+ // 2. Marketplace Authenticated Access (highest VUs: 157)
{
name: "Marketplace_Authenticated_Access_Max_Load",
- file: "tests/marketplace/library-access-test.js",
+ file: "tests/marketplace/library-access-test.js",
vus: 157,
duration: "3m",
- rampUp: "30s",
+ rampUp: "30s",
rampDown: "30s",
- description: "Authenticated marketplace/library operations at maximum load"
+ description: "Authenticated marketplace/library operations at maximum load",
},
// 3. Core API Load Test (highest VUs: 100)
@@ -52,19 +68,19 @@ const unifiedTestScenarios = [
vus: 100,
duration: "5m",
rampUp: "1m",
- rampDown: "1m",
- description: "Core authenticated API endpoints at maximum load"
+ rampDown: "1m",
+ description: "Core authenticated API endpoints at maximum load",
},
// 4. Graph Execution Load Test (highest VUs: 100)
{
- name: "Graph_Execution_Max_Load",
+ name: "Graph_Execution_Max_Load",
file: "tests/api/graph-execution-test.js",
vus: 100,
duration: "5m",
rampUp: "1m",
rampDown: "1m",
- description: "Graph workflow execution pipeline at maximum load"
+ description: "Graph workflow execution pipeline at maximum load",
},
// 5. Credits API Single Endpoint (upgraded to 100 VUs)
@@ -72,47 +88,47 @@ const unifiedTestScenarios = [
name: "Credits_API_Max_Load",
file: "tests/basic/single-endpoint-test.js",
vus: 100,
- duration: "3m",
+ duration: "3m",
rampUp: "30s",
rampDown: "30s",
env: { ENDPOINT: "credits", CONCURRENT_REQUESTS: "1" },
- description: "Credits API endpoint at maximum load"
+ description: "Credits API endpoint at maximum load",
},
// 6. Graphs API Single Endpoint (upgraded to 100 VUs)
{
name: "Graphs_API_Max_Load",
- file: "tests/basic/single-endpoint-test.js",
+ file: "tests/basic/single-endpoint-test.js",
vus: 100,
duration: "3m",
rampUp: "30s",
- rampDown: "30s",
+ rampDown: "30s",
env: { ENDPOINT: "graphs", CONCURRENT_REQUESTS: "1" },
- description: "Graphs API endpoint at maximum load"
+ description: "Graphs API endpoint at maximum load",
},
// 7. Blocks API Single Endpoint (upgraded to 100 VUs)
{
name: "Blocks_API_Max_Load",
file: "tests/basic/single-endpoint-test.js",
- vus: 100,
+ vus: 100,
duration: "3m",
rampUp: "30s",
rampDown: "30s",
env: { ENDPOINT: "blocks", CONCURRENT_REQUESTS: "1" },
- description: "Blocks API endpoint at maximum load"
+ description: "Blocks API endpoint at maximum load",
},
- // 8. Executions API Single Endpoint (upgraded to 100 VUs)
+ // 8. Executions API Single Endpoint (upgraded to 100 VUs)
{
name: "Executions_API_Max_Load",
file: "tests/basic/single-endpoint-test.js",
vus: 100,
duration: "3m",
- rampUp: "30s",
+ rampUp: "30s",
rampDown: "30s",
env: { ENDPOINT: "executions", CONCURRENT_REQUESTS: "1" },
- description: "Executions API endpoint at maximum load"
+ description: "Executions API endpoint at maximum load",
},
// 9. Comprehensive Platform Journey (highest VUs: 100)
@@ -122,8 +138,8 @@ const unifiedTestScenarios = [
vus: 100,
duration: "3m",
rampUp: "30s",
- rampDown: "30s",
- description: "End-to-end user journey simulation at maximum load"
+ rampDown: "30s",
+ description: "End-to-end user journey simulation at maximum load",
},
// 10. Marketplace Stress Test (highest VUs: 500)
@@ -132,231 +148,304 @@ const unifiedTestScenarios = [
file: "tests/marketplace/public-access-test.js",
vus: 500,
duration: "2m",
- rampUp: "1m",
+ rampUp: "1m",
rampDown: "1m",
- description: "Ultimate marketplace stress test"
+ description: "Ultimate marketplace stress test",
},
// 11. Core API Stress Test (highest VUs: 500)
{
- name: "Core_API_Stress_Test",
+ name: "Core_API_Stress_Test",
file: "tests/api/core-api-test.js",
vus: 500,
duration: "2m",
rampUp: "1m",
rampDown: "1m",
- description: "Ultimate core API stress test"
+ description: "Ultimate core API stress test",
},
// 12. Long Duration Core API Test (highest VUs: 100, longest duration)
{
name: "Long_Duration_Core_API_Test",
- file: "tests/api/core-api-test.js",
+ file: "tests/api/core-api-test.js",
vus: 100,
duration: "10m",
rampUp: "1m",
rampDown: "1m",
- description: "Extended duration core API endurance test"
- }
+ description: "Extended duration core API endurance test",
+ },
];
-// Configuration
-const K6_CLOUD_TOKEN = process.env.K6_CLOUD_TOKEN || '9347b8bd716cadc243e92f7d2f89107febfb81b49f2340d17da515d7b0513b51';
-const K6_CLOUD_PROJECT_ID = process.env.K6_CLOUD_PROJECT_ID || '4254406';
-const PAUSE_BETWEEN_TESTS = 30; // seconds
+// Problems that would fail every test, reported before the first one starts.
+function findSetupProblems() {
+ const problems = [];
+ if (!["PREVIEW", "DEV", "LOCAL", "PROD"].includes(environment)) {
+ problems.push(
+ `Unknown environment "${environment}": use PREVIEW, DEV, LOCAL or PROD`,
+ );
+ }
+ if (!["local", "cloud"].includes(executionMode)) {
+ problems.push(`Unknown mode "${executionMode}": use local or cloud`);
+ }
+ // Streaming results to Grafana Cloud is billed like a cloud run, and a
+ // K6_BIN binary would not go through k6.sh's opt-in check.
+ if (executionMode === "local" && /cloud/i.test(process.env.K6_OUT || "")) {
+ problems.push(
+ "local mode does not stream results to Grafana Cloud, which bills every virtual-user hour: unset K6_OUT, or use cloud mode for an agreed cloud run",
+ );
+ }
+ if (
+ environment === "PREVIEW" &&
+ !/^[1-9]\d{0,6}$/.test(process.env.PR_NUMBER || "")
+ ) {
+ problems.push("PREVIEW needs PR_NUMBER=<pull request number>");
+ }
+ // PREVIEW and LOCAL sign up throwaway accounts when this is unset.
+ if (
+ (environment === "DEV" || environment === "PROD") &&
+ !process.env.LOAD_TEST_ACCOUNTS
+ ) {
+ problems.push(
+ `${environment} needs LOAD_TEST_ACCOUNTS for the authenticated tests; see README.md`,
+ );
+ }
+ if (executionMode === "cloud") {
+ if (!process.env.K6_CLOUD_TOKEN) {
+ problems.push(
+ "cloud mode needs K6_CLOUD_TOKEN, a Grafana Cloud k6 API token",
+ );
+ }
+ if (!process.env.K6_CLOUD_STACK_ID) {
+ problems.push(
+ "cloud mode needs K6_CLOUD_STACK_ID, the Grafana Cloud stack's numeric ID",
+ );
+ }
+ if (process.env.LOAD_TEST_ACCOUNTS) {
+ problems.push(
+ "cloud mode does not send LOAD_TEST_ACCOUNTS, because Grafana Cloud stores a remote run's --env values with the run. " +
+ "Use local mode, or on a preview unset LOAD_TEST_ACCOUNTS so the tests sign up throwaway accounts.",
+ );
+ }
+ }
+ return problems;
+}
-/**
- * Sleep for specified milliseconds
- */
function sleep(ms) {
- return new Promise(resolve => setTimeout(resolve, ms));
+ return new Promise((resolve) => setTimeout(resolve, ms));
}
-/**
- * Run a single k6 test
- */
-async function runTest(test, index) {
+function minutes(duration) {
+ const unit = { h: 60, m: 1, s: 1 / 60 };
+ let total = 0;
+ for (const [, value, suffix] of duration.matchAll(
+ /(\d+(?:\.\d+)?)(h|m|s)/g,
+ )) {
+ total += Number(value) * unit[suffix];
+ }
+ return total;
+}
+
+// Grafana Cloud k6 bills each test as its peak VUs times its whole duration.
+function suiteVirtualUserHours() {
+ const vuMinutes = unifiedTestScenarios.reduce(
+ (sum, test) =>
+ sum +
+ test.vus *
+ (minutes(test.rampUp) +
+ minutes(test.duration) +
+ minutes(test.rampDown)),
+ 0,
+ );
+ return vuMinutes / 60;
+}
+
+function runTest(test, index) {
return new Promise((resolve, reject) => {
- console.log(`\nπŸš€ Test ${index + 1}/${unifiedTestScenarios.length}: ${test.name}`);
- console.log(`πŸ“Š Config: ${test.vus} VUs Γ— ${test.duration} (${executionMode} mode)`);
- console.log(`πŸ“ Script: ${test.file}`);
- console.log(`πŸ“‹ Description: ${test.description}`);
- console.log(`⏱️ Test started: ${new Date().toISOString()}`);
-
- const env = {
- K6_CLOUD_TOKEN,
- K6_CLOUD_PROJECT_ID,
+ console.log(
+ `\nTest ${index + 1}/${unifiedTestScenarios.length}: ${test.name}`,
+ );
+ console.log(
+ `Config: ${test.vus} VUs x ${test.duration} (${executionMode} mode)`,
+ );
+ console.log(`Script: ${test.file}`);
+ console.log(`Description: ${test.description}`);
+ console.log(`Started: ${new Date().toISOString()}`);
+
+ const testEnv = {
K6_ENVIRONMENT: environment,
+ K6_CLOUD_PROJECT_ID,
VUS: test.vus.toString(),
DURATION: test.duration,
RAMP_UP: test.rampUp,
RAMP_DOWN: test.rampDown,
- ...test.env
+ ...(process.env.PR_NUMBER ? { PR_NUMBER: process.env.PR_NUMBER } : {}),
+ ...(process.env.LOAD_TEST_ACCOUNT_COUNT
+ ? { LOAD_TEST_ACCOUNT_COUNT: process.env.LOAD_TEST_ACCOUNT_COUNT }
+ : {}),
+ ...test.env,
};
- let args;
- if (executionMode === 'cloud') {
- args = [
- 'cloud', 'run',
- ...Object.entries(env).map(([key, value]) => ['--env', `${key}=${value}`]).flat(),
- test.file
- ];
- } else {
- args = [
- 'run',
- ...Object.entries(env).map(([key, value]) => ['--env', `${key}=${value}`]).flat(),
- test.file
- ];
- }
-
- const k6Process = spawn('k6', args, {
- stdio: ['ignore', 'pipe', 'pipe'],
- env: { ...process.env, ...env }
+ // A remote cloud run sees only the variables passed with --env. A local
+ // run reads them, and any secrets, from the environment k6 inherits, so
+ // nothing secret ever goes on a command line.
+ const k6Args =
+ executionMode === "cloud"
+ ? [
+ "cloud",
+ "run",
+ ...Object.entries(testEnv).flatMap(([key, value]) => [
+ "--env",
+ `${key}=${value}`,
+ ]),
+ test.file,
+ ]
+ : ["run", test.file];
+ // k6.sh refuses Grafana Cloud unless its first argument is --cloud.
+ const wrapperArgs =
+ executionMode === "cloud" && !process.env.K6_BIN ? ["--cloud"] : [];
+
+ const k6Process = spawn(K6_BIN, [...wrapperArgs, ...k6Args], {
+ cwd: LOAD_TESTS_DIR,
+ stdio: ["ignore", "pipe", "pipe"],
+ env: { ...process.env, ...testEnv },
});
- let output = '';
- let testId = null;
+ let runUrl = null;
- k6Process.stdout.on('data', (data) => {
+ k6Process.stdout.on("data", (data) => {
const str = data.toString();
- output += str;
-
- // Extract test ID from k6 cloud output
- const testIdMatch = str.match(/Test created: .*\/(\d+)/);
- if (testIdMatch) {
- testId = testIdMatch[1];
- console.log(`πŸ”— Test URL: https://significantgravitas.grafana.net/a/k6-app/runs/${testId}`);
+
+ const runUrlMatch = str.match(/https?:\/\/\S+?\/runs\/\d+/);
+ if (runUrlMatch && !runUrl) {
+ runUrl = runUrlMatch[0];
+ console.log(`Run URL: ${runUrl}`);
}
- // Show progress updates
const progressMatch = str.match(/(\d+)%/);
if (progressMatch) {
- process.stdout.write(`\r⏳ Progress: ${progressMatch[1]}%`);
+ process.stdout.write(`\rProgress: ${progressMatch[1]}%`);
}
});
- k6Process.stderr.on('data', (data) => {
- output += data.toString();
+ let stderrTail = "";
+ k6Process.stderr.on("data", (data) => {
+ stderrTail = (stderrTail + data.toString()).slice(-2000);
});
- k6Process.on('close', (code) => {
- process.stdout.write('\n'); // Clear progress line
-
+ k6Process.on("close", (code) => {
+ process.stdout.write("\n");
+ const result = {
+ success: code === 0,
+ url: runUrl || "unknown",
+ vus: test.vus,
+ duration: test.duration,
+ };
if (code === 0) {
- console.log(`βœ… ${test.name} SUCCESS`);
- resolve({
- success: true,
- testId,
- url: testId ? `https://significantgravitas.grafana.net/a/k6-app/runs/${testId}` : 'unknown',
- vus: test.vus,
- duration: test.duration
- });
+ console.log(`${test.name} SUCCESS`);
} else {
- console.log(`❌ ${test.name} FAILED (exit code ${code})`);
- resolve({
- success: false,
- testId,
- url: testId ? `https://significantgravitas.grafana.net/a/k6-app/runs/${testId}` : 'unknown',
- exitCode: code,
- vus: test.vus,
- duration: test.duration
- });
+ console.log(`${test.name} FAILED (exit code ${code})`);
+ if (stderrTail) console.log(stderrTail.trim());
+ result.exitCode = code;
}
+ resolve(result);
});
- k6Process.on('error', (error) => {
- console.log(`❌ ${test.name} ERROR: ${error.message}`);
+ k6Process.on("error", (error) => {
+ console.log(`${test.name} ERROR: ${error.message}`);
reject(error);
});
});
}
-/**
- * Main execution
- */
async function main() {
- console.log(`\nπŸ“‹ UNIFIED TEST PLAN`);
- console.log(`πŸ“Š Total tests: ${unifiedTestScenarios.length} (reduced from 25 original tests)`);
- console.log(`⏱️ Estimated duration: ~60 minutes\n`);
+ console.log("AUTOGPT PLATFORM LOAD TEST ORCHESTRATOR\n");
+ console.log(`Target environment: ${environment}`);
+ console.log(`Execution mode: ${executionMode}`);
+
+ const problems = findSetupProblems();
+ if (problems.length > 0) {
+ for (const problem of problems) console.error(`Error: ${problem}`);
+ process.exit(2);
+ }
+ if (executionMode === "cloud") {
+ console.log(
+ `Grafana Cloud k6 bills every virtual-user hour, and this suite is about ${Math.round(suiteVirtualUserHours())} VUh a run; local mode is the default.`,
+ );
+ }
- console.log(`πŸ“‹ Test Summary:`);
+ console.log(`\nUNIFIED TEST PLAN`);
+ console.log(`Total tests: ${unifiedTestScenarios.length}`);
+ console.log(`Estimated duration: ~60 minutes\n`);
unifiedTestScenarios.forEach((test, i) => {
- console.log(` ${i + 1}. ${test.name} (${test.vus} VUs Γ— ${test.duration})`);
+ console.log(
+ ` ${i + 1}. ${test.name} (${test.vus} VUs x ${test.duration})`,
+ );
});
- console.log('');
const results = [];
for (let i = 0; i < unifiedTestScenarios.length; i++) {
const test = unifiedTestScenarios[i];
-
+
try {
const result = await runTest(test, i);
results.push({ ...test, ...result });
-
- // Pause between tests (except after the last one)
+
if (i < unifiedTestScenarios.length - 1) {
- console.log(`\n⏸️ Pausing ${PAUSE_BETWEEN_TESTS}s before next test...`);
+ console.log(
+ `\nPausing ${PAUSE_BETWEEN_TESTS}s before the next test...`,
+ );
await sleep(PAUSE_BETWEEN_TESTS * 1000);
}
} catch (error) {
- console.error(`πŸ’₯ Fatal error running ${test.name}:`, error.message);
+ console.error(`Fatal error running ${test.name}:`, error.message);
results.push({ ...test, success: false, error: error.message });
}
}
- // Summary
- console.log('\n' + '='.repeat(60));
- console.log('🏁 UNIFIED LOAD TEST RESULTS SUMMARY');
- console.log('='.repeat(60));
+ console.log("\n" + "=".repeat(60));
+ console.log("UNIFIED LOAD TEST RESULTS SUMMARY");
+ console.log("=".repeat(60));
- const successful = results.filter(r => r.success);
- const failed = results.filter(r => !r.success);
+ const successful = results.filter((r) => r.success);
+ const failed = results.filter((r) => !r.success);
- console.log(`βœ… Successful tests: ${successful.length}/${results.length} (${Math.round(successful.length / results.length * 100)}%)`);
- console.log(`❌ Failed tests: ${failed.length}/${results.length}`);
+ console.log(
+ `Successful tests: ${successful.length}/${results.length} (${Math.round((successful.length / results.length) * 100)}%)`,
+ );
+ console.log(`Failed tests: ${failed.length}/${results.length}`);
if (successful.length > 0) {
- console.log('\nβœ… SUCCESSFUL TESTS:');
- successful.forEach(test => {
- console.log(` β€’ ${test.name} (${test.vus} VUs) - ${test.url}`);
+ console.log("\nSUCCESSFUL TESTS:");
+ successful.forEach((test) => {
+ console.log(` - ${test.name} (${test.vus} VUs) - ${test.url}`);
});
}
if (failed.length > 0) {
- console.log('\n❌ FAILED TESTS:');
- failed.forEach(test => {
- console.log(` β€’ ${test.name} (${test.vus} VUs) - ${test.url || 'no URL'} (exit: ${test.exitCode || 'unknown'})`);
+ console.log("\nFAILED TESTS:");
+ failed.forEach((test) => {
+ console.log(
+ ` - ${test.name} (${test.vus} VUs) - ${test.url || "no URL"} (exit: ${test.exitCode || "unknown"})`,
+ );
});
}
- // Calculate total VU-minutes tested
- const totalVuMinutes = results.reduce((sum, test) => {
- const minutes = parseFloat(test.duration.replace(/[ms]/g, ''));
- const multiplier = test.duration.includes('m') ? 1 : (1/60); // convert seconds to minutes
- return sum + (test.vus * minutes * multiplier);
- }, 0);
-
- console.log(`\nπŸ“Š LOAD TESTING SUMMARY:`);
- console.log(` β€’ Total VU-minutes tested: ${Math.round(totalVuMinutes)}`);
- console.log(` β€’ Peak concurrent VUs: ${Math.max(...results.map(r => r.vus))}`);
- console.log(` β€’ Average test duration: ${(results.reduce((sum, r) => sum + parseFloat(r.duration.replace(/[ms]/g, '')), 0) / results.length).toFixed(1)}${results[0].duration.includes('m') ? 'm' : 's'}`);
-
- // Write results to file
- const timestamp = Math.floor(Date.now() / 1000);
- const resultsFile = `unified-results-${timestamp}.json`;
+ const resultsDir = path.join(LOAD_TESTS_DIR, "results");
+ fs.mkdirSync(resultsDir, { recursive: true });
+ const resultsFile = path.join(
+ resultsDir,
+ `unified-results-${Math.floor(Date.now() / 1000)}.json`,
+ );
fs.writeFileSync(resultsFile, JSON.stringify(results, null, 2));
- console.log(`\nπŸ“„ Detailed results saved to: ${resultsFile}`);
-
- console.log(`\nπŸŽ‰ UNIFIED LOAD TEST ORCHESTRATOR COMPLETE\n`);
+ console.log(`\nDetailed results saved to: ${resultsFile}`);
process.exit(failed.length === 0 ? 0 : 1);
}
-// Run if called directly
-if (process.argv[1] === new URL(import.meta.url).pathname) {
- main().catch(error => {
- console.error('πŸ’₯ Fatal error:', error);
+if (process.argv[1] === fileURLToPath(import.meta.url)) {
+ main().catch((error) => {
+ console.error("Fatal error:", error);
process.exit(1);
});
-}
\ No newline at end of file
+}
diff --git a/autogpt_platform/backend/load-tests/tests/api/core-api-test.js b/autogpt_platform/backend/load-tests/tests/api/core-api-test.js
index 2bf77d6c63..1341d1c19e 100644
--- a/autogpt_platform/backend/load-tests/tests/api/core-api-test.js
+++ b/autogpt_platform/backend/load-tests/tests/api/core-api-test.js
@@ -2,7 +2,7 @@
import http from "k6/http";
import { check } from "k6";
import { getEnvironmentConfig } from "../../configs/environment.js";
-import { getPreAuthenticatedHeaders } from "../../configs/pre-authenticated-tokens.js";
+import { authHeaders, authSetupTimeout, setupAuth } from "../../lib/auth.js";
const config = getEnvironmentConfig();
@@ -23,31 +23,22 @@ export const options = {
name: "AutoGPT Platform - Core API Validation Test",
},
// Timeout configurations to prevent early termination
- setupTimeout: "60s",
+ setupTimeout: authSetupTimeout(),
teardownTimeout: "60s",
noConnectionReuse: false,
userAgent: "k6-load-test/1.0",
};
-export default function () {
+export function setup() {
+ return setupAuth();
+}
+
+export default function (data) {
// Get load multiplier - how many concurrent requests each VU should make
const requestsPerVU = parseInt(__ENV.REQUESTS_PER_VU) || 1;
+ const headers = authHeaders(data);
try {
- // Step 1: Get pre-authenticated headers (no auth API calls during test)
- const headers = getPreAuthenticatedHeaders(__VU);
-
- // Handle missing token gracefully
- if (!headers || !headers.Authorization) {
- console.log(
- `⚠️ VU ${__VU} has no valid pre-authenticated token - skipping core API test`,
- );
- check(null, {
- "Core API: Failed gracefully without crashing VU": () => true,
- });
- return; // Exit iteration gracefully without crashing
- }
-
console.log(
`πŸš€ VU ${__VU} making ${requestsPerVU} concurrent API requests...`,
);
@@ -59,19 +50,19 @@ export default function () {
// Add core API requests that represent realistic user workflows
requests.push({
method: "GET",
- url: `${config.API_BASE_URL}/api/credits`,
+ url: `${config.API_BASE_URL}/credits`,
params: { headers },
});
requests.push({
method: "GET",
- url: `${config.API_BASE_URL}/api/graphs`,
+ url: `${config.API_BASE_URL}/graphs`,
params: { headers },
});
requests.push({
method: "GET",
- url: `${config.API_BASE_URL}/api/blocks`,
+ url: `${config.API_BASE_URL}/blocks`,
params: { headers },
});
}
diff --git a/autogpt_platform/backend/load-tests/tests/api/graph-execution-test.js b/autogpt_platform/backend/load-tests/tests/api/graph-execution-test.js
index 41d97241ba..541981300b 100644
--- a/autogpt_platform/backend/load-tests/tests/api/graph-execution-test.js
+++ b/autogpt_platform/backend/load-tests/tests/api/graph-execution-test.js
@@ -3,7 +3,7 @@ import http from "k6/http";
import { check, sleep, group } from "k6";
import { Rate, Trend, Counter } from "k6/metrics";
import { getEnvironmentConfig } from "../../configs/environment.js";
-import { getPreAuthenticatedHeaders } from "../../configs/pre-authenticated-tokens.js";
+import { authHeaders, authSetupTimeout, setupAuth } from "../../lib/auth.js";
// Test data generation functions
function generateTestGraph(name = null) {
const graphName =
@@ -53,25 +53,48 @@ function generateTestGraph(name = null) {
};
}
+// Graph inputs are keyed by the input block's `name`.
function generateExecutionInputs() {
return {
- "Load Test Input": {
- name: "Load Test Input",
- description: "Test input for load testing",
- value: `Test execution at ${new Date().toISOString()}`,
- },
+ "Load Test Input": `Test execution at ${new Date().toISOString()}`,
};
}
const config = getEnvironmentConfig();
+// A 402 is a billing refusal: the account has no credits, or no subscription
+// while the payment paywall applies to it. A new account can meet either,
+// depending on the target's flags. The route answered as designed, so it is
+// not a failed request, but no graph ran: graph_executions_total counts only
+// the runs that started, and each refusal has its own counter.
+const EXECUTION_STATUSES = http.expectedStatuses(200, 402);
+
// Custom metrics for graph execution testing
const graphCreations = new Counter("graph_creations_total");
const graphExecutions = new Counter("graph_executions_total");
+const graphExecutionsNoCredits = new Counter(
+ "graph_executions_refused_no_credits_total",
+);
+const graphExecutionsNoSubscription = new Counter(
+ "graph_executions_refused_no_subscription_total",
+);
const graphExecutionTime = new Trend("graph_execution_duration");
const graphCreationTime = new Trend("graph_creation_duration");
const executionErrors = new Rate("execution_errors");
+// Which billing refusal a 402 is, from its detail; null for any other 402.
+function billingRefusal(response) {
+ let detail = "";
+ try {
+ detail = String(response.json("detail") || "");
+ } catch (e) {
+ detail = "";
+ }
+ if (/insufficient balance/i.test(detail)) return "no credits";
+ if (/subscription is required/i.test(detail)) return "no subscription";
+ return null;
+}
+
// Configurable options for easy load adjustment
export const options = {
stages: [
@@ -92,7 +115,7 @@ export const options = {
name: "AutoGPT Platform - Graph Creation & Execution Test",
},
// Timeout configurations to prevent early termination
- setupTimeout: "60s",
+ setupTimeout: authSetupTimeout(),
teardownTimeout: "60s",
noConnectionReuse: false,
userAgent: "k6-load-test/1.0",
@@ -104,6 +127,7 @@ export function setup() {
`Configuration: VUs=${parseInt(__ENV.VUS) || 5}, Duration=${__ENV.DURATION || "2m"}`,
);
return {
+ ...setupAuth(),
timestamp: Date.now(),
};
}
@@ -111,20 +135,7 @@ export function setup() {
export default function (data) {
// Get load multiplier - how many concurrent operations each VU should perform
const requestsPerVU = parseInt(__ENV.REQUESTS_PER_VU) || 1;
-
- // Get pre-authenticated headers (no auth API calls during test)
- const headers = getPreAuthenticatedHeaders(__VU);
-
- // Handle missing token gracefully
- if (!headers || !headers.Authorization) {
- console.log(
- `⚠️ VU ${__VU} has no valid pre-authenticated token - skipping graph execution`,
- );
- check(null, {
- "Graph Execution: Failed gracefully without crashing VU": () => true,
- });
- return; // Exit iteration gracefully without crashing
- }
+ const headers = authHeaders(data);
console.log(
`πŸš€ VU ${__VU} performing ${requestsPerVU} concurrent graph operations...`,
@@ -140,7 +151,7 @@ export default function (data) {
// Add graph creation request
graphRequests.push({
method: "POST",
- url: `${config.API_BASE_URL}/api/graphs`,
+ url: `${config.API_BASE_URL}/graphs`,
body: JSON.stringify(graphData),
params: { headers },
});
@@ -195,12 +206,12 @@ export default function (data) {
executionRequests.push({
method: "POST",
- url: `${config.API_BASE_URL}/api/graphs/${graph.id}/execute/${graph.version}`,
+ url: `${config.API_BASE_URL}/graphs/${graph.id}/execute/${graph.version}`,
body: JSON.stringify({
inputs: executionInputs,
credentials_inputs: {},
}),
- params: { headers },
+ params: { headers, responseCallback: EXECUTION_STATUSES },
});
}
@@ -208,22 +219,31 @@ export default function (data) {
const executionResponses = http.batch(executionRequests);
let executionSuccessCount = 0;
+ const refusals = { "no credits": 0, "no subscription": 0 };
for (let i = 0; i < executionResponses.length; i++) {
const response = executionResponses[i];
+ const refusal = response.status === 402 ? billingRefusal(response) : null;
- const success = check(response, {
- [`Graph ${i + 1} execution initiated`]: (r) =>
- r.status === 200 || r.status === 402,
+ check(response, {
+ [`Graph ${i + 1} execution started, or refused for no credits or no subscription (402)`]:
+ (r) => r.status === 200 || refusal !== null,
});
- if (success) {
+ if (response.status === 200) {
executionSuccessCount++;
graphExecutions.add(1);
+ } else if (refusal === "no credits") {
+ refusals[refusal]++;
+ graphExecutionsNoCredits.add(1);
+ } else if (refusal === "no subscription") {
+ refusals[refusal]++;
+ graphExecutionsNoSubscription.add(1);
}
}
console.log(
- `βœ… VU ${__VU} executed ${executionSuccessCount}/${graphsToExecute.length} graphs`,
+ `βœ… VU ${__VU} executed ${executionSuccessCount}/${graphsToExecute.length} graphs ` +
+ `(refused: ${refusals["no credits"]} no credits, ${refusals["no subscription"]} no subscription)`,
);
}
@@ -231,13 +251,9 @@ export default function (data) {
sleep(Math.random() * 2 + 1); // 1-3 seconds
}
-// Legacy functions removed - replaced by concurrent execution in main function
-// These functions are no longer used since implementing http.batch() for true concurrency
-
export function teardown(data) {
+ // graph_creations_total and graph_executions_total are in k6's summary.
console.log("🧹 Cleaning up graph execution load test...");
- console.log(`Total graph creations: ${graphCreations.value || 0}`);
- console.log(`Total graph executions: ${graphExecutions.value || 0}`);
const testDuration = Date.now() - data.timestamp;
console.log(`Test completed in ${testDuration}ms`);
diff --git a/autogpt_platform/backend/load-tests/tests/basic/connectivity-test.js b/autogpt_platform/backend/load-tests/tests/basic/connectivity-test.js
index 8ad1a5961d..ef3c424a5b 100644
--- a/autogpt_platform/backend/load-tests/tests/basic/connectivity-test.js
+++ b/autogpt_platform/backend/load-tests/tests/basic/connectivity-test.js
@@ -1,14 +1,15 @@
/**
* Basic Connectivity Test
*
- * Tests basic connectivity and authentication without requiring backend API access
- * This test validates that the core infrastructure is working correctly
+ * Checks the pieces every authenticated test depends on: the backend answers
+ * /health, the frontend serves the JWKS the backend verifies tokens against,
+ * and the backend accepts a token minted by that frontend.
*/
import http from "k6/http";
import { check } from "k6";
import { getEnvironmentConfig } from "../../configs/environment.js";
-import { getPreAuthenticatedHeaders } from "../../configs/pre-authenticated-tokens.js";
+import { authHeaders, authSetupTimeout, setupAuth } from "../../lib/auth.js";
const config = getEnvironmentConfig();
@@ -19,119 +20,78 @@ export const options = {
{ duration: __ENV.RAMP_DOWN || "1m", target: 0 },
],
thresholds: {
- checks: ["rate>0.70"], // Reduced from 0.85 due to auth timeouts under load
- http_req_duration: ["p(95)<30000"], // Increased for cloud testing with high concurrency
- http_req_failed: ["rate<0.6"], // Increased to account for auth timeouts
+ checks: ["rate>0.70"],
+ http_req_duration: ["p(95)<30000"],
+ http_req_failed: ["rate<0.6"],
},
cloud: {
projectID: __ENV.K6_CLOUD_PROJECT_ID,
name: "AutoGPT Platform - Basic Connectivity & Auth Test",
},
- // Timeout configurations to prevent early termination
- setupTimeout: "60s",
+ setupTimeout: authSetupTimeout(),
teardownTimeout: "60s",
noConnectionReuse: false,
userAgent: "k6-load-test/1.0",
};
-export default function () {
+export function setup() {
+ return setupAuth();
+}
+
+export default function (data) {
// Get load multiplier - how many concurrent requests each VU should make
const requestsPerVU = parseInt(__ENV.REQUESTS_PER_VU) || 1;
+ const headers = authHeaders(data);
+
+ const requests = [];
+ for (let i = 0; i < requestsPerVU; i++) {
+ requests.push({ method: "GET", url: `${config.BACKEND_URL}/health` });
+ requests.push({
+ method: "GET",
+ url: `${config.FRONTEND_URL}/api/auth/jwks`,
+ });
+ requests.push({
+ method: "GET",
+ url: `${config.API_BASE_URL}/credits`,
+ params: { headers },
+ });
+ }
- try {
- // Get pre-authenticated headers
- const headers = getPreAuthenticatedHeaders(__VU);
-
- // Handle authentication failure gracefully
- if (!headers || !headers.Authorization) {
- console.log(
- `⚠️ VU ${__VU} has no valid pre-authentication token - skipping iteration`,
- );
- check(null, {
- "Authentication: Failed gracefully without crashing VU": () => true,
- });
- return; // Exit iteration gracefully without crashing
- }
-
- console.log(`πŸš€ VU ${__VU} making ${requestsPerVU} concurrent requests...`);
-
- // Create array of request functions to run concurrently
- const requests = [];
-
- for (let i = 0; i < requestsPerVU; i++) {
- requests.push({
- method: "GET",
- url: `${config.SUPABASE_URL}/rest/v1/`,
- params: { headers: { apikey: config.SUPABASE_ANON_KEY } },
- });
-
- requests.push({
- method: "GET",
- url: `${config.API_BASE_URL}/health`,
- params: { headers },
- });
- }
-
- // Execute all requests concurrently
- const responses = http.batch(requests);
-
- // Validate results
- let supabaseSuccesses = 0;
- let backendSuccesses = 0;
-
- for (let i = 0; i < responses.length; i++) {
- const response = responses[i];
+ const responses = http.batch(requests);
- if (i % 2 === 0) {
- // Supabase request
- const connectivityCheck = check(response, {
- "Supabase connectivity: Status is not 500": (r) => r.status !== 500,
- "Supabase connectivity: Response time < 5s": (r) =>
+ for (let i = 0; i < responses.length; i++) {
+ const response = responses[i];
+ switch (i % 3) {
+ case 0:
+ check(response, {
+ "Backend health: Status is 200": (r) => r.status === 200,
+ "Backend health: Response time < 5s": (r) =>
r.timings.duration < 5000,
});
- if (connectivityCheck) supabaseSuccesses++;
- } else {
- // Backend request
- const backendCheck = check(response, {
- "Backend server: Responds (any status)": (r) => r.status > 0,
- "Backend server: Response time < 5s": (r) =>
- r.timings.duration < 5000,
+ break;
+ case 1:
+ check(response, {
+ "Frontend JWKS: Status is 200": (r) => r.status === 200,
+ "Frontend JWKS: Has signing keys": (r) => {
+ try {
+ const keys = r.json("keys");
+ return Array.isArray(keys) && keys.length > 0;
+ } catch (e) {
+ return false;
+ }
+ },
+ });
+ break;
+ default:
+ check(response, {
+ "Authenticated request: Status is 200": (r) => r.status === 200,
+ "Authenticated request: Not Auth Error (401/403)": (r) =>
+ r.status !== 401 && r.status !== 403,
});
- if (backendCheck) backendSuccesses++;
- }
}
-
- console.log(
- `βœ… VU ${__VU} completed: ${supabaseSuccesses}/${requestsPerVU} Supabase, ${backendSuccesses}/${requestsPerVU} backend requests successful`,
- );
-
- // Basic auth validation (once per iteration)
- const authCheck = check(headers, {
- "Authentication: Pre-auth token available": (h) =>
- h && h.Authorization && h.Authorization.length > 0,
- });
-
- // JWT structure validation (once per iteration)
- const token = headers.Authorization.replace("Bearer ", "");
- const tokenParts = token.split(".");
- const tokenStructureCheck = check(tokenParts, {
- "JWT token: Has 3 parts (header.payload.signature)": (parts) =>
- parts.length === 3,
- "JWT token: Header is base64": (parts) =>
- parts[0] && parts[0].length > 10,
- "JWT token: Payload is base64": (parts) =>
- parts[1] && parts[1].length > 50,
- "JWT token: Signature exists": (parts) =>
- parts[2] && parts[2].length > 10,
- });
- } catch (error) {
- console.error(`πŸ’₯ Test failed: ${error.message}`);
- check(null, {
- "Test execution: No errors": () => false,
- });
}
}
-export function teardown(data) {
- console.log(`🏁 Basic connectivity test completed`);
+export function teardown() {
+ console.log("Basic connectivity test completed");
}
diff --git a/autogpt_platform/backend/load-tests/tests/basic/single-endpoint-test.js b/autogpt_platform/backend/load-tests/tests/basic/single-endpoint-test.js
index 36e0bca391..fbba170b85 100644
--- a/autogpt_platform/backend/load-tests/tests/basic/single-endpoint-test.js
+++ b/autogpt_platform/backend/load-tests/tests/basic/single-endpoint-test.js
@@ -2,7 +2,7 @@
import http from "k6/http";
import { check } from "k6";
import { getEnvironmentConfig } from "../../configs/environment.js";
-import { getPreAuthenticatedHeaders } from "../../configs/pre-authenticated-tokens.js";
+import { authHeaders, authSetupTimeout, setupAuth } from "../../lib/auth.js";
const config = getEnvironmentConfig();
@@ -21,29 +21,26 @@ export const options = {
projectID: parseInt(__ENV.K6_CLOUD_PROJECT_ID) || 4254406,
name: `AutoGPT Single Endpoint Test - ${__ENV.ENDPOINT || "credits"} API`,
},
+ setupTimeout: authSetupTimeout(),
};
-export default function () {
+export function setup() {
+ return setupAuth();
+}
+
+export default function (data) {
const endpoint = __ENV.ENDPOINT || "credits"; // credits, graphs, blocks, executions
const concurrentRequests = parseInt(__ENV.CONCURRENT_REQUESTS) || 1;
+ const headers = authHeaders(data);
try {
- const headers = getPreAuthenticatedHeaders(__VU);
-
- if (!headers || !headers.Authorization) {
- console.log(
- `⚠️ VU ${__VU} has no valid pre-authentication token - skipping test`,
- );
- return;
- }
-
console.log(
`πŸš€ VU ${__VU} testing /api/${endpoint} with ${concurrentRequests} concurrent requests`,
);
if (concurrentRequests === 1) {
// Single request mode (original behavior)
- const response = http.get(`${config.API_BASE_URL}/api/${endpoint}`, {
+ const response = http.get(`${config.API_BASE_URL}/${endpoint}`, {
headers,
});
@@ -68,7 +65,7 @@ export default function () {
for (let i = 0; i < concurrentRequests; i++) {
requests.push({
method: "GET",
- url: `${config.API_BASE_URL}/api/${endpoint}`,
+ url: `${config.API_BASE_URL}/${endpoint}`,
params: { headers },
});
}
diff --git a/autogpt_platform/backend/load-tests/tests/comprehensive/platform-journey-test.js b/autogpt_platform/backend/load-tests/tests/comprehensive/platform-journey-test.js
index 9571de41c7..2fd4a29e0b 100644
--- a/autogpt_platform/backend/load-tests/tests/comprehensive/platform-journey-test.js
+++ b/autogpt_platform/backend/load-tests/tests/comprehensive/platform-journey-test.js
@@ -1,37 +1,11 @@
import http from "k6/http";
-import { check, sleep, group } from "k6";
-import { Rate, Trend, Counter } from "k6/metrics";
+import { check, sleep } from "k6";
+import { Counter } from "k6/metrics";
import {
getEnvironmentConfig,
PERFORMANCE_CONFIG,
} from "../../configs/environment.js";
-import { getPreAuthenticatedHeaders } from "../../configs/pre-authenticated-tokens.js";
-
-// Inline test data generators (simplified from utils/test-data.js)
-function generateTestGraph(name = null) {
- const graphName =
- name || `Load Test Graph ${Math.random().toString(36).substr(2, 9)}`;
- return {
- name: graphName,
- description: "Generated graph for load testing purposes",
- graph: {
- nodes: [],
- links: [],
- },
- };
-}
-
-function generateExecutionInputs() {
- return { test_input: "load_test_value" };
-}
-
-function generateScheduleData() {
- return { enabled: false };
-}
-
-function generateAPIKeyRequest() {
- return { name: "Load Test API Key" };
-}
+import { authHeaders, authSetupTimeout, setupAuth } from "../../lib/auth.js";
const config = getEnvironmentConfig();
@@ -39,8 +13,6 @@ const config = getEnvironmentConfig();
const userOperations = new Counter("user_operations_total");
const graphOperations = new Counter("graph_operations_total");
const executionOperations = new Counter("execution_operations_total");
-const apiResponseTime = new Trend("api_response_time");
-const authErrors = new Rate("auth_errors");
// Test configuration for normal load testing
export const options = {
@@ -66,7 +38,7 @@ export const options = {
name: "AutoGPT Platform - Full Platform Integration Test",
},
// Timeout configurations to prevent early termination
- setupTimeout: "60s",
+ setupTimeout: authSetupTimeout(),
teardownTimeout: "60s",
noConnectionReuse: false,
userAgent: "k6-load-test/1.0",
@@ -75,6 +47,7 @@ export const options = {
export function setup() {
console.log("🎯 Setting up load test scenario...");
return {
+ ...setupAuth(),
timestamp: Date.now(),
};
}
@@ -82,28 +55,7 @@ export function setup() {
export default function (data) {
// Get load multiplier - how many concurrent user journeys each VU should simulate
const requestsPerVU = parseInt(__ENV.REQUESTS_PER_VU) || 1;
-
- let headers;
-
- try {
- headers = getPreAuthenticatedHeaders(__VU);
- } catch (error) {
- console.error(`❌ Authentication failed:`, error);
- authErrors.add(1);
- return;
- }
-
- // Handle authentication failure gracefully
- if (!headers || !headers.Authorization) {
- console.log(
- `⚠️ VU ${__VU} has no valid pre-authentication token - skipping comprehensive platform test`,
- );
- check(null, {
- "Comprehensive Platform: Failed gracefully without crashing VU": () =>
- true,
- });
- return; // Exit iteration gracefully without crashing
- }
+ const headers = authHeaders(data);
console.log(
`πŸš€ VU ${__VU} simulating ${requestsPerVU} realistic user workflows...`,
@@ -117,26 +69,26 @@ export default function (data) {
// Workflow 1: User checking their dashboard
requests.push({
method: "GET",
- url: `${config.API_BASE_URL}/api/credits`,
+ url: `${config.API_BASE_URL}/credits`,
params: { headers },
});
requests.push({
method: "GET",
- url: `${config.API_BASE_URL}/api/graphs`,
+ url: `${config.API_BASE_URL}/graphs`,
params: { headers },
});
// Workflow 2: User exploring available blocks for building agents
requests.push({
method: "GET",
- url: `${config.API_BASE_URL}/api/blocks`,
+ url: `${config.API_BASE_URL}/blocks`,
params: { headers },
});
// Workflow 3: User monitoring their recent executions
requests.push({
method: "GET",
- url: `${config.API_BASE_URL}/api/executions`,
+ url: `${config.API_BASE_URL}/executions`,
params: { headers },
});
}
@@ -200,7 +152,7 @@ export default function (data) {
})
) {
executionsSuccesses++;
- userOperations.add(1);
+ executionOperations.add(1);
}
break;
}
@@ -214,295 +166,8 @@ export default function (data) {
sleep(Math.random() * 3 + 1); // 1-4 seconds
}
-function userProfileJourney(headers) {
- const startTime = Date.now();
-
- // 1. Get user credits (JWT-only endpoint)
- const creditsResponse = http.get(`${config.API_BASE_URL}/api/credits`, {
- headers,
- });
-
- userOperations.add(1);
-
- check(creditsResponse, {
- "User credits loaded successfully": (r) => r.status === 200,
- });
-
- // 2. Check onboarding status
- const onboardingResponse = http.get(`${config.API_BASE_URL}/api/onboarding`, {
- headers,
- });
-
- userOperations.add(1);
-
- check(onboardingResponse, {
- "Onboarding status loaded": (r) => r.status === 200,
- });
-
- apiResponseTime.add(Date.now() - startTime);
-}
-
-function graphManagementJourney(headers) {
- const startTime = Date.now();
-
- // 1. List existing graphs
- const listResponse = http.get(`${config.API_BASE_URL}/api/graphs`, {
- headers,
- });
-
- graphOperations.add(1);
-
- const listSuccess = check(listResponse, {
- "Graphs list loaded successfully": (r) => r.status === 200,
- });
-
- // 2. Create a new graph (20% of users)
- if (Math.random() < 0.2) {
- const graphData = generateTestGraph();
-
- const createResponse = http.post(
- `${config.API_BASE_URL}/api/graphs`,
- JSON.stringify(graphData),
- { headers },
- );
-
- graphOperations.add(1);
-
- const createSuccess = check(createResponse, {
- "Graph created successfully": (r) => r.status === 200,
- });
-
- if (createSuccess && createResponse.status === 200) {
- try {
- const createdGraph = JSON.parse(createResponse.body);
-
- // 3. Get the created graph details
- const getResponse = http.get(
- `${config.API_BASE_URL}/api/graphs/${createdGraph.id}`,
- { headers },
- );
-
- graphOperations.add(1);
-
- check(getResponse, {
- "Graph details loaded": (r) => r.status === 200,
- });
-
- // 4. Execute the graph (50% chance)
- if (Math.random() < 0.5) {
- executeGraphScenario(createdGraph, headers);
- }
-
- // 5. Create schedule for graph (10% chance)
- if (Math.random() < 0.1) {
- createScheduleScenario(createdGraph.id, headers);
- }
- } catch (error) {
- console.error("Error handling created graph:", error);
- }
- }
- }
-
- // 3. Work with existing graphs (if any)
- if (listSuccess && listResponse.status === 200) {
- try {
- const existingGraphs = JSON.parse(listResponse.body);
-
- if (existingGraphs.length > 0) {
- // Pick a random existing graph
- const randomGraph =
- existingGraphs[Math.floor(Math.random() * existingGraphs.length)];
-
- // Get graph details
- const getResponse = http.get(
- `${config.API_BASE_URL}/api/graphs/${randomGraph.id}`,
- { headers },
- );
-
- graphOperations.add(1);
-
- check(getResponse, {
- "Existing graph details loaded": (r) => r.status === 200,
- });
-
- // Execute existing graph (30% chance)
- if (Math.random() < 0.3) {
- executeGraphScenario(randomGraph, headers);
- }
- }
- } catch (error) {
- console.error("Error working with existing graphs:", error);
- }
- }
-
- apiResponseTime.add(Date.now() - startTime);
-}
-
-function executeGraphScenario(graph, headers) {
- const startTime = Date.now();
-
- const executionInputs = generateExecutionInputs();
-
- const executeResponse = http.post(
- `${config.API_BASE_URL}/api/graphs/${graph.id}/execute/${graph.version}`,
- JSON.stringify({
- inputs: executionInputs,
- credentials_inputs: {},
- }),
- { headers },
- );
-
- executionOperations.add(1);
-
- const executeSuccess = check(executeResponse, {
- "Graph execution initiated": (r) => r.status === 200 || r.status === 402, // 402 = insufficient credits
- });
-
- if (executeSuccess && executeResponse.status === 200) {
- try {
- const execution = JSON.parse(executeResponse.body);
-
- // Monitor execution status (simulate user checking results)
- // Note: setTimeout doesn't work in k6, so we'll check status immediately
- const statusResponse = http.get(
- `${config.API_BASE_URL}/api/graphs/${graph.id}/executions/${execution.id}`,
- { headers },
- );
-
- executionOperations.add(1);
-
- check(statusResponse, {
- "Execution status retrieved": (r) => r.status === 200,
- });
- } catch (error) {
- console.error("Error monitoring execution:", error);
- }
- }
-
- apiResponseTime.add(Date.now() - startTime);
-}
-
-function createScheduleScenario(graphId, headers) {
- const scheduleData = generateScheduleData(graphId);
-
- const scheduleResponse = http.post(
- `${config.API_BASE_URL}/api/graphs/${graphId}/schedules`,
- JSON.stringify(scheduleData),
- { headers },
- );
-
- graphOperations.add(1);
-
- check(scheduleResponse, {
- "Schedule created successfully": (r) => r.status === 200,
- });
-}
-
-function blockOperationsJourney(headers) {
- const startTime = Date.now();
-
- // 1. Get available blocks
- const blocksResponse = http.get(`${config.API_BASE_URL}/api/blocks`, {
- headers,
- });
-
- userOperations.add(1);
-
- const blocksSuccess = check(blocksResponse, {
- "Blocks list loaded": (r) => r.status === 200,
- });
-
- // 2. Execute some blocks directly (simulate testing)
- if (blocksSuccess && Math.random() < 0.3) {
- // Execute GetCurrentTimeBlock (simple, fast block)
- const timeBlockResponse = http.post(
- `${config.API_BASE_URL}/api/blocks/a892b8d9-3e4e-4e9c-9c1e-75f8efcf1bfa/execute`,
- JSON.stringify({
- trigger: "test",
- format_type: {
- discriminator: "iso8601",
- timezone: "UTC",
- },
- }),
- { headers },
- );
-
- userOperations.add(1);
-
- check(timeBlockResponse, {
- "Time block executed or handled gracefully": (r) =>
- r.status === 200 || r.status === 500, // 500 = user_context missing (expected)
- });
- }
-
- apiResponseTime.add(Date.now() - startTime);
-}
-
-function systemOperationsJourney(headers) {
- const startTime = Date.now();
-
- // 1. Check executions list (simulate monitoring)
- const executionsResponse = http.get(`${config.API_BASE_URL}/api/executions`, {
- headers,
- });
-
- userOperations.add(1);
-
- check(executionsResponse, {
- "Executions list loaded": (r) => r.status === 200,
- });
-
- // 2. Check schedules (if any)
- const schedulesResponse = http.get(`${config.API_BASE_URL}/api/schedules`, {
- headers,
- });
-
- userOperations.add(1);
-
- check(schedulesResponse, {
- "Schedules list loaded": (r) => r.status === 200,
- });
-
- // 3. Check API keys (simulate user managing access)
- if (Math.random() < 0.1) {
- // 10% of users check API keys
- const apiKeysResponse = http.get(`${config.API_BASE_URL}/api/api-keys`, {
- headers,
- });
-
- userOperations.add(1);
-
- check(apiKeysResponse, {
- "API keys list loaded": (r) => r.status === 200,
- });
-
- // Occasionally create new API key (5% chance)
- if (Math.random() < 0.05) {
- const keyData = generateAPIKeyRequest();
-
- const createKeyResponse = http.post(
- `${config.API_BASE_URL}/api/api-keys`,
- JSON.stringify(keyData),
- { headers },
- );
-
- userOperations.add(1);
-
- check(createKeyResponse, {
- "API key created successfully": (r) => r.status === 200,
- });
- }
- }
-
- apiResponseTime.add(Date.now() - startTime);
-}
-
export function teardown(data) {
console.log("🧹 Cleaning up load test...");
- console.log(`Total user operations: ${userOperations.value}`);
- console.log(`Total graph operations: ${graphOperations.value}`);
- console.log(`Total execution operations: ${executionOperations.value}`);
-
const testDuration = Date.now() - data.timestamp;
console.log(`Test completed in ${testDuration}ms`);
}
diff --git a/autogpt_platform/backend/load-tests/tests/marketplace/library-access-test.js b/autogpt_platform/backend/load-tests/tests/marketplace/library-access-test.js
index b81f689797..689bb2115b 100644
--- a/autogpt_platform/backend/load-tests/tests/marketplace/library-access-test.js
+++ b/autogpt_platform/backend/load-tests/tests/marketplace/library-access-test.js
@@ -3,7 +3,7 @@ import http from "k6/http";
import { Counter } from "k6/metrics";
import { getEnvironmentConfig } from "../../configs/environment.js";
-import { getPreAuthenticatedHeaders } from "../../configs/pre-authenticated-tokens.js";
+import { authHeaders, authSetupTimeout, setupAuth } from "../../lib/auth.js";
const config = getEnvironmentConfig();
const BASE_URL = config.API_BASE_URL;
@@ -48,19 +48,19 @@ export const options = {
test_type: "marketplace_library_authorized",
environment: __ENV.K6_ENVIRONMENT || "DEV",
},
+ // handleSummary reports p(99), which k6 leaves out by default.
+ summaryTrendStats: ["avg", "min", "med", "max", "p(90)", "p(95)", "p(99)"],
+ setupTimeout: authSetupTimeout(),
};
-export default function () {
- console.log(`πŸ“š VU ${__VU} starting authenticated library journey...`);
+export function setup() {
+ return setupAuth();
+}
- // Get pre-authenticated headers
- const headers = getPreAuthenticatedHeaders(__VU);
- if (!headers || !headers.Authorization) {
- console.log(`❌ VU ${__VU} authentication failed, skipping iteration`);
- authenticationAttempts.add(1);
- return;
- }
+export default function (data) {
+ console.log(`πŸ“š VU ${__VU} starting authenticated library journey...`);
+ const headers = authHeaders(data);
authenticationAttempts.add(1);
authenticationSuccesses.add(1);
@@ -79,7 +79,7 @@ function authenticatedLibraryJourney(headers) {
// Step 1: Get user's library agents
console.log(`πŸ“– VU ${__VU} fetching user library agents...`);
const libraryAgentsResponse = http.get(
- `${BASE_URL}/api/library/agents?page=1&page_size=20`,
+ `${BASE_URL}/library/agents?page=1&page_size=20`,
{ headers },
);
@@ -109,7 +109,7 @@ function authenticatedLibraryJourney(headers) {
// Step 2: Get favorite agents
console.log(`⭐ VU ${__VU} fetching favorite library agents...`);
const favoriteAgentsResponse = http.get(
- `${BASE_URL}/api/library/agents/favorites?page=1&page_size=10`,
+ `${BASE_URL}/library/agents/favorites?page=1&page_size=10`,
{ headers },
);
@@ -141,21 +141,17 @@ function authenticatedLibraryJourney(headers) {
// First get available store agents to find one to add
const storeAgentsResponse = http.get(
- `${BASE_URL}/api/store/agents?page=1&page_size=5`,
+ `${BASE_URL}/store/agents?page=1&page_size=5`,
);
libraryRequests.add(1);
+ // A new preview's store is empty, so an empty list still counts.
const storeAgentsSuccess = check(storeAgentsResponse, {
"Store agents endpoint returns 200": (r) => r.status === 200,
"Store agents response has data": (r) => {
try {
const json = r.json();
- return (
- json &&
- json.agents &&
- Array.isArray(json.agents) &&
- json.agents.length > 0
- );
+ return json && Array.isArray(json.agents);
} catch {
return false;
}
@@ -172,18 +168,20 @@ function authenticatedLibraryJourney(headers) {
storeAgentsJson.agents[
Math.floor(Math.random() * storeAgentsJson.agents.length)
];
+ const storeListingVersionId =
+ getStoreListingVersionId(randomStoreAgent);
- if (randomStoreAgent?.store_listing_version_id) {
+ if (storeListingVersionId) {
console.log(
- `βž• VU ${__VU} adding agent "${randomStoreAgent.name || "Unknown"}" to library...`,
+ `βž• VU ${__VU} adding agent "${randomStoreAgent.agent_name || "Unknown"}" to library...`,
);
const addAgentPayload = {
- store_listing_version_id: randomStoreAgent.store_listing_version_id,
+ store_listing_version_id: storeListingVersionId,
};
const addAgentResponse = http.post(
- `${BASE_URL}/api/library/agents`,
+ `${BASE_URL}/library/agents`,
JSON.stringify(addAgentPayload),
{ headers },
);
@@ -218,7 +216,7 @@ function authenticatedLibraryJourney(headers) {
};
const updateAgentResponse = http.patch(
- `${BASE_URL}/api/library/agents/${addedAgentJson.id}`,
+ `${BASE_URL}/library/agents/${addedAgentJson.id}`,
JSON.stringify(updatePayload),
{ headers },
);
@@ -250,7 +248,7 @@ function authenticatedLibraryJourney(headers) {
// Step 5: Get specific library agent details
console.log(`πŸ“„ VU ${__VU} fetching agent details...`);
const agentDetailsResponse = http.get(
- `${BASE_URL}/api/library/agents/${addedAgentJson.id}`,
+ `${BASE_URL}/library/agents/${addedAgentJson.id}`,
{ headers },
);
@@ -281,7 +279,7 @@ function authenticatedLibraryJourney(headers) {
// Step 6: Fork the library agent (simulate user customization)
console.log(`🍴 VU ${__VU} forking agent for customization...`);
const forkAgentResponse = http.post(
- `${BASE_URL}/api/library/agents/${addedAgentJson.id}/fork`,
+ `${BASE_URL}/library/agents/${addedAgentJson.id}/fork`,
"",
{ headers },
);
@@ -342,7 +340,7 @@ function authenticatedLibraryJourney(headers) {
console.log(`πŸ” VU ${__VU} searching library for "${randomSearchTerm}"...`);
const searchLibraryResponse = http.get(
- `${BASE_URL}/api/library/agents?search_term=${encodeURIComponent(randomSearchTerm)}&page=1&page_size=10`,
+ `${BASE_URL}/library/agents?search_term=${encodeURIComponent(randomSearchTerm)}&page=1&page_size=10`,
{ headers },
);
@@ -384,7 +382,7 @@ function authenticatedLibraryJourney(headers) {
`πŸ”— VU ${__VU} fetching agent by graph ID "${randomLibraryAgent.graph_id}"...`,
);
const agentByGraphResponse = http.get(
- `${BASE_URL}/api/library/agents/by-graph/${randomLibraryAgent.graph_id}`,
+ `${BASE_URL}/library/agents/by-graph/${randomLibraryAgent.graph_id}`,
{ headers },
);
@@ -431,6 +429,30 @@ function authenticatedLibraryJourney(headers) {
);
}
+// Store list entries carry no listing version id; the agent's details do.
+function getStoreListingVersionId(storeAgent) {
+ if (!storeAgent?.creator || !storeAgent?.slug) return null;
+
+ const detailsResponse = http.get(
+ `${BASE_URL}/store/agents/${encodeURIComponent(storeAgent.creator)}/${encodeURIComponent(storeAgent.slug)}`,
+ );
+ libraryRequests.add(1);
+ const detailsSuccess = check(detailsResponse, {
+ "Store agent details returns 200": (r) => r.status === 200,
+ });
+ if (!detailsSuccess) {
+ failedRequests.add(1);
+ return null;
+ }
+ successfulRequests.add(1);
+
+ try {
+ return detailsResponse.json("store_listing_version_id") || null;
+ } catch {
+ return null;
+ }
+}
+
export function handleSummary(data) {
const summary = {
test_type: "Marketplace Library Authorized Access Load Test",
@@ -443,11 +465,11 @@ export function handleSummary(data) {
requests_per_vu: REQUESTS_PER_VU,
},
performance_metrics: {
- total_requests: data.metrics.http_reqs?.count || 0,
+ total_requests: data.metrics.http_reqs?.values?.count || 0,
failed_requests: data.metrics.http_req_failed?.values?.passes || 0,
avg_response_time: data.metrics.http_req_duration?.values?.avg || 0,
- p95_response_time: data.metrics.http_req_duration?.values?.p95 || 0,
- p99_response_time: data.metrics.http_req_duration?.values?.p99 || 0,
+ p95_response_time: data.metrics.http_req_duration?.values?.["p(95)"] || 0,
+ p99_response_time: data.metrics.http_req_duration?.values?.["p(99)"] || 0,
},
custom_metrics: {
library_requests: data.metrics.library_requests_total?.values?.count || 0,
@@ -461,9 +483,11 @@ export function handleSummary(data) {
},
thresholds_met: {
p95_threshold:
- (data.metrics.http_req_duration?.values?.p95 || 0) < THRESHOLD_P95,
+ (data.metrics.http_req_duration?.values?.["p(95)"] || 0) <
+ THRESHOLD_P95,
p99_threshold:
- (data.metrics.http_req_duration?.values?.p99 || 0) < THRESHOLD_P99,
+ (data.metrics.http_req_duration?.values?.["p(99)"] || 0) <
+ THRESHOLD_P99,
error_rate_threshold:
(data.metrics.http_req_failed?.values?.rate || 0) <
THRESHOLD_ERROR_RATE,
diff --git a/autogpt_platform/backend/load-tests/tests/marketplace/public-access-test.js b/autogpt_platform/backend/load-tests/tests/marketplace/public-access-test.js
index 97a7066431..aaab28c127 100644
--- a/autogpt_platform/backend/load-tests/tests/marketplace/public-access-test.js
+++ b/autogpt_platform/backend/load-tests/tests/marketplace/public-access-test.js
@@ -61,6 +61,8 @@ export const options = {
test_type: "marketplace_public_access",
environment: __ENV.K6_ENVIRONMENT || "DEV",
},
+ // handleSummary reports p(99), which k6 leaves out by default.
+ summaryTrendStats: ["avg", "min", "med", "max", "p(90)", "p(95)", "p(99)"],
};
export default function () {
@@ -76,7 +78,7 @@ function marketplaceBrowsingJourney() {
// Step 1: Browse marketplace homepage - get featured agents
console.log(`πŸͺ VU ${__VU} browsing marketplace homepage...`);
const featuredAgentsResponse = http.get(
- `${BASE_URL}/api/store/agents?featured=true&page=1&page_size=10`,
+ `${BASE_URL}/store/agents?featured=true&page=1&page_size=10`,
);
logHttpError(
featuredAgentsResponse,
@@ -108,22 +110,18 @@ function marketplaceBrowsingJourney() {
// Step 2: Browse all agents with pagination
console.log(`πŸ“‹ VU ${__VU} browsing all agents...`);
const allAgentsResponse = http.get(
- `${BASE_URL}/api/store/agents?page=1&page_size=20`,
+ `${BASE_URL}/store/agents?page=1&page_size=20`,
);
logHttpError(allAgentsResponse, "/api/store/agents", "GET");
marketplaceRequests.add(1);
+ // A new preview's store is empty, so an empty list still counts.
const allAgentsSuccess = check(allAgentsResponse, {
"All agents endpoint returns 200": (r) => r.status === 200,
"All agents response has data": (r) => {
try {
const json = r.json();
- return (
- json &&
- json.agents &&
- Array.isArray(json.agents) &&
- json.agents.length > 0
- );
+ return json && Array.isArray(json.agents);
} catch {
return false;
}
@@ -150,7 +148,7 @@ function marketplaceBrowsingJourney() {
console.log(`πŸ” VU ${__VU} searching for "${randomQuery}" agents...`);
const searchResponse = http.get(
- `${BASE_URL}/api/store/agents?search_query=${encodeURIComponent(randomQuery)}&page=1&page_size=10`,
+ `${BASE_URL}/store/agents?search_query=${encodeURIComponent(randomQuery)}&page=1&page_size=10`,
);
logHttpError(searchResponse, "/api/store/agents (search)", "GET");
@@ -176,13 +174,19 @@ function marketplaceBrowsingJourney() {
}
// Step 4: Browse agents by category
- const categories = ["AI", "PRODUCTIVITY", "COMMUNICATION", "DATA", "SOCIAL"];
+ const categories = [
+ "marketing",
+ "sales",
+ "research",
+ "content",
+ "development",
+ ];
const randomCategory =
categories[Math.floor(Math.random() * categories.length)];
console.log(`πŸ“‚ VU ${__VU} browsing "${randomCategory}" category...`);
const categoryResponse = http.get(
- `${BASE_URL}/api/store/agents?category=${randomCategory}&page=1&page_size=15`,
+ `${BASE_URL}/store/agents?category=${randomCategory}&page=1&page_size=15`,
);
logHttpError(categoryResponse, "/api/store/agents (category)", "GET");
@@ -217,12 +221,12 @@ function marketplaceBrowsingJourney() {
Math.floor(Math.random() * allAgentsJson.agents.length)
];
- if (randomAgent?.creator_username && randomAgent?.slug) {
+ if (randomAgent?.creator && randomAgent?.slug) {
console.log(
`πŸ“„ VU ${__VU} viewing agent details for "${randomAgent.slug}"...`,
);
const agentDetailsResponse = http.get(
- `${BASE_URL}/api/store/agents/${encodeURIComponent(randomAgent.creator_username)}/${encodeURIComponent(randomAgent.slug)}`,
+ `${BASE_URL}/store/agents/${encodeURIComponent(randomAgent.creator)}/${encodeURIComponent(randomAgent.slug)}`,
);
logHttpError(
agentDetailsResponse,
@@ -236,7 +240,12 @@ function marketplaceBrowsingJourney() {
"Agent details response has data": (r) => {
try {
const json = r.json();
- return json && json.id && json.name && json.description;
+ return (
+ json &&
+ json.store_listing_version_id &&
+ json.agent_name &&
+ typeof json.description === "string"
+ );
} catch {
return false;
}
@@ -263,7 +272,7 @@ function marketplaceBrowsingJourney() {
// Step 6: Browse creators
console.log(`πŸ‘₯ VU ${__VU} browsing creators...`);
const creatorsResponse = http.get(
- `${BASE_URL}/api/store/creators?page=1&page_size=20`,
+ `${BASE_URL}/store/creators?page=1&page_size=20`,
);
logHttpError(creatorsResponse, "/api/store/creators", "GET");
@@ -290,7 +299,7 @@ function marketplaceBrowsingJourney() {
// Step 7: Get featured creators
console.log(`⭐ VU ${__VU} browsing featured creators...`);
const featuredCreatorsResponse = http.get(
- `${BASE_URL}/api/store/creators?featured=true&page=1&page_size=10`,
+ `${BASE_URL}/store/creators?featured=true&page=1&page_size=10`,
);
logHttpError(
featuredCreatorsResponse,
@@ -334,11 +343,11 @@ function marketplaceBrowsingJourney() {
`πŸ‘€ VU ${__VU} viewing creator details for "${randomCreator.username}"...`,
);
const creatorDetailsResponse = http.get(
- `${BASE_URL}/api/store/creator/${encodeURIComponent(randomCreator.username)}`,
+ `${BASE_URL}/store/creators/${encodeURIComponent(randomCreator.username)}`,
);
logHttpError(
creatorDetailsResponse,
- "/api/store/creator/{username}",
+ "/api/store/creators/{username}",
"GET",
);
@@ -389,11 +398,11 @@ export function handleSummary(data) {
ramp_down: RAMP_DOWN,
},
performance_metrics: {
- total_requests: data.metrics.http_reqs?.count || 0,
+ total_requests: data.metrics.http_reqs?.values?.count || 0,
failed_requests: data.metrics.http_req_failed?.values?.passes || 0,
avg_response_time: data.metrics.http_req_duration?.values?.avg || 0,
- p95_response_time: data.metrics.http_req_duration?.values?.p95 || 0,
- p99_response_time: data.metrics.http_req_duration?.values?.p99 || 0,
+ p95_response_time: data.metrics.http_req_duration?.values?.["p(95)"] || 0,
+ p99_response_time: data.metrics.http_req_duration?.values?.["p(99)"] || 0,
},
custom_metrics: {
marketplace_requests:
@@ -404,9 +413,11 @@ export function handleSummary(data) {
},
thresholds_met: {
p95_threshold:
- (data.metrics.http_req_duration?.values?.p95 || 0) < THRESHOLD_P95,
+ (data.metrics.http_req_duration?.values?.["p(95)"] || 0) <
+ THRESHOLD_P95,
p99_threshold:
- (data.metrics.http_req_duration?.values?.p99 || 0) < THRESHOLD_P99,
+ (data.metrics.http_req_duration?.values?.["p(99)"] || 0) <
+ THRESHOLD_P99,
error_rate_threshold:
(data.metrics.http_req_failed?.values?.rate || 0) <
THRESHOLD_ERROR_RATE,