PostgreSQL backup tool with Point-In-Time-Recovery and restore verification
TypeScript backupbackupsdatabasedatabase-backupdevopsdockergolangkubernetesmariadbmongodbmysqlpgpostgrespostgresqls3self-hostedsystem-administrationtoolsweb-ui
FEATURE (dev): Seed a dev account with logical and WAL-streaming databases on just run, restart the backend on Go changes, and give Claude Code and Codex a pinned browser server, so an agent can open and check the UI without manual setup
Rostislav Dugin67541fb46b4b
diff --git a/.agents/skills/run-app/SKILL.md b/.agents/skills/run-app/SKILL.mdnew file mode 100644index 00000000..1b4374c4--- /dev/null+++ b/.agents/skills/run-app/SKILL.md@@ -0,0 +1,29 @@+---+name: run-app+description: Start, reuse and stop the local Databasus app, and check its pages in the browser through the chrome-devtools MCP server. Use when a task needs to see or click through the web interface, to check a UI change, or to launch the app locally.+---++# Run the app and check it in the browser++The browser server is registered for Claude Code and Codex as `chrome-devtools`. It opens a fresh browser profile in every session, so sign in at the start of each one. Whether the browser is headless or visible is the developer's choice in `.env`; leave `BROWSER_HEADLESS` as it is.++The ports to check and open are the ones the `links` process prints in [`process-compose.yaml`](../../../process-compose.yaml). The account, the seeded databases and their sources are listed in the package comment of the [seed command](../../../backend/cmd/seed_dev/doc.go).++## Before opening a page++Check both ports first, and act on what you find:++- **Both answer, and the app is from your worktree:** use it and leave it running when you hand the task over. The app is yours when the process listening on the backend port has its working directory inside your worktree; read it with `ss -ltnp` and `/proc/<pid>/cwd`.+- **Both answer, and the app is from another worktree:** leave it running, skip the browser check and tell the user it was skipped because the running app serves another worktree's code.+- **Only one answers:** start nothing, skip the browser check and tell the user which port is held.+- **Neither answers:** start `just run -t=false` at the repository root in the background, and wait until the backend port accepts connections and the seed process has finished.++## Working with the app++- Sign in with the seeded account.+- When the app of your worktree lacks the seeded account or a seeded database, run `go run ./cmd/seed_dev` in `backend/` instead of creating them by hand. It creates only what is missing.+- A browser check adds to the tests the task names and never replaces them.++## When you are done++Stop a run you started yourself before handing the task over. Send SIGINT to the `process-compose` process whose working directory is your worktree root, wait for it to exit, and confirm that both ports are free. A run killed with SIGKILL leaves processes and containers behind, so never use it. If a port stays held, tell the user which one.diff --git a/.claude/skills/run-app/SKILL.md b/.claude/skills/run-app/SKILL.mdnew file mode 120000index 00000000..a4ef2c65--- /dev/null+++ b/.claude/skills/run-app/SKILL.md@@ -0,0 +1 @@+../../../.agents/skills/run-app/SKILL.md\ No newline at end of filediff --git a/.codex/config.toml b/.codex/config.tomlnew file mode 100644index 00000000..b68729ef--- /dev/null+++ b/.codex/config.toml@@ -0,0 +1,4 @@+# The same browser server .mcp.json registers for Claude Code; scripts/browser-mcp.sh explains it.+[mcp_servers.chrome-devtools]+command = "bash"+args = ["-c", "exec \"$(git rev-parse --show-toplevel)/scripts/browser-mcp.sh\""]diff --git a/.env.example b/.env.exampleindex 3f728afc..d88b434d 100644--- a/.env.example+++ b/.env.example@@ -63,3 +63,6 @@ VITE_GITHUB_CLIENT_ID=VITE_GOOGLE_CLIENT_ID=VITE_IS_EMAIL_CONFIGURED=falseVITE_CLOUDFLARE_TURNSTILE_SITE_KEY=++# agent browser (true = headless, false = a visible window; applies when the agent client restarts the browser server)+BROWSER_HEADLESS=truediff --git a/.gitignore b/.gitignoreindex d361ec3d..d40f2205 100644--- a/.gitignore+++ b/.gitignore@@ -21,6 +21,7 @@ node_modules/.claude/skills/*!.claude/skills/git-commits/!.claude/skills/humanizer/+!.claude/skills/run-app/!.claude/skills/testing/backend/cleanup_test_db.exeplans/diff --git a/.mcp.json b/.mcp.jsonnew file mode 100644index 00000000..5f18595a--- /dev/null+++ b/.mcp.json@@ -0,0 +1,8 @@+{+ "mcpServers": {+ "chrome-devtools": {+ "command": "bash",+ "args": ["-c", "exec \"$(git rev-parse --show-toplevel)/scripts/browser-mcp.sh\""]+ }+ }+}diff --git a/AGENTS.md b/AGENTS.mdindex e49832e0..112cbc4f 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -32,7 +32,7 @@ Work happens on a Fedora host; any other host distro is on its own. Tool version| Command | Purpose ||---|---|-| `just run` | Start the app locally: dev database, test fixtures, WAL backup source with its load generator, backend and frontend |+| `just run` | Start the app locally: dev database, test fixtures, WAL backup source with its load generator, backend and frontend; seeds a dev account with databases and restarts the backend when its code changes || `just test` | Run the full backend suite || `just test-package <packages>`, run in `backend/` | Run the given backend test packages || `just test-filesystem` | Run the Docker filesystem suite, mostly left to CI |@@ -65,6 +65,12 @@ When you write, change, run or review tests, load the [testing skill](.agents/sk---+## Checking the app in a browser++When you start the app or look at its pages, load the [run-app skill](.agents/skills/run-app/SKILL.md) first. It says when to reuse, start and stop the app and how to sign in.++---+## Language in code**English only in code, comments, identifiers, log messages, API strings, test assertions, and commit messages.** No other language inside `backend/` or `agent/` at all, and none inside `frontend/src/` outside the interface dictionaries named below — not even for user-facing fallback copy or error messages.diff --git a/backend/.air.toml b/backend/.air.tomlnew file mode 100644index 00000000..8f171ccb--- /dev/null+++ b/backend/.air.toml@@ -0,0 +1,22 @@+# `just run` runs the backend under air: a saved Go file rebuilds it and replaces the running one.+# A build that fails leaves the previous backend serving, and the compile error shows in its output.+root = "."+tmp_dir = "tmp"++[build]+ cmd = "go build -o ./tmp/databasus ./cmd"+ entrypoint = ["./tmp/databasus"]+ include_ext = ["go"]+ # Tests and the dev-only commands are not part of the server binary, and the rest are generated+ # files or data the backend writes itself.+ exclude_dir = ["tmp", "swagger", "ui", "migrations", "cmd/seed_dev", "cmd/cleanup_test_db", "pgdata_test", "pgdata-for-restore", "temp"]+ exclude_regex = ["_test\\.go$"]+ # One rebuild for an editor that saves several files at once.+ delay = 1000+ stop_on_error = false+ # The backend shuts down on an interrupt the same way it does on Ctrl+C.+ send_interrupt = true+ kill_delay = "5s"++[misc]+ clean_on_exit = truediff --git a/backend/.gitignore b/backend/.gitignoreindex 15370304..794605c5 100644--- a/backend/.gitignore+++ b/backend/.gitignore@@ -10,3 +10,4 @@ ui/build/*pgdata-for-restore/temp/temp/+tmp/diff --git a/backend/README.md b/backend/README.mdindex b164cd35..f5d7ca72 100644--- a/backend/README.md+++ b/backend/README.md@@ -10,6 +10,10 @@ To run the backend together with the frontend, the dev database, the test fixtur> just run+The backend runs under [air](.air.toml), which rebuilds and restarts it when a Go file is saved; a build that fails leaves the previous backend serving. Once the backend accepts connections, the [seed command](cmd/seed_dev/doc.go) creates the dev account and the seeded databases if they are missing. To run the seed again against the running app, from `backend/`:++> go run ./cmd/seed_dev+To run tests:> just testdiff --git a/backend/cmd/seed_dev/api_client.go b/backend/cmd/seed_dev/api_client.gonew file mode 100644index 00000000..41b3721a--- /dev/null+++ b/backend/cmd/seed_dev/api_client.go@@ -0,0 +1,115 @@+package main++import (+ "bytes"+ "context"+ "encoding/json"+ "errors"+ "fmt"+ "io"+ "net/http"+ "strings"+)++// The backend answers a refusal with {"error": "..."} and an unhealthy check with+// {"status": "..."}; either text is the reason the developer needs to see.+type apiError struct {+ StatusCode int+ Reason string+}++func (e *apiError) Error() string {+ return e.Reason+}++type apiClient struct {+ httpClient *http.Client+ baseURL string+ token string+}++func (c *apiClient) get(ctx context.Context, path string, response any) error {+ return c.send(ctx, http.MethodGet, path, nil, response)+}++func (c *apiClient) post(ctx context.Context, path string, body, response any) error {+ return c.send(ctx, http.MethodPost, path, body, response)+}++func (c *apiClient) send(ctx context.Context, method, path string, body, response any) error {+ var requestBody io.Reader++ if body != nil {+ encodedBody, err := json.Marshal(body)+ if err != nil {+ return fmt.Errorf("encode request to %s: %w", path, err)+ }++ requestBody = bytes.NewReader(encodedBody)+ }++ request, err := http.NewRequestWithContext(ctx, method, c.baseURL+"/api/v1"+path, requestBody)+ if err != nil {+ return err+ }++ request.Header.Set("Content-Type", "application/json")++ if c.token != "" {+ request.Header.Set("Authorization", "Bearer "+c.token)+ }++ httpResponse, err := c.httpClient.Do(request)+ if err != nil {+ return err+ }+ defer func() { _ = httpResponse.Body.Close() }()++ responseBody, err := io.ReadAll(httpResponse.Body)+ if err != nil {+ return fmt.Errorf("read response from %s: %w", path, err)+ }++ if httpResponse.StatusCode >= http.StatusBadRequest {+ return &apiError{httpResponse.StatusCode, readRefusalReason(httpResponse.StatusCode, responseBody)}+ }++ if response == nil {+ return nil+ }++ if err := json.Unmarshal(responseBody, response); err != nil {+ return fmt.Errorf("decode response from %s: %w", path, err)+ }++ return nil+}++func readRefusalReason(statusCode int, responseBody []byte) string {+ var refusal struct {+ Error string `json:"error"`+ Status string `json:"status"`+ }++ if json.Unmarshal(responseBody, &refusal) == nil {+ if refusal.Error != "" {+ return refusal.Error+ }++ if refusal.Status != "" {+ return refusal.Status+ }+ }++ if text := strings.TrimSpace(string(responseBody)); text != "" {+ return fmt.Sprintf("HTTP %d: %s", statusCode, text)+ }++ return fmt.Sprintf("HTTP %d", statusCode)+}++func isAPIRefusal(err error) bool {+ var refusal *apiError++ return errors.As(err, &refusal)+}diff --git a/backend/cmd/seed_dev/doc.go b/backend/cmd/seed_dev/doc.gonew file mode 100644index 00000000..9598f1c3--- /dev/null+++ b/backend/cmd/seed_dev/doc.go@@ -0,0 +1,38 @@+// Command seed_dev fills the local development app with what a developer or an agent needs to+// look at a screen right after start: an account to sign in with, a workspace with a local storage+// and two PostgreSQL databases whose backups are already running. `just run` starts it once the+// backend accepts connections. Run it on its own with `go run ./cmd/seed_dev` in backend/ to+// restore an item that was deleted from the running app.+//+// It talks to the backend on http://localhost:4005 through the public API, so every item passes+// the same validation as in the web interface. The production image builds only ./cmd and never+// contains this command.+//+// What it makes sure exists, finding each item by its name:+//+// - the account [email protected] with the password Q1234567;+// - the workspace "Dev" and, in it, the local storage "Local";+// - "Logical PG16": logical backups of the shared logical test fixture on+// TEST_LOGICAL_POSTGRES_16_PORT (5004), database testdb, daily, keeping the 3 newest backups;+// - "WAL PG18": WAL-streaming physical backups of the live WAL backup source on+// WAL_BACKUP_POSTGRES_18_PORT (5018), signed in as WAL_BACKUP_DB_USER, keeping the 2 newest+// chains.+//+// Missing items are created; a database gets its backups switched on with "Local" and a first+// backup right away. An item that already exists is left exactly as it is, including one a+// developer edited, and an existing account is only ever signed in to. The seed refuses to run+// unless ENV_MODE in .env is development. It exits with a non-zero status and the backend's+// reason when the backend does not answer or reports itself unhealthy, when it cannot sign in as+// the account or register it, when the instance requires two-factor sign-in, and when the backend+// refuses to create an item.+//+// Fixing by hand what the seed leaves alone:+//+// - Recreating the wal-backup-postgres-18 container drops the replication slot of "WAL PG18",+// and its backups start failing. Delete "WAL PG18" in the web interface and run the seed again.+// - Recreating the backend-db container forgets "WAL PG18" but leaves its replication slot on+// the WAL backup source, which then keeps WAL without limit. Find the inactive slot named+// databasus_slot_* in pg_replication_slots there and drop it with pg_drop_replication_slot.+// - The backend reports itself unhealthy once 95% of the disk is used, and the seed then stops+// with that reason. Free disk space and run the seed again.+package maindiff --git a/backend/cmd/seed_dev/main.go b/backend/cmd/seed_dev/main.gonew file mode 100644index 00000000..212a1c2f--- /dev/null+++ b/backend/cmd/seed_dev/main.go@@ -0,0 +1,103 @@+package main++import (+ "context"+ "errors"+ "fmt"+ "io/fs"+ "log/slog"+ "net/http"+ "os"+ "strconv"+ "time"++ "github.com/joho/godotenv"+)++const (+ devBackendURL = "http://localhost:4005"++ // The command runs in backend/, next to the repository's .env.+ envFilePath = "../.env"++ // Creating a WAL-streaming database runs the replication readiness checks on the source,+ // which takes longer than an ordinary request.+ requestTimeout = 2 * time.Minute+)++func main() {+ log := slog.New(slog.NewTextHandler(os.Stdout, nil))++ params, err := loadSeedParams()+ if err != nil {+ log.Error("cannot read the seed settings", "error", err)+ os.Exit(1)+ }++ os.Exit(runSeed(context.Background(), &http.Client{Timeout: requestTimeout}, params, log))+}++func runSeed(ctx context.Context, httpClient *http.Client, params seedParams, log *slog.Logger) int {+ if err := seedDevApp(ctx, httpClient, params, log); err != nil {+ log.Error("seed failed", "error", err)++ return 1+ }++ return 0+}++// The defaults are the ones docker-compose.yml falls back to, so a .env without a variable still+// points at the container compose started.+func loadSeedParams() (seedParams, error) {+ if err := godotenv.Load(envFilePath); err != nil && !errors.Is(err, fs.ErrNotExist) {+ return seedParams{}, fmt.Errorf("load %s: %w", envFilePath, err)+ }++ if envMode := os.Getenv("ENV_MODE"); envMode != "development" {+ return seedParams{}, fmt.Errorf("ENV_MODE is %q, and the seed runs only in development", envMode)+ }++ logicalPort, err := strconv.Atoi(readEnvOrDefault("TEST_LOGICAL_POSTGRES_16_PORT", "5004"))+ if err != nil {+ return seedParams{}, fmt.Errorf("TEST_LOGICAL_POSTGRES_16_PORT: %w", err)+ }++ walPort, err := strconv.Atoi(readEnvOrDefault("WAL_BACKUP_POSTGRES_18_PORT", "5018"))+ if err != nil {+ return seedParams{}, fmt.Errorf("WAL_BACKUP_POSTGRES_18_PORT: %w", err)+ }++ sourceHost := readEnvOrDefault("TEST_LOCALHOST", "localhost")++ return seedParams{+ BaseURL: devBackendURL,+ Email: "[email protected]",+ Password: "Q1234567",+ WorkspaceName: "Dev",+ StorageName: "Local",+ LogicalDatabaseName: "Logical PG16",+ LogicalSource: postgresSource{+ Host: sourceHost,+ Port: logicalPort,+ Username: "testuser",+ Password: "testpassword",+ Database: "testdb",+ },+ WalDatabaseName: "WAL PG18",+ WalSource: postgresSource{+ Host: sourceHost,+ Port: walPort,+ Username: readEnvOrDefault("WAL_BACKUP_DB_USER", "testuser"),+ Password: readEnvOrDefault("WAL_BACKUP_DB_PASSWORD", "testpassword"),+ },+ }, nil+}++func readEnvOrDefault(name, defaultValue string) string {+ if value := os.Getenv(name); value != "" {+ return value+ }++ return defaultValue+}diff --git a/backend/cmd/seed_dev/main_test.go b/backend/cmd/seed_dev/main_test.gonew file mode 100644index 00000000..8b44d0fe--- /dev/null+++ b/backend/cmd/seed_dev/main_test.go@@ -0,0 +1,572 @@+package main++import (+ "bytes"+ "context"+ "log/slog"+ "net/http"+ "net/http/httptest"+ "strconv"+ "sync"+ "testing"+ "time"++ "github.com/gin-gonic/gin"+ "github.com/google/uuid"+ "github.com/stretchr/testify/assert"+ "github.com/stretchr/testify/require"++ "databasus-backend/internal/config"+ audit_logs "databasus-backend/internal/features/audit_logs"+ backuping_logical "databasus-backend/internal/features/backups/backups/backuping/logical"+ backuping_physical "databasus-backend/internal/features/backups/backups/backuping/physical"+ backups_controllers_logical "databasus-backend/internal/features/backups/backups/controllers/logical"+ backups_controllers_physical "databasus-backend/internal/features/backups/backups/controllers/physical"+ backups_dto_logical "databasus-backend/internal/features/backups/backups/dto/logical"+ backups_services "databasus-backend/internal/features/backups/backups/services"+ backups_config_logical "databasus-backend/internal/features/backups/config/logical"+ backups_config_physical "databasus-backend/internal/features/backups/config/physical"+ "databasus-backend/internal/features/databases"+ "databasus-backend/internal/features/email"+ "databasus-backend/internal/features/notifiers"+ "databasus-backend/internal/features/storages"+ local_storage "databasus-backend/internal/features/storages/models/local"+ system_healthcheck "databasus-backend/internal/features/system/healthcheck"+ users_controllers "databasus-backend/internal/features/users/controllers"+ users_dto "databasus-backend/internal/features/users/dto"+ users_enums "databasus-backend/internal/features/users/enums"+ users_middleware "databasus-backend/internal/features/users/middleware"+ users_services "databasus-backend/internal/features/users/services"+ users_testing "databasus-backend/internal/features/users/testing"+ workspaces_controllers "databasus-backend/internal/features/workspaces/controllers"+ workspaces_dto "databasus-backend/internal/features/workspaces/dto"+ workspaces_errors "databasus-backend/internal/features/workspaces/errors"+ test_utils "databasus-backend/internal/util/testing"+)++const (+ seedPassword = "Q1234567"+ otherPassword = "Other-password-1"++ firstBackupTimeout = 2 * time.Minute+)++func Test_Seed_WhenAccountIsNew_CreatesWorkspaceStorageAndDatabasesWithBackups(t *testing.T) {+ router, params := setUpSeedTest(t)++ exitCode, output := runSeedForTest(t, params)+ require.Equal(t, 0, exitCode, output)++ token := signInForTest(t, router, params.Email, seedPassword)+ workspaceID := findWorkspaceID(t, router, token, params.WorkspaceName)+ require.NotEqual(t, uuid.Nil, workspaceID, "workspace %q was not created", params.WorkspaceName)++ storageID := findStorageID(t, router, token, workspaceID, params.StorageName)+ require.NotEqual(t, uuid.Nil, storageID, "storage %q was not created", params.StorageName)++ logicalDatabaseID := findDatabaseID(t, router, token, workspaceID, params.LogicalDatabaseName)+ walDatabaseID := findDatabaseID(t, router, token, workspaceID, params.WalDatabaseName)+ require.NotEqual(t, uuid.Nil, logicalDatabaseID)+ require.NotEqual(t, uuid.Nil, walDatabaseID)++ var logicalConfig backups_config_logical.LogicalBackupConfig+ test_utils.MakeGetRequestAndUnmarshal(t, router,+ "/api/v1/backup-configs/database/"+logicalDatabaseID.String(),+ "Bearer "+token, http.StatusOK, &logicalConfig)++ assert.True(t, logicalConfig.IsBackupsEnabled)+ assert.Equal(t, storageID, *logicalConfig.StorageID)+ assert.Equal(t, backups_config_logical.RetentionPolicyTypeCount, logicalConfig.RetentionPolicyType)+ assert.Equal(t, 3, logicalConfig.RetentionCount)++ var logicalBackups backups_dto_logical.GetBackupsResponse+ test_utils.MakeGetRequestAndUnmarshal(t, router,+ "/api/v1/backups?database_id="+logicalDatabaseID.String(),+ "Bearer "+token, http.StatusOK, &logicalBackups)+ assert.NotEmpty(t, logicalBackups.Backups, "the first logical backup was not started")++ var walConfig backups_config_physical.PhysicalBackupConfig+ test_utils.MakeGetRequestAndUnmarshal(t, router,+ "/api/v1/backup-configs/physical/database/"+walDatabaseID.String(),+ "Bearer "+token, http.StatusOK, &walConfig)++ assert.True(t, walConfig.IsBackupsEnabled)+ assert.Equal(t, storageID, *walConfig.StorageID)+ assert.Equal(t, backups_config_physical.RetentionChains, walConfig.Retention)+ assert.Equal(t, 2, walConfig.ChainsRetention.Count)++ storedWalConfig, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(walDatabaseID)+ require.NoError(t, err)+ assert.NotNil(t, storedWalConfig.ForceFullRequestedAt, "the first WAL-streaming backup was not requested")+}++func Test_Seed_WhenRunTwice_CreatesNothingAndKeepsEditedDatabase(t *testing.T) {+ router, params := setUpSeedTest(t)++ exitCode, output := runSeedForTest(t, params)+ require.Equal(t, 0, exitCode, output)++ token := signInForTest(t, router, params.Email, seedPassword)+ workspaceID := findWorkspaceID(t, router, token, params.WorkspaceName)+ logicalDatabaseID := findDatabaseID(t, router, token, workspaceID, params.LogicalDatabaseName)+ waitForLogicalBackups(t, logicalDatabaseID)++ var editedConfig backups_config_logical.LogicalBackupConfig+ test_utils.MakeGetRequestAndUnmarshal(t, router,+ "/api/v1/backup-configs/database/"+logicalDatabaseID.String(),+ "Bearer "+token, http.StatusOK, &editedConfig)++ editedConfig.RetentionCount = 9+ editedConfig.IsBackupsEnabled = false+ test_utils.MakePostRequest(t, router, "/api/v1/backup-configs/save",+ "Bearer "+token, editedConfig, http.StatusOK)++ itemsBefore := listSeededItems(t, router, token)++ exitCode, output = runSeedForTest(t, params)+ require.Equal(t, 0, exitCode, output)++ assert.Equal(t, itemsBefore, listSeededItems(t, router, token))++ var configAfter backups_config_logical.LogicalBackupConfig+ test_utils.MakeGetRequestAndUnmarshal(t, router,+ "/api/v1/backup-configs/database/"+logicalDatabaseID.String(),+ "Bearer "+token, http.StatusOK, &configAfter)++ assert.Equal(t, 9, configAfter.RetentionCount)+ assert.False(t, configAfter.IsBackupsEnabled)+}++func Test_Seed_WhenSeededDatabaseWasDeleted_RecreatesOnlyThatDatabase(t *testing.T) {+ router, params := setUpSeedTest(t)++ exitCode, output := runSeedForTest(t, params)+ require.Equal(t, 0, exitCode, output)++ token := signInForTest(t, router, params.Email, seedPassword)+ workspaceID := findWorkspaceID(t, router, token, params.WorkspaceName)+ storageID := findStorageID(t, router, token, workspaceID, params.StorageName)+ logicalDatabaseID := findDatabaseID(t, router, token, workspaceID, params.LogicalDatabaseName)+ deletedWalDatabaseID := findDatabaseID(t, router, token, workspaceID, params.WalDatabaseName)++ test_utils.MakeDeleteRequest(t, router, "/api/v1/databases/"+deletedWalDatabaseID.String(),+ "Bearer "+token, http.StatusNoContent)++ exitCode, output = runSeedForTest(t, params)+ require.Equal(t, 0, exitCode, output)++ recreatedWalDatabaseID := findDatabaseID(t, router, token, workspaceID, params.WalDatabaseName)+ require.NotEqual(t, uuid.Nil, recreatedWalDatabaseID)+ assert.NotEqual(t, deletedWalDatabaseID, recreatedWalDatabaseID)++ var walConfig backups_config_physical.PhysicalBackupConfig+ test_utils.MakeGetRequestAndUnmarshal(t, router,+ "/api/v1/backup-configs/physical/database/"+recreatedWalDatabaseID.String(),+ "Bearer "+token, http.StatusOK, &walConfig)+ assert.True(t, walConfig.IsBackupsEnabled)++ assert.Equal(t, workspaceID, findWorkspaceID(t, router, token, params.WorkspaceName))+ assert.Equal(t, storageID, findStorageID(t, router, token, workspaceID, params.StorageName))+ assert.Equal(t, logicalDatabaseID, findDatabaseID(t, router, token, workspaceID, params.LogicalDatabaseName))+ assert.Len(t, listSeededItems(t, router, token).databaseIDs, 2)+}++func Test_Seed_WhenAccountHasAnotherPassword_FailsAndLeavesAccountUnchanged(t *testing.T) {+ router, params := setUpSeedTest(t)++ signUpForTest(t, router, params.Email, otherPassword)+ token := signInForTest(t, router, params.Email, otherPassword)+ profileBefore := getProfile(t, router, token)++ exitCode, output := runSeedForTest(t, params)++ assert.Equal(t, 1, exitCode)+ assert.Contains(t, output, params.Email)+ assert.Contains(t, output, "password is incorrect")++ tokenAfter := signInForTest(t, router, params.Email, otherPassword)+ assert.Equal(t, profileBefore, getProfile(t, router, tokenAfter))+ assert.Empty(t, listSeededItems(t, router, tokenAfter).workspaceIDs)+}++func Test_Seed_WhenAnotherAccountExists_LeavesThatAccountUnchanged(t *testing.T) {+ router, params := setUpSeedTest(t)++ otherEmail := "seed-other-" + uuid.NewString() + "@example.com"+ signUpForTest(t, router, otherEmail, otherPassword)+ t.Cleanup(func() { removeAccountWithItems(t, router, otherEmail) })++ otherProfileBefore := getProfile(t, router, signInForTest(t, router, otherEmail, otherPassword))++ exitCode, output := runSeedForTest(t, params)+ require.Equal(t, 0, exitCode, output)++ otherToken := signInForTest(t, router, otherEmail, otherPassword)+ assert.Equal(t, otherProfileBefore, getProfile(t, router, otherToken))+ assert.Empty(t, listSeededItems(t, router, otherToken).workspaceIDs)+}++func Test_Seed_WhenTwoFactorSignInIsRequired_FailsAndCreatesNothing(t *testing.T) {+ router, params := setUpSeedTest(t)++ token := signUpForTest(t, router, params.Email, seedPassword)++ users_services.GetUserService().SetEmailSender(users_testing.NewMockEmailSender())+ users_testing.EnableTwoFactorAuth(t.Context())+ t.Cleanup(func() {+ users_services.GetUserService().SetEmailSender(email.GetEmailSMTPSender())+ users_testing.ResetSettingsToDefaults(context.Background())+ })++ exitCode, output := runSeedForTest(t, params)++ assert.Equal(t, 1, exitCode)+ assert.Contains(t, output, params.Email)+ assert.Contains(t, output, "two-factor sign-in")+ assert.Empty(t, listSeededItems(t, router, token).workspaceIDs)+}++func Test_Seed_WhenMemberMayNotCreateWorkspaces_FailsNamingWorkspace(t *testing.T) {+ router, params := setUpSeedTest(t)++ token := signUpForTest(t, router, params.Email, seedPassword)+ require.Equal(t, users_enums.UserRoleMember, getProfile(t, router, token).Role)++ var existingWorkspace workspaces_dto.WorkspaceResponseDTO+ test_utils.MakePostRequestAndUnmarshal(t, router, "/api/v1/workspaces", "Bearer "+token,+ workspaces_dto.CreateWorkspaceRequestDTO{Name: "Existing"}, http.StatusOK, &existingWorkspace)++ test_utils.MakePostRequest(t, router, "/api/v1/storages", "Bearer "+token, storages.Storage{+ WorkspaceID: existingWorkspace.ID,+ Type: storages.StorageTypeLocal,+ Name: "Existing storage",+ LocalStorage: &local_storage.LocalStorage{},+ }, http.StatusOK)++ workspacesBefore := listWorkspaces(t, router, token)+ storagesBefore := listStorages(t, router, token, existingWorkspace.ID)++ users_testing.DisableMemberWorkspaceCreation(t.Context())+ t.Cleanup(func() { users_testing.ResetSettingsToDefaults(context.Background()) })++ exitCode, output := runSeedForTest(t, params)++ assert.Equal(t, 1, exitCode)+ assert.Contains(t, output, params.WorkspaceName)+ assert.Contains(t, output, workspaces_errors.ErrInsufficientPermissionsToCreateWorkspaces.Error())++ assert.Equal(t, workspacesBefore, listWorkspaces(t, router, token))+ assert.Equal(t, storagesBefore, listStorages(t, router, token, existingWorkspace.ID))+ assert.Equal(t, uuid.Nil, findWorkspaceID(t, router, token, params.WorkspaceName))+}++func Test_Seed_WhenBackendDoesNotAnswer_FailsWithJustRunHint(t *testing.T) {+ stoppedServer := httptest.NewServer(http.NotFoundHandler())+ stoppedServer.Close()++ params := buildTestSeedParams(stoppedServer.URL)++ exitCode, output := runSeedForTest(t, params)++ assert.Equal(t, 1, exitCode)+ assert.Contains(t, output, "just run")+}++// A full disk is unsafe to reproduce, so a stub stands in for the backend's health check.+func Test_Seed_WhenBackendReportsItselfUnhealthy_FailsWithReasonAndSendsNothingElse(t *testing.T) {+ const unhealthyReason = "more than 95% of the disk is used"++ var receivedPathsMutex sync.Mutex+ var receivedPaths []string++ stubServer := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {+ receivedPathsMutex.Lock()+ receivedPaths = append(receivedPaths, request.URL.Path)+ receivedPathsMutex.Unlock()++ writer.Header().Set("Content-Type", "application/json")+ writer.WriteHeader(http.StatusServiceUnavailable)+ _, _ = writer.Write([]byte(`{"status":"` + unhealthyReason + `"}`))+ }))+ t.Cleanup(stubServer.Close)++ exitCode, output := runSeedForTest(t, buildTestSeedParams(stubServer.URL))++ assert.Equal(t, 1, exitCode)+ assert.Contains(t, output, unhealthyReason)++ receivedPathsMutex.Lock()+ defer receivedPathsMutex.Unlock()+ assert.Equal(t, []string{"/api/v1/system/health"}, receivedPaths)+}++type seededItems struct {+ workspaceIDs []uuid.UUID+ storageIDs []uuid.UUID+ databaseIDs []uuid.UUID+}++// Every test claims the instance with an account of its own first, so the seeded account+// registers as a member, the way it does on a dev instance someone already uses.+func setUpSeedTest(t *testing.T) (*gin.Engine, seedParams) {+ t.Helper()++ router := newSeedTestRouter()+ server := httptest.NewServer(router)+ t.Cleanup(server.Close)++ users_testing.ResetSettingsToDefaults(t.Context())++ instanceOwner := users_testing.CreateTestUser(t.Context(), users_enums.UserRoleMember)+ t.Cleanup(func() { users_testing.DeleteTestUser(context.Background(), instanceOwner.UserID) })++ params := buildTestSeedParams(server.URL)+ t.Cleanup(func() { removeAccountWithItems(t, router, params.Email) })++ return router, params+}++func newSeedTestRouter() *gin.Engine {+ gin.SetMode(gin.TestMode)+ router := gin.New()++ v1 := router.Group("/api/v1")++ userController := users_controllers.GetUserController()+ userController.RegisterRoutes(v1)+ system_healthcheck.GetHealthcheckController().RegisterRoutes(v1)++ protected := v1.Group("")+ protected.Use(users_middleware.AuthMiddleware(users_services.GetUserService()))++ userController.RegisterProtectedRoutes(protected)+ workspaces_controllers.GetWorkspaceController().RegisterRoutes(protected)+ storages.GetStorageController().RegisterRoutes(protected)+ databases.GetDatabaseController().RegisterRoutes(protected)+ backups_config_logical.GetBackupConfigController().RegisterRoutes(protected)+ backups_config_physical.GetBackupConfigController().RegisterRoutes(protected)+ backups_controllers_logical.GetBackupController().RegisterRoutes(protected)+ backups_controllers_physical.GetPhysicalBackupController().RegisterRoutes(protected)++ databases.SetupDependencies()+ backups_services.SetupDependencies()+ audit_logs.SetupDependencies()+ notifiers.SetupDependencies()+ storages.SetupDependencies()+ backups_config_logical.SetupDependencies()+ backups_config_physical.SetupDependencies()+ backuping_physical.SetupDependencies()++ return router+}++func buildTestSeedParams(baseURL string) seedParams {+ env := config.GetEnv()++ logicalPort, err := strconv.Atoi(env.TestLogicalPostgres16Port)+ if err != nil {+ panic(err)+ }++ physicalPort, err := strconv.Atoi(env.TestPhysicalPostgres18Port)+ if err != nil {+ panic(err)+ }++ return seedParams{+ BaseURL: baseURL,+ Email: "seed-" + uuid.NewString() + "@example.com",+ Password: seedPassword,+ WorkspaceName: "Dev",+ StorageName: "Local",+ LogicalDatabaseName: "Logical PG16",+ LogicalSource: postgresSource{+ Host: env.TestLocalhost,+ Port: logicalPort,+ Username: "testuser",+ Password: "testpassword",+ Database: "testdb",+ },+ WalDatabaseName: "WAL PG18",+ WalSource: postgresSource{+ Host: env.TestLocalhost,+ Port: physicalPort,+ Username: "testuser",+ Password: "testpassword",+ },+ }+}++func runSeedForTest(t *testing.T, params seedParams) (int, string) {+ t.Helper()++ var output bytes.Buffer+ exitCode := runSeed(t.Context(), &http.Client{Timeout: requestTimeout}, params,+ slog.New(slog.NewTextHandler(&output, nil)))++ return exitCode, output.String()+}++func signUpForTest(t *testing.T, router *gin.Engine, email, password string) string {+ t.Helper()++ var signUp users_dto.SignInResponseDTO+ test_utils.MakePostRequestAndUnmarshal(t, router, "/api/v1/users/signup", "",+ users_dto.SignUpRequestDTO{Email: email, Password: password, Name: "Seed Test"},+ http.StatusOK, &signUp)++ return signUp.Token+}++func signInForTest(t *testing.T, router *gin.Engine, email, password string) string {+ t.Helper()++ var signIn users_dto.SignInResponseDTO+ test_utils.MakePostRequestAndUnmarshal(t, router, "/api/v1/users/signin", "",+ users_dto.SignInRequestDTO{Email: email, Password: password}, http.StatusOK, &signIn)+ require.NotEmpty(t, signIn.Token)++ return signIn.Token+}++func getProfile(t *testing.T, router *gin.Engine, token string) users_dto.UserProfileResponseDTO {+ t.Helper()++ var profile users_dto.UserProfileResponseDTO+ test_utils.MakeGetRequestAndUnmarshal(t, router, "/api/v1/users/me", "Bearer "+token,+ http.StatusOK, &profile)++ return profile+}++func listWorkspaces(t *testing.T, router *gin.Engine, token string) []workspaces_dto.WorkspaceResponseDTO {+ t.Helper()++ var listedWorkspaces workspaces_dto.ListWorkspacesResponseDTO+ test_utils.MakeGetRequestAndUnmarshal(t, router, "/api/v1/workspaces", "Bearer "+token,+ http.StatusOK, &listedWorkspaces)++ return listedWorkspaces.Workspaces+}++func findWorkspaceID(t *testing.T, router *gin.Engine, token, name string) uuid.UUID {+ t.Helper()++ for _, workspace := range listWorkspaces(t, router, token) {+ if workspace.Name == name {+ return workspace.ID+ }+ }++ return uuid.Nil+}++func findStorageID(t *testing.T, router *gin.Engine, token string, workspaceID uuid.UUID, name string) uuid.UUID {+ t.Helper()++ for _, storage := range listStorages(t, router, token, workspaceID) {+ if storage.Name == name {+ return storage.ID+ }+ }++ return uuid.Nil+}++func findDatabaseID(t *testing.T, router *gin.Engine, token string, workspaceID uuid.UUID, name string) uuid.UUID {+ t.Helper()++ for _, database := range listDatabases(t, router, token, workspaceID) {+ if database.Name == name {+ return database.ID+ }+ }++ return uuid.Nil+}++func listStorages(t *testing.T, router *gin.Engine, token string, workspaceID uuid.UUID) []storages.Storage {+ t.Helper()++ var listedStorages []storages.Storage+ test_utils.MakeGetRequestAndUnmarshal(t, router, "/api/v1/storages?workspace_id="+workspaceID.String(),+ "Bearer "+token, http.StatusOK, &listedStorages)++ return listedStorages+}++func listDatabases(t *testing.T, router *gin.Engine, token string, workspaceID uuid.UUID) []databases.Database {+ t.Helper()++ var listedDatabases []databases.Database+ test_utils.MakeGetRequestAndUnmarshal(t, router, "/api/v1/databases?workspace_id="+workspaceID.String(),+ "Bearer "+token, http.StatusOK, &listedDatabases)++ return listedDatabases+}++func listSeededItems(t *testing.T, router *gin.Engine, token string) seededItems {+ t.Helper()++ var items seededItems++ for _, workspace := range listWorkspaces(t, router, token) {+ items.workspaceIDs = append(items.workspaceIDs, workspace.ID)++ for _, storage := range listStorages(t, router, token, workspace.ID) {+ items.storageIDs = append(items.storageIDs, storage.ID)+ }++ for _, database := range listDatabases(t, router, token, workspace.ID) {+ items.databaseIDs = append(items.databaseIDs, database.ID)+ }+ }++ return items+}++// A logical backup the seed started still writes its file, and deleting its database before it+// finishes would leave the file behind.+func waitForLogicalBackups(t *testing.T, databaseID uuid.UUID) {+ t.Helper()++ backuping_logical.WaitForBackupCompletion(t, databaseID, 0, firstBackupTimeout)+}++func removeAccountWithItems(t *testing.T, router *gin.Engine, email string) {+ t.Helper()++ ctx := context.Background()++ user, err := users_services.GetUserService().GetUserByEmail(ctx, email)+ if err != nil || user == nil {+ return+ }++ access, err := users_services.GetUserService().GenerateAccessToken(ctx, user)+ require.NoError(t, err)++ token := access.Token+ items := listSeededItems(t, router, token)++ for _, workspaceID := range items.workspaceIDs {+ for _, database := range listDatabases(t, router, token, workspaceID) {+ if database.Type == databases.DatabaseTypePostgresLogical {+ waitForLogicalBackups(t, database.ID)+ }++ test_utils.MakeDeleteRequest(t, router, "/api/v1/databases/"+database.ID.String(),+ "Bearer "+token, http.StatusNoContent)+ }++ for _, storage := range listStorages(t, router, token, workspaceID) {+ test_utils.MakeDeleteRequest(t, router, "/api/v1/storages/"+storage.ID.String(),+ "Bearer "+token, http.StatusOK)+ }++ test_utils.MakeDeleteRequest(t, router, "/api/v1/workspaces/"+workspaceID.String(),+ "Bearer "+token, http.StatusOK)+ }++ users_testing.DeleteTestUser(ctx, user.ID)+}diff --git a/backend/cmd/seed_dev/seed.go b/backend/cmd/seed_dev/seed.gonew file mode 100644index 00000000..a1aef007--- /dev/null+++ b/backend/cmd/seed_dev/seed.go@@ -0,0 +1,336 @@+package main++import (+ "context"+ "fmt"+ "log/slog"+ "maps"+ "net/http"+ "slices"++ "github.com/google/uuid"+)++const (+ logicalBackupsRetentionCount = 3+ walChainsRetentionCount = 2+)++type postgresSource struct {+ Host string+ Port int+ Username string+ Password string+ Database string+}++type seedParams struct {+ BaseURL string++ Email string+ Password string++ WorkspaceName string+ StorageName string++ LogicalDatabaseName string+ LogicalSource postgresSource++ WalDatabaseName string+ WalSource postgresSource+}++// Sign-in and sign-up stay one attempt each: sign-in is rate-limited per address, and the seed+// must never take the password reset path that could change an existing account.+func seedDevApp(ctx context.Context, httpClient *http.Client, params seedParams, logger *slog.Logger) error {+ client := &apiClient{httpClient: httpClient, baseURL: params.BaseURL}++ if err := checkBackendHealth(ctx, client, params.BaseURL); err != nil {+ return err+ }++ token, err := signInOrSignUp(ctx, client, params, logger)+ if err != nil {+ return err+ }++ client.token = token++ workspaceID, err := findOrCreateWorkspace(ctx, client, params.WorkspaceName, logger)+ if err != nil {+ return err+ }++ storageID, err := findOrCreateLocalStorage(ctx, client, workspaceID, params.StorageName, logger)+ if err != nil {+ return err+ }++ var existingDatabases []namedItem+ if err := client.get(ctx, "/databases?workspace_id="+workspaceID.String(), &existingDatabases); err != nil {+ return fmt.Errorf("list databases of workspace %q: %w", params.WorkspaceName, err)+ }++ if !hasDatabaseNamed(existingDatabases, params.LogicalDatabaseName) {+ if err := createLogicalDatabase(ctx, client, workspaceID, storageID, params); err != nil {+ return err+ }++ logger.Info("created database with logical backups", "name", params.LogicalDatabaseName)+ }++ if !hasDatabaseNamed(existingDatabases, params.WalDatabaseName) {+ if err := createWalDatabase(ctx, client, workspaceID, storageID, params); err != nil {+ return err+ }++ logger.Info("created database with WAL-streaming backups", "name", params.WalDatabaseName)+ }++ logger.Info(fmt.Sprintf("dev app is seeded: sign in as %s", params.Email))++ return nil+}++func checkBackendHealth(ctx context.Context, client *apiClient, baseURL string) error {+ err := client.get(ctx, "/system/health", nil)+ if err == nil {+ return nil+ }++ if isAPIRefusal(err) {+ return fmt.Errorf("backend at %s reports itself unhealthy: %w", baseURL, err)+ }++ return fmt.Errorf("backend at %s does not answer, start the app with `just run`: %w", baseURL, err)+}++func signInOrSignUp(+ ctx context.Context,+ client *apiClient,+ params seedParams,+ logger *slog.Logger,+) (string, error) {+ var signIn signInResponse++ signInErr := client.post(ctx, "/users/signin",+ signInRequest{Email: params.Email, Password: params.Password}, &signIn)+ if signInErr == nil {+ if signIn.Token == "" {+ return "", fmt.Errorf(+ "cannot sign in as %s: the instance requires two-factor sign-in", params.Email)+ }++ return signIn.Token, nil+ }++ if !isAPIRefusal(signInErr) {+ return "", fmt.Errorf("sign in as %s: %w", params.Email, signInErr)+ }++ var signUp signInResponse++ signUpErr := client.post(ctx, "/users/signup", signUpRequest{+ Email: params.Email,+ Password: params.Password,+ Name: "Dev",+ }, &signUp)+ if signUpErr != nil {+ return "", fmt.Errorf("cannot sign in as %s (%w) and cannot register it (%w)",+ params.Email, signInErr, signUpErr)+ }++ logger.Info("registered the dev account")++ return signUp.Token, nil+}++func findOrCreateWorkspace(+ ctx context.Context,+ client *apiClient,+ workspaceName string,+ logger *slog.Logger,+) (uuid.UUID, error) {+ var listedWorkspaces listWorkspacesResponse+ if err := client.get(ctx, "/workspaces", &listedWorkspaces); err != nil {+ return uuid.Nil, fmt.Errorf("list workspaces: %w", err)+ }++ for _, workspace := range listedWorkspaces.Workspaces {+ if workspace.Name == workspaceName {+ return workspace.ID, nil+ }+ }++ var createdWorkspace namedItem+ if err := client.post(ctx, "/workspaces",+ createWorkspaceRequest{Name: workspaceName}, &createdWorkspace); err != nil {+ return uuid.Nil, fmt.Errorf("cannot create workspace %q: %w", workspaceName, err)+ }++ logger.Info("created workspace", "name", workspaceName)++ return createdWorkspace.ID, nil+}++func findOrCreateLocalStorage(+ ctx context.Context,+ client *apiClient,+ workspaceID uuid.UUID,+ storageName string,+ logger *slog.Logger,+) (uuid.UUID, error) {+ var listedStorages []namedItem+ if err := client.get(ctx, "/storages?workspace_id="+workspaceID.String(), &listedStorages); err != nil {+ return uuid.Nil, fmt.Errorf("list storages: %w", err)+ }++ for _, storage := range listedStorages {+ if storage.Name == storageName {+ return storage.ID, nil+ }+ }++ var createdStorage namedItem+ if err := client.post(ctx, "/storages", createLocalStorageRequest{+ WorkspaceID: workspaceID,+ Type: "LOCAL",+ Name: storageName,+ }, &createdStorage); err != nil {+ return uuid.Nil, fmt.Errorf("cannot create storage %q: %w", storageName, err)+ }++ logger.Info("created storage", "name", storageName)++ return createdStorage.ID, nil+}++func createLogicalDatabase(+ ctx context.Context,+ client *apiClient,+ workspaceID, storageID uuid.UUID,+ params seedParams,+) error {+ source := params.LogicalSource++ databaseID, err := createDatabase(ctx, client, createDatabaseRequest{+ WorkspaceID: workspaceID,+ Name: params.LogicalDatabaseName,+ Type: "POSTGRES_LOGICAL",+ PostgresqlLogical: &postgresqlLogicalSource{+ Version: "16",+ Host: source.Host,+ Port: source.Port,+ Username: source.Username,+ Password: source.Password,+ Database: source.Database,+ CpuCount: 1,+ },+ })+ if err != nil {+ return err+ }++ err = switchOnBackups(ctx, client, params.LogicalDatabaseName,+ "/backup-configs/database/"+databaseID.String(), "/backup-configs/save",+ map[string]any{+ "storage": map[string]any{"id": storageID},+ "storageId": storageID,+ "retentionPolicyType": "COUNT",+ "retentionCount": logicalBackupsRetentionCount,+ })+ if err != nil {+ return err+ }++ if err := client.post(ctx, "/backups", makeLogicalBackupRequest{DatabaseID: databaseID}, nil); err != nil {+ return fmt.Errorf("cannot start the first backup of database %q: %w", params.LogicalDatabaseName, err)+ }++ return nil+}++func createWalDatabase(+ ctx context.Context,+ client *apiClient,+ workspaceID, storageID uuid.UUID,+ params seedParams,+) error {+ source := params.WalSource++ databaseID, err := createDatabase(ctx, client, createDatabaseRequest{+ WorkspaceID: workspaceID,+ Name: params.WalDatabaseName,+ Type: "POSTGRES_PHYSICAL",+ PostgresqlPhysical: &postgresqlPhysicalSource{+ Version: "18",+ BackupType: "FULL_INCREMENTAL_WAL_STREAM",+ Host: source.Host,+ Port: source.Port,+ Username: source.Username,+ Password: source.Password,+ },+ })+ if err != nil {+ return err+ }++ err = switchOnBackups(ctx, client, params.WalDatabaseName,+ "/backup-configs/physical/database/"+databaseID.String(), "/backup-configs/physical/save",+ map[string]any{+ "storage": map[string]any{"id": storageID},+ "storageId": storageID,+ "retention": "CHAINS",+ "chainsRetention": map[string]any{"count": walChainsRetentionCount},+ "fullBackupsRetention": map[string]any{},+ })+ if err != nil {+ return err+ }++ if err := client.post(ctx, "/backups/physical/database/"+databaseID.String()+"/trigger",+ triggerPhysicalBackupRequest{Type: "auto"}, nil); err != nil {+ return fmt.Errorf("cannot start the first backup of database %q: %w", params.WalDatabaseName, err)+ }++ return nil+}++func createDatabase(ctx context.Context, client *apiClient, database createDatabaseRequest) (uuid.UUID, error) {+ var createdDatabase namedItem+ if err := client.post(ctx, "/databases/create", database, &createdDatabase); err != nil {+ return uuid.Nil, fmt.Errorf("cannot create database %q: %w", database.Name, err)+ }++ return createdDatabase.ID, nil+}++// The backend fills a new database's backup settings with its defaults on the first read. The seed+// changes only the fields it owns and sends the rest back as read, so it never restates defaults+// such as the schedule or the WAL lag threshold.+func switchOnBackups(+ ctx context.Context,+ client *apiClient,+ databaseName, readPath, savePath string,+ seededFields map[string]any,+) error {+ var backupConfig map[string]any+ if err := client.get(ctx, readPath, &backupConfig); err != nil {+ return fmt.Errorf("read backup settings of database %q: %w", databaseName, err)+ }++ backupConfig["isBackupsEnabled"] = true+ maps.Copy(backupConfig, seededFields)++ if err := client.post(ctx, savePath, backupConfig, nil); err != nil {+ return fmt.Errorf("cannot switch on backups of database %q: %w", databaseName, err)+ }++ return nil+}++func hasDatabaseNamed(existingDatabases []namedItem, name string) bool {+ return slices.ContainsFunc(existingDatabases, func(database namedItem) bool {+ return database.Name == name+ })+}diff --git a/backend/cmd/seed_dev/wire.go b/backend/cmd/seed_dev/wire.gonew file mode 100644index 00000000..d5ceeacd--- /dev/null+++ b/backend/cmd/seed_dev/wire.go@@ -0,0 +1,81 @@+package main++import "github.com/google/uuid"++// The seed declares only the fields it sends and reads instead of importing the backend's request+// types: importing a feature package runs its dependency wiring, which loads the full backend+// configuration and refuses to start without the database client libraries. The seed test drives+// the real controllers, so a renamed field fails there.++type signInRequest struct {+ Email string `json:"email"`+ Password string `json:"password"`+}++type signUpRequest struct {+ Email string `json:"email"`+ Password string `json:"password"`+ Name string `json:"name"`+}++// A sign-in answers with a token, or with a pending sign-in when the instance requires a second+// factor; a sign-up always answers with a token.+type signInResponse struct {+ Token string `json:"token"`+ PendingSignInID string `json:"pendingSignInId"`+}++type namedItem struct {+ ID uuid.UUID `json:"id"`+ Name string `json:"name"`+}++type listWorkspacesResponse struct {+ Workspaces []namedItem `json:"workspaces"`+}++type createWorkspaceRequest struct {+ Name string `json:"name"`+}++type createLocalStorageRequest struct {+ WorkspaceID uuid.UUID `json:"workspaceId"`+ Type string `json:"type"`+ Name string `json:"name"`+ LocalStorage struct{} `json:"localStorage"`+}++type createDatabaseRequest struct {+ WorkspaceID uuid.UUID `json:"workspaceId"`+ Name string `json:"name"`+ Type string `json:"type"`+ PostgresqlLogical *postgresqlLogicalSource `json:"postgresqlLogical,omitempty"`+ PostgresqlPhysical *postgresqlPhysicalSource `json:"postgresqlPhysical,omitempty"`+}++type postgresqlLogicalSource struct {+ Version string `json:"version"`+ Host string `json:"host"`+ Port int `json:"port"`+ Username string `json:"username"`+ Password string `json:"password"`+ Database string `json:"database"`+ CpuCount int `json:"cpuCount"`+}++type postgresqlPhysicalSource struct {+ Version string `json:"version"`+ BackupType string `json:"backupType"`+ Host string `json:"host"`+ Port int `json:"port"`+ Username string `json:"username"`+ Password string `json:"password"`+}++type makeLogicalBackupRequest struct {+ DatabaseID uuid.UUID `json:"database_id"`+}++type triggerPhysicalBackupRequest struct {+ Type string `json:"type"`+}diff --git a/mise.toml b/mise.tomlindex 38d0109d..7a25938e 100644--- a/mise.toml+++ b/mise.toml@@ -8,5 +8,7 @@ process-compose = "1.122.0""aqua:pressly/goose" = "3.27.3"swag = "1.16.4"golangci-lint = "2.11.3"+"aqua:air-verse/air" = "1.67.4"+"npm:chrome-devtools-mcp" = "1.11.0"python = "3.14.7""npm:@fission-ai/openspec" = "1.11.0"diff --git a/openspec/changes/archive/2026-10-21-agent-dev-environment/.openspec.yaml b/openspec/changes/archive/2026-10-21-agent-dev-environment/.openspec.yamlnew file mode 100644index 00000000..dc36fb1b--- /dev/null+++ b/openspec/changes/archive/2026-10-21-agent-dev-environment/.openspec.yaml@@ -0,0 +1,2 @@+schema: spec-driven+created: 2026-10-21diff --git a/openspec/changes/archive/2026-10-21-agent-dev-environment/design.md b/openspec/changes/archive/2026-10-21-agent-dev-environment/design.mdnew file mode 100644index 00000000..5d142b58--- /dev/null+++ b/openspec/changes/archive/2026-10-21-agent-dev-environment/design.md@@ -0,0 +1,187 @@+## Context++See `proposal.md` for the motivation and `specs/` for the behavior. This section covers only the current state that shapes the approach.++- `just run` calls `scripts/dev.sh`. That script starts the dev containers and runs process-compose with `--no-server` (`scripts/dev.sh:49`). Nothing outside the TUI can restart a process.+- The backend process is a plain `go run ./cmd` (`process-compose.yaml:9-11`), so it never sees a saved edit.+- The frontend is the Vite dev server (`process-compose.yaml:13-14`), which already updates the page on its own.+- The backend applies migrations on every start (`backend/cmd/main.go:92`). It listens on the fixed `:4005` (`backend/cmd/main.go:71`), and the dev frontend calls that port (`frontend/src/constants.ts:22`).+- `free_backend_port` kills whatever listens on 4005 before a run (`scripts/dev.sh:20-33`). The EXIT trap stops the dev database, the WAL backup source and its load generator (`scripts/dev.sh:13-15`).+- The public API covers everything a seed needs, and in development nothing blocks sign-up:+ - Sign-up, sign-in and the user check are at `backend/internal/features/users/controllers/user_controller.go:26-31`.+ - The first account becomes the root admin (`backend/internal/features/users/services/user_services.go:728-786`).+ - Turnstile only runs when its secret is set.+ - Workspaces are created at `workspace_controller.go:22-30`, storages at `storages/controller.go:20-21`, and databases are created and listed at `databases/controller.go:23-27`.+- Creating a database tests the connection on the server itself. For physical databases that includes the replication readiness checks. The live WAL backup source in `docker-compose.yml` passes them.+- The web interface saves the database first, then its backup config, then triggers the first backup when backups are on (`frontend/src/features/databases/ui/CreateDatabaseComponent.tsx:87-120`).+- The physical backup config's defaults keep 7 chains (`backend/internal/features/backups/config/physical/constants.go:4-5`). They already pass the WAL-stream validation (`backend/internal/features/backups/config/physical/model.go:149-196`).+- Dev-only Go commands already live next to the server in `backend/cmd/`: `cleanup_test_db` runs from `scripts/test.sh:49`. The production image builds only `./cmd` (`Dockerfile:62`).+- `chrome-devtools-mcp` is registered only in one developer's personal Claude Code config, as `npx chrome-devtools-mcp@latest`. Codex has no browser server.+- The test fixtures on 5007 and 5008 are reset by `cleanup_test_db` (`backend/cmd/cleanup_test_db/main.go:104-106`). The physical backend tests also use them for replication slots.++Rules that constrain the approach:++- From the root `AGENTS.md`:+ - "Running commands": a long-lived process the agent started is stopped before the task ends.+ - "Documentation consistency" and the `agent-instructions` spec: a skill holds rules of agent behavior, and lists of what exists live next to the code.+- From `backend/AGENTS.md`:+ - "Prefer controller tests over unit tests", and the test cleanup rules.+- From the testing skill: test what a caller observes, and mock only external systems.++## Goals / Non-Goals++**Goals:**++- A just-started local app is ready to sign in to and holds databases with live backups, with no clicks needed.+- The backend picks up a saved Go edit without anyone touching the process window.+- Claude Code and Codex start the same pinned browser server from repository config.+- Every new piece is checked by an automated test or a local command the agent runs.++**Non-Goals:**++- Making the dev environment safe to expose on a network.+- Seeding a state that matches production data, or many databases for load.+- Recovering a seeded database whose replication slot was lost when its container was recreated. See Risks.++## Decisions++### 1. The seed is a dev-only Go command that calls the public HTTP API++`backend/cmd/seed_dev/main.go` is a separate main package, like `cleanup_test_db`. It reads the repository `.env` with `godotenv`, signs in or signs up, and then creates what is missing through `/api/v1`. It declares the few request and response fields it uses instead of importing the backend's types. Importing any feature package runs its dependency wiring, which loads the full backend configuration and exits when the bundled database clients cannot start. That happens on Fedora without the `libedit` shim `just run` applies, so a seed run on its own would fail before sending a request. The backup settings are read and sent back as a JSON object with only the seeded fields changed, so the server defaults survive without being restated. A renamed field still fails the seed test, which drives the real controllers. Its test sits in the same package. The test serves the real controllers through `httptest.NewServer`, against the test database and fixtures. Its router is built the way `backend/cmd/main.go:366-376` and `createE2ETestRouter` (`backend/internal/features/users/controllers/e2e_test.go:250-262`) build theirs. The user controller's public routes (sign-up, sign-in) and the healthcheck controller sit on the bare `/api/v1` group. Everything else sits on a group behind `AuthMiddleware`: the workspace, storage and database controllers, both backup config controllers, and the logical and physical backup controllers that trigger the first backup. The `workspaces_testing.CreateTestRouter` helper does not fit, because it puts every controller behind auth and registers no healthcheck. The seed takes its account, names and database endpoints as parameters. `main` fills them from `.env`, falling back to the defaults of `docker-compose.yml`, and the test passes a unique email and the test fixture ports.++Rejected alternatives:++- **A Python or Bash script.** Its test would have to fake our own backend, which the testing skill forbids. Bash also needs `jq`, and mise does not pin it.+- **Extending `--create-user` in the server binary** (`backend/cmd/main.go:175-180`). That puts fixture data and known credentials into the production binary, against the requirement "Seed credentials exist only in local development".+- **Writing to the metadata database directly.** The direct path skips validation, version detection and the server-side connection check. It also skips the replication slot name the physical model assigns on create, and it bypasses the running backend's in-memory cache.++### 2. Items are found by name, created when missing and never changed++The seed uses these names:++- the account `[email protected]`;+- the workspace `Dev`;+- the storage `Local`;+- the databases `Logical PG16` and `WAL PG18`.++The seed lists what exists in the workspace and creates only the missing items. An item that exists is left exactly as it is, including its backup config. This keeps whatever a developer changed while working on a feature.++Rejected alternatives:++- **Reconciling every item to the seeded state on each run.** Every `just run` would undo the developer's edits, and on a physical database it could change the backup type mid-chain.+- **A marker row recording that seeding happened.** It needs a migration for dev tooling. It would also stop the seed from restoring a deleted database, which the "One seeded database was deleted" scenario requires.++### 3. Sign in first, sign up only when sign-in fails++The seed signs in with `[email protected]` and `Q1234567`. Sign-in has three outcomes. A token means success. A pending two-factor sign-in, returned when the instance requires two-factor authentication (`backend/internal/features/users/services/user_services.go:198-204`), makes the seed exit non-zero with a message that names the account and says two-factor sign-in is required. A failed sign-in leads to sign-up. When sign-up is refused for any reason, the seed exits non-zero with the account and the backend's reason. Possible reasons are that the account already exists or that the registration policy closes sign-up for a non-first account (`user_services.go:728-786`). It never calls the password reset path (`--new-password`), so it cannot change an existing account, as the security requirement demands. Sign-in is rate-limited to 10 attempts per minute per email (`user_controller.go:107-113`). One attempt per run stays far below that.++On an empty database the dev account becomes the root admin. On a database where someone signed up first, it becomes a member. Creating a workspace then depends on the workspace-creation setting. If the backend refuses this or any other create call, the seed exits non-zero with the name of the item and the backend's reason, and it never retries by changing an existing item. The developer then either signs in as that admin or allows members to create workspaces.++### 4. Seeded databases and their backup settings++| Name | Type | Source | Backups |+|---|---|---|---|+| `Logical PG16` | `POSTGRES_LOGICAL` | shared logical fixture, `TEST_LOGICAL_POSTGRES_16_PORT`, `testdb` | daily, `COUNT` retention of 3 |+| `WAL PG18` | `POSTGRES_PHYSICAL`, `FULL_INCREMENTAL_WAL_STREAM` | WAL backup source, `WAL_BACKUP_POSTGRES_18_PORT` and `WAL_BACKUP_DB_*` | server defaults, retention `CHAINS` with 2 chains |++For each database the seed fetches the default backup config, switches backups on, sets the `Local` storage as `storage.id` and the retention above, and saves it. It then triggers the first backup the same way the web interface does after its wizard. Everything else comes from the server's defaults, so the seed does not restate the WAL lag threshold or the interval rules.++The logical source stays on 5004 even though `just test` resets that container. The reset recreates `testdb` with the same user (`backend/cmd/cleanup_test_db/main.go:104`), so later logical backups keep working, and `pg_dump` only reads.++Rejected alternatives:++- **The physical fixtures on 5007 or 5008.** The supervisor of a dev WAL-streaming database keeps a replication slot on its source, and the physical backend tests own those containers.+- **A separate dev container for logical backups.** The shared fixture already serves this purpose, and one more always-on container costs memory for nothing.+- **Leaving backups off.** The user chose backups on, so the Monitoring tab and the backup list show live data right after start.++### 5. The seed runs as a process-compose process gated on the backend port++`process-compose.yaml` gains:++- a readiness probe on the backend that checks that port 4005 accepts connections. process-compose keeps probing for the whole run, and every request the backend serves writes an access log line (`backend/internal/middleware/access_log.go`), so an HTTP probe would fill the backend output with a health line every few seconds. The seed asks the public health endpoint (`backend/internal/features/system/healthcheck/controller.go:14`) itself, once;+- a `seed` process that runs `go run ./cmd/seed_dev` in `backend/`, with `depends_on: backend: condition: process_healthy` and no restart.++A failed seed ends only its own process, as the spec requires. To seed an app that is already running, the same command runs on its own as `go run ./cmd/seed_dev` in `backend/`. It always targets `http://localhost:4005`, because the backend port is fixed. When the health endpoint does not answer, the seed exits at once with the message the spec requires, and it never waits for an app that is not there. When the endpoint answers with an error status, the seed prints the reason from the response body, for example the disk usage check.++Rejected alternatives:++- **A `just seed` recipe with a backend address parameter.** The backend port is fixed and the agent never has a backend of its own on another port, so the parameter has no user today. `just run` seeds on its own, and the rare rerun goes through the command directly. The seed already takes the base URL as a parameter for its test, so a flag can be added if ports per worktree ever appear.+- **Seeding from `scripts/dev.sh`** before process-compose starts. The backend is not up yet, so the script needs its own wait loop, and its output would land outside the process window.+- **A seeding hook inside the backend's startup.** That is production code.++### 6. The backend runs under air, pinned in mise++`mise.toml` adds `"aqua:air-verse/air"` at the latest release, 1.67.4 at the time of writing. `process-compose.yaml` runs `air` in `backend/` instead of `go run ./cmd`. `backend/.air.toml` sets the following:++- **Build.** `go build -o ./tmp/databasus ./cmd`. `tmp/` is added to `backend/.gitignore`.+- **What triggers a rebuild.** `.go` files only. Files ending in `_test.go` are excluded, and so are `tmp`, `swagger`, `ui`, `migrations` and the dev-only commands `cmd/seed_dev` and `cmd/cleanup_test_db`.+- **Debounce.** About one second, so an editor's multi-file save causes one rebuild.+- **On a failed build.** `stop_on_error = false`, so the old binary keeps serving.+- **On restart.** The old process gets an interrupt and a short grace period, so the backend shuts down the way it does on Ctrl+C.++Each restart reruns the migrations, just as `just run` does today.++air does not exit when its child dies: it keeps watching and rebuilds on the next save. The current `free_backend_port` (`scripts/dev.sh:17-33`) relies on the opposite, that killing the listener also ends its parent `go run`. So an air left behind by a killed run would survive the port cleanup and later fight the new run for :4005. `scripts/dev.sh` therefore gains a step before process-compose starts. It stops every `air` process whose working directory is this worktree's `backend/` and only then frees the port. The comment above `free_backend_port` changes to match. Watchers of other worktrees are left alone.++Rejected alternatives:++- **watchexec around `go run`.** It kills the old process before building, so a compile error leaves no backend on 4005.+- **Removing `--no-server`** and restarting the backend through the process-compose CLI. This keeps the restart manual, while the user chose automatic restarts.+- **gow.** Not in the mise registry, so it cannot be pinned the same way.++### 7. The browser server is chrome-devtools-mcp, pinned in mise and started through one launcher++`mise.toml` adds `"npm:chrome-devtools-mcp" = "1.11.0"`. The launcher `scripts/browser-mcp.sh` works as follows:++1. It finds the repository root from its own location and changes into it. mise resolves the pinned version from the `mise.toml` of the current directory, and the client may start the launcher from anywhere.+2. It reads `BROWSER_HEADLESS` from the environment, then from the root `.env`, and defaults to `true`. It parses `.env` without sourcing it, because `.env` holds quoted DSNs.+3. A value other than `true` or `false` makes it exit with an error on stderr. The client then reports that the server failed to start.+4. It runs `exec mise exec -- chrome-devtools-mcp --isolated --viewport 1440x900`, adding `--headless` when the value is `true`. `mise exec` resolves the pinned version whatever `PATH` the agent client was started with.++`.mcp.json` and `.codex/config.toml` both register the server as `chrome-devtools`. Each calls the launcher through `bash -c` and `git rev-parse --show-toplevel`, the same pattern the hooks in `.claude/settings.json` and `.codex/hooks.json` use. `.env.example` gets `BROWSER_HEADLESS=true` under a `# agent browser` heading, and the local `.env` gets the same line, as the user asked.++`--isolated` gives every session a temporary profile, so two agents never fight over `~/.cache/chrome-devtools-mcp/chrome-profile`. The cost is that each session signs in again, which takes one form with the seeded account. The name `chrome-devtools` keeps the tool names agents already know.++Rejected alternatives:++- **Playwright MCP.** It downloads its own Chromium, and Playwright does not officially support Fedora. Its debugging tools for console, network and performance are thinner. Registering both servers would give the agent two near-identical toolsets.+- **`npx chrome-devtools-mcp@latest`.** Unpinned, and it fetches from npm whenever a session starts.+- **Expanding `${BROWSER_HEADLESS}` inside `.mcp.json`.** Claude Code supports that, but Codex does not expand variables in arguments. The launcher gives both clients one behavior.+- **A persistent shared profile.** The second concurrent session fails because the profile is locked.++### 8. The skill holds rules, and the facts live next to the seed++`.agents/skills/run-app/SKILL.md` describes when and how the agent checks the UI, and nothing else:++- reuse the app only when both ports answer and the process listening on the backend port has its working directory inside the agent's own worktree;+- when the app serves another worktree, leave it alone, skip the browser check and tell the user it was skipped and why;+- when only one of the two ports answers, start nothing, skip the browser check and tell the user which port is held;+- when neither port answers, start `just run -t=false` in the background;+- sign in with the seeded account;+- stop the run with an interrupt and confirm that both ports are free;+- a browser check never replaces the tests a task names.++The account, the names and the ports are a list of what exists, so the `agent-instructions` spec keeps them out of the skill. The account and the database names live in the seed command's package comment, and the ports in the `links` process of `process-compose.yaml`; the skill links to both as its sources of values. `.claude/skills/run-app/SKILL.md` is a symlink, like the other repository skills. The skill also tells the agent that it may rerun the seed command when the app of its own worktree lacks the dev account or a seeded database. The root `AGENTS.md` updates its `just run` row to mention the seed and the restart on change. It also gains a two-line "Checking the app in a browser" section that points to the skill, the same way "Tests" points to the testing skill.++The skill is named `run-app` so that the built-in `run` skill, which first looks for a project skill about launching the app, finds it.++Rejected alternative:++- **Putting the account and database list straight into `AGENTS.md` or the skill.** The `agent-instructions` spec forbids lists of what exists there, and the diff reviewer would reject them.++## Risks / Trade-offs++- **[Risk]** Each backend restart interrupts a full backup or a WAL receive in progress. → The user accepted this. The receiver already resumes after a restart, and the next scheduled run repeats an interrupted full backup.+- **[Risk]** An agent's background `just run` killed with SIGKILL skips the EXIT trap. That can leave a Vite server on 5173, and the next Vite then silently moves to 5174. → The skill tells the agent to stop the run with an interrupt and to confirm that 4005 and 5173 are free afterwards. If one port is still held, the next check finds only one port answering, and the skill makes the agent report it instead of starting a second run. A stale backend and a stale air of the same worktree are stopped by `scripts/dev.sh` (decision 6).+- **[Risk]** Recreating the WAL backup source container drops the slot of `WAL PG18`, and its backups start failing. In the other direction, recreating `backend-db` orphans the slot on the source, which then keeps WAL without limit. → The seed does not repair either case, because existing items stay untouched. The developer deletes `WAL PG18` and reruns the seed command. An orphaned slot is dropped on the source with `pg_drop_replication_slot`. The seed command's package comment records both fixes.+- **[Risk]** Codex reads a project `.codex/config.toml` only for a trusted project. → Codex already trusts this repository for `.codex/hooks.json`. The task verifies the registration with `codex mcp list` from the repository root.+- **[Risk]** The health endpoint also fails when disk usage reaches 95% (`backend/internal/features/system/healthcheck/controller.go:36-50`). On a nearly full dev disk the seed would create nothing. → The seed prints the health endpoint's reason in its process window, under `just run` and on its own alike. The seed command's package comment names this case.+- **[Trade-off]** A seed test against the real API needs the shared test fixtures and the test database. It is slower than a faked test, but it catches a changed API.+- **[Trade-off]** The account `[email protected]` / `Q1234567` exists on any dev instance that ran `just run`, and the dev backend listens on all interfaces. The dev database already exposes its port with the same password, so the exposure does not grow. Binding to localhost stays out of scope.++## Migration Plan++1. After pulling the change, run `mise install` to get `air` and `chrome-devtools-mcp`.+2. Remove the personal `chrome-devtools` entry from `~/.claude.json` with `claude mcp remove chrome-devtools -s local`. A local-scope server overrides a project-scope server of the same name, so the old `npx …@latest` entry would otherwise keep running.+3. Approve the project `.mcp.json` server when Claude Code asks for it for the first time.+4. Rollback is a revert. The seeded items stay in the dev database and can be deleted in the UI.diff --git a/openspec/changes/archive/2026-10-21-agent-dev-environment/proposal.md b/openspec/changes/archive/2026-10-21-agent-dev-environment/proposal.mdnew file mode 100644index 00000000..3a2e2574--- /dev/null+++ b/openspec/changes/archive/2026-10-21-agent-dev-environment/proposal.md@@ -0,0 +1,71 @@+## Why++An agent working on the frontend cannot see what it built. No browser tool is registered in the repository: the only one is a `chrome-devtools` entry in one developer's personal Claude Code config, which pulls the latest package from npm on every start, and Codex has none at all. A fresh `just run` also opens an empty app: before anyone can look at a screen, someone has to sign up, create a workspace and a storage, and add a database through the wizard. The backend also ignores edits until someone restarts it by hand in the process window. The frontend already picks up edits on its own.++The result is that every UI check needs the developer's hands. This change makes the local app ready to look at as soon as it starts and keeps it running the latest code. It also gives Claude Code and Codex a browser to see and click through it, either visibly or headless.++## Governing docs++This change answers to the root [`AGENTS.md`](../../../AGENTS.md) and to [`backend/AGENTS.md`](../../../backend/AGENTS.md), because the seed command and the restart-on-change configuration live under `backend/`. It touches nothing under `frontend/`, `agent/` or `website/`.++## What Changes++- `just run` runs the backend under a file watcher. A saved Go source file rebuilds the backend and restarts it. When the new code fails to compile, the previous backend keeps serving and the compile error shows in the backend's output.+- `just run` seeds the local app once the backend answers. The seed creates whatever is missing among the following:+ - a dev account, `[email protected]` with password `Q1234567`;+ - a workspace with a local storage;+ - a PostgreSQL database with logical backups of the shared logical test fixture;+ - a PostgreSQL database with WAL-streaming backups of the live WAL backup source.+- Both databases have backups switched on, keep only a few backups, and get a first backup right away. The seed never modifies anything that already exists, so it runs on every start without touching data a developer changed.+- The seed command can also be run on its own against the app already running on the backend port, for example after a seeded database was deleted. There is no separate `just` recipe for it: the backend port is fixed, and `just run` covers the usual case.+- The seed is a dev-only command in the backend module, like the existing `cleanup_test_db`. The production image never builds it.+- `mise.toml` pins the backend file watcher and the Chrome DevTools MCP server.+- The repository registers that browser server for Claude Code (`.mcp.json`) and Codex (`.codex/config.toml`) through one launcher script. Each agent session gets its own temporary browser profile, so parallel sessions do not block each other.+- `BROWSER_HEADLESS` in `.env` and `.env.example` chooses headless or visible Chrome, with headless as the default. A value set in the environment wins over the file.+- A new repository skill tells the agent how to check the UI:+ - reuse the running app only when it serves the agent's own worktree;+ - skip the check and tell the user when the app serves another worktree or only one of its two ports answers;+ - start `just run` without its window when nothing runs;+ - rerun the seed command when the app of its own worktree lacks the dev account or a seeded database;+ - sign in with the seeded account;+ - stop what it started.+- The skill also says that a browser check adds to tests and never replaces them. The skill is exposed to Claude Code through a symlink in `.claude/skills`, like the others.+- The root `AGENTS.md` says in its `just run` row that the run seeds the app and restarts the backend on change, and points to the new skill.++### Out of scope++- Separate ports per worktree, and a backend address or port parameter for the seed. Only one worktree runs `just run` at a time, so the fixed backend and frontend ports stay.+- Playwright and its MCP server. One browser toolset is enough, and two would give the agent near-duplicate tools.+- Regenerating the API documentation when controller annotations change. It still happens once at the start of `just run`.+- Restarting the backend when a migration file alone changes. The next Go edit or the next `just run` applies it.+- Seeding MySQL, MariaDB, MongoDB, notifiers or the physical test fixtures on ports 5007 and 5008. Those fixtures belong to the physical backend tests, and a dev replication slot on them would interfere with those tests.+- Binding the dev servers to localhost only. The dev database already listens on all interfaces with a known password, and that does not change here.+- Changing the rule in `openspec/config.yaml` that keeps browser clicking out of task verification.+- Removing the developer's personal `chrome-devtools` entry from `~/.claude.json`. It lives outside the repository. The design explains why it has to go.++## Capabilities++### New Capabilities++None.++### Modified Capabilities++- `local-development-workflow`:+ - `just run` restarts the backend on source changes, updates the frontend without a restart and seeds the dev account and databases;+ - the seed command can be rerun against the running app;+ - the seed is dev-only;+ - the pinned tool set gains the file watcher and the browser server.+- `agent-workflow`: the agent can see and drive the local app in a browser, both in Claude Code and in Codex, and follows the rules for reusing, starting and stopping the app.++## Impact++- `backend/cmd/seed_dev/`: new dev-only command and its test, which drives the real API on the test database.+- `backend/.air.toml` and the backend's `.gitignore`: watcher configuration and its build directory.+- `process-compose.yaml`, `scripts/dev.sh`: the watcher, stopping a watcher left over from a killed run, the readiness probe and the seed process.+- `mise.toml`: two new pinned tools.+- `scripts/browser-mcp.sh`, `.mcp.json`, `.codex/config.toml`: browser server launcher and registration.+- `.env.example` and the local `.env`: `BROWSER_HEADLESS`.+- `.agents/skills/run-app/SKILL.md` and its `.claude/skills/run-app/SKILL.md` symlink.+- `AGENTS.md`: command table row and a pointer to the skill.+- No production code, API, schema or migration changes.diff --git a/openspec/changes/archive/2026-10-21-agent-dev-environment/specs/agent-workflow/spec.md b/openspec/changes/archive/2026-10-21-agent-dev-environment/specs/agent-workflow/spec.mdnew file mode 100644index 00000000..fe4c8285--- /dev/null+++ b/openspec/changes/archive/2026-10-21-agent-dev-environment/specs/agent-workflow/spec.md@@ -0,0 +1,79 @@+## ADDED Requirements++### Requirement: The agent can see and drive the local app++The repository SHALL register one browser automation server for both Claude Code and Codex. With it, an agent opens pages of the local app and reads what the page shows, including a screenshot, the page structure, console messages and network requests. It also clicks, types and navigates. Both clients SHALL start the server from the same repository configuration. The server SHALL drive the Chrome browser installed on the host. Each agent session SHALL get its own temporary browser profile, which is discarded when the session's browser closes, so that two sessions running at once do not block each other and no sign-in carries over between sessions.++The repository SHALL hold a skill that tells the agent how to check the app in the browser:++- When the backend and frontend ports already answer from a `just run` started in the agent's own worktree, the agent uses that app.+- When they answer from another worktree, the agent leaves that app running, does not run the browser check, and tells the user the check was skipped because the running app serves another worktree's code.+- When only one of the two ports answers, the agent does not start `just run` and does not run the browser check. It tells the user which port is held and that the check was skipped.+- When neither port answers, the agent starts `just run` without its interactive window and stops it before it finishes the task, leaving neither port held.+- The agent signs in with the seeded dev account.+- When the app of the agent's own worktree lacks the dev account or a seeded database, the agent reruns the seed command against it instead of creating the items by hand.++A browser check SHALL add to the tests a task names and SHALL NOT replace them.++#### Scenario: Agent checks a page with no app running++- **WHEN** an agent needs to see a page and nothing answers on the backend and frontend ports+- **THEN** the agent starts `just run` without its interactive window, signs in as the dev account, opens the page and reads what it shows+- **AND** no process of that run remains when the agent hands the task over++#### Scenario: The developer's app runs in the same worktree++- **WHEN** an agent needs to see a page and the running app was started in the agent's own worktree+- **THEN** the agent uses that app and leaves it running when it hands the task over++#### Scenario: The running app belongs to another worktree++- **WHEN** an agent needs to see a page and the running app was started in another worktree+- **THEN** the agent does not stop that app and does not open the page+- **AND** the agent tells the user that the browser check was skipped because the app serves another worktree's code++#### Scenario: Only one port answers++- **WHEN** an agent needs to see a page, the frontend port answers and the backend port does not+- **THEN** the agent does not start `just run` and does not open the page+- **AND** the agent tells the user that the frontend port is held and the browser check was skipped++#### Scenario: A seeded database is missing++- **WHEN** an agent uses the app of its own worktree and the WAL-streaming database is missing from it+- **THEN** the agent reruns the seed command, and the database appears with backups switched on++#### Scenario: Two agents use the browser at once++- **WHEN** two agent sessions open the browser at the same time+- **THEN** both browsers start, each with its own empty profile++#### Scenario: Codex opens a page++- **WHEN** Codex is started in the repository+- **THEN** it lists the same browser server Claude Code uses, at the same pinned version++### Requirement: The developer chooses a visible or headless browser++The browser the agent drives SHALL run headless unless the developer asks otherwise. The `BROWSER_HEADLESS` setting in the developer's `.env` file SHALL choose headless when `true` and a visible window when `false`. A value set in the environment the agent client starts from SHALL take precedence over the file. When neither sets it, the browser SHALL run headless. Any value other than `true` or `false` SHALL keep the browser server from starting, with an error that names `BROWSER_HEADLESS`. The example environment file SHALL contain the setting with its default value. The choice SHALL take effect the next time the agent client starts the browser server.++#### Scenario: Default setting++- **WHEN** neither the developer's `.env` file nor the environment sets `BROWSER_HEADLESS` and an agent opens a page+- **THEN** no browser window appears on the developer's screen++#### Scenario: Developer wants to watch++- **WHEN** the developer sets `BROWSER_HEADLESS=false` in `.env` and the agent client restarts the browser server+- **THEN** the agent's browser opens as a visible window++#### Scenario: Environment overrides the file++- **WHEN** `.env` sets `BROWSER_HEADLESS=true` and the agent client is started with `BROWSER_HEADLESS=false` in its environment+- **THEN** the browser opens as a visible window++#### Scenario: Invalid value++- **WHEN** `.env` sets `BROWSER_HEADLESS=maybe` and the agent client starts the browser server+- **THEN** the server does not start+- **AND** the error the client reports names `BROWSER_HEADLESS`diff --git a/openspec/changes/archive/2026-10-21-agent-dev-environment/specs/local-development-workflow/spec.md b/openspec/changes/archive/2026-10-21-agent-dev-environment/specs/local-development-workflow/spec.mdnew file mode 100644index 00000000..0a7506ca--- /dev/null+++ b/openspec/changes/archive/2026-10-21-agent-dev-environment/specs/local-development-workflow/spec.md@@ -0,0 +1,155 @@+## MODIFIED Requirements++### Requirement: One command starts the local app++The repository SHALL provide `just run` at its root. It SHALL start the development database, the shared backend test fixtures, the live WAL backup source with its load generator, the backend and the frontend development server, and SHALL show the database, the load generator, the backend, the frontend and the seed in one terminal window that lists every process separately, with each process's output available on its own. It SHALL work in a fresh worktree that has a `.env` file but has never generated the backend's API documentation or installed frontend dependencies. When a backend from an earlier run still holds the backend port, the command SHALL stop that process before starting the new backend, so that requests reach the code just started. It SHALL also stop a backend file watcher left over from an earlier run in the same worktree, so that no old watcher rebuilds and restarts a backend after the new run starts. A saved change to a frontend source file SHALL reach the open page without restarting any process.++#### Scenario: Developer starts the app++- **WHEN** a developer runs `just run` on a Fedora host with a `.env` file+- **THEN** the development database, the shared test fixtures, the WAL backup source, its load generator, the backend and the frontend start+- **AND** the window lists the database, the load generator, the backend, the frontend and the seed as separate processes with their own output++#### Scenario: Fresh worktree++- **WHEN** a developer runs `just run` in a worktree that has a `.env` file and nothing else prepared+- **THEN** the backend compiles and starts without a separate generation command+- **AND** the frontend development server starts without a separate dependency installation command++#### Scenario: Stale backend holds the port++- **WHEN** a backend from an earlier run still listens on the backend port+- **THEN** `just run` stops that process and the new backend serves requests on that port++#### Scenario: Watcher from a killed run survives++- **WHEN** an earlier `just run` in the same worktree was killed without cleanup and its backend file watcher is still running+- **THEN** `just run` stops that watcher before starting its own+- **AND** a later saved Go file restarts only the backend of the new run++#### Scenario: Frontend file is saved++- **WHEN** a developer saves a change to a frontend source file while `just run` is running+- **THEN** the open page shows the change without any process being restarted++### Requirement: Tool versions are pinned in the repository++The repository SHALL pin the exact versions of a defined tool set, so that two hosts set up from the same commit run the same versions of those tools. The pinned set SHALL cover the backend and frontend toolchains, the command runner, the commit hook runner, the process window, the migration tool, the API documentation generator, the Go linter, the backend file watcher and the browser automation server agents use. Docker, the shell, the Chrome browser and system libraries such as `libedit` come from the host and are not pinned.++#### Scenario: New host setup++- **WHEN** a developer installs the pinned tools on a new Fedora host and runs `just lint`+- **THEN** every tool from the pinned set that the recipe calls resolves to its pinned version++#### Scenario: Browser server starts at its pinned version++- **WHEN** an agent client starts the browser automation server registered in the repository+- **THEN** the server runs at the version pinned in the repository, without downloading a newer one++## ADDED Requirements++### Requirement: The backend restarts when its source changes++While `just run` is running, saving a change to a backend Go source file SHALL rebuild the backend and replace the running backend with the new build, without the developer touching the process window. A change to a test file alone SHALL NOT restart the backend. When the new code fails to compile, the backend that was already running SHALL keep serving requests, and the compile error SHALL appear in the backend's output in the process window. Interrupting `just run` SHALL leave no backend process from that run, including one started by a restart.++#### Scenario: Developer saves a backend change++- **WHEN** a developer changes the response of an endpoint and saves the Go file while `just run` is running+- **THEN** a request to that endpoint returns the new response once the rebuild finishes, without any manual restart++#### Scenario: The change does not compile++- **WHEN** a developer saves a Go file that does not compile+- **THEN** the backend output shows the compile error+- **AND** the previous backend still answers requests on the backend port++#### Scenario: Only a test file changes++- **WHEN** a developer saves a change to a Go test file and nothing else+- **THEN** the backend is not restarted++#### Scenario: Interrupt after restarts++- **WHEN** the backend has restarted several times and the developer presses Ctrl+C in the `just run` window+- **THEN** no backend process from that run remains++### Requirement: The local app is seeded with a dev account and databases++When the backend started by `just run` begins answering requests, the seed SHALL make sure the local app contains:++- the dev account `[email protected]` with password `Q1234567`;+- a workspace owned by that account;+- a local storage in that workspace;+- a PostgreSQL database with logical backups of the shared logical test fixture;+- a PostgreSQL database with WAL-streaming physical backups of the live WAL backup source.++The seed SHALL go through the same public API and validation as the web interface. It SHALL find each item by its name, SHALL create only the items that are missing and SHALL NOT change an item that already exists. When the seed creates a database, it SHALL switch its backups on with the local storage and SHALL start the first backup. The logical database SHALL keep its 3 newest backups, and the WAL-streaming database SHALL keep its 2 newest backup chains.++The seed command, `go run ./cmd/seed_dev` in the backend directory, SHALL also run on its own against the app already answering on the backend port, with the same behavior. The repository SHALL NOT provide a separate `just` recipe for it. When nothing answers on the backend port, the seed command SHALL exit with a non-zero status and say that the app has to be started with `just run`. When the backend answers its health check with an error status, for example because disk usage is too high, the seed SHALL exit with a non-zero status and show the reason the backend gave, and SHALL create nothing.++When the seed cannot sign in as the dev account and cannot create it, for example because the account exists with a different password or registration is closed, it SHALL exit with a non-zero status and a message that names the account and the reason the backend gave, and SHALL create nothing. When the instance requires two-factor sign-in, the seed SHALL exit with a non-zero status and a message that names the account and says that two-factor sign-in is required, and SHALL create nothing. When the backend refuses to create a seeded item after sign-in, for example because the dev account is a member that may not create workspaces, the seed SHALL exit with a non-zero status, show the name of that item and the reason the backend gave, and SHALL NOT change any item that already exists. A seed that fails SHALL NOT stop the other processes of `just run`.++#### Scenario: First run on an empty app++- **WHEN** a developer runs `just run` against a development database that has no users+- **THEN** after the backend starts, signing in as `[email protected]` with `Q1234567` succeeds+- **AND** the account's workspace lists the local storage and both databases, each with backups switched on and a first backup started+- **AND** the logical database's backup settings keep 3 backups, and the WAL-streaming database's keep 2 chains++#### Scenario: Second run++- **WHEN** a developer runs `just run` again with the seeded items already present+- **THEN** no item is created a second time+- **AND** no existing item changes, including one the developer edited after the first run++#### Scenario: One seeded database was deleted++- **WHEN** a developer deleted the WAL-streaming database and runs the seed command on its own+- **THEN** the seed creates that database again with backups switched on+- **AND** the other items stay unchanged++#### Scenario: Dev account has another password++- **WHEN** `[email protected]` exists with a password other than `Q1234567` and the seed runs+- **THEN** the seed exits with a non-zero status and a message naming the account and the failed sign-in+- **AND** no workspace, storage or database was created+- **AND** the account's password and role are unchanged+- **AND** the backend and the frontend keep running++#### Scenario: Backend refuses to create the workspace++- **WHEN** the dev account is a member, members may not create workspaces, and the seed runs+- **THEN** the seed exits with a non-zero status and a message naming the workspace and the backend's reason+- **AND** no existing item changed+- **AND** the backend and the frontend keep running++#### Scenario: Two-factor sign-in is required++- **WHEN** the instance requires two-factor sign-in and the seed runs+- **THEN** the seed exits with a non-zero status and a message naming the account and saying that two-factor sign-in is required+- **AND** no workspace, storage or database was created++#### Scenario: Backend reports itself unhealthy++- **WHEN** the backend answers its health check with an error status and the reason "disk usage is above 95%", and the seed command runs on its own+- **THEN** the seed exits with a non-zero status and shows that reason+- **AND** nothing is created++#### Scenario: App is not running++- **WHEN** a developer runs the seed command on its own while no backend answers on the backend port+- **THEN** the command exits with a non-zero status and tells the developer to start the app with `just run`++### Requirement: Seed credentials exist only in local development++The dev account, its password and the seeded databases SHALL be created only by `just run` and by the seed command run on its own in a development checkout. The production image and the production backend SHALL neither contain the seed nor run it, whatever the environment variables, and SHALL create no account with known credentials. The seed SHALL NOT change the password, the role or the settings of any existing account, including the dev account.++#### Scenario: Production image starts++- **WHEN** the production image starts with an empty data directory+- **THEN** no account exists until someone signs up, and `[email protected]` cannot sign in++#### Scenario: Another account exists++- **WHEN** the development database holds an account other than the dev account and the seed runs+- **THEN** that account's password, role and settings are unchanged afterwardsdiff --git a/openspec/changes/archive/2026-10-21-agent-dev-environment/tasks.md b/openspec/changes/archive/2026-10-21-agent-dev-environment/tasks.mdnew file mode 100644index 00000000..d32b637a--- /dev/null+++ b/openspec/changes/archive/2026-10-21-agent-dev-environment/tasks.md@@ -0,0 +1,56 @@+## 1. Seed command++- [x] 1.1 Create `backend/cmd/seed_dev/main.go` as a dev-only main package, following design decisions 1 to 4:+ - The seed takes the account, the names and both database endpoints as parameters, and `main` fills them from the repository `.env`, falling back to the defaults in `docker-compose.yml`. It imports no backend package, per design decision 1, so it starts without the `libedit` shim.+ - It checks `GET /api/v1/system/health` first. It exits non-zero with "start the app with `just run`" when the backend does not answer, and with the reason from the response body when the endpoint returns an error status.+ - It signs in, or signs up when sign-in fails, and stops naming the account when both fail.+ - It finds or creates the workspace `Dev`, the storage `Local` and the databases `Logical PG16` and `WAL PG18`. When the backend refuses a create call, the seed exits non-zero with the item's name and the backend's reason.+ - For each database it creates, it saves the backup config with backups on, the storage and the retention (3 logical backups, 2 WAL chains), then triggers the first backup the way `CreateDatabaseComponent.tsx` does.+ - The package comment lists the account, the names, the sources, the two slot recovery steps and the disk usage case from design.md "Risks".+ - Verify: in `backend/`, `go build ./cmd/seed_dev` and `just lint` pass, and `go list -deps ./cmd | grep seed_dev` prints nothing. At the root, `grep -n 'go build' Dockerfile` shows that the server binary is built only from `./cmd`.+- [x] 1.2 Add `backend/cmd/seed_dev/main_test.go`. It drives the seed against `httptest.NewServer` over a router built per design decision 1. The user controller's public routes and the healthcheck controller go on `/api/v1` without auth, as `backend/cmd/main.go` registers them. The workspace, storage and database controllers, both backup config controllers, and the logical and physical backup controllers go on a group behind `AuthMiddleware`. It uses the test database, a unique email per test, the logical fixture on `TEST_LOGICAL_POSTGRES_16_PORT` and the physical fixture on `TEST_PHYSICAL_POSTGRES_18_PORT`, and cleans up everything it created as `backend/AGENTS.md` "Clean up test data" requires. The tests cover:+ - a fresh account gets the workspace, the storage and both databases with backups enabled, a first backup started, a logical retention of 3 backups and a physical retention of 2 chains;+ - a second run creates nothing and leaves a database the test edited unchanged;+ - a deleted database is recreated while the rest stays unchanged;+ - an account with another password makes the seed fail with the account in the message, creates nothing and leaves that account's password and role unchanged;+ - a separate account created before the seed runs keeps its password, role and settings afterwards;+ - with two-factor sign-in required by the instance settings, the seed exits non-zero with the account and the two-factor reason and creates nothing;+ - a failed sign-in or sign-up exits non-zero, which the test asserts for every failure case;+ - a dev account that is a member, with member workspace creation switched off, makes the seed fail with the workspace name and the backend's reason, and changes no existing item;+ - an unreachable base URL makes the seed fail with the `just run` hint.+ - a stub `httptest.NewServer` whose `/api/v1/system/health` answers 503 with a `status` reason makes the seed exit non-zero with that reason in the message, and no other request reaches the stub. A full disk is a failure the testing skill allows to fake, because it is unsafe to reproduce.+ - Verify: `just test-package ./cmd/seed_dev/...` in `backend/`.++## 2. Backend restart on change and seeding in `just run`++- [x] 2.1 Add `"aqua:air-verse/air" = "1.67.4"` to `mise.toml`. Verify: `mise install` succeeds and `mise exec -- air -v` prints 1.67.4.+- [x] 2.2 Add `backend/.air.toml` per design decision 6, and add `tmp/` to `backend/.gitignore`. Verify: in `backend/`, `go build -o ./tmp/databasus ./cmd` (the configured build command) succeeds, and `git status --short --ignored tmp` shows `tmp/` as ignored. Then `grep -nE '^\s*(include_ext|exclude_regex|stop_on_error|send_interrupt|kill_delay)\b' .air.toml` shows `include_ext = ["go"]`, an `exclude_regex` with `_test\.go`, `stop_on_error = false`, `send_interrupt = true` and a non-zero `kill_delay`.+- [x] 2.3 In `process-compose.yaml`:+ - run the backend with `air` in `backend/`;+ - give the backend a readiness probe that checks that port 4005 accepts connections, per design decision 5;+ - add a `seed` process that runs `go run ./cmd/seed_dev` in `backend/` after the backend is healthy, without restarts.+ - In `scripts/dev.sh`, before `free_backend_port`, stop every `air` process whose working directory is this worktree's `backend/`, and update the comment above `free_backend_port` per design decision 6.+ - Verify: `process-compose --config process-compose.yaml up --dry-run` exits 0; `grep -n -A8 '^ seed:' process-compose.yaml` shows `depends_on` with `backend` and `condition: process_healthy` and `restart: "no"`; `grep -nE 'exit_on_end|exit_on_skipped|exit_on_failure' process-compose.yaml` prints nothing, so a failed seed does not end `just run`; `bash -n scripts/dev.sh` passes; and `grep -n 'air' scripts/dev.sh` shows the step comparing each process's working directory with `$root/backend`.++## 3. Browser server for both agent clients++- [x] 3.1 Add `"npm:chrome-devtools-mcp" = "1.11.0"` to `mise.toml`. Verify: `mise install` succeeds and `mise exec -- chrome-devtools-mcp --version` prints 1.11.0.+- [x] 3.2 Add `BROWSER_HEADLESS=true` under a `# agent browser` comment to `.env.example` and to the local `.env`, which the user approved. Verify: `grep -x 'BROWSER_HEADLESS=true' .env .env.example` matches in both files.+- [x] 3.3 Add the executable `scripts/browser-mcp.sh` per design decision 7: change into the repository root; read the environment first, then `.env` without sourcing it, then the default `true`; error on any other value; `exec mise exec -- chrome-devtools-mcp --isolated --viewport 1440x900`, adding `--headless` when the value is `true`. Verify with `bash -n scripts/browser-mcp.sh`, then with a copy of the launcher at `$tmp/scripts/browser-mcp.sh`, a temporary `$tmp/.env` and a stub `mise` first on `PATH` that prints its arguments:+ - with no `BROWSER_HEADLESS` in `.env` or the environment, the printed arguments contain `--headless` and `--isolated`;+ - with `BROWSER_HEADLESS=false` in `.env`, they do not contain `--headless`;+ - with `BROWSER_HEADLESS=true` in `.env` and `BROWSER_HEADLESS=false` in the environment, they do not contain `--headless`;+ - with `BROWSER_HEADLESS=maybe` in `.env`, the launcher exits non-zero, the stub is not called, and stderr names `BROWSER_HEADLESS`.+- [x] 3.4 Register the server as `chrome-devtools` in `.mcp.json` and in `.codex/config.toml`, both calling the launcher through `bash -c` and `git rev-parse --show-toplevel`. Verify: `python3 -m json.tool .mcp.json` succeeds, and `codex mcp list`, run from the repository root, lists `chrome-devtools` with the launcher command.++## 4. Agent instructions++- [x] 4.1 Add `.agents/skills/run-app/SKILL.md` with the rules from design decision 8 and spec requirement "The agent can see and drive the local app". It includes the rule to rerun `go run ./cmd/seed_dev` in `backend/` when the app of the agent's own worktree lacks the dev account or a seeded database. It links to the seed command's package comment for the account, names and ports instead of listing them. Add the symlink `.claude/skills/run-app/SKILL.md` -> `../../../.agents/skills/run-app/SKILL.md`. Verify: `readlink .claude/skills/run-app/SKILL.md` prints that target, `grep -c '[email protected]' .agents/skills/run-app/SKILL.md` prints 0, and `grep -n 'cmd/seed_dev' .agents/skills/run-app/SKILL.md` matches the rerun rule.+- [x] 4.2 In the root `AGENTS.md`, update the `just run` row to mention the seed and the backend restart on change, and add a short "Checking the app in a browser" section that points to the `run-app` skill. Search the repository's Markdown for other descriptions of `just run` and bring them in line. Verify: `grep -rn "just run\|just seed" --include=*.md . | grep -v openspec/` finds no `just seed` and every `just run` mention consistent with the new behavior.+- [x] 4.3 Run the root linters for the touched modules. Verify: `just lint` exits 0.++## Change verification++```sh+just test-package ./cmd/seed_dev/...+```diff --git a/openspec/specs/agent-workflow/spec.md b/openspec/specs/agent-workflow/spec.mdindex a55dfde9..d517f6af 100644--- a/openspec/specs/agent-workflow/spec.md+++ b/openspec/specs/agent-workflow/spec.md@@ -74,3 +74,81 @@ Each of the repository's own skills, that is every skill except the generated `o#### Scenario: The commit skill in Claude Code- **WHEN** Claude Code lists the skills it can load in this repository- **THEN** the list contains the Git commits skill, and it reads the same file Codex reads++### Requirement: The agent can see and drive the local app++The repository SHALL register one browser automation server for both Claude Code and Codex. With it, an agent opens pages of the local app and reads what the page shows, including a screenshot, the page structure, console messages and network requests. It also clicks, types and navigates. Both clients SHALL start the server from the same repository configuration. The server SHALL drive the Chrome browser installed on the host. Each agent session SHALL get its own temporary browser profile, which is discarded when the session's browser closes, so that two sessions running at once do not block each other and no sign-in carries over between sessions.++The repository SHALL hold a skill that tells the agent how to check the app in the browser:++- When the backend and frontend ports already answer from a `just run` started in the agent's own worktree, the agent uses that app.+- When they answer from another worktree, the agent leaves that app running, does not run the browser check, and tells the user the check was skipped because the running app serves another worktree's code.+- When only one of the two ports answers, the agent does not start `just run` and does not run the browser check. It tells the user which port is held and that the check was skipped.+- When neither port answers, the agent starts `just run` without its interactive window and stops it before it finishes the task, leaving neither port held.+- The agent signs in with the seeded dev account.+- When the app of the agent's own worktree lacks the dev account or a seeded database, the agent reruns the seed command against it instead of creating the items by hand.++A browser check SHALL add to the tests a task names and SHALL NOT replace them.++#### Scenario: Agent checks a page with no app running++- **WHEN** an agent needs to see a page and nothing answers on the backend and frontend ports+- **THEN** the agent starts `just run` without its interactive window, signs in as the dev account, opens the page and reads what it shows+- **AND** no process of that run remains when the agent hands the task over++#### Scenario: The developer's app runs in the same worktree++- **WHEN** an agent needs to see a page and the running app was started in the agent's own worktree+- **THEN** the agent uses that app and leaves it running when it hands the task over++#### Scenario: The running app belongs to another worktree++- **WHEN** an agent needs to see a page and the running app was started in another worktree+- **THEN** the agent does not stop that app and does not open the page+- **AND** the agent tells the user that the browser check was skipped because the app serves another worktree's code++#### Scenario: Only one port answers++- **WHEN** an agent needs to see a page, the frontend port answers and the backend port does not+- **THEN** the agent does not start `just run` and does not open the page+- **AND** the agent tells the user that the frontend port is held and the browser check was skipped++#### Scenario: A seeded database is missing++- **WHEN** an agent uses the app of its own worktree and the WAL-streaming database is missing from it+- **THEN** the agent reruns the seed command, and the database appears with backups switched on++#### Scenario: Two agents use the browser at once++- **WHEN** two agent sessions open the browser at the same time+- **THEN** both browsers start, each with its own empty profile++#### Scenario: Codex opens a page++- **WHEN** Codex is started in the repository+- **THEN** it lists the same browser server Claude Code uses, at the same pinned version++### Requirement: The developer chooses a visible or headless browser++The browser the agent drives SHALL run headless unless the developer asks otherwise. The `BROWSER_HEADLESS` setting in the developer's `.env` file SHALL choose headless when `true` and a visible window when `false`. A value set in the environment the agent client starts from SHALL take precedence over the file. When neither sets it, the browser SHALL run headless. Any value other than `true` or `false` SHALL keep the browser server from starting, with an error that names `BROWSER_HEADLESS`. The example environment file SHALL contain the setting with its default value. The choice SHALL take effect the next time the agent client starts the browser server.++#### Scenario: Default setting++- **WHEN** neither the developer's `.env` file nor the environment sets `BROWSER_HEADLESS` and an agent opens a page+- **THEN** no browser window appears on the developer's screen++#### Scenario: Developer wants to watch++- **WHEN** the developer sets `BROWSER_HEADLESS=false` in `.env` and the agent client restarts the browser server+- **THEN** the agent's browser opens as a visible window++#### Scenario: Environment overrides the file++- **WHEN** `.env` sets `BROWSER_HEADLESS=true` and the agent client is started with `BROWSER_HEADLESS=false` in its environment+- **THEN** the browser opens as a visible window++#### Scenario: Invalid value++- **WHEN** `.env` sets `BROWSER_HEADLESS=maybe` and the agent client starts the browser server+- **THEN** the server does not start+- **AND** the error the client reports names `BROWSER_HEADLESS`diff --git a/openspec/specs/local-development-workflow/spec.md b/openspec/specs/local-development-workflow/spec.mdindex 0aaa1c91..fac10f06 100644--- a/openspec/specs/local-development-workflow/spec.md+++ b/openspec/specs/local-development-workflow/spec.md@@ -7,13 +7,13 @@ Defines the commands a developer or an agent uses on a local Fedora host to run### Requirement: One command starts the local app-The repository SHALL provide `just run` at its root. It SHALL start the development database, the shared backend test fixtures, the live WAL backup source with its load generator, the backend and the frontend development server, and SHALL show the database, the load generator, the backend and the frontend in one terminal window that lists every process separately, with each process's output available on its own. It SHALL work in a fresh worktree that has a `.env` file but has never generated the backend's API documentation or installed frontend dependencies. When a backend from an earlier run still holds the backend port, the command SHALL stop that process before starting the new backend, so that requests reach the code just started.+The repository SHALL provide `just run` at its root. It SHALL start the development database, the shared backend test fixtures, the live WAL backup source with its load generator, the backend and the frontend development server, and SHALL show the database, the load generator, the backend, the frontend and the seed in one terminal window that lists every process separately, with each process's output available on its own. It SHALL work in a fresh worktree that has a `.env` file but has never generated the backend's API documentation or installed frontend dependencies. When a backend from an earlier run still holds the backend port, the command SHALL stop that process before starting the new backend, so that requests reach the code just started. It SHALL also stop a backend file watcher left over from an earlier run in the same worktree, so that no old watcher rebuilds and restarts a backend after the new run starts. A saved change to a frontend source file SHALL reach the open page without restarting any process.#### Scenario: Developer starts the app- **WHEN** a developer runs `just run` on a Fedora host with a `.env` file- **THEN** the development database, the shared test fixtures, the WAL backup source, its load generator, the backend and the frontend start-- **AND** the window lists the database, the load generator, the backend and the frontend as separate processes with their own output+- **AND** the window lists the database, the load generator, the backend, the frontend and the seed as separate processes with their own output#### Scenario: Fresh worktree@@ -26,6 +26,17 @@ The repository SHALL provide `just run` at its root. It SHALL start the developm- **WHEN** a backend from an earlier run still listens on the backend port- **THEN** `just run` stops that process and the new backend serves requests on that port+#### Scenario: Watcher from a killed run survives++- **WHEN** an earlier `just run` in the same worktree was killed without cleanup and its backend file watcher is still running+- **THEN** `just run` stops that watcher before starting its own+- **AND** a later saved Go file restarts only the backend of the new run++#### Scenario: Frontend file is saved++- **WHEN** a developer saves a change to a frontend source file while `just run` is running+- **THEN** the open page shows the change without any process being restarted+### Requirement: Interrupting the local app stops what it startedWhen a developer interrupts `just run` with Ctrl+C, the backend and the frontend development server SHALL exit, and the development database, the WAL backup source and its load generator SHALL stop. These containers SHALL be stopped rather than removed, so the database's data survives into the next run. The shared backend test fixtures SHALL keep running, because the backend suite and other worktrees use them.@@ -111,13 +122,18 @@ The repository SHALL provide `just lint` and `just format` at its root, covering### Requirement: Tool versions are pinned in the repository-The repository SHALL pin the exact versions of a defined tool set, so that two hosts set up from the same commit run the same versions of those tools. The pinned set SHALL cover the backend and frontend toolchains, the command runner, the commit hook runner, the process window, the migration tool, the API documentation generator and the Go linter. Docker, the shell and system libraries such as `libedit` come from the host and are not pinned.+The repository SHALL pin the exact versions of a defined tool set, so that two hosts set up from the same commit run the same versions of those tools. The pinned set SHALL cover the backend and frontend toolchains, the command runner, the commit hook runner, the process window, the migration tool, the API documentation generator, the Go linter, the backend file watcher and the browser automation server agents use. Docker, the shell, the Chrome browser and system libraries such as `libedit` come from the host and are not pinned.#### Scenario: New host setup- **WHEN** a developer installs the pinned tools on a new Fedora host and runs `just lint`- **THEN** every tool from the pinned set that the recipe calls resolves to its pinned version+#### Scenario: Browser server starts at its pinned version++- **WHEN** an agent client starts the browser automation server registered in the repository+- **THEN** the server runs at the version pinned in the repository, without downloading a newer one+### Requirement: Commits are checked before they are createdCreating a commit SHALL format the staged files of each module the commit touches and add the formatted result to the commit. It SHALL run that module's lint checks, and for the frontend its type check as well, and SHALL refuse the commit when any of them fails. A module the commit does not touch SHALL NOT be checked.@@ -136,3 +152,109 @@ Creating a commit SHALL format the staged files of each module the commit touche- **WHEN** a commit touches only the backend- **THEN** no frontend or verification agent check runs++### Requirement: The backend restarts when its source changes++While `just run` is running, saving a change to a backend Go source file SHALL rebuild the backend and replace the running backend with the new build, without the developer touching the process window. A change to a test file alone SHALL NOT restart the backend. When the new code fails to compile, the backend that was already running SHALL keep serving requests, and the compile error SHALL appear in the backend's output in the process window. Interrupting `just run` SHALL leave no backend process from that run, including one started by a restart.++#### Scenario: Developer saves a backend change++- **WHEN** a developer changes the response of an endpoint and saves the Go file while `just run` is running+- **THEN** a request to that endpoint returns the new response once the rebuild finishes, without any manual restart++#### Scenario: The change does not compile++- **WHEN** a developer saves a Go file that does not compile+- **THEN** the backend output shows the compile error+- **AND** the previous backend still answers requests on the backend port++#### Scenario: Only a test file changes++- **WHEN** a developer saves a change to a Go test file and nothing else+- **THEN** the backend is not restarted++#### Scenario: Interrupt after restarts++- **WHEN** the backend has restarted several times and the developer presses Ctrl+C in the `just run` window+- **THEN** no backend process from that run remains++### Requirement: The local app is seeded with a dev account and databases++When the backend started by `just run` begins answering requests, the seed SHALL make sure the local app contains:++- the dev account `[email protected]` with password `Q1234567`;+- a workspace owned by that account;+- a local storage in that workspace;+- a PostgreSQL database with logical backups of the shared logical test fixture;+- a PostgreSQL database with WAL-streaming physical backups of the live WAL backup source.++The seed SHALL go through the same public API and validation as the web interface. It SHALL find each item by its name, SHALL create only the items that are missing and SHALL NOT change an item that already exists. When the seed creates a database, it SHALL switch its backups on with the local storage and SHALL start the first backup. The logical database SHALL keep its 3 newest backups, and the WAL-streaming database SHALL keep its 2 newest backup chains.++The seed command, `go run ./cmd/seed_dev` in the backend directory, SHALL also run on its own against the app already answering on the backend port, with the same behavior. The repository SHALL NOT provide a separate `just` recipe for it. When nothing answers on the backend port, the seed command SHALL exit with a non-zero status and say that the app has to be started with `just run`. When the backend answers its health check with an error status, for example because disk usage is too high, the seed SHALL exit with a non-zero status and show the reason the backend gave, and SHALL create nothing.++When the seed cannot sign in as the dev account and cannot create it, for example because the account exists with a different password or registration is closed, it SHALL exit with a non-zero status and a message that names the account and the reason the backend gave, and SHALL create nothing. When the instance requires two-factor sign-in, the seed SHALL exit with a non-zero status and a message that names the account and says that two-factor sign-in is required, and SHALL create nothing. When the backend refuses to create a seeded item after sign-in, for example because the dev account is a member that may not create workspaces, the seed SHALL exit with a non-zero status, show the name of that item and the reason the backend gave, and SHALL NOT change any item that already exists. A seed that fails SHALL NOT stop the other processes of `just run`.++#### Scenario: First run on an empty app++- **WHEN** a developer runs `just run` against a development database that has no users+- **THEN** after the backend starts, signing in as `[email protected]` with `Q1234567` succeeds+- **AND** the account's workspace lists the local storage and both databases, each with backups switched on and a first backup started+- **AND** the logical database's backup settings keep 3 backups, and the WAL-streaming database's keep 2 chains++#### Scenario: Second run++- **WHEN** a developer runs `just run` again with the seeded items already present+- **THEN** no item is created a second time+- **AND** no existing item changes, including one the developer edited after the first run++#### Scenario: One seeded database was deleted++- **WHEN** a developer deleted the WAL-streaming database and runs the seed command on its own+- **THEN** the seed creates that database again with backups switched on+- **AND** the other items stay unchanged++#### Scenario: Dev account has another password++- **WHEN** `[email protected]` exists with a password other than `Q1234567` and the seed runs+- **THEN** the seed exits with a non-zero status and a message naming the account and the failed sign-in+- **AND** no workspace, storage or database was created+- **AND** the account's password and role are unchanged+- **AND** the backend and the frontend keep running++#### Scenario: Backend refuses to create the workspace++- **WHEN** the dev account is a member, members may not create workspaces, and the seed runs+- **THEN** the seed exits with a non-zero status and a message naming the workspace and the backend's reason+- **AND** no existing item changed+- **AND** the backend and the frontend keep running++#### Scenario: Two-factor sign-in is required++- **WHEN** the instance requires two-factor sign-in and the seed runs+- **THEN** the seed exits with a non-zero status and a message naming the account and saying that two-factor sign-in is required+- **AND** no workspace, storage or database was created++#### Scenario: Backend reports itself unhealthy++- **WHEN** the backend answers its health check with an error status and the reason "disk usage is above 95%", and the seed command runs on its own+- **THEN** the seed exits with a non-zero status and shows that reason+- **AND** nothing is created++#### Scenario: App is not running++- **WHEN** a developer runs the seed command on its own while no backend answers on the backend port+- **THEN** the command exits with a non-zero status and tells the developer to start the app with `just run`++### Requirement: Seed credentials exist only in local development++The dev account, its password and the seeded databases SHALL be created only by `just run` and by the seed command run on its own in a development checkout. The production image and the production backend SHALL neither contain the seed nor run it, whatever the environment variables, and SHALL create no account with known credentials. The seed SHALL NOT change the password, the role or the settings of any existing account, including the dev account.++#### Scenario: Production image starts++- **WHEN** the production image starts with an empty data directory+- **THEN** no account exists until someone signs up, and `[email protected]` cannot sign in++#### Scenario: Another account exists++- **WHEN** the development database holds an account other than the dev account and the seed runs+- **THEN** that account's password, role and settings are unchanged afterwardsdiff --git a/process-compose.yaml b/process-compose.yamlindex a6297667..da6ef535 100644--- a/process-compose.yaml+++ b/process-compose.yaml@@ -8,8 +8,28 @@ processes:echo "API docs http://localhost:4005/api/v1/docs/swagger/index.html"backend:- command: go run ./cmd+ command: airworking_dir: backend+ # The probe only checks that the port accepts connections: process-compose keeps probing for+ # the whole run, and an HTTP probe would add an access log line to the backend output every+ # time. The seed asks the health endpoint itself. The first build of a fresh worktree takes a+ # while, so the probe keeps waiting instead of giving up.+ readiness_probe:+ exec:+ command: "bash -c '</dev/tcp/localhost/4005'"+ initial_delay_seconds: 2+ period_seconds: 2+ timeout_seconds: 5+ failure_threshold: 100000++ seed:+ command: go run ./cmd/seed_dev+ working_dir: backend+ depends_on:+ backend:+ condition: process_healthy+ availability:+ restart: "no"frontend:command: pnpm --dir frontend devdiff --git a/scripts/browser-mcp.sh b/scripts/browser-mcp.shnew file mode 100755index 00000000..34f2a427--- /dev/null+++ b/scripts/browser-mcp.sh@@ -0,0 +1,31 @@+#!/usr/bin/env bash+# Starts the Chrome DevTools MCP server that Claude Code (.mcp.json) and Codex (.codex/config.toml)+# register, at the version pinned in mise.toml. BROWSER_HEADLESS chooses a headless browser (true,+# the default) or a visible window (false); a value in the client's environment wins over .env.+# --isolated gives every session a temporary profile, so parallel sessions never share a lock.+set -euo pipefail++# mise picks the pinned version from the mise.toml of the current directory, and the client may+# start this script from anywhere.+cd "$(dirname "$0")/.."++# .env holds quoted connection strings, so the value is read out of it instead of sourcing it.+read_env_file_value() {+ local name=$1+ [[ -f .env ]] || return 0+ sed -n "s/^${name}=//p" .env | tail -n 1 | tr -d "\"'\r"+}++headless=${BROWSER_HEADLESS:-$(read_env_file_value BROWSER_HEADLESS)}+headless=${headless:-true}++case "$headless" in+ true) headless_flags=(--headless) ;;+ false) headless_flags=() ;;+ *)+ echo "BROWSER_HEADLESS must be true or false, got '$headless'" >&2+ exit 1+ ;;+esac++exec mise exec -- chrome-devtools-mcp --isolated --viewport 1440x900 "${headless_flags[@]}"diff --git a/scripts/dev.sh b/scripts/dev.shindex 8fb08061..435cd150 100755--- a/scripts/dev.sh+++ b/scripts/dev.sh@@ -14,9 +14,22 @@ stop_services() {docker compose -f "$root/docker-compose.yml" stop "${own_services[@]}"}+# air keeps watching after the backend it started dies, so a watcher left by a killed run of this+# worktree would rebuild a backend next to the new run and fight it for the port. Watchers of other+# worktrees work in their own backend/ and are left alone.+stop_stale_backend_watchers() {+ local air_pid+ for air_pid in $(pgrep -x air); do+ [[ "$(readlink "/proc/$air_pid/cwd" 2>/dev/null)" == "$root/backend" ]] || continue+ echo "stopping stale backend watcher pid $air_pid"+ kill "$air_pid" 2>/dev/null || true+ timeout 10 tail --pid="$air_pid" -f /dev/null || kill -9 "$air_pid" 2>/dev/null || true+ done+}+# A stale backend keeps holding the port and answers with the previously built code; a new one that-# fails to bind only logs the error, so requests silently hit the old binary. Killing the listener-# also makes its parent `go run` exit.+# fails to bind only logs the error, so requests silently hit the old binary. Its watcher is stopped+# first by stop_stale_backend_watchers, because air would otherwise start the backend again.free_backend_port() {local port_hex inode pidport_hex=$(printf ':%04X' "$backend_port")@@ -43,6 +56,7 @@ timeout 60 bash -c 'until docker exec backend-db pg_isready -U postgres >/dev/nu(cd "$root/backend" && swag init -d . -g cmd/main.go -o swagger --quiet)pnpm --dir "$root/frontend" install --frozen-lockfile+stop_stale_backend_watchersfree_backend_portcd "$root"