Skip to content

Database governance built for humans and agents — controlling changes and access across every major database.

docs(design): non-interactive recovery beside a running Bytebase (#21586)

zchpeter53fb4815dc64
diff --git a/docs/design/non-interactive-recovery.md b/docs/design/non-interactive-recovery.md
new file mode 100644
index 0000000000..a570b5a418
--- /dev/null
+++ b/docs/design/non-interactive-recovery.md
@@ -0,0 +1,261 @@
+# Non-interactive recovery beside a running Bytebase
+
+Status: proposal · 2026-10-13 · [BYT-10271](https://linear.app/bytebase/issue/BYT-10271)
+
+`bytebase recovery` restores password sign-in for a self-hosted workspace nobody can sign in to.
+Today it runs only as an interactive menu, and its runbook stops every Bytebase server first and
+restarts them after. Teams that operate only through Kubernetes jobs have no supported way to run
+it, and edit the metadata database by hand instead. **Recovery runs as one command with arguments
+beside a running Bytebase, and its change is live within one minute, with no stop and no
+restart.**
+
+## Problem
+
+Checked on `main` at `ca70649de6`, and on a throwaway server on 2026-09-29 and 2026-10-08:
+
+- **Recovery needs a terminal.** `newRecoveryCommand` in `backend/bin/server/cmd/recovery.go`
+ returns `recovery requires terminal input and output` unless input and output are both
+ terminals. `TestRecoveryCommandContract` fails if the command gains a subcommand or an email,
+ password, workspace or yes flag.
+- **A running server does not see the change.** Unless it runs with `--ha`, the server caches users,
+ settings and policies with no expiry (`store.New(ctx, pgURL, !profile.HA)` in
+ `backend/server/server.go`). Recovery opens its own store, so it cannot refresh them. After a
+ password reset beside a running server, the new password was rejected and the old one still
+ worked until a restart. A workspace profile field changed in the database stayed invisible
+ through the API until a restart. With `--ha` the reset applied at once.
+- **So the runbook requires downtime.** `docs/operations/admin-recovery.md` and the first lines the
+ command prints say to stop every server before recovery and restart them after.
+
+What recovery changes, and where a running server reads it:
+
+| Recovery changes | A running server reads it through | Expires today |
+|---|---|---|
+| A user's password | `userEmailCache`, 32,768 entries | Never |
+| `disallowPasswordSignin` in the workspace profile | `settingCache`, 1,024 entries; the profile is read at startup | Never |
+| A user's workspace role | `policyCache`, 4,096 entries | Never |
+
+The lockout counter that a reset also clears is read from the database on every attempt.
+
+Running beside a server is already safe for the data, under the two conditions in D4. Recovery
+starts no server, runs no migration and writes no replica heartbeat. Its setting and IAM writes
+lock the row and merge inside the transaction (`UpdateSettingAtomic`, `PatchIamPolicy`), as the
+server's own writes to those rows do, and the password change updates one column. The stop-first
+rule guarded an IAM lost-update race that #21285 has since fixed. What remains is the cache.
+
+## Principle
+
+> Recovery is a break-glass tool, so it has to work when nothing can be prepared: one command, no
+> terminal, no downtime. A running server may serve a cached copy for at most one minute, then it
+> reads the database again.
+
+## Decisions
+
+**D1 · Each recovery action is also a subcommand.**
+
+```text
+bytebase recovery the interactive menu, unchanged
+bytebase recovery enable-password-signin
+bytebase recovery reset-password --email <email> --password <password>
+```
+
+A subcommand runs one action, prints the result and exits. It needs no terminal and asks for no
+confirmation: naming the action is the confirmation. It keeps every check the menu runs (exactly
+one workspace, a usable administrator for `enable-password-signin`, the domain and password
+restrictions for `reset-password`) and writes the same audit record. The menu stays and still needs
+a terminal. Rejected: one flat flag set on `bytebase recovery` with an `--action` flag, where which
+flags are required would depend on another flag's value.
+
+**D2 · The password is a flag.** `--password` takes the new password on the command line, as
+`redis-cli -a`, `docker login --password`, `grafana cli admin reset-admin-password` and
+`gitea admin user change-password --password` do. It is the form people expect. A Kubernetes Job
+fills it from a Secret without writing the value in the manifest (`$(NEW_PASSWORD)` in `args`). A
+value on the command line is visible in the process list and the shell history, so the runbook
+tells the user to change the password after signing in. Rejected: a password file or standard
+input only, as `psql` and `docker login --password-stdin` do. It is safer, but it adds a step to
+every use, and it can be added beside the flag later. This replaces the original recovery design's
+rule that the password never comes from an argument.
+
+**D3 · The subcommands do not add a workspace role.** After a reset, the menu offers a role from a
+numbered list to a user who has none. A flag would need the role's ID (`roles/workspaceAdmin`, or a
+custom role's ID), which nobody has to hand during an outage. `reset-password` reports that the
+user has no workspace role and points at the menu. Rejected: `--add-role <role>`.
+
+**D4 · Recovery runs beside the running server.** The runbook and the command stop telling the
+operator to stop and restart Bytebase. Recovery reaches the metadata database exactly as the server
+does: the same image, flags, environment, network and mounted files. On Kubernetes that is a Job
+built from the server's pod template, running `bytebase recovery` where the template runs the
+server. A literal `PG_URL` is not always enough. It can be a file path. The connection can rely on
+the pod's service account (AWS RDS and Cloud SQL IAM authentication) or on mounted TLS files. With
+the chart's `escapePassword`, the container's start script builds `PG_URL`, so the Job keeps that
+script. The runbook gives the Job for each of these, and the example under States is the simplest
+one. Two conditions stay:
+
+- **The same build as the running server.** Recovery does not migrate the schema, and an older
+ build drops setting fields it does not know (`ProtojsonUnmarshaler` discards unknown fields).
+ The Job takes the image the running pod reports in its status (`imageID`, a digest), not the
+ tag from the manifest. A tag can move: the Helm chart defaults to `latest` and always pulls, so
+ a Job started later can get a newer build than the server runs.
+- **With the embedded database, run the command inside the server's container.** A second
+ container cannot reach the embedded PostgreSQL. If it mounts the same data directory, it starts
+ its own PostgreSQL on it.
+
+**D5 · The user, setting and policy caches expire after one minute.** `userEmailCache`,
+`settingCache` and `policyCache` become `expirable.LRU` with a one-minute lifetime, the type and
+lifetime `rolesCache` and the group caches already use. An entry is dropped one minute after it was
+stored, and a read does not extend it, so a recovery change is live on every running server within
+about one minute. All three need it: with the setting cache alone, a password reset would still
+wait for a restart, and so would a role the menu adds. `settingCache` and `policyCache` hold every
+setting and every policy type, so all of them are read again once a minute.
+
+The minute runs from when the server stored its copy, not from recovery's commit. A read already in
+progress when recovery commits returns the old value and stores it with a full lifetime, so that
+server is late by the length of that one read: milliseconds for a setting or a policy, the
+user-list query for a user. "Within one minute" in this document and in the command's output means
+that lifetime.
+
+Rejected:
+
+- **A message to running servers,** as a new type on the `bytebase_signal` channel that clears the
+ caches. It is instant, but PostgreSQL delivers a notification only to sessions listening when it
+ is sent. One sent while the listener reconnects is lost, PgBouncer does not support `LISTEN` in
+ transaction pooling, and recovery cannot tell whether any server heard it. A missed message
+ leaves the server stale until a restart, silently.
+- **A counter in the database that servers check on the heartbeat.** It cannot be missed and lets
+ recovery confirm each server, but it needs a schema change and more code than a break-glass path
+ justifies.
+- **Reading the database directly at sign-in.** It covers the password and the setting, not the
+ role.
+- **Running with `--ha`,** which turns these caches off. Every authenticated request then reloads
+ the full user list (`resolvePrincipal` reaches `GetUserByEmail` through `GetAccountByEmail`).
+- **Two minutes for the user cache.** It halves the user-list reloads, but then the wait differs
+ by action and a leaked password works a minute longer.
+- **Counting the lifetime from the start of the read.** It makes the limit exact, but it replaces
+ the stock expiring cache with a custom one at every place that fills these caches. The gain is
+ the length of one read, and only when that read overlaps recovery's commit.
+
+The message can be added on top later if a minute proves too slow. A missed one would then cost a
+minute and no longer a restart.
+
+**D6 · Recovery states the limit and does not wait.** A successful subcommand prints what changed
+and `Live within one minute.`, then exits 0. Any failure exits 1 and says whether anything was
+saved. Recovery cannot confirm that a server applied the change; the limit holds without a check.
+Rejected: a `--wait` flag that sleeps for a minute so that a finished Job means the change is live.
+The caller can wait.
+
+**D7 · Simultaneous misses each reload.** A `userEmailCache` miss reloads every user
+(`listAndCacheAllUsers`). With expiry that reload runs about once a minute, and requests that miss
+at the same moment each run their own. `settingCache` serializes its fills, so simultaneous misses
+on one setting read one after another. That is accepted for now: a server with `--ha` already
+reloads the user list on every request. Rejected for now: sharing one reload between simultaneous
+misses, as `LicenseService` does with `singleflight`. It is the first thing to add if a large
+workspace shows a spike.
+
+## States
+
+What each command prints and returns:
+
+| Command and situation | Output | Exit |
+|---|---|---|
+| `enable-password-signin`, an administrator has a password | `Password sign-in is enabled. Live within one minute.` | 0 |
+| `enable-password-signin`, already enabled | The same | 0 |
+| `enable-password-signin`, no administrator has a password | The existing error | 1 |
+| `reset-password`, an active user with a workspace role | `Password updated for user <email>. Live within one minute.` | 0 |
+| `reset-password`, an active user with no workspace role | The same, then `User <email> has no workspace role. Run "bytebase recovery" to add one.` | 0 |
+| `reset-password`, an unknown, deactivated or non-user account | The existing error | 1 |
+| `reset-password`, the password breaks the workspace restriction, or the email is outside the allowed domains | The existing error | 1 |
+| Either, a required flag is missing | A usage error. Nothing is read or written. | 1 |
+| Either, the change is saved but the audit record fails | The result line, then the error | 1 |
+| Either, the metadata database has no workspace, or more than one | The existing error | 1 |
+| `bytebase recovery` without a terminal | The existing error, now naming the two subcommands | 1 |
+
+When a recovery change is live on a running server:
+
+| Server | Today | With this design |
+|---|---|---|
+| Default, caches on | After a restart | Within one minute |
+| `--ha`, caches off | At once | At once |
+
+What the operator does:
+
+| | Today | With this design |
+|---|---|---|
+| Steps | Stop every server, run the menu from a terminal, start every server | Run one command beside the running server |
+| Downtime | The whole recovery | None |
+| From a pipeline | Not supported | A Job |
+| A leaked password | Works until the restart | Works for up to one more minute |
+
+On Kubernetes, in the simplest case of a server that reads `PG_URL` from a Secret:
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: bytebase-recovery
+ namespace: <the server's namespace>
+spec:
+ backoffLimit: 0
+ template:
+ spec:
+ restartPolicy: Never
+ serviceAccountName: <the server pod's service account>
+ # Copy the server pod's volumes, init containers and imagePullSecrets, if it has any.
+ containers:
+ - name: recovery
+ image: <the running pod's imageID: its registry, repository and digest>
+ args: ["recovery", "reset-password", "--email", "<email>", "--password", "$(NEW_PASSWORD)"]
+ # The server container's env and volumeMounts, plus the new password.
+ env:
+ - name: PG_URL
+ valueFrom:
+ secretKeyRef: { name: <secret>, key: <pg_url_key> }
+ - name: NEW_PASSWORD
+ valueFrom:
+ secretKeyRef: { name: <secret>, key: <password_key> }
+```
+
+The command exits when it is done, and the Job does not start it again after it succeeds. A pod
+deleted before the Job records the result can be replaced, so an action can run twice. Both
+actions are safe to repeat: the second run changes nothing and writes one more audit record. With
+`backoffLimit: 0` a failed run is not retried. A Deployment or StatefulSet would restart the
+container every time it exits and repeat the action without end.
+
+## What changes outside recovery
+
+- **Every server with the caches on reads each cached entry again once a minute.** Settings and
+ policies are single-row reads. The user list is one query a minute (D7).
+- **What these three caches hold is live within a minute after a direct change to the metadata
+ database.** Today it waits for a restart. That bounds a stale entry in any of them, including
+ the policies that permission checks read. It does not reach what the server copies out of the
+ workspace profile when it starts, such as the audit-log-to-stdout flag and the debug level, or
+ the caches that still have no expiry (projects, instances, databases). Recovery changes none of
+ those.
+- **The menu keeps its prompts and its checks.** It loses only the lines that say to stop and
+ restart Bytebase.
+
+## Scope
+
+| File | Change |
+|---|---|
+| `backend/bin/server/cmd/recovery.go` | The two subcommands (D1 to D3), the result lines and exit status (D6), and no stop or restart lines (D4) |
+| `backend/store/store.go` | `userEmailCache`, `settingCache` and `policyCache` as `expirable.LRU` with a one-minute lifetime that tests can shorten (D5) |
+| `backend/store/setting.go` | The comments that say the setting cache has no TTL |
+| `docs/operations/admin-recovery.md`, `docs/operations/high-availability.md` | The runbook without the stop and the restart, with the one-minute wait, the subcommands, and the Job for each way a deployment supplies `PG_URL` |
+
+Tests:
+
+- `TestRecoveryCommandContract` changes. The menu still refuses a non-terminal, and the help lists
+ the two subcommands and their flags. New cases cover every row of the first States table through
+ the existing `fakeRecoveryRunner`.
+- A store test with the caches on and a short lifetime. It warms the three caches, changes the rows
+ through a second store as recovery does, waits past the lifetime, and reads the new values. A
+ second case reads before the lifetime ends and gets the cached value.
+- Once by hand: run `reset-password` beside a running server and confirm that the new password
+ works, and the old one stops, within a minute and without a restart.
+
+## Not in this PR
+
+- The implementation.
+- The public recovery page (bytebase/bytebase.com, `docs/administration/admin-recovery.mdx`), which
+ needs the same changes as the runbook.
+- A message or a counter that makes the change instant or confirmed (D5).
+- A role flag, a password file, a wait flag and a shared user reload (D2, D3, D6, D7).