Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
docs: login attempt lockout design (v1 API audit T9) (#21230)
Danny Xu4fc8b61ff9a5
diff --git a/docs/design/login-attempt-lockout.md b/docs/design/login-attempt-lockout.mdnew file mode 100644index 0000000000..294926b043--- /dev/null+++ b/docs/design/login-attempt-lockout.md@@ -0,0 +1,265 @@+# Login attempt lockout++One table, `login_attempt`, gives every credential — password, emailed code, second factor — an+attempt limit keyed on the identity under attack, identical on Cloud and self-hosted. It replaces+the two counters that fail today: the audit-log count, which reads zero on Cloud, and the per-code+`attempts` column, which an attacker resets by exhausting it. Closes T9 in+[`v1-api-audit-2026-08.md`](v1-api-audit-2026-08.md).++## Background++Guessing is throttled at three points by three mechanisms:++| Credential | Today |+|---|---|+| Password | count failed-login rows in `audit_log`; 10 in 10 minutes → `ResourceExhausted` |+| Emailed code (Cloud's primary factor; password reset) | `attempts` column on the code row; 5 wrong guesses → row deleted |+| MFA (TOTP, recovery code) | the same audit-log count; 5 in 5 minutes → `ResourceExhausted` |++### Problem++- **The audit-log counter reads zero on Cloud.** A failed login's audit row needs a workspace. On+ SaaS the only source is the request's own `workspace` field, which an attacker omits, so the row+ is never written and the password and MFA lockouts never fire. One MFA temp token buys five+ minutes of unmetered TOTP guesses. Self-hosted was fixed in #21189 but still reads the audit+ log, so a retention purge or the BYT-10124 rework can switch it off again.+- **The per-code counter defeats the resend cooldown.** The 60-second cooldown is a predicate on+ the code row, and the row is deleted after the fifth wrong guess. Send, guess five times, trigger+ the delete, send again: five fresh guesses at a 10⁶-space code per seven requests, bounded only+ by throughput. This is the first factor of every MFA-less Cloud account. Even without the+ delete, five guesses per code at one code a minute is 7,200 a day.+- **The code row stores caller-supplied context.** `workspace` is whatever the send request+ named, kept so verify can re-apply that workspace's policies. It binds nothing — the membership+ check at verify is what stops a user choosing a weaker policy — and by verify time the server+ knows the user's workspaces itself.++## Goals++- **G1** One behavior on Cloud and self-hosted, keyed on the identity under attack — never on+ optional request context a caller can omit to make the record vanish.+- **G2** Guessing is bounded per identity across codes and tokens, independent of request rate.+- **G3** No existence oracle: unknown and known emails lock at the same attempt with the same error.+- **G4** Locks expire on their own; no admin unlock.+- **G5** Correct under replicas: state in the metadata database, one atomic statement per attempt.+- **G6** Password and MFA thresholds, error codes, and messages unchanged.++### Non-goals++- **Per-IP throttling and send caps.** Edge concerns: Cloudflare rules on Cloud, the reverse proxy+ on self-hosted. See [Security and performance](#security-and-performance).+- **A per-tenant failed-login record on Cloud.** Accepted in T9.+- **Single-use MFA temp tokens.** Unnecessary once guesses are bounded per identity.+- **The SaaS password oracle** (wrong password → `Unauthenticated`, right password →+ `PermissionDenied` after bcrypt). Now bounded by the `PASSWORD` lock; the reorder must keep+ service-account password login working, so it is its own change.++## Design++### Table++```sql+-- One row per (identity, kind): attempts since the last success, and when the latest was.+CREATE TABLE login_attempt (+ identity text NOT NULL, -- the identity under attack, server-resolved and globally unique (see Semantics); not a FK, so unknown ones count (G3)+ kind text NOT NULL, -- LoginAttemptKind: PASSWORD | EMAIL_CODE | MFA+ attempts int NOT NULL,+ last_attempt_at timestamptz NOT NULL,+ PRIMARY KEY (identity, kind)+);++CREATE INDEX idx_login_attempt_last_attempt_at ON login_attempt (last_attempt_at);+```++No `workspace` column: the credential is per identity and opens every workspace that identity+belongs to. No `ip` column: the caller IP is forgeable, and per-source throttling is an edge concern.++### Semantics++| Kind | Attempts `N` | Duration `D` |+|---|---|---|+| `PASSWORD` | 10 | 10 min |+| `EMAIL_CODE` | 5 | 10 min |+| `MFA` | 5 | 5 min |++1. **An attempt claims a slot before the credential is checked.** `N` attempts are granted. If+ the identity already holds `N` and the latest was under `D` ago, the next gets no slot:+ `ResourceExhausted`, before any bcrypt, TOTP, or hash comparison.+2. **The counter forgets after `D` of quiet.** A claim more than `D` after the latest attempt+ restarts at one. Locked attempts do not claim, so a lock lasts exactly `D` from the `N`th.+3. **Success deletes the row.**++The `identity` is the globally-unique subject of the attack, resolved by the server so a caller+cannot merge two identities into one bucket or split one across buckets (G1):++- local password and emailed code: the normalized email;+- MFA: the email inside the signed temp token;+- LDAP: the resolved identity-provider ID joined with the submitted username. The same username in+ another directory is a different person and must be a different row — the point sharpened by+ success deleting the row: keyed by bare username, an attacker who controls that username in a+ directory of their own could clear a victim's counter by logging in there.++Identities over 254 characters — and, where the identity is an email, invalid syntax — are rejected+before the claim, so garbage never writes a row.++This is Vault's threshold / duration / counter-reset model with the last two collapsed, as Vault's+own defaults do. It is at least as strict as a sliding window everywhere, and stricter against an+attacker who spaces guesses just under `D` apart.++### One statement per attempt++The same guarded-upsert shape the store already uses for the resend cooldown. `$3` is `N`, `$4`+is `D` in seconds:++```sql+INSERT INTO login_attempt (identity, kind, attempts, last_attempt_at)+VALUES ($1, $2, 1, now())+ON CONFLICT (identity, kind) DO UPDATE SET+ attempts = CASE+ WHEN login_attempt.last_attempt_at < now() - make_interval(secs => $4) THEN 1+ ELSE login_attempt.attempts + 1+ END,+ last_attempt_at = now()+WHERE login_attempt.attempts < $3+ OR login_attempt.last_attempt_at < now() - make_interval(secs => $4)+RETURNING 1;+```++A row back is a slot; no row is a lockout. The row lock serializes concurrent claims, so the+`N`th slot goes to exactly one of them (G5). `now()` is database time, so replicas cannot+disagree. A slot spent on a non-credential error (database failure mid-verification) is not+refunded.++The store exposes exactly three operations: claim an attempt, clear on success, purge stale rows.++### Where the claims happen++Password login claims `PASSWORD` once, before the credential is checked — under the email for a+local password, under the provider-scoped identity for an LDAP bind — and clears it on success. Verifying an emailed code claims `EMAIL_CODE` before the code row is even loaded, for+login and password-reset codes alike, and clears it on a match. Completing MFA — during login or+when switching workspaces — claims `MFA` for the email inside the signed temp token and clears it+on success. A successful password reset also clears `PASSWORD`, as Grafana and Mattermost do, so+a user who locked themselves out is not still locked with the new password. The three+audit-counting checks they replace are deleted.++### `email_verification_code`++Becomes `(email, purpose, code_hash, expires_at, last_sent_at)`: a code, its expiry, and the+resend cooldown.++- **Loses `attempts`**, `IncrementEmailVerificationCodeAttempts`, and the delete on exhaustion.+ The row lives until it expires or is consumed, so the cooldown always has a row to evaluate: the+ bypass is closed structurally. Sending is unaffected by a lock — a new code can be requested+ (cooldown applies) but not verified until the lock expires, so the send path needs no check. The+ lockout error becomes `ResourceExhausted` / `too many attempts, please try again later` (was+ `Unauthenticated`); nothing depends on the old form. Deliberately, five wrong codes now lock the+ email for ten minutes, where today a fresh code was one request away — the price of a bound that+ survives code rotation.+- **Loses `workspace`.** Verify never reads a caller-supplied workspace; it uses what the server+ knows about the email. Password reset: once the code verifies, the user is known, so the+ password policy and the audit workspace come from the user's own memberships — the singleton on+ self-hosted. Email-code signup: the gates that run before creating a new user check the singleton on+ self-hosted, or on SaaS the workspace whose invitation the email holds — the same lookup+ provisioning already uses; a brand-new signup has neither and gets the SaaS defaults, as today.+ Existing users are checked after authentication against their resolved workspace, as today.++### Audit log and cleanup++The audit interceptor is untouched: self-hosted keeps writing the failed and locked-out login+rows it writes today, so the existing end-to-end lockout test holds. The hourly data cleaner also+purges rows older than an hour — the longest `D` is ten minutes, so they are dead — which caps the+table at one row per `(identity, kind)` attempted in the last hour.++### Security and performance++**Security: better everywhere except one regression.** Cloud gets working lockouts and the bypass+closes. The regression: the lockout itself becomes a silent denial of service — five requests per+ten minutes hold an email's `EMAIL_CODE` lock, no code pending, no email sent. No rate limit stops+one request every two minutes, keying on IP would hand a botnet unlimited guesses, and Clerk,+Stytch, and GitLab ship the same exposure — so this ships accepted. Remedy if a targeted lockout is+observed: OWASP's device-cookie bucket — a signed cookie set on web login routes a returning+browser's attempts to a trusted bucket an attacker cannot reach; the key grows to+`(identity, kind, trusted)`, a free change on a table purged hourly. One soft spot: an LDAP filter+that accepts several identifier forms gets one bucket per form; the directory's own lockout backs+it.++**Performance: net improvement.** An indexed single-row upsert replaces an audit-log scan per+attempt, locked attempts return before bcrypt, and one identifier's attempts serialize on its own+row, throttling only that attacker. The one new cost — a small row per distinct identifier+attempted, an unauthenticated write path Cloud did not have — is bounded by pre-claim validation,+the hourly purge, and edge rate caps.++**Edge rules (configuration, not code).** Cloudflare does nothing for these paths until rules+exist. `Login` / `ResetPassword`: ~30 per 10 min per IP, counting only `401`/`429` responses, so+service accounts and CI logging in successfully from shared IPs never trip it. For that filter to+catch LDAP spraying, an invalid-credential bind must surface as `Unauthenticated`, not the+`Internal` (500) it returns today — a small pre-existing fix, without which the edge misses failed+LDAP binds and the per-username rows they write. The two send paths, which always answer OK to+avoid enumeration: the same limit on all requests — tolerates an office NAT, caps single-IP+mail-bombing. No challenges on auth paths; CLI and CI cannot solve them.+Preconditions: origin locked to Cloudflare's ranges, `CF-Connecting-IP` as the audit caller IP.+Self-hosted: the reverse proxy in front, as Gitea and Kratos do.++## Alternatives++- **Announce the principal's workspace so the audit row is written on Cloud** (T9's original+ fix). Still reads the audit log; nothing for emails with no workspace; leaves the code path.+- **Centralize password and MFA, keep the per-code counter.** Two mechanisms; 7,200 guesses a+ day even with the bypass patched (fails G2).+- **Counter on the `principal` row** (Devise, ASP.NET). No row for unknown emails → existence+ oracle on the Nth attempt (fails G3).+- **Log-count table** (Grafana, Authelia). N× the rows, a purge that must never lag, no atomic slot.+- **Reuse `email_verification_code` with new purposes.** Saves one `CREATE TABLE`; costs+ placeholder `code_hash`/`last_sent_at` and a table whose name lies.+- **In-memory or Redis counters.** Wrong under replicas (fails G5); no Redis dependency to add.+- **Exact sliding window.** Needs per-failure rows; differs only against a drip-feeding attacker.+- **Lock until admin unlock** (Mattermost, pgAdmin, Zitadel). Fails G4; no admin on Cloud.+- **Keep `workspace` on the code row, or read it from the verify request.** Both keep a+ caller-supplied value in a policy decision, and the second needs a new `ResetPasswordRequest`+ field. The server knows the user's workspaces after the code verifies.+- **Claim only when a code is pending.** Makes the lockout denial noisy (the attacker must trigger+ a send every ten minutes) without preventing it, and adds a pending-code oracle.+- **Key LDAP by the bare username.** Since success deletes the row, an attacker who controls the+ same username in a directory of their own clears the victim's counter by logging in there+ (fails G1).++## Reference++Surveyed 29 products for lockout and 18 for emailed codes (2026-08-23). Every element here has+precedent:++| Element | Same as |+|---|---|+| Lockout state in dedicated state, not derived from an audit log | every product that has a lockout |+| Separate table keyed by the submitted identifier, unknown IDs counted | Grafana, Nextcloud, django-axes, Laravel, Authentik |+| One record for password and second factor | GitLab, Keycloak, Zitadel, Clerk, Stytch |+| Threshold + duration, counter forgets after idle | Vault, Sourcegraph, ASP.NET Identity |+| Temporary self-expiring lock, reset on success | GitLab 10/10 min, ASP.NET 5/5 min, Keycloak, Vault, Entra |+| Slot claimed atomically before verification | Mattermost |+| Emailed-code failures feed the per-identity lockout, no per-code cap | Zitadel, Keycloak (same user lock as passwords), Clerk (10 → 1 h), Stytch (10 → 1 h), Okta (5 → authenticator blocked), Cognito |+| 6 digits, ≤10-minute validity, single use, hashed | NIST 800-63B r4, Clerk, WorkOS, Twilio |++The other standard shapes for codes are a cap per code (Auth0: 3; Hanko: 3; Twilio Verify: 5+checks, 5 sends per 10 minutes) and a counter on the flow that survives resends (SuperTokens,+Kratos). A bound that survives code rotation is what the bypass needs; per identity is the simpler+place to keep it, and the one that also covers passwords and MFA.++Related: [`v1-api-audit-2026-08.md`](v1-api-audit-2026-08.md) T9 ·+[BYT-10068](https://linear.app/bytebase/issue/BYT-10068) / #21189 (self-hosted half) ·+[BYT-10124](https://linear.app/bytebase/issue/BYT-10124).++Sources: [Vault user lockout](https://developer.hashicorp.com/vault/docs/concepts/user-lockout) ·+[Keycloak `LOGIN_FAILURE`](https://github.com/keycloak/keycloak/blob/main/model/jpa/src/main/resources/META-INF/jpa-changelog-26.7.0.xml) ·+[GitLab devise config](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/8_devise.rb) ·+[Mattermost claim](https://raw.githubusercontent.com/mattermost/mattermost/master/server/channels/app/authentication.go) ·+[Grafana `login_attempt`](https://raw.githubusercontent.com/grafana/grafana/main/pkg/services/loginattempt/loginattemptimpl/store.go) ·+[django-axes](https://github.com/jazzband/django-axes/blob/master/axes/models.py) ·+[ASP.NET LockoutOptions](https://github.com/dotnet/aspnetcore/blob/main/src/Identity/Extensions.Core/src/LockoutOptions.cs) ·+[Zitadel `checkOTP`](https://github.com/zitadel/zitadel/blob/main/internal/command/user_human_otp.go) ·+[SuperTokens passwordless](https://github.com/supertokens/supertokens-core/blob/master/src/main/java/io/supertokens/passwordless/Passwordless.java) ·+[Clerk user lockout](https://clerk.com/docs/guides/secure/user-lockout) ·+[Stytch user locks](https://stytch.com/docs/resources/platform/user-locks) ·+[Okta OTP lockout](https://support.okta.com/help/s/article/configurable-lockout-settings-for-multifactor-authentication-failure-attempts) ·+[Auth0 email OTP](https://auth0.com/docs/authenticate/passwordless/authentication-methods/email-otp) ·+[Twilio Verify check](https://www.twilio.com/docs/verify/api/verification-check) ·+[OWASP device cookies](https://owasp.org/www-community/Slow_Down_Online_Guessing_Attacks_with_Device_Cookies) ·+[NIST SP 800-63B r4](https://pages.nist.gov/800-63-4/sp800-63b.html).diff --git a/docs/design/v1-api-audit-2026-08.md b/docs/design/v1-api-audit-2026-08.mdindex 86d4dbb085..53d2c15313 100644--- a/docs/design/v1-api-audit-2026-08.md+++ b/docs/design/v1-api-audit-2026-08.md@@ -15,7 +15,7 @@ noted. Nothing here is patched yet. Fixed findings are removed and listed at the| T15 | Rotating a leaked cloud credential can silently do nothing | HIGH || T13 | "Waiting for my approval" can show an empty page with a live Load More | HIGH || T14 | Paging through issues across projects skips some and repeats others | HIGH |-| T9 | On Cloud, 2FA codes can be guessed without limit, and no failed login is recorded | MED |+| T9 | On Cloud, login codes and 2FA codes can be guessed without limit | HIGH || T18a | Test-environment rights can cancel a running production migration | MED || T12 | Anonymous callers can enumerate emails, relay mail, and read your LDAP config | MED || T16 | Four `BatchGet` RPCs behave four different ways when an item is missing | MED |@@ -28,33 +28,39 @@ noted. Nothing here is patched yet. Fixed findings are removed and listed at the## Getting in, and staying in-### T9 · On Cloud, 2FA codes can be guessed without limit — MED+### T9 · On Cloud, login codes and 2FA codes can be guessed without limit — HIGH-The password and MFA lockouts work by counting failed-login rows in the audit log. On Cloud the-sign-in page doesn't send a workspace ID, and with no workspace the audit row is never written — so-the counter reads zero forever and the limits never fire. `auth_service.go:180-189`+The password and MFA lockouts count failed-login rows in the audit log, and a failed login's audit+row needs a workspace to be written under. Self-hosted always has one — the singleton+(`resolveAuthenticationWorkspaceID`, `auth_service.go:167-190`). On Cloud the only source is the+request's own `workspace` field, which an attacker omits; with none the interceptor skips the row+(`audit.go:398-400`), the counter reads zero forever, and the limits never fire. A lockout keyed on+something the caller controls is not a lockout.**This does not expose passwords on Cloud.** SaaS forces `DisallowPasswordSignin = true`-unconditionally (`auth_service.go:1624-1627`), and Cloud accounts are created with a random 32-byte-bcrypt hash (`:2049-2056`), so there is no guessable password to find. Email-code login — the actual-Cloud primary factor — carries its own attempt counter in the database, independent of the audit log:-5 tries per code, code destroyed on exhaustion, 10-minute expiry, constant-time compare, hashed at-rest (`verifyEmailCode`, `:1962-1984`). That path is properly protected.--**What is exposed is the second factor.** `completeMFALogin` gates on `checkMFALockout`-(`auth_service.go:1076`), which is the audit-row counter — inert on Cloud for the same reason. The-MFA temp token is a stateless JWT valid for 5 minutes and is not single-use, so an attacker holding-one can spend the whole window guessing TOTP codes (a 10⁶ space) or recovery codes with nothing-counting the misses. Reaching that point requires already passing the email-code step, i.e. access to-the victim's inbox — which is precisely the situation the second factor exists to survive.--Secondary: **no failed-login record is written on Cloud at all**, so there is nothing to alert or-investigate on.--Self-hosted is fixed and tested ([#21189](https://github.com/bytebase/bytebase/pull/21189)) — there-the finding was genuinely HIGH, because password signin is allowed. **Fix:** in SaaS, resolve the-workspace from the request host before authenticating, or count MFA failures somewhere other than-the audit log.+(`auth_service.go:1624-1627`) and Cloud accounts carry a random 32-byte bcrypt hash (`:2049-2056`).+**What is exposed is the second factor.** `completeMFALogin` gates on `checkMFALockout` (`:1076`) —+inert on Cloud — and the MFA temp token is a reusable five-minute JWT, so an attacker who has passed+the email-code step (i.e. holds the inbox, the situation the second factor exists to survive) can+guess TOTP or recovery codes for the whole window with nothing counting the misses.++Self-hosted is fixed and tested ([#21189](https://github.com/bytebase/bytebase/pull/21189)), but the+control still reads the audit log. The email-code path, traced for the fix, has its own hole: the+resend cooldown is a predicate on the code row and the row is deleted on attempt exhaustion, so the+cooldown can be bypassed and the Cloud primary factor guessed at request rate. That raises the+finding to HIGH: it is the first factor of every MFA-less Cloud account, not a second factor behind+a compromised inbox.++**Fix:** [`login-attempt-lockout.md`](login-attempt-lockout.md) — one `login_attempt` table for+password, email-code, and MFA failures, replacing both the audit-log counter and the per-code+attempt column. Re-verified 2026-08-23 against `37339d73c8`.++**Accepted, not fixed:** no per-tenant failed-login record on Cloud (a control, not a record, is what+the fix adds; SIEM consumers are self-hosted, where rows already exist, and attribution is ambiguous+once a user spans workspaces), and lockout-as-denial-of-service (inherent to per-identity lockout —+with the fix it covers Cloud's email-code factor too, where the same denial is already reachable by+burning a victim's codes; bounded by the short window; the remedy is per-IP throttling, scoped out+with password spraying in BYT-10068).### T10 · A borrowed session becomes permanent account takeover — HIGH@@ -249,15 +255,15 @@ constraints.## What I'd do, in order-1. **Require re-authentication for password change and 2FA disable (T10).** It's the one finding- here that turns a temporary compromise into a permanent one, and it affects both deployments.-2. **Fix the ACL class, not the instance (T2).** Teach the interceptor to collect every annotated+1. **Bound credential guessing (T9)** per [`login-attempt-lockout.md`](login-attempt-lockout.md):+ one attempt table for password, email-code, and MFA failures on both deployments. It needs no+ stolen session, only a target email, and the design is ready to implement.+2. **Require re-authentication for password change and 2FA disable (T10).** The one finding that+ turns a temporary compromise into a permanent one, and it affects both deployments.+3. **Fix the ACL class, not the instance (T2).** Teach the interceptor to collect every annotatedresource field, then annotate `DiffSchema`. Otherwise the next second-resource field reopens it.-3. **Make credential rotation fail loudly (T15).** A security operation that returns 200 and does+4. **Make credential rotation fail loudly (T15).** A security operation that returns 200 and doesnothing is worse than one that errors.-4. **Close the SaaS half of T9.** Lower severity than it first looked — Cloud's primary factor is- protected independently — but the second factor currently has no attempt limit, and no failed- login is recorded anywhere.5. **Fail closed on auth annotations, and put api-linter in CI.** Reject`AUTH_METHOD_UNSPECIFIED` at startup; make a `permission` on a CUSTOM RPC either enforced or abuild error, since 15 are decorative. None of Tier 5 was visible to `buf lint`'s BASIC profile,