Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(security): document password admission recovery tradeoff

Josué571765771bf8
diff --git a/docs/how-to/password-work-ownership-recovery.md b/docs/how-to/password-work-ownership-recovery.md
index b08d12376..e6da1a21c 100644
--- a/docs/how-to/password-work-ownership-recovery.md
+++ b/docs/how-to/password-work-ownership-recovery.md
@@ -6,9 +6,15 @@ The slot has no expiry. Database connection loss, a missed heartbeat, capture ex
or a process restart does not prove that the old computation stopped.
A crashed owner or an ambiguous acquisition/release can therefore leave password
-verification unavailable. Other authentication methods and credential recovery keep
-their existing policies. Never clear this record from a request, cleanup job, startup
-hook, or automated timeout. Do not reset the pace or identifier history to recover it.
+verification unavailable. Other authentication methods keep their existing policies.
+Verified password-reset recovery also uses this singleton to bound password hashing,
+but does not reserve the anonymous login pace. If the owner is busy or unavailable,
+recovery fails closed and can be retried after the sanitized `Retry-After` interval.
+Never clear this record from a request, cleanup job, startup hook, or automated
+timeout. Do not reset the pace or identifier history to recover it.
+
+See [password admission availability and monitoring](../operations/password-admission.md)
+for the intentionally retained anonymous-login starvation tradeoff and its signals.
## Roll out the first ownership-aware version
diff --git a/docs/operations/password-admission.md b/docs/operations/password-admission.md
new file mode 100644
index 000000000..6638e5808
--- /dev/null
+++ b/docs/operations/password-admission.md
@@ -0,0 +1,107 @@
+# Password admission availability and monitoring
+
+## Approved admission policy
+
+Anonymous password login retains the Spec 043 controls:
+
+- Five attempts per trusted source per minute.
+- Five admitted attempts per normalized identifier per 15-minute window.
+- A 15-minute failure cooldown after five password failures.
+- One cost-12 password verification permit every 7.5 seconds, concurrency one,
+ with no queue and no pace refund.
+
+These anonymous-login limits protect account-wide guessing and pace public
+cost-12 password verification. The durable work owner also keeps verification and
+recovery hashing at global concurrency one. Recovery hashes bypass the anonymous
+1-per-7.5-second start pace, so that rate limit does not cover aggregate bcrypt
+starts across both lanes. Distributed sources can consume the public pace while
+staying below their individual source limits. Ordinary anonymous logins can
+consequently receive HTTP 429 during sustained pressure. This availability tradeoff
+is retained; the global starvation mechanism has not been removed, and monitoring
+alone does not mitigate it.
+
+## Verified recovery
+
+A password-reset invitation issued by the trusted member-registration flow can set a
+new password without reserving the anonymous login pace. The reset credential is
+validated for signature, registered identity, email, invitation purpose, expiry and
+pending state. The password update and sibling reset-link revocation commit together.
+Only after that commit does Control API issue the normal signed session, which
+External REST stores in the profile session cookie.
+
+Recovery hashes use the existing durable `password_verification_work` singleton.
+This keeps bcrypt concurrency at one across service processes and fails closed when
+that owner is busy or database authority is unavailable. There is no recovery queue
+and no second bcrypt lane. A concurrent request may receive 429 while the owner is
+held; retry the verified recovery request after its sanitized `Retry-After` interval.
+
+Password reset keeps the public identifier attempt timestamps. The existing
+credential-generation fence clears failure cooldown state, while any remaining
+attempt-budget exhaustion continues to deny anonymous login until its normal window
+expires. A reset link cannot be exchanged through the generic invitation-accept
+endpoint, reused after redemption, or used to create a session before the password
+transaction commits.
+
+Reset-link issuance retains the existing five-per-minute per-email limiter in
+Control API and the five-per-minute External REST limiter. Completing a link is
+also source-limited at ten requests per minute and uses the one-time pending-link
+state plus the global durable bcrypt owner; the endpoint does not reserve the
+anonymous login pace.
+
+## Cross-service rollout
+
+The Control API reset-completion behavior and External REST consumer form one
+contract. Roll them out as a coordinated change, with the Control API route
+available before the updated External REST route serves reset completions, then
+deploy the Profile UI navigation update. During a mixed-version window, reset
+completion may fail closed: the old consumer cannot complete through the new
+Control API, and the new consumer cannot find the endpoint on an old Control API.
+Retry recovery after the coordinated rollout is healthy. Roll back External REST
+and Profile UI together before rolling Control API back; do not leave the new
+consumer pointed at a Control API without the password-token endpoint.
+
+## Signals and operator response
+
+Control API exports `password_admission_denials_total{reason}` on its internal
+`/metrics` endpoint. Its finite, low-cardinality reasons are `source_rate`,
+`identifier_attempts`, `identifier_cooldown`, `global_pace`, `verification_busy`,
+and `authority_failure`. No identifier, source address, credential or account
+existence is used as a metric label. The source limiter also exports its existing
+`rate_limit_hits_total{bucket_type="external_authentication_attempt",result="unavailable"}`
+series when its PostgreSQL backend cannot enforce the source limit.
+
+Use this query for a dashboard grouped by rejection class:
+
+```promql
+sum by (reason) (increase(password_admission_denials_total[5m]))
+```
+
+Suggested alert policy for a Prometheus-compatible monitor:
+
+- Notify operators when `global_pace` denials occur in a five-minute window. Treat
+ this as a saturation/availability signal and correlate it with request volume,
+ CPU throttling and login success rates; it is not proof of an attack by itself.
+- Alert immediately on any `authority_failure` in a five-minute window. Check
+ PostgreSQL reachability and Control API logs without logging request identities.
+- Alert on any `external_authentication_attempt` limiter `unavailable` result;
+ source pacing is fail-closed, so this means login requests are being rejected
+ because the limiter cannot establish authoritative counts.
+- Review `identifier_attempts`, `identifier_cooldown`, `source_rate` and
+ `verification_busy` together to distinguish targeted guessing, source abuse and
+ work-owner contention.
+
+The repository currently provides the Control API metrics endpoint and Grafana/Loki
+starter configuration, but does not configure a Prometheus scraper or alert manager.
+Install the query and alert policy in the deployment's authorized monitoring system;
+do not treat an uninstalled rule as deployed coverage.
+
+Before changing global throughput, measure production per-replica CPU requests and
+limits, effective container CPU quota and throttling, Node.js runtime allocation,
+cost-12 bcrypt latency/throughput at concurrency one, and normal login arrival rates.
+The checked-in base Control API manifest requests 50m CPU and 128 MiB memory, with
+limits of 300m CPU and 768 MiB memory. Those values do not establish effective
+production sizing: the active overlay, replica/HPA count, container cgroup quota,
+throttling, runtime architecture/version, production bcrypt latency, and normal
+arrival rate/SLO still need measurement. The current local measurements are not
+production sizing evidence. No higher pace or additional concurrency is authorized
+by this policy.
diff --git a/monitoring/README.md b/monitoring/README.md
index 6f01df536..a80da5b81 100644
--- a/monitoring/README.md
+++ b/monitoring/README.md
@@ -2,15 +2,19 @@
Helm values and Grafana dashboard configs for the Clerum observability stack. This directory does not contain applications -- it provides configuration for Grafana and Loki deployed via their official Helm charts.
+This starter does not configure a Prometheus scraper or Alertmanager. Control API
+password-admission metrics and suggested alert queries are documented in
+[password admission availability and monitoring](../docs/operations/password-admission.md).
+
## Contents
-| Path | Description |
-|------|-------------|
-| `namespace.yaml` | Creates the `monitoring` Kubernetes namespace |
-| `grafana/values.yaml` | Grafana Helm values (Loki datasource, dashboard provider, persistence, security context) |
-| `loki/values.yaml` | Loki-stack Helm values (Loki + Promtail, 100 Gi persistence) |
-| `dashboards/loki-grafana-dashboard.json` | Pre-built Grafana dashboard for Loki log exploration |
-| `Makefile` | Install/upgrade/uninstall targets for both charts |
+| Path | Description |
+| ---------------------------------------- | ---------------------------------------------------------------------------------------- |
+| `namespace.yaml` | Creates the `monitoring` Kubernetes namespace |
+| `grafana/values.yaml` | Grafana Helm values (Loki datasource, dashboard provider, persistence, security context) |
+| `loki/values.yaml` | Loki-stack Helm values (Loki + Promtail, 100 Gi persistence) |
+| `dashboards/loki-grafana-dashboard.json` | Pre-built Grafana dashboard for Loki log exploration |
+| `Makefile` | Install/upgrade/uninstall targets for both charts |
## Prerequisites
@@ -32,21 +36,22 @@ make install_grafana
## Makefile Targets
-| Target | Description |
-|--------|-------------|
-| `setup` | Add Grafana Helm repo + create namespace |
-| `install_loki` | Install loki-stack chart (v2.9.10) |
-| `upgrade_loki` | Upgrade loki-stack release |
-| `uninstall_loki` | Remove loki-stack release |
-| `install_grafana` | Install Grafana chart |
-| `upgrade_grafana` | Upgrade Grafana release |
-| `uninstall_grafana` | Remove Grafana release |
-| `port_forward_grafana` | Forward Grafana UI to `localhost:3000` |
-| `status` | Show pods, DaemonSets, and PVCs in the monitoring namespace |
+| Target | Description |
+| ---------------------- | ----------------------------------------------------------- |
+| `setup` | Add Grafana Helm repo + create namespace |
+| `install_loki` | Install loki-stack chart (v2.9.10) |
+| `upgrade_loki` | Upgrade loki-stack release |
+| `uninstall_loki` | Remove loki-stack release |
+| `install_grafana` | Install Grafana chart |
+| `upgrade_grafana` | Upgrade Grafana release |
+| `uninstall_grafana` | Remove Grafana release |
+| `port_forward_grafana` | Forward Grafana UI to `localhost:3000` |
+| `status` | Show pods, DaemonSets, and PVCs in the monitoring namespace |
## Grafana Configuration
The `grafana/values.yaml` configures:
+
- Loki as a datasource at `http://loki.monitoring:3100`
- Dashboard auto-provisioning from the `grafana-dashboards` ConfigMap
- 10 Gi persistent storage
@@ -54,8 +59,8 @@ The `grafana/values.yaml` configures:
## Dashboard Inventory
-| File | Description |
-|------|-------------|
+| File | Description |
+| ---------------------------------------- | ----------------------------------------------------------------------- |
| `dashboards/loki-grafana-dashboard.json` | Log exploration dashboard sourced from Grafana community (gnetId 15141) |
To load the dashboard, create a ConfigMap and reference it in the Grafana values: