Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs(security): document password admission recovery tradeoff
Josué571765771bf8
diff --git a/docs/how-to/password-work-ownership-recovery.md b/docs/how-to/password-work-ownership-recovery.mdindex b08d12376..e6da1a21c 100644--- a/docs/how-to/password-work-ownership-recovery.md+++ b/docs/how-to/password-work-ownership-recovery.md@@ -6,9 +6,15 @@ The slot has no expiry. Database connection loss, a missed heartbeat, capture exor a process restart does not prove that the old computation stopped.A crashed owner or an ambiguous acquisition/release can therefore leave password-verification unavailable. Other authentication methods and credential recovery keep-their existing policies. Never clear this record from a request, cleanup job, startup-hook, or automated timeout. Do not reset the pace or identifier history to recover it.+verification unavailable. Other authentication methods keep their existing policies.+Verified password-reset recovery also uses this singleton to bound password hashing,+but does not reserve the anonymous login pace. If the owner is busy or unavailable,+recovery fails closed and can be retried after the sanitized `Retry-After` interval.+Never clear this record from a request, cleanup job, startup hook, or automated+timeout. Do not reset the pace or identifier history to recover it.++See [password admission availability and monitoring](../operations/password-admission.md)+for the intentionally retained anonymous-login starvation tradeoff and its signals.## Roll out the first ownership-aware versiondiff --git a/docs/operations/password-admission.md b/docs/operations/password-admission.mdnew file mode 100644index 000000000..6638e5808--- /dev/null+++ b/docs/operations/password-admission.md@@ -0,0 +1,107 @@+# Password admission availability and monitoring++## Approved admission policy++Anonymous password login retains the Spec 043 controls:++- Five attempts per trusted source per minute.+- Five admitted attempts per normalized identifier per 15-minute window.+- A 15-minute failure cooldown after five password failures.+- One cost-12 password verification permit every 7.5 seconds, concurrency one,+ with no queue and no pace refund.++These anonymous-login limits protect account-wide guessing and pace public+cost-12 password verification. The durable work owner also keeps verification and+recovery hashing at global concurrency one. Recovery hashes bypass the anonymous+1-per-7.5-second start pace, so that rate limit does not cover aggregate bcrypt+starts across both lanes. Distributed sources can consume the public pace while+staying below their individual source limits. Ordinary anonymous logins can+consequently receive HTTP 429 during sustained pressure. This availability tradeoff+is retained; the global starvation mechanism has not been removed, and monitoring+alone does not mitigate it.++## Verified recovery++A password-reset invitation issued by the trusted member-registration flow can set a+new password without reserving the anonymous login pace. The reset credential is+validated for signature, registered identity, email, invitation purpose, expiry and+pending state. The password update and sibling reset-link revocation commit together.+Only after that commit does Control API issue the normal signed session, which+External REST stores in the profile session cookie.++Recovery hashes use the existing durable `password_verification_work` singleton.+This keeps bcrypt concurrency at one across service processes and fails closed when+that owner is busy or database authority is unavailable. There is no recovery queue+and no second bcrypt lane. A concurrent request may receive 429 while the owner is+held; retry the verified recovery request after its sanitized `Retry-After` interval.++Password reset keeps the public identifier attempt timestamps. The existing+credential-generation fence clears failure cooldown state, while any remaining+attempt-budget exhaustion continues to deny anonymous login until its normal window+expires. A reset link cannot be exchanged through the generic invitation-accept+endpoint, reused after redemption, or used to create a session before the password+transaction commits.++Reset-link issuance retains the existing five-per-minute per-email limiter in+Control API and the five-per-minute External REST limiter. Completing a link is+also source-limited at ten requests per minute and uses the one-time pending-link+state plus the global durable bcrypt owner; the endpoint does not reserve the+anonymous login pace.++## Cross-service rollout++The Control API reset-completion behavior and External REST consumer form one+contract. Roll them out as a coordinated change, with the Control API route+available before the updated External REST route serves reset completions, then+deploy the Profile UI navigation update. During a mixed-version window, reset+completion may fail closed: the old consumer cannot complete through the new+Control API, and the new consumer cannot find the endpoint on an old Control API.+Retry recovery after the coordinated rollout is healthy. Roll back External REST+and Profile UI together before rolling Control API back; do not leave the new+consumer pointed at a Control API without the password-token endpoint.++## Signals and operator response++Control API exports `password_admission_denials_total{reason}` on its internal+`/metrics` endpoint. Its finite, low-cardinality reasons are `source_rate`,+`identifier_attempts`, `identifier_cooldown`, `global_pace`, `verification_busy`,+and `authority_failure`. No identifier, source address, credential or account+existence is used as a metric label. The source limiter also exports its existing+`rate_limit_hits_total{bucket_type="external_authentication_attempt",result="unavailable"}`+series when its PostgreSQL backend cannot enforce the source limit.++Use this query for a dashboard grouped by rejection class:++```promql+sum by (reason) (increase(password_admission_denials_total[5m]))+```++Suggested alert policy for a Prometheus-compatible monitor:++- Notify operators when `global_pace` denials occur in a five-minute window. Treat+ this as a saturation/availability signal and correlate it with request volume,+ CPU throttling and login success rates; it is not proof of an attack by itself.+- Alert immediately on any `authority_failure` in a five-minute window. Check+ PostgreSQL reachability and Control API logs without logging request identities.+- Alert on any `external_authentication_attempt` limiter `unavailable` result;+ source pacing is fail-closed, so this means login requests are being rejected+ because the limiter cannot establish authoritative counts.+- Review `identifier_attempts`, `identifier_cooldown`, `source_rate` and+ `verification_busy` together to distinguish targeted guessing, source abuse and+ work-owner contention.++The repository currently provides the Control API metrics endpoint and Grafana/Loki+starter configuration, but does not configure a Prometheus scraper or alert manager.+Install the query and alert policy in the deployment's authorized monitoring system;+do not treat an uninstalled rule as deployed coverage.++Before changing global throughput, measure production per-replica CPU requests and+limits, effective container CPU quota and throttling, Node.js runtime allocation,+cost-12 bcrypt latency/throughput at concurrency one, and normal login arrival rates.+The checked-in base Control API manifest requests 50m CPU and 128 MiB memory, with+limits of 300m CPU and 768 MiB memory. Those values do not establish effective+production sizing: the active overlay, replica/HPA count, container cgroup quota,+throttling, runtime architecture/version, production bcrypt latency, and normal+arrival rate/SLO still need measurement. The current local measurements are not+production sizing evidence. No higher pace or additional concurrency is authorized+by this policy.diff --git a/monitoring/README.md b/monitoring/README.mdindex 6f01df536..a80da5b81 100644--- a/monitoring/README.md+++ b/monitoring/README.md@@ -2,15 +2,19 @@Helm values and Grafana dashboard configs for the Clerum observability stack. This directory does not contain applications -- it provides configuration for Grafana and Loki deployed via their official Helm charts.+This starter does not configure a Prometheus scraper or Alertmanager. Control API+password-admission metrics and suggested alert queries are documented in+[password admission availability and monitoring](../docs/operations/password-admission.md).+## Contents-| Path | Description |-|------|-------------|-| `namespace.yaml` | Creates the `monitoring` Kubernetes namespace |-| `grafana/values.yaml` | Grafana Helm values (Loki datasource, dashboard provider, persistence, security context) |-| `loki/values.yaml` | Loki-stack Helm values (Loki + Promtail, 100 Gi persistence) |-| `dashboards/loki-grafana-dashboard.json` | Pre-built Grafana dashboard for Loki log exploration |-| `Makefile` | Install/upgrade/uninstall targets for both charts |+| Path | Description |+| ---------------------------------------- | ---------------------------------------------------------------------------------------- |+| `namespace.yaml` | Creates the `monitoring` Kubernetes namespace |+| `grafana/values.yaml` | Grafana Helm values (Loki datasource, dashboard provider, persistence, security context) |+| `loki/values.yaml` | Loki-stack Helm values (Loki + Promtail, 100 Gi persistence) |+| `dashboards/loki-grafana-dashboard.json` | Pre-built Grafana dashboard for Loki log exploration |+| `Makefile` | Install/upgrade/uninstall targets for both charts |## Prerequisites@@ -32,21 +36,22 @@ make install_grafana## Makefile Targets-| Target | Description |-|--------|-------------|-| `setup` | Add Grafana Helm repo + create namespace |-| `install_loki` | Install loki-stack chart (v2.9.10) |-| `upgrade_loki` | Upgrade loki-stack release |-| `uninstall_loki` | Remove loki-stack release |-| `install_grafana` | Install Grafana chart |-| `upgrade_grafana` | Upgrade Grafana release |-| `uninstall_grafana` | Remove Grafana release |-| `port_forward_grafana` | Forward Grafana UI to `localhost:3000` |-| `status` | Show pods, DaemonSets, and PVCs in the monitoring namespace |+| Target | Description |+| ---------------------- | ----------------------------------------------------------- |+| `setup` | Add Grafana Helm repo + create namespace |+| `install_loki` | Install loki-stack chart (v2.9.10) |+| `upgrade_loki` | Upgrade loki-stack release |+| `uninstall_loki` | Remove loki-stack release |+| `install_grafana` | Install Grafana chart |+| `upgrade_grafana` | Upgrade Grafana release |+| `uninstall_grafana` | Remove Grafana release |+| `port_forward_grafana` | Forward Grafana UI to `localhost:3000` |+| `status` | Show pods, DaemonSets, and PVCs in the monitoring namespace |## Grafana ConfigurationThe `grafana/values.yaml` configures:+- Loki as a datasource at `http://loki.monitoring:3100`- Dashboard auto-provisioning from the `grafana-dashboards` ConfigMap- 10 Gi persistent storage@@ -54,8 +59,8 @@ The `grafana/values.yaml` configures:## Dashboard Inventory-| File | Description |-|------|-------------|+| File | Description |+| ---------------------------------------- | ----------------------------------------------------------------------- || `dashboards/loki-grafana-dashboard.json` | Log exploration dashboard sourced from Grafana community (gnetId 15141) |To load the dashboard, create a ConfigMap and reference it in the Grafana values: