Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

fix(helm): create the runtime role managed mode needs, instead of documenting the crash (CHOO-2623)

Wojciech Wojtyniakdd5e14c381dc
diff --git a/deploy/remote/helm/switch/templates/_helpers.tpl b/deploy/remote/helm/switch/templates/_helpers.tpl
index ba433ccde..53bb96948 100644
--- a/deploy/remote/helm/switch/templates/_helpers.tpl
+++ b/deploy/remote/helm/switch/templates/_helpers.tpl
@@ -91,7 +91,16 @@ two modes stay in lockstep.
{{- end -}}
{{- end }}
-{{- define "switch.postgresUser" -}}
+{{/*
+The bootstrapped Postgres superuser: "postgres" in managed mode (the account
+POSTGRES_USER creates on the StatefulSet below), or postgresql.external.username
+in existing mode. This is the identity Mattermost connects as — it owns its
+own "mattermost" database and runs no row-level security, so it has nothing to
+be restricted from there — and the identity the managed StatefulSet and its
+initdb scripts bootstrap and administer with. Distinct from switch.postgresUser
+below, which is DB_USER: the restricted role the RLS policies apply to.
+*/}}
+{{- define "switch.postgresSuperUser" -}}
{{- if eq .Values.postgresql.mode "existing" -}}
{{- .Values.postgresql.external.username | default "postgres" -}}
{{- else -}}
@@ -99,6 +108,34 @@ postgres
{{- end -}}
{{- end }}
+{{/*
+DB_USER: the runtime role every request is actually served over, and the one
+the row-level-security policies apply to.
+
+ mode: existing -> postgresql.external.username, unchanged from what this
+ chart has always sent. The chart does not manage that
+ database, so it cannot create a role in it — DB_USER keeps
+ pointing at whatever account you configure until you
+ create a restricted one yourself and repoint this at it.
+ mode: managed -> postgresql.managed.runtimeUsername, a role this chart
+ creates itself (fresh installs: templates/postgresql/
+ configmap-init.yaml; upgrades of a deployment that
+ predates it: the create-runtime-role pre-upgrade hook Job).
+ Managed mode is the one mode where the chart owns the
+ Postgres it points at, so it is the only mode where it
+ *can* create the role — leaving this at the superuser by
+ default there would make the chart's own default install
+ the one deployment shape with no tenant isolation in it,
+ which is the opposite of what this change is for.
+*/}}
+{{- define "switch.postgresUser" -}}
+{{- if eq .Values.postgresql.mode "existing" -}}
+{{- .Values.postgresql.external.username | default "postgres" -}}
+{{- else -}}
+{{- .Values.postgresql.managed.runtimeUsername -}}
+{{- end -}}
+{{- end }}
+
{{- define "switch.postgresDatabase" -}}
{{- if eq .Values.postgresql.mode "existing" -}}
{{- .Values.postgresql.external.database | default "switch" -}}
@@ -124,10 +161,42 @@ external secret (e.g. one synced by external-secrets / sealed-secrets).
{{- .Values.postgresql.existingSecretKey | default "POSTGRES_PASSWORD" -}}
{{- end }}
+{{/*
+DB_OWNER_USER: the schema owner switch-core migrates and grants as.
+
+ owner.username set -> exactly that value, in either postgresql mode.
+ owner.username unset,
+ mode: managed -> "postgres", the bootstrapped superuser: it
+ already exists, already owns the schema, and
+ its password is already tracked as
+ secrets.postgresPassword, so a managed install
+ needs no separate owner credential configured
+ at all.
+ owner.username unset,
+ mode: existing -> empty. The chart has no external database
+ superuser to default to, so an unconfigured
+ owner there really does mean "no owner
+ configured" — see the values.yaml comment on
+ postgresql.owner for what that implies at boot.
+*/}}
+{{- define "switch.postgresOwnerUser" -}}
+{{- if .Values.postgresql.owner.username -}}
+{{- .Values.postgresql.owner.username -}}
+{{- else if eq .Values.postgresql.mode "managed" -}}
+postgres
+{{- end -}}
+{{- end }}
+
{{/*
Name/key of the Secret holding the schema owner's password, mirroring
-switch.postgresSecretName/Key above for the runtime role. Only referenced when
-postgresql.owner.username is set.
+switch.postgresSecretName/Key above for the runtime role.
+
+Only consulted when postgresql.owner.username is explicitly set — that is the
+only case with a DB_OWNER_PASSWORD key of its own (see switch.secretData). The
+managed default owner (switch.postgresOwnerUser resolving to "postgres" with
+no owner.username set) has no such key: it is the bootstrapped superuser, so
+its password is whatever switch.postgresSecretName/Key already resolve to, and
+callers must fall back to those directly rather than through this helper.
*/}}
{{- define "switch.postgresOwnerSecretName" -}}
{{- if .Values.postgresql.owner.existingSecret -}}
@@ -341,14 +410,20 @@ Include with `nindent 12`.
key: {{ include "switch.postgresSecretKey" . }}
- name: DB_NAME
value: {{ include "switch.postgresDatabase" . | quote }}
-{{- if .Values.postgresql.owner.username }}
+{{- $ownerUser := include "switch.postgresOwnerUser" . }}
+{{- if $ownerUser }}
- name: DB_OWNER_USER
- value: {{ .Values.postgresql.owner.username | quote }}
+ value: {{ $ownerUser | quote }}
- name: DB_OWNER_PASSWORD
valueFrom:
secretKeyRef:
+ {{- if .Values.postgresql.owner.username }}
name: {{ include "switch.postgresOwnerSecretName" . }}
key: {{ include "switch.postgresOwnerSecretKey" . }}
+ {{- else }}
+ name: {{ include "switch.postgresSecretName" . }}
+ key: {{ include "switch.postgresSecretKey" . }}
+ {{- end }}
{{- end }}
{{- if not .Values.postgresql.requireRestrictedRole }}
- name: DB_REQUIRE_RESTRICTED_ROLE
diff --git a/deploy/remote/helm/switch/templates/mattermost/deployment.yaml b/deploy/remote/helm/switch/templates/mattermost/deployment.yaml
index f009ff521..b95288d65 100644
--- a/deploy/remote/helm/switch/templates/mattermost/deployment.yaml
+++ b/deploy/remote/helm/switch/templates/mattermost/deployment.yaml
@@ -42,7 +42,7 @@ spec:
name: {{ include "switch.postgresSecretName" . }}
key: {{ include "switch.postgresSecretKey" . }}
- name: MM_SQLSETTINGS_DATASOURCE
- value: "postgres://{{ include "switch.postgresUser" . }}:$(POSTGRES_PASSWORD)@{{ include "switch.postgresHost" . }}:{{ include "switch.postgresPort" . }}/mattermost?sslmode={{ .Values.postgresql.sslMode }}{{ if include "switch.dbCaBundleConfigMap" . }}&sslrootcert={{ include "switch.dbCaBundlePath" . }}{{ end }}"
+ value: "postgres://{{ include "switch.postgresSuperUser" . }}:$(POSTGRES_PASSWORD)@{{ include "switch.postgresHost" . }}:{{ include "switch.postgresPort" . }}/mattermost?sslmode={{ .Values.postgresql.sslMode }}{{ if include "switch.dbCaBundleConfigMap" . }}&sslrootcert={{ include "switch.dbCaBundlePath" . }}{{ end }}"
- name: MM_SERVICESETTINGS_SITEURL
value: {{ .Values.mattermost.siteUrl | default "" | quote }}
- name: MM_SERVICESETTINGS_ENABLELOCALMODE
diff --git a/deploy/remote/helm/switch/templates/postgresql/configmap-init.yaml b/deploy/remote/helm/switch/templates/postgresql/configmap-init.yaml
index 962fa1222..eeb6e00b1 100644
--- a/deploy/remote/helm/switch/templates/postgresql/configmap-init.yaml
+++ b/deploy/remote/helm/switch/templates/postgresql/configmap-init.yaml
@@ -8,6 +8,43 @@ metadata:
data:
02-create-mattermost-db.sh: |
#!/bin/bash
- psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname='mattermost'" | grep -q 1 || \
- psql -U postgres -c "CREATE DATABASE mattermost;"
+ psql -U {{ include "switch.postgresSuperUser" . }} -tc "SELECT 1 FROM pg_database WHERE datname='mattermost'" | grep -q 1 || \
+ psql -U {{ include "switch.postgresSuperUser" . }} -c "CREATE DATABASE mattermost;"
+ # /docker-entrypoint-initdb.d only runs once, against a brand-new PGDATA —
+ # Postgres never reruns it on a volume that already holds a cluster. So this
+ # is the fresh-install half of creating the runtime role switch-core
+ # connects to as DB_USER; templates/switch-core/create-runtime-role-job.yaml
+ # is the other half, for a managed deployment upgrading from before this
+ # file existed, where initdb never runs again to create it.
+ #
+ # No grants here, and no ALTER ROLE for an existing role: on a fresh volume
+ # there is nothing yet to grant — the tables this role needs access to don't
+ # exist until Alembic creates them — and switch-core issues the grants
+ # itself at boot, from the owner connection, immediately after that
+ # migration (core/switch_core/db/runtime_role.py::grant_runtime_role).
+ #
+ # The role's password is deliberately the bootstrapped superuser's own
+ # ($POSTGRES_PASSWORD, already in this container's environment): see
+ # postgresql.managed.runtimeUsername in values.yaml for why that is not a
+ # second secret to manage.
+ #
+ # No `set` line, deliberately. A ConfigMap volume mounts its files without
+ # the executable bit, so the Postgres entrypoint *sources* this rather than
+ # running it, and any shell option set here outlives the script and governs
+ # the rest of the entrypoint. That entrypoint runs `set -Eeo pipefail` and
+ # carries a standing note that it cannot add `-u` until every unset variable
+ # it reads has been dealt with — so a `set -euo pipefail` here would impose
+ # exactly the option its author has ruled out, on their shell, from a file
+ # they never see. `-e` and `pipefail` are already in force for the same
+ # reason, and `psql -v ON_ERROR_STOP=1` is what actually makes a failing
+ # statement fail the boot.
+ 03-create-runtime-role.sh: |
+ #!/bin/bash
+ psql -v ON_ERROR_STOP=1 -U {{ include "switch.postgresSuperUser" . }} -d postgres \
+ -v role="{{ include "switch.postgresUser" . }}" \
+ -v pw="$POSTGRES_PASSWORD" <<'SQL'
+ SELECT format('CREATE ROLE %I LOGIN PASSWORD %L NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS', :'role', :'pw')
+ WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'role')
+ \gexec
+ SQL
{{- end }}
diff --git a/deploy/remote/helm/switch/templates/postgresql/statefulset.yaml b/deploy/remote/helm/switch/templates/postgresql/statefulset.yaml
index 49ceba5a3..340b1986b 100644
--- a/deploy/remote/helm/switch/templates/postgresql/statefulset.yaml
+++ b/deploy/remote/helm/switch/templates/postgresql/statefulset.yaml
@@ -37,7 +37,7 @@ spec:
protocol: TCP
env:
- name: POSTGRES_USER
- value: postgres
+ value: {{ include "switch.postgresSuperUser" . | quote }}
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
diff --git a/deploy/remote/helm/switch/templates/switch-core/create-runtime-role-job.yaml b/deploy/remote/helm/switch/templates/switch-core/create-runtime-role-job.yaml
new file mode 100644
index 000000000..1e6ffd66a
--- /dev/null
+++ b/deploy/remote/helm/switch/templates/switch-core/create-runtime-role-job.yaml
@@ -0,0 +1,100 @@
+{{- if eq .Values.postgresql.mode "managed" }}
+{{- $secretOverride := "" }}
+{{/*
+The pre-upgrade copy of the Secret, on exactly the condition that makes it
+exist. A hook runs before helm applies the release's ordinary resources, so a
+chart-managed Secret read here is the *previous* release's — which is why
+migrate-prereqs.yaml renders a copy at a lower hook weight, and why this Job
+reads that copy when there is one.
+
+`switchCore.migrations.preUpgradeJob` has to be part of the condition rather
+than only `secrets.existingSecret`: the copy is rendered by that same
+template, so turning the migration Job off (to run migrations out of band)
+also removes the copy, and a Job naming a Secret nobody creates does not fail
+somewhere later — its pod never starts, the hook never succeeds, and every
+upgrade of the release is blocked. Falling back to the release's own Secret is
+correct there: in managed mode the target is always the in-cluster
+StatefulSet, whose superuser password was fixed when its volume was
+bootstrapped and which no later values change can move, so the previous
+release's copy of it is the one that still authenticates.
+*/}}
+{{- if and .Values.switchCore.migrations.preUpgradeJob (not .Values.secrets.existingSecret) }}
+{{- $secretOverride = printf "%s-migrate" (include "switch.secretName" .) }}
+{{- end }}
+{{- $ctx := merge (dict "secretNameOverride" $secretOverride) . }}
+# The fresh-install path creates the runtime role from
+# templates/postgresql/configmap-init.yaml, but /docker-entrypoint-initdb.d
+# only runs once, against a brand-new PGDATA volume — Postgres never reruns it
+# on a volume that already has a cluster in it. A managed deployment that
+# existed before that file did would upgrade straight into
+# db/runtime_role.py's self-check with no restricted role to connect as,
+# which is exactly the crashloop this Job exists to close: the same
+# idempotent CREATE-ROLE-if-missing, run over the Postgres Service instead of
+# inside the initdb container, plus an ALTER ROLE so a rotated
+# secrets.postgresPassword reaches the role even when it already existed.
+#
+# pre-upgrade only, deliberately, and at a weight before the migration Job's
+# (see its own header comment): on a fresh install the managed Postgres
+# StatefulSet has not even been created yet when pre-install hooks run, and
+# initdb creates the role for that case instead — this Job would have nothing
+# to connect to.
+apiVersion: batch/v1
+kind: Job
+metadata:
+ name: {{ include "switch.fullname" . }}-create-runtime-role
+ labels:
+ {{- include "switch.labels" . | nindent 4 }}
+ {{- include "switch.selectorLabels" (dict "Release" .Release "component" "create-runtime-role") | nindent 4 }}
+ annotations:
+ "helm.sh/hook": pre-upgrade
+ "helm.sh/hook-weight": "-1"
+ "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
+spec:
+ backoffLimit: 3
+ template:
+ metadata:
+ labels:
+ {{- include "switch.labels" . | nindent 8 }}
+ {{- include "switch.selectorLabels" (dict "Release" .Release "component" "create-runtime-role") | nindent 8 }}
+ spec:
+ {{- if .Values.global.imagePullSecrets }}
+ {{- include "switch.imagePullSecrets" . | nindent 6 }}
+ {{- end }}
+ restartPolicy: OnFailure
+ initContainers:
+ {{- include "switch.waitFor" (dict "name" "postgres" "host" (include "switch.postgresHost" .) "port" (include "switch.postgresPort" .)) | nindent 8 }}
+ containers:
+ - name: create-runtime-role
+ image: {{ .Values.postgresql.image }}
+ imagePullPolicy: {{ .Values.postgresql.imagePullPolicy }}
+ command:
+ - sh
+ - -c
+ - |
+ set -eu
+ psql -v ON_ERROR_STOP=1 --host "$PGHOST" --port "$PGPORT" \
+ --username {{ include "switch.postgresSuperUser" . }} --dbname postgres \
+ -v role="$RUNTIME_ROLE" -v pw="$PGPASSWORD" <<'SQL'
+ SELECT format('CREATE ROLE %I LOGIN PASSWORD %L NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS', :'role', :'pw')
+ WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'role')
+ \gexec
+ ALTER ROLE :"role" PASSWORD :'pw';
+ SQL
+ env:
+ - name: PGHOST
+ value: {{ include "switch.postgresHost" . | quote }}
+ - name: PGPORT
+ value: {{ include "switch.postgresPort" . | quote }}
+ - name: RUNTIME_ROLE
+ value: {{ include "switch.postgresUser" . | quote }}
+ # The same secret PGPASSWORD authenticates this connection as the
+ # superuser AND the value ALTER ROLE sets the runtime role's
+ # password to — see the values.yaml comment on
+ # postgresql.managed.runtimeUsername for why that is one password,
+ # not two.
+ - name: PGPASSWORD
+ valueFrom:
+ secretKeyRef:
+ name: {{ include "switch.postgresSecretName" $ctx }}
+ key: {{ include "switch.postgresSecretKey" $ctx }}
+{{- end }}
diff --git a/deploy/remote/helm/switch/values.yaml b/deploy/remote/helm/switch/values.yaml
index 758a21219..cbc9c5c59 100644
--- a/deploy/remote/helm/switch/values.yaml
+++ b/deploy/remote/helm/switch/values.yaml
@@ -417,37 +417,62 @@ postgresql:
# runtime role's grants immediately afterwards, so a table a migration just
# added is usable by the role that is about to need it.
#
- # Leave owner.username empty to opt this deployment out of both — boot then
- # applies no migrations and expects the schema to already be at head (see
- # config.py's owner_database_url / runs_migrations_at_boot).
+ # mode: managed defaults this to the bootstrapped "postgres" superuser (the
+ # same account POSTGRES_USER creates on the StatefulSet below) whenever
+ # owner.username is left empty, because that account already exists in
+ # every managed install and its password is already the one this chart
+ # tracks as secrets.postgresPassword — there is nothing new to configure.
+ # Set owner.username only to point a managed deployment's migrations at some
+ # other DDL-capable role instead.
#
- # This chart does NOT create the runtime role, and does not grant it
- # anything — that is application code, run once by switch-core at boot from
- # this connection, and it only works once the role already exists. Create it
- # out of band before pointing a deployment at it: docs/old/rds-migration.md
- # has the exact SQL for an RDS master user; for mode: managed the equivalent
- # is execing into the postgresql-0 pod as the "postgres" superuser the
- # StatefulSet below already bootstraps.
+ # mode: existing has no such account to default to — the chart has no
+ # visibility into what the external database's superuser is called — so
+ # owner.username stays genuinely empty there until you set it. That does NOT
+ # mean boot skips migrations: alembic upgrade head still runs, over whatever
+ # DB_USER resolves to (see migrations/env.py's fallback), and for a
+ # restricted runtime role that fails loudly with a permissions error rather
+ # than quietly doing nothing. That failure is deliberate — it is boot
+ # telling you this deployment moved to a restricted role without saying who
+ # owns the schema, rather than starting up unable to apply its own
+ # migrations and only finding out later.
+ #
+ # mode: managed can create the runtime role itself (see
+ # postgresql.managed.runtimeUsername, templates/postgresql/configmap-init.yaml
+ # for fresh installs and templates/switch-core/create-runtime-role-job.yaml
+ # for an upgrade of a deployment that predates it) precisely because it owns
+ # the Postgres it points at. mode: existing does not, and cannot: the chart
+ # has no way to run CREATE ROLE against a database it does not manage, so
+ # create the role out of band before pointing a deployment at it —
+ # docs/old/rds-migration.md has the exact SQL for an RDS master user.
owner:
username: ""
# Password source, mirroring existingSecret/existingSecretKey above. Only
- # consulted when owner.username is set.
+ # consulted when owner.username is set; the managed default above reads
+ # its password from wherever secrets.postgresPassword does instead.
existingSecret: ""
existingSecretKey: DB_OWNER_PASSWORD
# Refuse to serve when the runtime connection (DB_USER) is not actually
# subject to row-level security — a superuser, a BYPASSRLS role, or the
- # owner of the policied tables. Matches switch-core's own default (on),
- # because the failure it guards against is silent: a deployment that
- # believes it is isolating tenants and is not looks identical to one that
- # is, right up until a second tenant reads the first's rooms.
+ # owner of the policied tables. On by default in both modes, matching
+ # switch-core's own default, because the failure it guards against is
+ # silent: a deployment that believes it is isolating tenants and is not
+ # looks identical to one that is, right up until a second tenant reads the
+ # first's rooms.
+ #
+ # mode: managed creates DB_USER's role itself — see
+ # postgresql.managed.runtimeUsername above and the owner comment's
+ # explanation of why managed mode can do this and existing mode cannot — so
+ # the default install already has a restricted role for this check to pass
+ # against, and there is no reason to relax it there.
#
- # mode: managed ships DB_USER as the bootstrapped "postgres" superuser (see
- # POSTGRES_USER on the StatefulSet below) and this chart does not yet offer
- # a way to point it at a separately created role, so a managed-mode
- # deployment left at the default here will refuse to start. Set this false
- # until that role exists; switch-core then logs an error on every boot as
- # the visible reminder that tenant isolation is not active.
+ # mode: existing is the case this flag exists for. The chart cannot run
+ # CREATE ROLE against a database it does not manage, so a fresh
+ # existing-mode deployment, or one upgrading before its operator has run the
+ # CREATE ROLE in docs/old/rds-migration.md, has no restricted role yet. Set
+ # this false until that role exists; switch-core then logs an error on every
+ # boot instead of refusing to serve, as the visible reminder that tenant
+ # isolation is not active yet.
requireRestrictedRole: true
# ── mode: existing ──────────────────────────────────────────────────────────
@@ -473,6 +498,27 @@ postgresql:
limits:
memory: 512Mi
+ managed:
+ # DB_USER for mode: managed — the restricted role this chart creates for
+ # switch-core to serve requests as, so the row-level-security policies
+ # apply to the connection actually handling traffic rather than to the
+ # bootstrapped superuser (see the postgresql.owner and requireRestrictedRole
+ # comments above for the full picture). Created on a fresh install by
+ # templates/postgresql/configmap-init.yaml and, for an upgrade of a
+ # deployment that predates it, by the create-runtime-role pre-upgrade hook.
+ #
+ # Its password is deliberately the bootstrapped superuser's own —
+ # secrets.postgresPassword, or whatever postgresql.existingSecret resolves
+ # to — rather than a password of its own. That adds no new required value,
+ # keeps working unmodified with postgresql.existingSecret, and exposes
+ # nothing new: this pod already holds that password as DB_OWNER_PASSWORD.
+ # The point of splitting DB_USER from the superuser is that the
+ # row-level-security policies apply to the connection serving requests,
+ # not to compartmentalise passwords between two accounts that already
+ # trust each other — so change the username here if you want, but there is
+ # no separate password field to set alongside it.
+ runtimeUsername: switch_app
+
# Postgres server parameters, passed as `-c name=value`. Each client
# connection is a backend process costing several MB of non-reclaimable
# memory, so raising the pool above without also raising resources.limits