Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript
59.1 KB
Raw Blame
/**
 * Self-provisioning of the `evenfire-registry-pull` image-pull Secret.
 *
 * On a self-hosted cluster there is no managed operator to drop the dockerconfigjson
 * Secret that local-mode plugins on the private registry need, so control-api
 * provisions it itself. Design: docs/architecture/registry-pull-secret-self-provisioning.md.
 *
 * Contract (must match what the managed operator writes, so either satisfies the same
 * imagePullSecrets reference):
 *   name  = 'evenfire-registry-pull'         (EVENFIRE_REGISTRY_PULL_SECRET_NAME)
 *   ns    = EVERY platform workload namespace (see below; imagePullSecrets are ns-local)
 *   type  = kubernetes.io/dockerconfigjson
 *   data['.dockerconfigjson'] = base64 payload, keyed on OUR image host (registryUrl),
 *                               NOT the registry's token-issuer
 *   label clerum.io/managed-by = control-api  (ownership marker; absent ⇒ externally owned)
 *
 * Namespaces — the reference is NOT confined to an McpServer's own namespace. WRC injects
 * `imagePullSecrets: [evenfire-registry-pull]` into every workload whose image host equals
 * the configured registry host, wherever CLERUM_REGISTRY_URL is set, and recipe workloads
 * split across all three platform workload namespaces by kind (transport → mcp-server,
 * spec.ui.workloadRef → sandbox-ui, everything else → sandbox-recipes). So the Secret must
 * exist in ALL of them — `platformWorkloadNamespaces()`, the one allowlist writes are
 * confined to.
 *
 * MANAGED clusters: VERIFY, never write. control-api provisions nothing there — the
 * platform operator owns the Secret — but WRC still injects the reference, so a namespace
 * the operator has not populated yields a workload that can only ImagePullBackOff, behind
 * a 201. Managed mode therefore runs a read-only presence check over the platform
 * namespaces (`verifyOperatorProvisioned`) and fails the caller BEFORE the CRD is
 * persisted when one it needs has no usable copy (`operator_secret_missing`). This asserts
 * nothing about what any operator does; it only refuses to persist a reference that cannot
 * resolve on THIS cluster right now. The check is scoped by `required` for the same reason
 * everything else here is: today's managed operator populates mcp-server, so McpServer
 * installs must keep working while recipe installs — which land in all three — do not
 * silently half-work.
 *
 * A recipe caller passes the whole platform set and requires all of it, which is stricter
 * than that recipe may strictly need: control-api does not model WRC's per-workload
 * kind→namespace split, so it cannot tell which of the three a given recipe will land in.
 * Requiring all three is the conservative side of that gap — it refuses an install that
 * MIGHT have pulled rather than persisting one that might not.
 *
 * Two invariants make this safe to run repeatedly:
 *   1. NEVER write a Secret we do not own. An unlabeled Secret belongs to an external
 *      operator; we return `exists-foreign` and leave it alone. Not owning it does not
 *      mean waving the caller through, though: if it cannot serve an image pull at all we
 *      fail the install (`foreign_secret_unusable`) rather than let a CRD reference it.
 *   2. Mint ONLY when we are already committed to a write we know can succeed. The
 *      registry's pull-credential mint is rotate-on-call — it revokes the org's previous
 *      key — so a mint that is followed by a failed write strands every other namespace's
 *      Secret on a revoked credential.
 *
 * Those two combine into an ALL-OR-NOTHING rule on external pre-provisioning, because the
 * credential is per-ORG while the Secret is per-NAMESPACE. Minting for a namespace we own
 * also revokes the key sealed inside a foreign Secret we deliberately refused to touch —
 * silently: the foreign bytes never change, foreign copies are excluded from the
 * fingerprint set so the divergence check can never see it, and invariant 1 forbids
 * repairing it. So if ANY target in a pass holds a USABLE foreign Secret, the org's pull
 * credential is externally managed and this service mints NOTHING, in any namespace. An
 * operator therefore provisions ALL of `platformWorkloadNamespaces()` or none of them; a
 * half-external cluster fails the installs that need a namespace we cannot fill
 * (`foreign_secret_would_be_revoked`) instead of quietly breaking the ones that work.
 *
 * "Usable" is the gate, not "foreign": an unusable foreign Secret (wrong type, or keyed on
 * another host) is not serving any pull for this registry, so rotating the org key cannot
 * break a working path through it and it does not block the mint. It is still fatal to a
 * caller that references its namespace — via `foreign_secret_unusable`, per invariant 1.
 *
 * Invariant 1 is about a moment, not a name. Ownership is classified once per pass, and the
 * mutations that follow (the wrong-type delete, the credential write) happen later, with a
 * mint in between. `withPullSecretLock` does not help: it serializes control-api replicas
 * against each other, not against an operator, another controller or a `kubectl` replacing
 * the same NAME — and the reconcile cron reopens that window on every tick rather than once
 * per install. So every mutation re-proves ownership immediately before it acts, comparing
 * `metadata.uid` and the ownership marker against what was classified (`recheckOwnership`),
 * and then CARRIES that identity into the write itself as a precondition, so the apiserver
 * refuses a replace or delete against an object that moved in the gap (`mutateOwned`).
 * A namespace that changed hands is recorded (`ownership_changed`) and left alone; one that
 * merely vanished is created, since the mint has already revoked whatever key it held.
 */
import { createHash } from 'node:crypto'
import { config } from '../config.js'
import { withTransaction } from '../db.js'
import type { K8sGateway } from '../k8s.js'
import { rootLogger } from '../observability/logger.js'
import {
  EVENFIRE_REGISTRY_PULL_SECRET_NAME,
  registryHostFromUrl,
} from '../routes/admin/registryImagePullSecret.js'
import type { SecretPreconditions, SecretUpsertRequest } from '../types.js'
import { mintOrgPullCredential, resolvePublishScope } from './registryClient.js'
import { isRegistryAuthActive } from './registryConnectionDb.js'

const logger = rootLogger.child({ module: 'registry-pull-secret' })

/**
 * Audit trail for minting + writing a registry credential, matching the `[AUDIT]` shape
 * the admin registry routes emit. NEVER include the key or the dockerconfigjson payload.
 */
function auditPullSecret(action: string, details: Record<string, unknown>): void {
  console.log(
    `[AUDIT] ${JSON.stringify({ timestamp: new Date().toISOString(), action, ...details })}`
  )
}

const MANAGED_BY_LABEL = 'clerum.io/managed-by'
const MANAGED_BY_VALUE = 'control-api'
const DOCKERCONFIG_KEY = '.dockerconfigjson'
const DOCKERCONFIG_TYPE = 'kubernetes.io/dockerconfigjson'
const FINGERPRINT_ANNOTATION = 'clerum.io/pull-key-fingerprint'

/**
 * This service is a PLATFORM writer, not a user-facing route: it stamps
 * `clerum.io/`-prefixed metadata (the ownership label and the key fingerprint) that
 * `secretConstraints` deliberately blocks by default, so a caller cannot forge platform
 * ownership through `/admin/secrets` or the recipe-secret routes. Every write here declares
 * the exact pull-secret capability — dropping it makes the real writer reject the fingerprint,
 * and `MockGateway` (which bypasses `SecretService`) will NOT catch it; see
 * test/services.pullSecretConstraints.test.ts.
 */
const PLATFORM_WRITE = { capability: 'registryPullSecret' } as const

export type EnsurePullSecretResult =
  | 'created'
  | 'repaired'
  | 'exists-ours'
  | 'exists-foreign'
  | 'skipped'

/**
 * Raised when the pull Secret cannot be provisioned but the caller is about to attach a
 * reference to it. The caller MUST fail the install rather than persist an McpServer
 * whose imagePullSecrets can never resolve — that is the silent ImagePullBackOff this
 * whole mechanism exists to remove.
 */
export class PullSecretProvisionError extends Error {
  constructor(
    message: string,
    readonly reason: string,
    readonly status: number = 500
  ) {
    super(message)
    this.name = 'PullSecretProvisionError'
  }
}

/** Extract an HTTP-ish status from a K8s client error (mirrors extractK8sError). */
function k8sStatus(err: unknown): number | null {
  if (err && typeof err === 'object') {
    const e = err as { code?: number; statusCode?: number; httpStatus?: number }
    const s = e.code ?? e.statusCode ?? e.httpStatus
    if (typeof s === 'number') return s
  }
  return null
}

/**
 * Build a kubernetes.io/dockerconfigjson payload (base64) for `key` against `host`.
 * Keyed on OUR configured registry host (= the image host, per the attach invariant)
 * so the kubelet selects this credential — the registry's server-built blob is keyed
 * on its own token-issuer and cannot be trusted here.
 */
function buildDockerconfigjson(host: string, key: string): string {
  const auth = Buffer.from(`_:${key}`).toString('base64')
  const cfg = { auths: { [host]: { username: '_', password: key, auth } } }
  return Buffer.from(JSON.stringify(cfg)).toString('base64')
}

/**
 * True when the stored blob actually carries an entry for `host`. Presence alone is not
 * enough: a Secret minted before CLERUM_REGISTRY_URL changed is keyed on the old host and
 * the kubelet will never select it. Unparseable ⇒ false (treat as broken, repair it).
 */
function dockerconfigMatchesHost(blob: string | undefined, host: string): boolean {
  if (!blob) return false
  try {
    const parsed = JSON.parse(Buffer.from(blob, 'base64').toString('utf8')) as {
      auths?: Record<string, unknown>
    }
    if (!parsed?.auths || !Object.prototype.hasOwnProperty.call(parsed.auths, host)) return false
    return dockerconfigEntryCarriesCredential(parsed.auths[host])
  } catch {
    return false
  }
}

/**
 * Does an `auths[<host>]` entry carry something the kubelet can actually present?
 *
 * Host presence alone is not enough. `{"auths":{"registry.example":{}}}` parses cleanly,
 * matches the host, and is entirely unusable — the kubelet finds no credential to send and
 * the pull goes out anonymous. Accepting it is wrong in BOTH directions:
 *
 *  - for a Secret we own, it classifies an empty copy as healthy instead of repairing it;
 *  - for a FOREIGN copy it is worse, because the all-or-nothing gate reads "usable foreign"
 *    as "somebody's live credential is sealed in here" and refuses to mint at all. Nothing
 *    was ever served through an empty entry, so that is a self-inflicted outage: every
 *    private install fails closed on account of a Secret that could never pull.
 *
 * Accepted forms, all of which the kubelet understands: `auth` (base64 `user:password`, what
 * Docker writes), the split `username`/`password` form some tools emit, and the
 * `identitytoken` / `registrytoken` fields credential helpers use. Any ONE non-empty string
 * suffices — this is a usability check, not a validity check. Whether the credential still
 * WORKS is the registry's answer to give, costs a round trip per install, and belongs to
 * the out-of-band-revocation problem, which is tracked separately.
 */
function dockerconfigEntryCarriesCredential(entry: unknown): boolean {
  if (!entry || typeof entry !== 'object') return false
  const e = entry as Record<string, unknown>
  const nonEmpty = (v: unknown): boolean => typeof v === 'string' && v.trim() !== ''
  return (
    nonEmpty(e.auth) ||
    nonEmpty(e.password) ||
    nonEmpty(e.identitytoken) ||
    nonEmpty(e.registrytoken)
  )
}

/**
 * Which OBJECT a read saw, as opposed to what was in it.
 *
 * `uid` is the only thing that survives a delete-and-recreate of the same name: labels,
 * annotations and data can all be reproduced by whoever took the name over, and
 * `resourceVersion` only says "something changed", not "this is a different object". It is
 * what lets a mutation prove it is acting on the thing `classify` decided about rather than
 * on whatever now answers to that name (see `recheckOwnership`).
 */
interface SecretIdentity {
  uid: string | undefined
  resourceVersion: string | undefined
}

interface SecretView extends SecretIdentity {
  labels: Record<string, string>
  annotations: Record<string, string>
  type: string | undefined
  dockerconfig: string | undefined
}

function readSecret(raw: unknown): SecretView {
  const s = (raw ?? {}) as {
    metadata?: {
      labels?: Record<string, string>
      annotations?: Record<string, string>
      uid?: string
      resourceVersion?: string
    }
    type?: string
    data?: Record<string, string>
  }
  return {
    labels: s.metadata?.labels ?? {},
    annotations: s.metadata?.annotations ?? {},
    uid: s.metadata?.uid,
    resourceVersion: s.metadata?.resourceVersion,
    type: s.type,
    dockerconfig: s.data?.[DOCKERCONFIG_KEY],
  }
}

function buildSecretReq(targetNs: string, cred: MintedCredential): SecretUpsertRequest {
  return {
    name: EVENFIRE_REGISTRY_PULL_SECRET_NAME,
    namespace: targetNs,
    type: DOCKERCONFIG_TYPE,
    labels: { [MANAGED_BY_LABEL]: MANAGED_BY_VALUE },
    annotations: { [FINGERPRINT_ANNOTATION]: cred.fingerprint },
    data: { [DOCKERCONFIG_KEY]: cred.dockerconfigjson },
  }
}

interface MintedCredential {
  dockerconfigjson: string
  /** Short digest of the key, so copies in different namespaces can be compared. */
  fingerprint: string
}

/**
 * Mint ONE credential. Call this at most once per ensure pass: the registry's mint is
 * rotate-on-call, so a second mint revokes the first and would strand any namespace
 * already written with it.
 */
async function mintCredential(orgName: string, host: string): Promise<MintedCredential> {
  const { key } = await mintOrgPullCredential(orgName)
  return {
    dockerconfigjson: buildDockerconfigjson(host, key),
    fingerprint: createHash('sha256').update(key).digest('hex').slice(0, 12),
  }
}

// Collapse concurrent ensure passes WITHIN this replica so two in-flight installs cannot
// both mint. Keyed on the whole target set, since a pass is atomic over its namespaces —
// which means every caller must pass the SAME set or the dedupe silently does nothing.
// That is why the single-namespace wrapper below delegates to the full platform set: two
// mints against a rotate-on-call registry leave the loser's namespaces on a revoked key,
// and (because the loser rewrites the winner's namespaces too) with FINGERPRINTS THAT
// AGREE — so the divergence check in ensureInner cannot see it and no later pass re-mints.
//
// Correct only at ONE replica (deploy/base/control-plane/control-api.yaml pins
// `replicas: 1`). An in-process Map cannot see another replica's pass, so scaling out
// would require an org-scoped distributed lock around the mint — control-api has Postgres
// available for that. Not implemented: single-replica is the deployed shape today.
//
// The cached value carries the per-namespace FAILURE REASONS alongside the results rather
// than throwing for them inside the pass. Whether an unusable foreign Secret — or a
// namespace we could not fill without revoking a foreign key — is fatal depends on which
// namespaces the CALLER needs, and the dedupe key deliberately does not encode that:
// folding the required set into the key would stop an MCP install and a recipe install from
// collapsing, which is the very race this map exists to prevent. So the pass records, and
// each caller applies its own policy on the shared result.
const inflight = new Map<string, Promise<EnsurePass>>()

/**
 * One pass's outcome: per-namespace results, plus the two per-namespace reasons a caller
 * may have to fail on.
 *
 * `unusable` — a FOREIGN Secret the kubelet cannot use (`foreign_secret_unusable`).
 * `blocked`  — a namespace we OWN that needed a mint we refused to perform, because some
 *              other target in the pass holds a USABLE foreign Secret and the mint would
 *              revoke its key (`foreign_secret_would_be_revoked`). Full caller-facing
 *              sentences, not fragments: the remedy names the foreign namespace, which only
 *              the pass knows. A blocked namespace has NO entry in `results` — nothing
 *              happened to it, and every caller that could observe the gap fails on it
 *              first.
 * `missing`   — MANAGED mode only: a platform namespace with no usable operator-provisioned
 *              copy (`operator_secret_missing`). Fragments ("it does not exist", "it exists
 *              but …"), because the caller-facing sentence names every missing namespace at
 *              once and only `ensureRegistryPullSecrets` knows which ones the caller needs.
 * `taken`    — a namespace we OWNED at classify time whose Secret changed hands before we
 *              could mutate it (`ownership_changed`). Fragments, like `unusable`: the
 *              caller-facing sentence is assembled by `ensureRegistryPullSecrets`.
 *
 * `missing` is mutually exclusive with the rest by mode: it comes from the verification
 * pass, the others from the provisioning pass, and a pass is one or the other.
 */
type EnsurePass = {
  results: Map<string, EnsurePullSecretResult>
  unusable: Map<string, string>
  blocked: Map<string, string>
  missing: Map<string, string>
  taken: Map<string, string>
}

/** The platform workload namespaces a pull credential may legitimately be written into. */
export function platformWorkloadNamespaces(): string[] {
  return [
    ...new Set([config.mcpServersNamespace, config.sandboxNamespace, config.sandboxUiNamespace]),
  ]
}

/**
 * Ensure the `evenfire-registry-pull` Secret exists, and is current, in EVERY given
 * namespace — minting at most ONE credential for the whole pass.
 *
 * The credential is per-ORG but the Secret is per-NAMESPACE, and the registry's mint is
 * rotate-on-call (it revokes the org's previous key). Minting per namespace would
 * therefore have each mint invalidate the namespaces already written. So: read all, mint
 * once if any target needs it, then write that single credential to every target we own.
 *
 * Post-condition: every non-foreign target holds the same, current credential — UNLESS some
 * target holds a USABLE foreign Secret, in which case the pass mints and writes nothing at
 * all (see the all-or-nothing rule in the file header) and every owned target that needed a
 * mint is recorded as blocked instead.
 *
 * Returns `'skipped'` for all targets ONLY when provisioning is legitimately not our job
 * (managed mode, or no registry configured). Every other "cannot provision" condition
 * THROWS `PullSecretProvisionError`, so a caller about to attach an imagePullSecrets
 * reference fails the install loudly instead of persisting an unresolvable one. In managed
 * mode `'skipped'` is now a VERIFIED answer rather than an assumed one: the operator's
 * copies are read first, and a required namespace without a usable one throws too.
 */
export async function ensureRegistryPullSecrets(
  gateway: K8sGateway,
  namespaces: string[],
  opts: { required?: string[] } = {}
): Promise<Map<string, EnsurePullSecretResult>> {
  const targets = [...new Set(namespaces)].sort()
  const dedupeKey = targets.join(',')
  let run = inflight.get(dedupeKey)
  if (!run) {
    run = ensureInner(gateway, targets).finally(() => inflight.delete(dedupeKey))
    inflight.set(dedupeKey, run)
  }
  const { results, unusable, blocked, missing, taken } = await run

  // A recorded failure is fatal only to a caller that needs THAT namespace. A pass covers
  // the whole platform set (for the mint-once collapse), so an MCP install must not be
  // failed by a squatter in a sandbox namespace it never references — while a recipe
  // install, which does land there, still fails loudly. `required` defaults to every target,
  // so a caller that asks for a set is asserting it needs all of it.
  const required = opts.required ?? targets

  // MANAGED mode: the operator owns the Secret, so the only thing we can do for a caller
  // is refuse to persist a reference that cannot resolve. Report every missing namespace
  // at once — an operator fixing one at a time would otherwise walk the whole set through
  // three failed installs. (Order against the two loops below is not load-bearing: this
  // map and those are populated by different passes; see EnsurePass.)
  const absent = required.filter(ns => missing.has(ns))
  if (absent.length > 0) {
    const host = registryHostFromUrl(config.registryUrl)
    const detail = absent.map(ns => `"${ns}" (${missing.get(ns)})`).join(', ')
    throw new PullSecretProvisionError(
      `the ${EVENFIRE_REGISTRY_PULL_SECRET_NAME} Secret cannot serve an image pull in ${detail}. This cluster runs in "${config.registryConnectionMode}" registry mode, where the platform operator — not control-api — owns that Secret, so control-api will not create it here. Ask the operator to provision ${EVENFIRE_REGISTRY_PULL_SECRET_NAME} (type ${DOCKERCONFIG_TYPE}, with a ${DOCKERCONFIG_KEY} entry for "${host}") in the namespace(s) above, then retry. Proceeding would persist a workload referencing a credential no pod can resolve.`,
      'operator_secret_missing',
      409
    )
  }

  // `unusable` first, across the WHOLE required set, before any `blocked`. Both can apply
  // at once (a malformed foreign copy in one namespace, a well-shaped one blocking the mint
  // in another), and `unusable` names a concrete defect in a specific Secret that has to be
  // fixed or removed either way — deleting the copy that blocked the mint would just
  // surface it on the next install. Leading with the defect is the shorter path out.
  for (const ns of required) {
    const why = unusable.get(ns)
    if (why) {
      throw new PullSecretProvisionError(
        `the ${EVENFIRE_REGISTRY_PULL_SECRET_NAME} Secret in namespace "${ns}" is externally owned and cannot serve an image pull: ${why}. Fix it in place, or delete it to let control-api manage that namespace. (A well-shaped foreign Secret is accepted even if its key has since been revoked — checking that would take a registry round-trip.)`,
        'foreign_secret_unusable',
        409
      )
    }
  }
  // A namespace that changed hands mid-pass. Same scoping as `unusable` — the pass sweeps
  // the whole platform set, so only a caller that references this namespace may be failed by
  // it — and after `unusable`, which names a defect that is there to stay: a takeover is a
  // race, and the retry this message asks for may well find a settled cluster.
  // Order against `blocked` is not load-bearing: `blocked` is recorded on a path that
  // returns before anything is mutated, so the two maps are never both populated.
  for (const ns of required) {
    const why = taken.get(ns)
    if (why) {
      throw new PullSecretProvisionError(
        `the ${EVENFIRE_REGISTRY_PULL_SECRET_NAME} Secret in namespace "${ns}" changed hands while this pass was running: ${why}. control-api classified it as its own, then found a different object under that name before writing, so it wrote nothing — it never touches a Secret it does not own. Retry: the next pass classifies what is actually there, so a usable external copy settles into the normal externally-provisioned case and an unusable one comes back as foreign_secret_unusable, naming the concrete defect.`,
        'ownership_changed',
        409
      )
    }
  }
  for (const ns of required) {
    const why = blocked.get(ns)
    if (why) throw new PullSecretProvisionError(why, 'foreign_secret_would_be_revoked', 409)
  }
  return results
}

/**
 * Single-namespace convenience wrapper, for the McpServer install/upgrade paths that
 * provision exactly the namespace their workload lands in.
 *
 * It still runs the FULL platform-namespace pass and then picks its own namespace out of
 * the result, because the inflight dedupe is keyed on the target set: a pass over just
 * `[targetNs]` would never collapse with a concurrent recipe install over all three, and
 * both would mint (see the note on `inflight`). `targetNs` stays in the list so an
 * unsupported namespace is still rejected by ensureInner's allowlist — same reason code,
 * same status, same point in the sequence.
 *
 * Costs two extra Secret reads per MCP install. Buys the mint-once guarantee across route
 * paths, and makes every install a full-set pass, so a diverged cluster converges on
 * whichever install runs next.
 */
export async function ensureRegistryPullSecret(
  gateway: K8sGateway,
  targetNs: string
): Promise<EnsurePullSecretResult> {
  const results = await ensureRegistryPullSecrets(
    gateway,
    [targetNs, ...platformWorkloadNamespaces()],
    // Only this caller's own namespace is load-bearing: the siblings are swept along for
    // the mint-once collapse, so an unusable foreign squatter in one of them must not fail
    // an install that never references it. A USABLE foreign copy in a sibling is different
    // — it forbids the mint outright (see the all-or-nothing rule in the file header), so
    // if `targetNs` needed one it comes back `foreign_secret_would_be_revoked`.
    { required: [targetNs] }
  )
  return results.get(targetNs) ?? 'skipped'
}

/**
 * Classification of one namespace's current Secret, decided before anything is minted.
 *
 * The two states that lead to a MUTATION carry `observed`: the identity of the object the
 * decision was made about. Everything downstream of `classify` acts on a snapshot — the
 * mint sits between the read and the write — so a mutation has to be able to say which
 * object it meant. `foreign` and `absent` carry none: nothing is ever mutated on a foreign
 * target, and an absent one is created (a create is its own compare-and-swap — it 409s if
 * the name is taken, which `writeCredential` already handles).
 */
type TargetState =
  | { kind: 'foreign'; unusable: string | null }
  | { kind: 'valid'; fingerprint: string; observed: SecretIdentity }
  | { kind: 'absent' }
  | { kind: 'broken'; wrongType: boolean; observed: SecretIdentity }

/**
 * Why a Secret we do NOT own cannot serve an image pull, or null when it can.
 *
 * Read-only by construction — the same two checks `classify` applies to our own copies,
 * with no fingerprint requirement (that annotation is ours; an external operator has no
 * reason to write it) and no write of any kind.
 *
 * Deliberately shape-only: a well-formed blob whose key the registry has since revoked
 * still passes here. Proving liveness would take a registry round-trip on every install,
 * which this check does not do.
 */
function foreignUsabilityProblem(current: SecretView, host: string): string | null {
  if (current.type !== DOCKERCONFIG_TYPE) {
    return `its type is "${current.type ?? 'unset'}", not ${DOCKERCONFIG_TYPE} — the kubelet ignores it for image pulls`
  }
  if (!dockerconfigMatchesHost(current.dockerconfig, host)) {
    // Two different repairs, so name which one it is: a missing entry is usually the wrong
    // registry host, an empty one is usually a half-written or template-generated Secret.
    return `its ${DOCKERCONFIG_KEY} carries no usable credential for "${host}" (the entry is absent, or present but carries no auth/password/token), so the kubelet will never select it`
  }
  return null
}

function classify(current: SecretView | null, host: string): TargetState {
  if (!current) return { kind: 'absent' }
  // INVARIANT 1: never write a Secret we do not own — even a broken one. Adopting it would
  // seize an external operator's credential; repairing it in place would rotate the org
  // key out from under them. We still inspect it: not-ours decides who WRITES, not whether
  // the caller may attach a reference to it (see ensureInner).
  if (current.labels[MANAGED_BY_LABEL] !== MANAGED_BY_VALUE) {
    return { kind: 'foreign', unusable: foreignUsabilityProblem(current, host) }
  }
  const observed: SecretIdentity = {
    uid: current.uid,
    resourceVersion: current.resourceVersion,
  }
  const wrongType = current.type !== DOCKERCONFIG_TYPE
  // Presence alone is not enough: a blob keyed on a previous registry host can never be
  // selected by the kubelet, and a copy with no fingerprint cannot be compared across
  // namespaces.
  const fingerprint = current.annotations[FINGERPRINT_ANNOTATION]
  if (wrongType || !dockerconfigMatchesHost(current.dockerconfig, host) || !fingerprint) {
    return { kind: 'broken', wrongType, observed }
  }
  return { kind: 'valid', fingerprint, observed }
}

/**
 * What a pre-mutation re-read concluded about the object still sitting under that name.
 *
 * `same` carries the identity THAT READ saw, not the one `classify` saw: it is what the
 * following mutation binds itself to, so the API server can reject the write if anything
 * moves in between (see `mutateOwned`). Passing the older identity would pin the write to a
 * version the object may legitimately have left behind.
 */
type OwnershipRecheck =
  | { verdict: 'same'; current: SecretIdentity }
  | { verdict: 'gone' }
  | { verdict: 'taken'; why: string }

/**
 * Re-prove, immediately before mutating `ns`, that the Secret there is still the object
 * `classify` decided about and still carries our ownership marker.
 *
 * WHY IT IS NEEDED. Everything after `classify` acts on a snapshot: the wrong-type delete
 * and the credential write both happen later, with a mint in between. `withPullSecretLock`
 * does not cover this — it serializes control-api replicas against each other, not against
 * an operator, another controller or a `kubectl` replacing the same NAME. And
 * `SecretService.updateSecret` cannot cover it either: it re-reads to pick up the latest
 * `resourceVersion`, which makes it always WIN the write; it never asks whether it is still
 * the same object. Without this check, invariant 1 ("never write a Secret we do not own")
 * only holds for as long as nobody else moves — and the reconcile cron reopens the window
 * every tick rather than once per install.
 *
 * WHAT IT IS — AND IS NOT. On its own this is only half a compare-and-swap: the proof read
 * is a separate API request from the mutation, so a takeover landing between the two would
 * still slip through. It is never called on its own for that reason. `mutateOwned` pairs it
 * with an `If-Match`-style precondition on the write itself — the identity returned here is
 * carried into `metadata.resourceVersion` on the replace and `preconditions.uid` on the
 * delete — so the API server, not this function, is what finally refuses a write to an
 * object that moved. This read still earns its place: it produces the fresh identity the
 * write binds to, and it distinguishes `gone` (create instead) from `taken` (surrender),
 * neither of which a bare 409 can tell apart.
 *
 * WHAT IT COMPARES. `uid` and the ownership label, per the two ways a name changes hands:
 *   - `uid` differs ⇒ deleted and recreated. Data, labels and annotations can all be
 *     reproduced by the new owner; the uid cannot.
 *   - the marker is gone ⇒ adopted in place by an external owner, and `classify` would now
 *     call it `foreign`.
 * `resourceVersion` is deliberately NOT a gate HERE, though it is a precondition on the
 * write. It changes on any edit, including benign ones by something that left our marker in
 * place, and refusing on that basis would be worse than writing: the mint has already
 * revoked the key that copy holds, so declining to overwrite leaves the namespace on a dead
 * credential. That is why `mutateOwned` treats the resulting 409 as "re-read and try again",
 * not as "surrender" — only this function's ownership verdict may end the attempt. Here the
 * version difference is logged, which is where "still ours, but someone edited it" is worth
 * seeing.
 *
 * `uid` is compared only when both reads surfaced one. The API server always sets it; a
 * caller that cannot see it (an older gateway, a fake) falls back to the ownership label,
 * which is exactly the evidence `classify` itself used.
 *
 * A read failure other than 404 propagates, same as everywhere else here. Before the mint
 * that aborts the pass having changed nothing; after it, it leaves the remaining namespaces
 * unwritten — which is the pre-existing behaviour of any post-mint write failure, and the
 * fingerprint divergence check converges it on the next pass.
 */
async function recheckOwnership(
  gateway: K8sGateway,
  ns: string,
  observed: SecretIdentity
): Promise<OwnershipRecheck> {
  const now = await readCurrent(gateway, ns)
  if (!now) return { verdict: 'gone' }
  if (now.labels[MANAGED_BY_LABEL] !== MANAGED_BY_VALUE) {
    return {
      verdict: 'taken',
      why: `its ${MANAGED_BY_LABEL} label is no longer "${MANAGED_BY_VALUE}", so an external owner has adopted that name`,
    }
  }
  if (observed.uid && now.uid && observed.uid !== now.uid) {
    return {
      verdict: 'taken',
      why: 'it was deleted and recreated by something else (metadata.uid changed), so it is no longer the Secret this pass classified',
    }
  }
  if (observed.resourceVersion && now.resourceVersion !== observed.resourceVersion) {
    logger.info(
      { namespace: ns, from: observed.resourceVersion, to: now.resourceVersion },
      'evenfire-registry-pull Secret was modified between classification and write; still ours, proceeding'
    )
  }
  return { verdict: 'same', current: { uid: now.uid, resourceVersion: now.resourceVersion } }
}

/**
 * How many times an ownership-bound mutation may lose a benign race before we stop.
 *
 * Each attempt is one read plus one write, and the only thing that makes an attempt fail
 * without ending the loop is a 409 from something else writing the same Secret in that gap.
 * Three is chosen to absorb a genuine race, not to wait out a controller that is fighting us
 * for the name: with a persistent writer no number of retries converges, and the honest
 * answer is to stop and report, which is what exhausting these does.
 */
const OWNERSHIP_CAS_ATTEMPTS = 3

/** The outcome of an ownership-bound mutation. `done` means it landed. */
type MutateOutcome = { verdict: 'done' } | { verdict: 'gone' } | { verdict: 'taken'; why: string }

/**
 * Mutate the Secret in `ns` ONLY IF it is still the object this pass classified and still
 * ours — a real compare-and-swap, decided by the API server rather than by a read we hope
 * nothing raced.
 *
 * `recheckOwnership` alone cannot do this: it proves ownership in one request and the
 * mutation happens in the next, so a takeover in that gap is invisible to it. Here the
 * identity it observed is carried INTO the mutation as a precondition (`resourceVersion` on
 * a replace, `preconditions.uid` on a delete), so a write to an object that moved is
 * refused with 409 by the apiserver — the check and the use become one operation.
 *
 * WHY 409 IS NOT AN ANSWER BY ITSELF. It says "this object is not what you thought", which
 * conflates two very different situations: someone took the name over, and someone edited a
 * Secret that is still ours. Surrendering on both would be wrong — the mint upstream has
 * already revoked the key the existing copy holds, so refusing to write over a benign edit
 * strands that namespace on a dead credential, an ImagePullBackOff we caused ourselves. So a
 * 409 sends us back through `recheckOwnership`, which CAN tell the two apart, and only its
 * `taken` verdict ends the loop. That is also why the retry drops the stale
 * `resourceVersion` but keeps the `uid`: the version is expected to have moved, the identity
 * is not.
 *
 * `gone` is returned rather than retried — the object is absent, and what to do about that
 * differs per call site (create the credential, or accept that the delete is already done).
 */
async function mutateOwned(
  gateway: K8sGateway,
  ns: string,
  observed: SecretIdentity,
  mutate: (precondition: SecretPreconditions) => Promise<unknown>
): Promise<MutateOutcome> {
  let expected = observed
  for (let attempt = 1; attempt <= OWNERSHIP_CAS_ATTEMPTS; attempt++) {
    const recheck = await recheckOwnership(gateway, ns, expected)
    if (recheck.verdict !== 'same') return recheck
    try {
      await mutate({
        uid: recheck.current.uid,
        resourceVersion: recheck.current.resourceVersion,
      })
      return { verdict: 'done' }
    } catch (err) {
      if (k8sStatus(err) !== 409) throw err
      logger.info(
        { namespace: ns, attempt },
        'evenfire-registry-pull Secret changed under an ownership-bound write; re-proving ownership before retrying'
      )
      // Keep the identity, drop the version: the next `recheckOwnership` must still catch a
      // delete-and-recreate, but the version it compares against is known to be stale and
      // would only log noise.
      expected = { uid: recheck.current.uid, resourceVersion: undefined }
    }
  }
  return {
    verdict: 'taken',
    why: `something else rewrote it during each of ${OWNERSHIP_CAS_ATTEMPTS} ownership-bound write attempts, so this pass never managed to write it without risking somebody else's object`,
  }
}

/**
 * Serialize the read->mint->write pass ACROSS PROCESSES.
 *
 * The in-process dedupe map only covers one replica, and `replicas: 1` with the default
 * RollingUpdate means two control-api pods coexist on every deploy. Without this, their
 * mints interleave: A mints k1, B mints k2 (revoking k1), and whichever writes last leaves
 * some namespaces holding a revoked key. The fingerprint check makes that self-healing,
 * but only on the NEXT pass — pulls fail until then.
 *
 * `pg_advisory_xact_lock` blocks rather than failing, which is what we want: the second
 * pod waits, then re-reads and finds a valid, agreeing credential, so it returns
 * `exists-ours` without minting at all. The lock auto-releases on commit/rollback, so it
 * cannot leak — the same mechanism the registry uses inside `mintPullKey`, and that
 * `traceMaintenance` uses for cross-replica dedup.
 *
 * The transaction is held across the K8s and registry calls inside `work`. That is
 * deliberate and bounded: the mint carries a timeout and the K8s calls are few.
 */
async function withPullSecretLock<T>(orgName: string, work: () => Promise<T>): Promise<T> {
  return withTransaction(async db => {
    await db.query('SELECT pg_advisory_xact_lock(hashtext($1))', [
      `registry-pull-secret:${orgName}`,
    ])
    return work()
  })
}

/** An empty pass, ready to be recorded into. */
function newPass(): EnsurePass {
  return {
    results: new Map(),
    unusable: new Map(),
    blocked: new Map(),
    missing: new Map(),
    taken: new Map(),
  }
}

async function ensureInner(gateway: K8sGateway, targets: string[]): Promise<EnsurePass> {
  // One record-and-decide accumulator threaded through the pass, rather than a growing list
  // of positional Maps: which reason a namespace lands in is the whole output of this
  // service, and every writer needs to reach more than one of them.
  const pass = newPass()

  // ── Legitimate no-ops ─────────────────────────────────────────────────────
  // No registry host: nothing references it, and there is no host to check a blob
  // against either — the pull secret is not in play at all.
  const host = registryHostFromUrl(config.registryUrl)
  if (!host) {
    for (const ns of targets) pass.results.set(ns, 'skipped')
    return pass
  }
  // Managed clusters: the operator owns this Secret; never contend with it — but do check
  // that what the caller is about to reference is actually there.
  if (config.registryConnectionMode !== 'self-hosted') {
    return verifyOperatorProvisioned(gateway, targets, host)
  }

  // ── Hard preconditions: from here on we MUST provision or fail loudly ─────
  // The pull Secret is a registry credential. Confine writes to the platform workload
  // namespaces so a caller-supplied `body.namespace` can never plant it elsewhere (and
  // can never rotate the org key as a side effect).
  const allowed = new Set(platformWorkloadNamespaces())
  const rogue = targets.filter(ns => !allowed.has(ns))
  if (rogue.length > 0) {
    throw new PullSecretProvisionError(
      `refusing to provision the registry pull secret outside the platform workload namespaces (got ${rogue.map(n => `"${n}"`).join(', ')}, allowed ${[...allowed].map(n => `"${n}"`).join(', ')})`,
      'unsupported_namespace',
      400
    )
  }
  if (!(await isRegistryAuthActive())) {
    throw new PullSecretProvisionError(
      'registry connection is not active; complete the registry connect flow before installing a private plugin',
      'registry_not_connected',
      409
    )
  }
  // A cold-started control-api can cache a null org from before the registry bound this
  // client. Force one refresh before failing, mirroring the self-service key routes.
  let { orgName } = await resolvePublishScope()
  if (!orgName) ({ orgName } = await resolvePublishScope({ force: true }))
  if (!orgName) {
    throw new PullSecretProvisionError(
      'registry has not bound this deployment to an organization yet; cannot mint an image-pull credential',
      'org_unresolved',
      409
    )
  }

  // Everything from here — read, classify, mint, write — must be serialized across
  // replicas, not just within this process (see withPullSecretLock).
  return withPullSecretLock(orgName, () =>
    ensureLocked(gateway, targets, host, orgName as string, pass)
  )
}

/**
 * MANAGED clusters: read-only verification that the operator's Secret is where the caller
 * is about to point a workload at it. Mints nothing, writes nothing, takes no lock, and
 * touches neither Postgres nor the registry — invariant 1 admits no exception here, and
 * "the operator owns it" is precisely why we may only look.
 *
 * Reuses `foreignUsabilityProblem`, the same shape-only test applied to externally-owned
 * Secrets on the self-hosted path: correct type, and a dockerconfig entry for our host. A
 * second notion of "usable" would drift from the one the kubelet actually acts on. Ownership
 * is deliberately NOT consulted — whoever wrote it, all that matters is whether it can serve
 * the pull — and neither is liveness of the sealed key, which would cost a registry
 * round-trip per install.
 *
 * Scoped to `platformWorkloadNamespaces()`: that set is the operator contract. A target
 * outside it stays `'skipped'` rather than becoming a failure, because control-api has no
 * claim about namespaces no operator ever agreed to populate, and failing there would break
 * managed installs that work today.
 */
async function verifyOperatorProvisioned(
  gateway: K8sGateway,
  targets: string[],
  host: string
): Promise<EnsurePass> {
  const pass = newPass()
  const platform = new Set(platformWorkloadNamespaces())
  for (const ns of targets) {
    // Provisioning is genuinely not our job on this cluster, so the result stays 'skipped'
    // whichever way the check goes; the verdict travels in `missing`.
    pass.results.set(ns, 'skipped')
    if (!platform.has(ns)) continue
    const current = await readCurrent(gateway, ns)
    if (!current) {
      pass.missing.set(ns, 'it does not exist')
      continue
    }
    const problem = foreignUsabilityProblem(current, host)
    if (problem) pass.missing.set(ns, `it exists but ${problem}`)
  }
  // Deliberately not logged at warn: this runs on every reconcile tick as well as every
  // install, and on a managed cluster an unpopulated namespace is a normal state until
  // something actually asks for it. The caller that asks gets the loud failure.
  if (pass.missing.size > 0) {
    logger.debug(
      { namespaces: [...pass.missing.keys()] },
      'managed cluster: no usable operator-provisioned evenfire-registry-pull Secret in these namespaces'
    )
  }
  return pass
}

/**
 * The read -> classify -> mint -> write pass. Runs while holding the cross-process lock,
 * so it may assume no other replica is minting for this org concurrently.
 */
async function ensureLocked(
  gateway: K8sGateway,
  targets: string[],
  host: string,
  orgName: string,
  // `pass.missing` is always empty on this path: a namespace with nothing in it is one we
  // PROVISION, not one we report as missing. It rides along so both passes return the same
  // shape.
  pass: EnsurePass
): Promise<EnsurePass> {
  const { results: out, unusable, blocked } = pass
  // ── Read + classify every target BEFORE minting anything ──────────────────
  const states = new Map<string, TargetState>()
  for (const ns of targets) {
    states.set(ns, classify(await readCurrent(gateway, ns), host))
  }

  // ── Foreign targets: hands off, but not a free pass ───────────────────────
  // Every caller of this service is about to attach an `imagePullSecrets` reference to
  // this name and persist a CRD, so a foreign Secret the kubelet cannot use is exactly as
  // fatal as a missing one — and we may not repair it. We RECORD that here rather than
  // throwing: the pass spans the whole platform set for the mint-once collapse, but only a
  // caller that actually references the namespace should be failed by it (see
  // `ensureRegistryPullSecrets`, which raises `foreign_secret_unusable` for its own
  // required set). A well-shaped foreign Secret proceeds untouched either way.
  const foreign = targets.filter(ns => states.get(ns)!.kind === 'foreign')
  const ours = targets.filter(ns => states.get(ns)!.kind !== 'foreign')
  for (const ns of foreign) {
    const state = states.get(ns) as { kind: 'foreign'; unusable: string | null }
    if (state.unusable) {
      unusable.set(ns, state.unusable)
      logger.warn(
        { namespace: ns, problem: state.unusable },
        'externally-owned evenfire-registry-pull Secret cannot serve an image pull; leaving it untouched — installs that reference this namespace will fail until it is fixed or removed'
      )
    } else {
      logger.warn(
        { namespace: ns },
        'externally-owned evenfire-registry-pull Secret present; leaving it untouched (remove it to let control-api manage this namespace)'
      )
    }
    out.set(ns, 'exists-foreign')
  }
  if (ours.length === 0) return pass

  // A mint is needed when any namespace we own lacks a usable credential, OR when the
  // copies have diverged — divergence means an earlier pass wrote some namespaces and
  // failed on others, and we cannot recover the live key from a fingerprint.
  const valid = ours.filter(ns => states.get(ns)!.kind === 'valid')
  const fingerprints = new Set(
    valid.map(ns => (states.get(ns) as { fingerprint: string }).fingerprint)
  )
  const diverged = fingerprints.size > 1
  const needsMint = valid.length !== ours.length || diverged

  if (!needsMint) {
    for (const ns of ours) out.set(ns, 'exists-ours')
    return pass
  }

  // ── All-or-nothing: a USABLE foreign copy anywhere forbids the mint ───────
  // The credential is per-ORG and the mint is rotate-on-call, so minting for a namespace we
  // own would revoke the key inside the foreign Secret we just agreed not to touch. That
  // damage is invisible — the foreign bytes are unchanged, the foreign fingerprint is not
  // in the divergence set above, and invariant 1 forbids repairing it — so the only safe
  // move is to mint nothing and report the namespaces we could not fill. Owned copies that
  // are already current are untouched by this: they needed no mint, so `needsMint` above
  // has already returned them as `exists-ours`.
  //
  // The gate is USABILITY, not foreignness. An UNUSABLE foreign Secret cannot be serving a
  // pull for this registry by construction — the kubelet ignores a wrong `type` outright,
  // and never selects a blob carrying no entry for our host — so rotating the org key
  // cannot break a working path through it. Blocking on it would only fail installs that
  // never reference its namespace, which is the scoping this service already decided
  // against. It stays recorded in `unusable`, so the callers that DO reference it still
  // fail on it (`foreign_secret_unusable`).
  const blocking = foreign.filter(
    ns => (states.get(ns) as { unusable: string | null }).unusable === null
  )
  if (blocking.length > 0) {
    // Under divergence every owned copy needs the rewrite, not just the invalid ones.
    const unfillable = diverged ? ours : ours.filter(ns => states.get(ns)!.kind !== 'valid')
    const externals = blocking.map(n => `"${n}"`).join(', ')
    const all = platformWorkloadNamespaces()
      .map(n => `"${n}"`)
      .join(', ')
    for (const ns of unfillable) {
      blocked.set(
        ns,
        `cannot provision the ${EVENFIRE_REGISTRY_PULL_SECRET_NAME} Secret in namespace "${ns}": an externally-owned copy of it exists in ${externals}, and this organization's pull credential is rotate-on-call — minting one for "${ns}" would revoke the key inside that external Secret and break every pod already pulling with it. Either provision ${EVENFIRE_REGISTRY_PULL_SECRET_NAME} externally in ALL of ${all}, or delete the external copy in ${externals} so control-api can manage every namespace itself.`
      )
    }
    // The owned copies we did NOT block are genuinely current — nothing rotated their key —
    // so report them honestly. Leaving them out would fall through the wrapper's
    // `?? 'skipped'` and claim provisioning was never our job.
    for (const ns of ours) {
      if (!blocked.has(ns)) out.set(ns, 'exists-ours')
    }
    logger.warn(
      { external: blocking, unfillable },
      'usable externally-owned evenfire-registry-pull Secret present; refusing to mint (the mint is org-wide and rotate-on-call, so it would revoke the external key) — installs that need the unfillable namespaces will fail'
    )
    return pass
  }

  if (diverged) {
    logger.warn(
      { namespaces: ours },
      'registry pull secret copies diverged across namespaces; re-minting to converge'
    )
  }

  // `Secret.type` is immutable, so a wrong-typed Secret must be deleted and recreated.
  // Do every delete BEFORE minting: a delete that fails must not leave us holding a
  // freshly-minted key that has already revoked the credential the cluster was using.
  //
  // A delete acts on a name, not on the object we classified, so re-prove ownership first
  // (`recheckOwnership`). This is the destructive half of the TOCTOU: without it, a Secret
  // replaced between the classify read and here is deleted on the strength of a decision
  // made about something that no longer exists — an external operator's credential removed
  // by a reconcile tick.
  const recreate = new Set<string>()
  const surrendered = new Set<string>()
  for (const ns of ours) {
    const state = states.get(ns)!
    if (state.kind !== 'broken' || !state.wrongType) continue
    const outcome = await mutateOwned(gateway, ns, state.observed, precondition =>
      gateway.deleteSecret(EVENFIRE_REGISTRY_PULL_SECRET_NAME, ns, precondition)
    )
    if (outcome.verdict === 'taken') {
      recordTakeover(pass, ns, outcome.why)
      surrendered.add(ns)
      continue
    }
    // 'gone': already deleted by someone else, so there is nothing to delete — but the
    // namespace still needs the credential, and `recreate` is exactly "create, do not
    // update" downstream. Falling through to the update path would 404.
    recreate.add(ns)
  }
  // Namespaces that changed hands are no longer ours to write. Every caller that references
  // one fails on `ownership_changed`; the rest of the pass proceeds for the namespaces we
  // still own, because they need the credential regardless.
  //
  // KNOWN RESIDUAL: a copy that turns up here is NOT fed back into the all-or-nothing mint
  // block above, so if it is a usable foreign Secret the mint below can still revoke the key
  // sealed in it. Closing that would mean re-classifying the whole set and rebuilding the
  // `blocked` messages mid-pass, for a window measured in one round trip — and the same
  // residual already exists for a foreign Secret that appears any time after the classify
  // read, including on the create-race path in `writeCredential`. What is fixed here is the
  // part that is unrecoverable: we no longer DELETE or OVERWRITE somebody else's object.
  const writable = ours.filter(ns => !surrendered.has(ns))
  if (writable.length === 0) return pass

  // ── Mint ONCE, then fan the same credential out to every namespace we own ──
  // Namespaces that were already valid are rewritten too: the mint just revoked the key
  // they were holding.
  const cred = await mintCredential(orgName, host)
  for (const ns of writable) {
    const state = states.get(ns)!
    const outcome = await writeCredential(gateway, ns, state, cred, recreate.has(ns), {
      host,
      pass,
    })
    out.set(ns, outcome)
  }

  auditPullSecret('registry_pull_secret_provisioned', {
    namespaces: writable,
    org: orgName,
    fingerprint: cred.fingerprint,
  })
  logger.info({ namespaces: writable, orgName }, 'registry pull secret provisioned')
  return pass
}

/**
 * Record a namespace whose Secret changed hands between classification and mutation.
 *
 * Recorded, not thrown, for the same reason `unusable` is: the pass spans the whole platform
 * set so the mint can be collapsed across callers, and only a caller that actually
 * references this namespace should be failed by what happens in it.
 *
 * The result is `'exists-foreign'` because that is now literally true — someone else owns
 * that name — and it keeps `results` complete, so a caller that does NOT require this
 * namespace gets a truthful answer instead of falling through the single-namespace
 * wrapper's `?? 'skipped'` (which would claim provisioning was never our job).
 */
function recordTakeover(pass: EnsurePass, ns: string, why: string): void {
  pass.taken.set(ns, why)
  pass.results.set(ns, 'exists-foreign')
  logger.warn(
    { namespace: ns, problem: why },
    'evenfire-registry-pull Secret changed owner between classification and write; leaving it untouched — installs that reference this namespace will fail until the cluster settles'
  )
}

/** Write `cred` into one namespace, choosing create/update/recreate from its prior state. */
async function writeCredential(
  gateway: K8sGateway,
  ns: string,
  state: TargetState,
  cred: MintedCredential,
  alreadyDeleted: boolean,
  // The whole pass, so a foreign Secret discovered on either race path below is recorded
  // exactly as one discovered by `classify` would be — see the comments there.
  ctx: { host: string; pass: EnsurePass }
): Promise<EnsurePullSecretResult> {
  // Wrong-typed Secrets were deleted before the mint (see ensureInner), so they are now
  // absent and must be created, not updated.
  if (alreadyDeleted) {
    await gateway.createSecret(buildSecretReq(ns, cred), PLATFORM_WRITE)
    return 'repaired'
  }
  if (state.kind === 'absent') {
    try {
      await gateway.createSecret(buildSecretReq(ns, cred), PLATFORM_WRITE)
      return 'created'
    } catch (err) {
      if (k8sStatus(err) !== 409) throw err
      // Lost a create race — but NOT against ourselves: concurrent passes in this replica
      // collapse on the `inflight` key. The winner is therefore another writer (a second
      // replica, which this design does not support, or an out-of-band actor), and which
      // key is live is not knowable here — ours revoked theirs only if we minted last.
      // Write ours anyway, so this pass leaves every namespace it owns on ONE key;
      // adopting the winner's would split this namespace from the rest for a credential we
      // can verify no better. Skip if the winner is foreign — invariant 1 still holds.
      const winner = await readCurrent(gateway, ns)
      if (!winner) {
        // Created and deleted again before this read. The Secret is absent, and an update
        // against an absent Secret 404s — so create, do not fall through.
        await gateway.createSecret(buildSecretReq(ns, cred), PLATFORM_WRITE)
        return 'created'
      }
      if (winner.labels[MANAGED_BY_LABEL] !== MANAGED_BY_VALUE) {
        // The all-or-nothing scan in ensureInner ran before this Secret existed, so the
        // mint has already happened and cannot be taken back. What we CAN still do is give
        // this namespace the same verdict the classify path would: a foreign Secret the
        // kubelet will ignore must fail the callers that reference it, not sail past them
        // and leave a CRD pointing at a credential no pod can use.
        const problem = foreignUsabilityProblem(winner, ctx.host)
        if (problem) {
          ctx.pass.unusable.set(ns, problem)
          logger.warn(
            { namespace: ns, problem },
            'lost the create race to an externally-owned evenfire-registry-pull Secret that cannot serve an image pull; leaving it untouched — installs that reference this namespace will fail until it is fixed or removed'
          )
        }
        return 'exists-foreign'
      }
      // The winner is ours, so adopt it — but bind the write to the object we just read.
      // "Read `winner`, then update by name" is the same TOCTOU as everywhere else on this
      // path: the ownership check and the replace are two requests, and `updateSecret`
      // without a precondition re-reads and wins regardless of what landed in between. A
      // create race is precisely when a third writer is known to be active on this name, so
      // this is the LAST place to assume the gap is safe.
      const adopted = await mutateOwned(gateway, ns, winner, precondition =>
        gateway.updateSecret(buildSecretReq(ns, cred), precondition, PLATFORM_WRITE)
      )
      if (adopted.verdict === 'taken') {
        recordTakeover(ctx.pass, ns, adopted.why)
        return 'exists-foreign'
      }
      if (adopted.verdict === 'gone') {
        // Deleted again in the gap. Nobody holds the name and the namespace still needs a
        // credential, so create rather than leaving it empty.
        await gateway.createSecret(buildSecretReq(ns, cred), PLATFORM_WRITE)
      }
      return 'created'
    }
  }
  if (state.kind === 'foreign') {
    // Unreachable: foreign targets are filtered out of `ours` before the write loop. Kept as
    // a typed dead end rather than a cast, so a future caller that forgets the filter cannot
    // turn invariant 1 into a runtime accident.
    return 'exists-foreign'
  }
  // The update path. Left to itself `SecretService.updateSecret` re-reads to pick up the
  // latest `resourceVersion` and therefore ALWAYS wins the write — it proves the object is
  // current, never that it is still ours. `mutateOwned` binds the replace to the identity we
  // classified instead, so the apiserver refuses it if the object moved.
  const outcome = await mutateOwned(gateway, ns, state.observed, precondition =>
    gateway.updateSecret(buildSecretReq(ns, cred), precondition, PLATFORM_WRITE)
  )
  if (outcome.verdict === 'taken') {
    recordTakeover(ctx.pass, ns, outcome.why)
    return 'exists-foreign'
  }
  if (outcome.verdict === 'gone') {
    // Deleted under us. NOT a takeover — nobody else holds the name — and the mint above has
    // already revoked whatever key that copy carried, so leaving the namespace empty is the
    // one outcome guaranteed to ImagePullBackOff. Create, because an update against an
    // absent Secret 404s.
    await gateway.createSecret(buildSecretReq(ns, cred), PLATFORM_WRITE)
    return 'created'
  }
  return state.kind === 'valid' ? 'exists-ours' : 'repaired'
}

/** Read the Secret, mapping 404 to null. Any other error propagates (e.g. 403 = wrong ns). */
async function readCurrent(gateway: K8sGateway, targetNs: string): Promise<SecretView | null> {
  try {
    return readSecret(await gateway.getSecret(EVENFIRE_REGISTRY_PULL_SECRET_NAME, targetNs))
  } catch (err) {
    if (k8sStatus(err) === 404) return null
    throw err
  }
}