Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

chore(control-api,docs): close PR 231 audit follow-ups

Alfredo Lopez02147a0c634c
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index bc6af46e0..02142093e 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -184,3 +184,11 @@ observe a newer same-binding access token but must never refresh/reissue or
consume the Host refresh-token lineage.
Do not widen the command, switch clusters, reset PVCs, or delete locks with a
live owner. Code-by-code guidance is in `reference.md`.
+
+The active profile lock is `$T2_LOCK_ROOT/<profile>.lock`; stale reclaim uses
+the sibling `$T2_LOCK_ROOT/<profile>.reclaim`. A killed reclaimer can leave the
+sibling claim. After verifying that the owner PID, reclaimer process, and all
+other profile-mutating sessions are gone, recover only the exact paths: use
+`rmdir -- "$T2_LOCK_ROOT/<profile>.reclaim"` for an empty claim and
+`rm -rf -- "$T2_LOCK_ROOT/<profile>.lock"` for the lock. If only one exists,
+operate only on that one. Never remove a live owner/reclaimer or the lock root.
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index f628734ec..4e76a9da0 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -91,6 +91,31 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,
| `NP08_HCC_AUTHORIZATION_FAILED` | The deployed Host-to-HCC authorization journey or cleanup failed after T0/T1 passed. | Repair the first NP08 failure, then use `minikube-t2-runtime` on the exact same profile/HEAD; never refresh/reissue in the test. |
| `PLAYWRIGHT_FAILED` | The opt-in user-visible journey failed after runtime and lane evidence passed. | Repair the journey and retry with `minikube-t2-runtime` on the same exact tuple. |
+## Stale-lock reclaim layout
+
+The active profile lock is `$T2_LOCK_ROOT/<profile>.lock`. The atomic stale
+reclaim claim is its sibling `$T2_LOCK_ROOT/<profile>.reclaim`, never a child of
+the active lock. A reclaimer terminated abruptly may leave that sibling
+directory, so it is intentionally fail-closed rather than silently adopted.
+
+Before manual recovery, verify all of the following:
+
+- the recorded lock owner PID is not alive;
+- no stale-lock reclaimer process is alive;
+- no other session is mutating the profile.
+
+After those checks, operate only on the two exact paths. Prefer `rmdir` for the
+empty claim:
+
+```bash
+rmdir -- "$T2_LOCK_ROOT/<profile>.reclaim"
+rm -rf -- "$T2_LOCK_ROOT/<profile>.lock"
+```
+
+If only the claim exists, run only the `rmdir`; if only the lock exists, run
+only the `rm -rf`. Never remove either path with a live owner/reclaimer and
+never remove all of `T2_LOCK_ROOT`.
+
## GFS restore pointers (high level, no secrets)
GFS T1 real-Postgres coverage has its own gate
diff --git a/AGENTS.md b/AGENTS.md
index 38e23d3a3..f30cede07 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -142,6 +142,17 @@ T2 with `make minikube-t2-runtime` (`T2_RUN_T0=false T2_RUN_T1=false`). That
path is valid only when the pre-gate marker already matches HEAD
(`already-synced`). Do not set those flags to skip an uncertified lane.
+The active profile mutation lock is `$T2_LOCK_ROOT/<profile>.lock`; stale-lock
+reclaim coordination uses the sibling directory
+`$T2_LOCK_ROOT/<profile>.reclaim`. A killed reclaimer may leave that sibling
+claim behind. Before manual recovery, verify that the recorded owner PID, any
+reclaimer process, and every other profile-mutating session are gone. Operate
+only on those two exact paths: use `rmdir -- "$T2_LOCK_ROOT/<profile>.reclaim"`
+for an empty claim and
+`rm -rf -- "$T2_LOCK_ROOT/<profile>.lock"` for the stale lock. If only one
+path exists, remove only that path; never remove either path while its owner or
+reclaimer is live, and never remove the lock root.
+
A fresh or uninitialized profile must complete the supported bootstrap. The
standalone preflight refuses to call `pre-gate-sync` on an incomplete profile;
`make minikube-t2` runs the supported setup when its planner selects
diff --git a/control-api/package-lock.json b/control-api/package-lock.json
index 41d8754c6..7c22a97b4 100644
--- a/control-api/package-lock.json
+++ b/control-api/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-control-api",
- "version": "0.1.493",
+ "version": "0.1.494",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-control-api",
- "version": "0.1.493",
+ "version": "0.1.494",
"dependencies": {
"@clerum/display-field": "file:../packages/display-field",
"@clerum/image-policy": "file:../packages/image-policy",
diff --git a/control-api/package.json b/control-api/package.json
index e27bd427d..e1c0171a7 100644
--- a/control-api/package.json
+++ b/control-api/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-control-api",
- "version": "0.1.493",
+ "version": "0.1.494",
"description": "Internal Clerum control plane REST API + UI for managing CRDs and secrets",
"main": "dist/main.js",
"engines": {
diff --git a/control-api/src/routes/admin/communicationChannelCredentials.ts b/control-api/src/routes/admin/communicationChannelCredentials.ts
index a86fed956..ead3db77a 100644
--- a/control-api/src/routes/admin/communicationChannelCredentials.ts
+++ b/control-api/src/routes/admin/communicationChannelCredentials.ts
@@ -121,7 +121,7 @@ export function registerCommunicationChannelCredentialsRoutes(
secretName: existingSecretName,
namespace: ns,
rotated: true,
- rotatedKeys: Object.keys(data),
+ rotatedKeys: Object.keys(data).sort((a, b) => a.localeCompare(b)),
})
return
}
@@ -146,7 +146,7 @@ export function registerCommunicationChannelCredentialsRoutes(
secretName,
namespace: ns,
rotated: false,
- rotatedKeys: Object.keys(data),
+ rotatedKeys: Object.keys(data).sort((a, b) => a.localeCompare(b)),
})
})
)
diff --git a/control-api/test/routes.adminCommunicationChannelCredentials.test.ts b/control-api/test/routes.adminCommunicationChannelCredentials.test.ts
index 33722490a..9c3a5c154 100644
--- a/control-api/test/routes.adminCommunicationChannelCredentials.test.ts
+++ b/control-api/test/routes.adminCommunicationChannelCredentials.test.ts
@@ -231,12 +231,12 @@ describe('admin communicationchannels — credentials cascade', () => {
gatewayMock.mergeSecret.mockResolvedValue({
name: 'cc-foo-credentials',
namespace: 'channels',
- keys: ['telegram-bot-token'],
+ keys: ['slack-bot-token', 'telegram-bot-token'],
})
const res = await request(makeApp())
.put('/admin/communication-channels/foo/credentials')
- .send({ 'telegram-bot-token': 'new-token' })
+ .send({ 'telegram-bot-token': 'new-token', 'slack-bot-token': 'slack-token' })
expect(res.status).toBe(200)
expect(res.body).toEqual({
@@ -244,13 +244,13 @@ describe('admin communicationchannels — credentials cascade', () => {
secretName: 'cc-foo-credentials',
namespace: 'channels',
rotated: true,
- rotatedKeys: ['telegram-bot-token'],
+ rotatedKeys: ['slack-bot-token', 'telegram-bot-token'],
})
expect(gatewayMock.mergeSecret).toHaveBeenCalledWith(
expect.objectContaining({
name: 'cc-foo-credentials',
namespace: 'channels',
- stringData: { 'telegram-bot-token': 'new-token' },
+ stringData: { 'telegram-bot-token': 'new-token', 'slack-bot-token': 'slack-token' },
})
)
// Existing-ref branch must NOT recreate the Secret or rewrite the CC.
@@ -293,9 +293,10 @@ describe('admin communicationchannels — credentials cascade', () => {
},
})
- const res = await request(makeApp())
- .put('/admin/communication-channels/foo/credentials')
- .send({ 'telegram-bot-token': NEW_TOKEN_PLAIN })
+ const res = await request(makeApp()).put('/admin/communication-channels/foo/credentials').send({
+ 'telegram-bot-token': NEW_TOKEN_PLAIN,
+ 'slack-bot-token': SLACK_TOKEN_PLAIN,
+ })
expect(res.status).toBe(200)
// Names-only contract: metadata + the rotated key NAMES the caller sent.
@@ -304,7 +305,7 @@ describe('admin communicationchannels — credentials cascade', () => {
secretName: 'cc-foo-credentials',
namespace: 'channels',
rotated: true,
- rotatedKeys: ['telegram-bot-token'],
+ rotatedKeys: ['slack-bot-token', 'telegram-bot-token'],
})
expect(res.body).not.toHaveProperty('result')
// Belt-and-suspenders: no secret value (base64 or plaintext) at ANY depth,
@@ -335,7 +336,7 @@ describe('admin communicationchannels — credentials cascade', () => {
const res = await request(makeApp())
.put('/admin/communication-channels/foo/credentials')
- .send({ 'telegram-bot-token': 'tok' })
+ .send({ 'telegram-bot-token': 'tok', 'slack-bot-token': 'slack-tok' })
expect(res.status).toBe(200)
// Names-only for the rotated:false branch too: return the written key NAMES,
@@ -345,14 +346,14 @@ describe('admin communicationchannels — credentials cascade', () => {
secretName: 'cc-foo-credentials',
namespace: 'channels',
rotated: false,
- rotatedKeys: ['telegram-bot-token'],
+ rotatedKeys: ['slack-bot-token', 'telegram-bot-token'],
})
expect(res.body).not.toHaveProperty('result')
expect(gatewayMock.createSecret).toHaveBeenCalledWith(
expect.objectContaining({
name: 'cc-foo-credentials',
namespace: 'channels',
- stringData: { 'telegram-bot-token': 'tok' },
+ stringData: { 'telegram-bot-token': 'tok', 'slack-bot-token': 'slack-tok' },
})
)
expect(gatewayMock.updateResource).toHaveBeenCalledWith(
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index 9c192d18b..f4702e599 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -97,12 +97,14 @@ after the reconcile.
The lock is keyed by repository, branch, `HEAD`, and profile, while the
profile-level lock prevents two processes from mutating one profile at the
-same time. The lock record contains only local metadata and is removed by
-success, failure, timeout, and interrupt traps. A live owner produces
-`PROFILE_BUSY`; stale metadata may be reclaimed only after its recorded PID is
-no longer running. Reclaimers serialize through an atomic `.reclaim` directory
-inside that exact profile lock. A concurrent loser fails closed and never
-removes either the stale directory or the winner's replacement lock.
+same time. The active lock is `$T2_LOCK_ROOT/<profile>.lock`; its record
+contains only local metadata and is removed by success, failure, timeout, and
+interrupt traps. A live owner produces `PROFILE_BUSY`; stale metadata may be
+reclaimed only after its recorded PID is no longer running. Reclaimers
+serialize through the atomic sibling directory
+`$T2_LOCK_ROOT/<profile>.reclaim`, never a child of the active lock. A
+concurrent loser fails closed and never removes either the stale directory or
+the winner's replacement lock.
The pre-gate marker must contain the current worktree identifier, exact `HEAD`,
cluster fingerprint, image coordinate, and the exact `imagesGeneratedAt` value
@@ -125,17 +127,22 @@ retries to 1-10, and retry delay to 0-300 seconds; empty successful
### Orphaned lock recovery
If `PROFILE_BUSY` reports that the lock has no valid owner PID, or that a stale
-reclaim is already in progress, first verify that neither the recorded owner
-nor a reclaimer process owns the profile. A `.reclaim` directory can remain
-after a reclaimer is killed and is intentionally not auto-reclaimed. After
-those checks, remove only the exact profile lock directory and retry the same
-command:
+reclaim is already in progress, first verify that the recorded owner PID is not
+alive, that no reclaimer process is alive, and that no other session is
+mutating the profile. A `$T2_LOCK_ROOT/<profile>.reclaim` sibling directory can
+remain after a reclaimer is killed and is intentionally not auto-reclaimed.
+After those checks, operate only on the two exact profile paths:
```bash
+rmdir -- "$T2_LOCK_ROOT/<profile>.reclaim"
rm -rf -- "$T2_LOCK_ROOT/<profile>.lock"
```
-Never remove a lock with a live owner, and never remove the whole lock root.
+Use only the command for a path that exists: if only the claim exists, run
+`rmdir`; if only the lock exists, run `rm -rf`; if both exist, remove the empty
+claim first and then the stale lock. `rmdir` is preferred for the claim because
+it must be empty. Never remove either path with a live owner/reclaimer, and
+never remove the whole lock root.
## Preconditions and secrets
diff --git a/scripts/minikube/profile-owner.sh b/scripts/minikube/profile-owner.sh
index f44e81573..d24abeb27 100755
--- a/scripts/minikube/profile-owner.sh
+++ b/scripts/minikube/profile-owner.sh
@@ -93,12 +93,13 @@ profile_owner_validate_branch() {
}
profile_owner_validate_profile_name() {
- local profile="${1:-}"
+ local profile="${1:-}" normalized_profile
if [[ ${#profile} -gt 63 || ! "${profile}" =~ ^[a-zA-Z0-9][a-zA-Z0-9._-]*$ ]]; then
profile_owner_error PROFILE_METADATA_INVALID 'profile name is not a safe local Minikube identifier'
return 1
fi
- case "${profile}" in
+ normalized_profile="$(printf '%s' "${profile}" | LC_ALL=C tr '[:upper:]' '[:lower:]')"
+ case "${normalized_profile}" in
*gke*|*prod*|*staging*|clerum-test|default|minikube)
profile_owner_error PROFILE_METADATA_INVALID "profile is shared or protected: ${profile}"
return 1 ;;
diff --git a/scripts/minikube/t2-common.sh b/scripts/minikube/t2-common.sh
index f38e7c08e..10fc64d86 100755
--- a/scripts/minikube/t2-common.sh
+++ b/scripts/minikube/t2-common.sh
@@ -972,7 +972,7 @@ t2_lock_acquire() {
t2_fail PROFILE_BUSY "profile lock path is not a directory: $T2_LOCK_DIR"
return 1
fi
- local existing_pid existing_token existing_start reclaim_dir
+ local existing_pid existing_token existing_start
existing_pid="$(t2_lock_owner_value PID || true)"
existing_token="$(t2_lock_owner_value TOKEN || true)"
existing_start="$(t2_lock_owner_value PROCESS_START || true)"
diff --git a/scripts/tests/test-minikube-profile-owner.sh b/scripts/tests/test-minikube-profile-owner.sh
index b33132b6d..140cb3b3a 100755
--- a/scripts/tests/test-minikube-profile-owner.sh
+++ b/scripts/tests/test-minikube-profile-owner.sh
@@ -188,6 +188,26 @@ ports_only_legacy="$("${OWNER_SCRIPT}" resolve --repo-dir "${repo_a}" --branch "
--profile-root "${v1_root}" --ports-env "${legacy_dir}/ports.env")"
[[ "$(value_from "${ports_only_legacy}" PROFILE)" == "${legacy_profile}" ]] || fail 'explicit ports.env did not bind to its sibling profile metadata'
+valid_profile='clerum-feature-local'
+valid_dir="${TMP_ROOT}/${valid_profile}"
+mkdir -p "${valid_dir}"
+write_v2_profile "${valid_dir}/profile.env" "${valid_profile}" "${repo_a}" "${branch}" "${head_a}"
+write_ports "${valid_dir}/ports.env" 26500
+"${OWNER_SCRIPT}" validate --repo-dir "${repo_a}" --branch "${branch}" \
+ --profile "${valid_profile}" --profile-env "${valid_dir}/profile.env" \
+ --ports-env "${valid_dir}/ports.env" >/dev/null || fail 'valid safe profile was rejected'
+
+for protected_profile in Prod PROD-local GKE-local Staging CLERUM-TEST Default MINIKUBE; do
+ protected_dir="${TMP_ROOT}/protected-${protected_profile}"
+ mkdir -p "${protected_dir}"
+ write_v2_profile "${protected_dir}/profile.env" "${protected_profile}" "${repo_a}" "${branch}" "${head_a}"
+ write_ports "${protected_dir}/ports.env" 26600
+ expect_failure PROFILE_METADATA_INVALID "case-insensitive protected profile ${protected_profile}" \
+ "${OWNER_SCRIPT}" validate --repo-dir "${repo_a}" --branch "${branch}" \
+ --profile "${protected_profile}" --profile-env "${protected_dir}/profile.env" \
+ --ports-env "${protected_dir}/ports.env"
+done
+
missing_ports_root="${TMP_ROOT}/missing-ports"
missing_ports_dir="${missing_ports_root}/${legacy_profile}"
mkdir -p "${missing_ports_dir}"
diff --git a/scripts/tests/test-minikube-t2-lock-race.sh b/scripts/tests/test-minikube-t2-lock-race.sh
index ff7bfc8b4..dcd81dc03 100755
--- a/scripts/tests/test-minikube-t2-lock-race.sh
+++ b/scripts/tests/test-minikube-t2-lock-race.sh
@@ -7,6 +7,7 @@ trap 'rm -rf "${TMP_DIR}"' EXIT
LOCK_ROOT="${TMP_DIR}/locks"
LOCK_DIR="${LOCK_ROOT}/race-profile.lock"
+RECLAIM_DIR="${LOCK_ROOT}/race-profile.reclaim"
READY_FILE="${TMP_DIR}/ready"
RESULT_FILE="${TMP_DIR}/result"
RELEASE_FILE="${TMP_DIR}/release"
@@ -14,6 +15,25 @@ mkdir -p "${LOCK_DIR}"
: >"${READY_FILE}"
: >"${RESULT_FILE}"
+for doc in \
+ "${ROOT}/AGENTS.md" \
+ "${ROOT}/.cursor/skills/minikube-t0-t1-t2/SKILL.md" \
+ "${ROOT}/.cursor/skills/minikube-t0-t1-t2/reference.md" \
+ "${ROOT}/docs/testing/minikube-t2-runbook.md"; do
+ grep -Fq '$T2_LOCK_ROOT/<profile>.lock' "${doc}" || {
+ echo "FAIL: stale-lock documentation omits the active lock path: ${doc}" >&2
+ exit 1
+ }
+ grep -Fq '$T2_LOCK_ROOT/<profile>.reclaim' "${doc}" || {
+ echo "FAIL: stale-lock documentation omits the sibling reclaim path: ${doc}" >&2
+ exit 1
+ }
+ if grep -Eq '\.lock/\.reclaim|inside that exact profile lock' "${doc}"; then
+ echo "FAIL: stale-lock documentation nests the reclaim claim under the lock: ${doc}" >&2
+ exit 1
+ fi
+done
+
owner_token_key=TOKEN
{
printf '%s=%s\n' "${owner_token_key}" stale-owner-token
@@ -110,6 +130,10 @@ winner_pid="$(awk '{print $2}' "${RESULT_FILE}")"
echo 'FAIL: losing reclaimer left a claim inside the active replacement lock' >&2
exit 1
}
+[[ ! -e "${RECLAIM_DIR}" ]] || {
+ echo 'FAIL: winning reclaimer left the sibling claim behind after acquiring the lock' >&2
+ exit 1
+}
: >"${RELEASE_FILE}"
set +e