Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Admit human action dispatch under the policy coordinator

rcourtmanf00ab32a32e3
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 53b2a60b58..b74ff73ecf 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -5330,6 +5330,36 @@ transport fails closed when the connected agent cannot answer it. Registry or
readiness-check infrastructure failures remain
nonterminal internal errors rather than false permanent refusals.
+Human `Execute` admits through the same policy admission coordinator as
+`ExecuteUnderPolicy`. It holds the admission read lock from the live-readiness
+gates, including the operator remediation lock (`NeverAutoRemediate`) and the
+Retired lifecycle state, through the committed `executing` transition, and
+releases it before the executor's `ExecuteAction` is called, so a slow
+dispatch never blocks a policy save. Executor readiness and binding calls and
+the admission hooks still run under the lock, so they can delay a save and must
+never request a policy write or re-enter `Execute`/`ExecuteUnderPolicy`. An
+operator lock saved through `WithPolicyMutation` therefore either lands before
+the readiness check and refuses the approved action with a persisted
+refused-before-dispatch failure, or queues behind the committed admission. A
+save that has been acknowledged to the operator is never followed by a new
+lifecycle admission (`Execute` or `ExecuteUnderPolicy`) for that resource.
+The committed `executing` transition is the policy linearization point for both
+paths: a lock refuses new admissions and does not recall an admitted
+dispatch. That includes an attempt that was admitted but never sent (a crash
+between admission and send), which `RecoverExecutingActions` resumes without
+re-reading the lock. Recalling known-unsent work would need an atomic,
+conditional pre-send terminalization that neither store has
+(`RecordActionExecutionRefusal` accepts only planned, pending, and approved
+rows, and `ForceFail` records inconclusive truth), so operator-facing copy must
+say that an action already admitted can still run. Proof:
+`TestExecuteAdmissionLinearizesWithOperatorLockSave*` asserts the coordinator
+excludes a policy writer between the readiness check and the admission commit
+and orders a real concurrent lock save against the admission,
+`TestExecuteAdmittedDispatchIsNotRecalledByLaterOperatorLock` pins release
+before dispatch and no recall, and
+`TestRecoverExecutingActionsDoesNotRecallQueuedAttemptAfterOperatorLock` pins
+the unsent-attempt boundary.
+
The public Patrol investigation boundary now carries independent
`max_turns` and `max_evidence_calls` request limits and returns `model_turns`,
`evidence_calls`, and total `tool_calls`. Persisted investigation sessions keep
diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md
index 7831ddf0d7..39579156bc 100644
--- a/docs/release-control/v6/internal/subsystems/security-privacy.md
+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md
@@ -796,8 +796,13 @@ the `white_label` branding entitlement.
action emergency stop are revalidated under one admission coordinator and
committed with `executing`. Missing or unreadable policy denies dispatch.
Human approval remains separate, but `NeverAutoRemediate`, plan drift, and
- emergency stop are universal. Emergency stop after `executing` is only
- best-effort cancellation and must never be described as rollback.
+ emergency stop are universal for every new admission, and human dispatch
+ revalidates them under the same admission coordinator rather than before
+ it, so an acknowledged lock save is never followed by a new lifecycle
+ admission. An already admitted dispatch, including one resumed by restart
+ recovery, is not revalidated. Emergency stop after
+ `executing` is only best-effort cancellation and must never be described
+ as rollback.
The typed host-update executor wired here is likewise not generic command
authority. It may dispatch only the closed `install_os_updates` operation,
bound to the server-observed package inventory fingerprint and canonical
diff --git a/internal/actionlifecycle/service.go b/internal/actionlifecycle/service.go
index a9818bd7e7..a6ca0de5f6 100644
--- a/internal/actionlifecycle/service.go
+++ b/internal/actionlifecycle/service.go
@@ -166,8 +166,14 @@ func (s *Service) admissionCoordinator() *PolicyAdmissionCoordinator {
// lifecycle holds the admission read lock.
type PolicyAuthorizer func(ctx context.Context, record unified.ActionAuditRecord, now time.Time) (unified.ActionPolicyAuthorizationLease, string, error)
-// WithPolicyMutation serializes a policy write against automatic admission.
-// Writers must persist their change before returning.
+// WithPolicyMutation serializes a policy write against dispatch admission,
+// automatic and human alike. Both hold the read side while they revalidate and
+// commit the executing transition, and fire the admission transition hook (and
+// the completion hook for an admission refusal) before releasing it. A write
+// must therefore never be requested from inside those hooks, and a hook must
+// not re-enter Execute or ExecuteUnderPolicy: a second read lock behind a
+// pending writer would deadlock. Writers must persist their change before
+// returning.
func (s *Service) WithPolicyMutation(write func() error) error {
if s == nil || write == nil {
return errors.New("policy mutation unavailable")
@@ -1133,6 +1139,12 @@ func (s *Service) validateApprovalFloor(ctx context.Context, record unified.Acti
// locks are all persisted as refused executions (never silently dropped)
// and published to the completion hook. There is no bypass that reaches
// the executor without passing every gate.
+//
+// The gates and the executing transition run under the same admission
+// coordinator as ExecuteUnderPolicy, so the committed admission is the
+// linearization point against an operator policy save: a lock saved before it
+// refuses the dispatch, a lock saved after it queues behind it and does not
+// recall a dispatch that is already admitted.
func (s *Service) Execute(ctx context.Context, orgID, actionID string, actor unified.ActionActor, reason string) (unified.ActionAuditRecord, error) {
actionID = strings.TrimSpace(actionID)
if actionID == "" {
@@ -1172,25 +1184,47 @@ func (s *Service) Execute(ctx context.Context, orgID, actionID string, actor uni
return record, nil
}
+ // Human admission shares the automatic path's boundary: the operator
+ // remediation lock is read and the executing transition committed under
+ // the admission read lock, so a lock save (which takes the write side via
+ // WithPolicyMutation) either lands before the readiness check below and
+ // refuses the dispatch, or queues behind the committed admission. The lock
+ // is released before the executor runs.
+ coordinator := s.admissionCoordinator()
+ coordinator.mu.RLock()
+ started, admitted, admissionErr := s.beginHumanExecution(ctx, orgID, store, record, actorID, reason)
+ coordinator.mu.RUnlock()
+ if !admitted {
+ return started, admissionErr
+ }
+ return s.dispatchCommitted(ctx, orgID, store, started, actorID)
+}
+
+// beginHumanExecution revalidates every dispatch gate and commits the
+// executing transition for a human-approved action. Callers must hold the
+// admission read lock. A true result means the action is executing and its
+// durable dispatch attempt may be driven, including when a concurrent caller
+// won the admission.
+func (s *Service) beginHumanExecution(ctx context.Context, orgID string, store Store, record unified.ActionAuditRecord, actorID, reason string) (unified.ActionAuditRecord, bool, error) {
now := s.now()
- record, err = s.materializeExpiry(store, orgID, record)
+ record, err := s.materializeExpiry(store, orgID, record)
if err != nil {
- return unified.ActionAuditRecord{}, err
+ return unified.ActionAuditRecord{}, false, err
}
if record.State == unified.ActionStateExpired {
- return record, unified.ErrActionPlanExpired
+ return record, false, unified.ErrActionPlanExpired
}
if err := unified.ValidateActionExecutionStart(record, now); err != nil {
if unified.IsPermanentActionExecutionRefusal(err) {
failed, persistErr := RecordRefusedExecution(store, record, actorID, now, err)
if persistErr != nil {
- return unified.ActionAuditRecord{}, &PersistError{Op: "refused action execution", Err: persistErr}
+ return unified.ActionAuditRecord{}, false, &PersistError{Op: "refused action execution", Err: persistErr}
}
s.publishTransition(orgID, failed)
s.publishCompleted(failed)
- return failed, err
+ return failed, false, err
}
- return unified.ActionAuditRecord{}, err
+ return unified.ActionAuditRecord{}, false, err
}
// Reuse the exact gate set applied before approval. This second check is
// the dispatch-side half of the boundary: it closes the race between the
@@ -1200,46 +1234,43 @@ func (s *Service) Execute(ctx context.Context, orgID, actionID string, actor uni
if unified.IsPermanentActionExecutionRefusal(err) {
failed, persistErr := RecordRefusedExecution(store, record, actorID, now, err)
if persistErr != nil {
- return unified.ActionAuditRecord{}, &PersistError{Op: "refused action execution", Err: persistErr}
+ return unified.ActionAuditRecord{}, false, &PersistError{Op: "refused action execution", Err: persistErr}
}
s.publishTransition(orgID, failed)
s.publishCompleted(failed)
- return failed, err
+ return failed, false, err
}
- return unified.ActionAuditRecord{}, err
+ return unified.ActionAuditRecord{}, false, err
}
started, startEvent, err := unified.BeginActionExecution(record, actorID, now)
if err != nil {
- return unified.ActionAuditRecord{}, err
+ return unified.ActionAuditRecord{}, false, err
}
if reason != "" {
startEvent.Message = "Action execution started: " + reason
}
attempt, err := unified.NewActionDispatchAttempt(started.ID, now)
if err != nil {
- return unified.ActionAuditRecord{}, err
+ return unified.ActionAuditRecord{}, false, err
}
if binder, ok := s.Executor.(DispatchBinder); ok {
attempt, err = binder.BindActionDispatch(ctx, started, attempt)
if err != nil {
- return unified.ActionAuditRecord{}, err
+ return unified.ActionAuditRecord{}, false, err
}
}
if err := store.RecordActionExecutionAdmission(started, startEvent, attempt); err != nil {
if errors.Is(err, unified.ErrActionAlreadyExecuting) || errors.Is(err, unified.ErrActionExecutionFinal) {
- current, found, queryErr := store.GetActionAudit(actionID)
+ current, found, queryErr := store.GetActionAudit(record.ID)
if queryErr == nil && found {
- if current.State == unified.ActionStateExecuting {
- return s.dispatchCommitted(ctx, orgID, store, current, actorID)
- }
- return current, nil
+ return current, current.State == unified.ActionStateExecuting, nil
}
}
- return unified.ActionAuditRecord{}, &PersistError{Op: "action execution start", Err: err}
+ return unified.ActionAuditRecord{}, false, &PersistError{Op: "action execution start", Err: err}
}
s.publishTransition(orgID, started)
- return s.dispatchCommitted(ctx, orgID, store, started, actorID)
+ return started, true, nil
}
// ExecuteUnderPolicy is the only automatic dispatch boundary. It revalidates
diff --git a/internal/actionlifecycle/service_test.go b/internal/actionlifecycle/service_test.go
index fe4dd6ca22..d1d7cc7f5f 100644
--- a/internal/actionlifecycle/service_test.go
+++ b/internal/actionlifecycle/service_test.go
@@ -1500,6 +1500,254 @@ func TestExecuteRefusesRetiredResourceWithoutDispatch(t *testing.T) {
}
}
+// admissionOrderStore stamps the moment the audit store commits an execution
+// admission, so a test can order it against a concurrent policy save.
+type admissionOrderStore struct {
+ unified.ResourceStore
+ seq *atomic.Int64
+ admission atomic.Int64
+}
+
+func (s *admissionOrderStore) RecordActionExecutionAdmission(record unified.ActionAuditRecord, event unified.ActionLifecycleEvent, attempt unified.ActionDispatchAttempt) error {
+ err := s.ResourceStore.RecordActionExecutionAdmission(record, event, attempt)
+ if err == nil {
+ s.admission.Store(s.seq.Add(1))
+ }
+ return err
+}
+
+// windowExecutor runs one hook inside the readiness phase: after the operator
+// lock was read, before admission is committed.
+type windowExecutor struct {
+ *stubExecutor
+ window func()
+}
+
+func (e *windowExecutor) CheckActionAvailable(ctx context.Context, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness {
+ if window := e.window; window != nil {
+ e.window = nil
+ window()
+ }
+ return e.stubExecutor.CheckActionAvailable(ctx, req, resource)
+}
+
+// dispatchHookExecutor runs one hook once the dispatch is durably admitted,
+// immediately before the executor would act.
+type dispatchHookExecutor struct {
+ *stubExecutor
+ onDispatch func()
+}
+
+func (e *dispatchHookExecutor) ExecuteAction(ctx context.Context, record unified.ActionAuditRecord) (*unified.ExecutionResult, error) {
+ if hook := e.onDispatch; hook != nil {
+ e.onDispatch = nil
+ hook()
+ }
+ return e.stubExecutor.ExecuteAction(ctx, record)
+}
+
+func approveForExecution(t *testing.T, service *Service, actionID string) {
+ t.Helper()
+ if _, err := service.Decide(context.Background(), "default", actionID, testActionDecision(t, service, "default", actionID, unified.ActionApprovalRecord{
+ Actor: "[email protected]", Method: unified.MethodAPI, Outcome: unified.OutcomeApproved,
+ })); err != nil {
+ t.Fatalf("Decide: %v", err)
+ }
+}
+
+func saveRemediationLock(store unified.ResourceStore) error {
+ return store.SetResourceOperatorState(unified.ResourceOperatorState{CanonicalID: "vm:42", NeverAutoRemediate: true})
+}
+
+// A lock saved while a human-approved action is between its readiness check and
+// its admission commit must not be missed: the save either lands before the
+// readiness check (the dispatch is refused) or queues behind the admission (the
+// dispatch is already committed). What it may never do is acknowledge to the
+// operator before the admission commits and still let the dispatch through.
+func runExecuteAdmissionLinearizesWithOperatorLockSave(t *testing.T, newStore func(t *testing.T) unified.ResourceStore) {
+ now := time.Now().UTC()
+ var seq atomic.Int64
+ store := &admissionOrderStore{ResourceStore: newStore(t), seq: &seq}
+ executor := &windowExecutor{stubExecutor: &stubExecutor{result: &unified.ExecutionResult{Success: true, Output: "restarted"}}}
+ service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), executor)
+
+ // Both actions are approved up front: once the lock is saved, Decide itself
+ // refuses to record an approval for the locked resource.
+ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
+ if err != nil {
+ t.Fatalf("Plan: %v", err)
+ }
+ approveForExecution(t, service, plan.ActionID)
+ second, err := service.Plan(context.Background(), "default", unified.ActionRequest{
+ RequestID: "req-2", ResourceID: "vm:42", CapabilityName: "restart",
+ Params: map[string]any{"mode": "graceful"}, Reason: "Second restart", RequestedBy: "agent:test",
+ }, testActionActor("requester", "default"))
+ if err != nil {
+ t.Fatalf("second Plan: %v", err)
+ }
+ approveForExecution(t, service, second.ActionID)
+
+ var ackSeq atomic.Int64
+ acked := make(chan struct{})
+ entered := make(chan struct{})
+ saveErr := make(chan error, 1)
+ executor.window = func() {
+ // Deterministic half: a policy writer must be excluded while Execute sits
+ // between the lock read and the admission commit.
+ coordinator := service.admissionCoordinator()
+ if coordinator.mu.TryLock() {
+ coordinator.mu.Unlock()
+ t.Error("a policy writer could enter between the readiness check and the admission commit: Execute is not holding the admission coordinator")
+ }
+ // Concurrent half: a real writer gets its chance to finish inside the
+ // window. A lifecycle holding the coordinator keeps it queued, so the
+ // bounded wait only elapses on the correct path.
+ go func() {
+ err := service.WithPolicyMutation(func() error {
+ close(entered)
+ return saveRemediationLock(store)
+ })
+ ackSeq.Store(seq.Add(1))
+ saveErr <- err
+ close(acked)
+ }()
+ select {
+ case <-acked:
+ case <-entered:
+ waitFor(t, acked, "lock save to acknowledge")
+ case <-time.After(100 * time.Millisecond):
+ }
+ }
+
+ completed, execErr := service.Execute(context.Background(), "default", plan.ActionID, testActionActor("[email protected]", "default"), "approved restart")
+ waitFor(t, acked, "lock save to acknowledge after Execute")
+ if err := <-saveErr; err != nil {
+ t.Fatalf("lock save: %v", err)
+ }
+
+ if executor.calls > 0 && ackSeq.Load() < store.admission.Load() {
+ t.Fatalf("operator lock save was acknowledged (seq %d) before the dispatch was admitted (seq %d), yet the action still reached the executor (calls=%d, state=%q, err=%v)",
+ ackSeq.Load(), store.admission.Load(), executor.calls, completed.State, execErr)
+ }
+ if execErr != nil || completed.State != unified.ActionStateCompleted || executor.calls != 1 {
+ t.Fatalf("admitted dispatch must complete once the save queues behind it: state=%q calls=%d err=%v", completed.State, executor.calls, execErr)
+ }
+
+ // The queued save is durable and refuses every later dispatch, including an
+ // action that was approved before the lock existed.
+ refused, err := service.Execute(context.Background(), "default", second.ActionID, testActionActor("[email protected]", "default"), "")
+ if !errors.Is(err, unified.ErrResourceRemediationLocked) || refused.State != unified.ActionStateFailed || executor.calls != 1 {
+ t.Fatalf("later dispatch after the lock save: state=%q calls=%d err=%v, want refused with ErrResourceRemediationLocked", refused.State, executor.calls, err)
+ }
+}
+
+func waitFor(t *testing.T, done <-chan struct{}, what string) {
+ t.Helper()
+ select {
+ case <-done:
+ case <-time.After(10 * time.Second):
+ t.Fatalf("timed out waiting for %s", what)
+ }
+}
+
+func TestExecuteAdmissionLinearizesWithOperatorLockSaveMemoryStore(t *testing.T) {
+ runExecuteAdmissionLinearizesWithOperatorLockSave(t, func(*testing.T) unified.ResourceStore { return unified.NewMemoryStore() })
+}
+
+func TestExecuteAdmissionLinearizesWithOperatorLockSaveSQLite(t *testing.T) {
+ runExecuteAdmissionLinearizesWithOperatorLockSave(t, func(t *testing.T) unified.ResourceStore {
+ store, err := unified.NewSQLiteResourceStore(t.TempDir(), "default")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ return store
+ })
+}
+
+// Admission is the linearization point: a lock saved after the dispatch is
+// durably admitted never recalls it, and the save must not wait for the
+// dispatch, so Execute may not hold the admission coordinator across the
+// executor call.
+func TestExecuteAdmittedDispatchIsNotRecalledByLaterOperatorLock(t *testing.T) {
+ now := time.Now().UTC()
+ store := unified.NewMemoryStore()
+ executor := &dispatchHookExecutor{stubExecutor: &stubExecutor{result: &unified.ExecutionResult{Success: true, Output: "restarted"}}}
+ service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), executor)
+
+ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
+ if err != nil {
+ t.Fatalf("Plan: %v", err)
+ }
+ approveForExecution(t, service, plan.ActionID)
+
+ saved := make(chan error, 1)
+ executor.onDispatch = func() {
+ go func() {
+ saved <- service.WithPolicyMutation(func() error { return saveRemediationLock(store) })
+ }()
+ select {
+ case err := <-saved:
+ if err != nil {
+ t.Errorf("lock save during dispatch: %v", err)
+ }
+ case <-time.After(5 * time.Second):
+ t.Error("lock save blocked behind an admitted dispatch: Execute must release the admission coordinator before calling the executor")
+ }
+ }
+
+ completed, err := service.Execute(context.Background(), "default", plan.ActionID, testActionActor("[email protected]", "default"), "approved restart")
+ if err != nil || completed.State != unified.ActionStateCompleted || executor.calls != 1 {
+ t.Fatalf("admitted dispatch: state=%q calls=%d err=%v, want completed exactly once", completed.State, executor.calls, err)
+ }
+ if state, found, getErr := store.GetResourceOperatorState("vm:42"); getErr != nil || !found || !state.BlocksRemediation() {
+ t.Fatalf("lock save during dispatch must persist: found=%v state=%#v err=%v", found, state, getErr)
+ }
+}
+
+// The no-recall boundary also covers an attempt that was admitted but never
+// sent (a crash between admission and send): restart recovery drives the
+// committed authority and does not re-read the operator lock. Recalling known
+// unsent work would need an atomic pre-send terminalization the stores do not
+// have, so this pins the contract instead of leaving it implicit.
+func TestRecoverExecutingActionsDoesNotRecallQueuedAttemptAfterOperatorLock(t *testing.T) {
+ now := time.Now().UTC()
+ store := unified.NewMemoryStore()
+ executor := &stubExecutor{result: &unified.ExecutionResult{Success: true, Output: "restarted"}}
+ service := serviceForStore(t, store, testResource(now, unified.ApprovalNone), executor)
+
+ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
+ if err != nil {
+ t.Fatalf("Plan: %v", err)
+ }
+ record, found, err := store.GetActionAudit(plan.ActionID)
+ if err != nil || !found {
+ t.Fatalf("GetActionAudit: found=%v err=%v", found, err)
+ }
+ started, event, err := unified.BeginActionExecution(record, "operator", now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ attempt, err := unified.NewActionDispatchAttempt(started.ID, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := store.RecordActionExecutionAdmission(started, event, attempt); err != nil {
+ t.Fatal(err)
+ }
+ if queued, ok, getErr := store.GetActionDispatchAttempt(started.ID); getErr != nil || !ok || queued.State != unified.ActionDispatchQueued {
+ t.Fatalf("setup attempt = %#v found=%v err=%v, want queued", queued, ok, getErr)
+ }
+
+ if err := service.WithPolicyMutation(func() error { return saveRemediationLock(store) }); err != nil {
+ t.Fatalf("lock save: %v", err)
+ }
+ recovered, err := service.RecoverExecutingActions(context.Background(), "default", "system:recovery", 10)
+ if err != nil || len(recovered) != 1 || recovered[0].State != unified.ActionStateCompleted || executor.calls != 1 {
+ t.Fatalf("recovered=%#v calls=%d err=%v, want the admitted attempt dispatched once", recovered, executor.calls, err)
+ }
+}
+
func TestExecuteRefusesDriftedPlan(t *testing.T) {
now := time.Now().UTC()
env := newServiceEnv(t, testResource(now, unified.ApprovalNone))