Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Admit human action dispatch under the policy coordinator
rcourtmanf00ab32a32e3
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 53b2a60b58..b74ff73ecf 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -5330,6 +5330,36 @@ transport fails closed when the connected agent cannot answer it. Registry orreadiness-check infrastructure failures remainnonterminal internal errors rather than false permanent refusals.+Human `Execute` admits through the same policy admission coordinator as+`ExecuteUnderPolicy`. It holds the admission read lock from the live-readiness+gates, including the operator remediation lock (`NeverAutoRemediate`) and the+Retired lifecycle state, through the committed `executing` transition, and+releases it before the executor's `ExecuteAction` is called, so a slow+dispatch never blocks a policy save. Executor readiness and binding calls and+the admission hooks still run under the lock, so they can delay a save and must+never request a policy write or re-enter `Execute`/`ExecuteUnderPolicy`. An+operator lock saved through `WithPolicyMutation` therefore either lands before+the readiness check and refuses the approved action with a persisted+refused-before-dispatch failure, or queues behind the committed admission. A+save that has been acknowledged to the operator is never followed by a new+lifecycle admission (`Execute` or `ExecuteUnderPolicy`) for that resource.+The committed `executing` transition is the policy linearization point for both+paths: a lock refuses new admissions and does not recall an admitted+dispatch. That includes an attempt that was admitted but never sent (a crash+between admission and send), which `RecoverExecutingActions` resumes without+re-reading the lock. Recalling known-unsent work would need an atomic,+conditional pre-send terminalization that neither store has+(`RecordActionExecutionRefusal` accepts only planned, pending, and approved+rows, and `ForceFail` records inconclusive truth), so operator-facing copy must+say that an action already admitted can still run. Proof:+`TestExecuteAdmissionLinearizesWithOperatorLockSave*` asserts the coordinator+excludes a policy writer between the readiness check and the admission commit+and orders a real concurrent lock save against the admission,+`TestExecuteAdmittedDispatchIsNotRecalledByLaterOperatorLock` pins release+before dispatch and no recall, and+`TestRecoverExecutingActionsDoesNotRecallQueuedAttemptAfterOperatorLock` pins+the unsent-attempt boundary.+The public Patrol investigation boundary now carries independent`max_turns` and `max_evidence_calls` request limits and returns `model_turns`,`evidence_calls`, and total `tool_calls`. Persisted investigation sessions keepdiff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex 7831ddf0d7..39579156bc 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -796,8 +796,13 @@ the `white_label` branding entitlement.action emergency stop are revalidated under one admission coordinator andcommitted with `executing`. Missing or unreadable policy denies dispatch.Human approval remains separate, but `NeverAutoRemediate`, plan drift, and- emergency stop are universal. Emergency stop after `executing` is only- best-effort cancellation and must never be described as rollback.+ emergency stop are universal for every new admission, and human dispatch+ revalidates them under the same admission coordinator rather than before+ it, so an acknowledged lock save is never followed by a new lifecycle+ admission. An already admitted dispatch, including one resumed by restart+ recovery, is not revalidated. Emergency stop after+ `executing` is only best-effort cancellation and must never be described+ as rollback.The typed host-update executor wired here is likewise not generic commandauthority. It may dispatch only the closed `install_os_updates` operation,bound to the server-observed package inventory fingerprint and canonicaldiff --git a/internal/actionlifecycle/service.go b/internal/actionlifecycle/service.goindex a9818bd7e7..a6ca0de5f6 100644--- a/internal/actionlifecycle/service.go+++ b/internal/actionlifecycle/service.go@@ -166,8 +166,14 @@ func (s *Service) admissionCoordinator() *PolicyAdmissionCoordinator {// lifecycle holds the admission read lock.type PolicyAuthorizer func(ctx context.Context, record unified.ActionAuditRecord, now time.Time) (unified.ActionPolicyAuthorizationLease, string, error)-// WithPolicyMutation serializes a policy write against automatic admission.-// Writers must persist their change before returning.+// WithPolicyMutation serializes a policy write against dispatch admission,+// automatic and human alike. Both hold the read side while they revalidate and+// commit the executing transition, and fire the admission transition hook (and+// the completion hook for an admission refusal) before releasing it. A write+// must therefore never be requested from inside those hooks, and a hook must+// not re-enter Execute or ExecuteUnderPolicy: a second read lock behind a+// pending writer would deadlock. Writers must persist their change before+// returning.func (s *Service) WithPolicyMutation(write func() error) error {if s == nil || write == nil {return errors.New("policy mutation unavailable")@@ -1133,6 +1139,12 @@ func (s *Service) validateApprovalFloor(ctx context.Context, record unified.Acti// locks are all persisted as refused executions (never silently dropped)// and published to the completion hook. There is no bypass that reaches// the executor without passing every gate.+//+// The gates and the executing transition run under the same admission+// coordinator as ExecuteUnderPolicy, so the committed admission is the+// linearization point against an operator policy save: a lock saved before it+// refuses the dispatch, a lock saved after it queues behind it and does not+// recall a dispatch that is already admitted.func (s *Service) Execute(ctx context.Context, orgID, actionID string, actor unified.ActionActor, reason string) (unified.ActionAuditRecord, error) {actionID = strings.TrimSpace(actionID)if actionID == "" {@@ -1172,25 +1184,47 @@ func (s *Service) Execute(ctx context.Context, orgID, actionID string, actor unireturn record, nil}+ // Human admission shares the automatic path's boundary: the operator+ // remediation lock is read and the executing transition committed under+ // the admission read lock, so a lock save (which takes the write side via+ // WithPolicyMutation) either lands before the readiness check below and+ // refuses the dispatch, or queues behind the committed admission. The lock+ // is released before the executor runs.+ coordinator := s.admissionCoordinator()+ coordinator.mu.RLock()+ started, admitted, admissionErr := s.beginHumanExecution(ctx, orgID, store, record, actorID, reason)+ coordinator.mu.RUnlock()+ if !admitted {+ return started, admissionErr+ }+ return s.dispatchCommitted(ctx, orgID, store, started, actorID)+}++// beginHumanExecution revalidates every dispatch gate and commits the+// executing transition for a human-approved action. Callers must hold the+// admission read lock. A true result means the action is executing and its+// durable dispatch attempt may be driven, including when a concurrent caller+// won the admission.+func (s *Service) beginHumanExecution(ctx context.Context, orgID string, store Store, record unified.ActionAuditRecord, actorID, reason string) (unified.ActionAuditRecord, bool, error) {now := s.now()- record, err = s.materializeExpiry(store, orgID, record)+ record, err := s.materializeExpiry(store, orgID, record)if err != nil {- return unified.ActionAuditRecord{}, err+ return unified.ActionAuditRecord{}, false, err}if record.State == unified.ActionStateExpired {- return record, unified.ErrActionPlanExpired+ return record, false, unified.ErrActionPlanExpired}if err := unified.ValidateActionExecutionStart(record, now); err != nil {if unified.IsPermanentActionExecutionRefusal(err) {failed, persistErr := RecordRefusedExecution(store, record, actorID, now, err)if persistErr != nil {- return unified.ActionAuditRecord{}, &PersistError{Op: "refused action execution", Err: persistErr}+ return unified.ActionAuditRecord{}, false, &PersistError{Op: "refused action execution", Err: persistErr}}s.publishTransition(orgID, failed)s.publishCompleted(failed)- return failed, err+ return failed, false, err}- return unified.ActionAuditRecord{}, err+ return unified.ActionAuditRecord{}, false, err}// Reuse the exact gate set applied before approval. This second check is// the dispatch-side half of the boundary: it closes the race between the@@ -1200,46 +1234,43 @@ func (s *Service) Execute(ctx context.Context, orgID, actionID string, actor uniif unified.IsPermanentActionExecutionRefusal(err) {failed, persistErr := RecordRefusedExecution(store, record, actorID, now, err)if persistErr != nil {- return unified.ActionAuditRecord{}, &PersistError{Op: "refused action execution", Err: persistErr}+ return unified.ActionAuditRecord{}, false, &PersistError{Op: "refused action execution", Err: persistErr}}s.publishTransition(orgID, failed)s.publishCompleted(failed)- return failed, err+ return failed, false, err}- return unified.ActionAuditRecord{}, err+ return unified.ActionAuditRecord{}, false, err}started, startEvent, err := unified.BeginActionExecution(record, actorID, now)if err != nil {- return unified.ActionAuditRecord{}, err+ return unified.ActionAuditRecord{}, false, err}if reason != "" {startEvent.Message = "Action execution started: " + reason}attempt, err := unified.NewActionDispatchAttempt(started.ID, now)if err != nil {- return unified.ActionAuditRecord{}, err+ return unified.ActionAuditRecord{}, false, err}if binder, ok := s.Executor.(DispatchBinder); ok {attempt, err = binder.BindActionDispatch(ctx, started, attempt)if err != nil {- return unified.ActionAuditRecord{}, err+ return unified.ActionAuditRecord{}, false, err}}if err := store.RecordActionExecutionAdmission(started, startEvent, attempt); err != nil {if errors.Is(err, unified.ErrActionAlreadyExecuting) || errors.Is(err, unified.ErrActionExecutionFinal) {- current, found, queryErr := store.GetActionAudit(actionID)+ current, found, queryErr := store.GetActionAudit(record.ID)if queryErr == nil && found {- if current.State == unified.ActionStateExecuting {- return s.dispatchCommitted(ctx, orgID, store, current, actorID)- }- return current, nil+ return current, current.State == unified.ActionStateExecuting, nil}}- return unified.ActionAuditRecord{}, &PersistError{Op: "action execution start", Err: err}+ return unified.ActionAuditRecord{}, false, &PersistError{Op: "action execution start", Err: err}}s.publishTransition(orgID, started)- return s.dispatchCommitted(ctx, orgID, store, started, actorID)+ return started, true, nil}// ExecuteUnderPolicy is the only automatic dispatch boundary. It revalidatesdiff --git a/internal/actionlifecycle/service_test.go b/internal/actionlifecycle/service_test.goindex fe4dd6ca22..d1d7cc7f5f 100644--- a/internal/actionlifecycle/service_test.go+++ b/internal/actionlifecycle/service_test.go@@ -1500,6 +1500,254 @@ func TestExecuteRefusesRetiredResourceWithoutDispatch(t *testing.T) {}}+// admissionOrderStore stamps the moment the audit store commits an execution+// admission, so a test can order it against a concurrent policy save.+type admissionOrderStore struct {+ unified.ResourceStore+ seq *atomic.Int64+ admission atomic.Int64+}++func (s *admissionOrderStore) RecordActionExecutionAdmission(record unified.ActionAuditRecord, event unified.ActionLifecycleEvent, attempt unified.ActionDispatchAttempt) error {+ err := s.ResourceStore.RecordActionExecutionAdmission(record, event, attempt)+ if err == nil {+ s.admission.Store(s.seq.Add(1))+ }+ return err+}++// windowExecutor runs one hook inside the readiness phase: after the operator+// lock was read, before admission is committed.+type windowExecutor struct {+ *stubExecutor+ window func()+}++func (e *windowExecutor) CheckActionAvailable(ctx context.Context, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness {+ if window := e.window; window != nil {+ e.window = nil+ window()+ }+ return e.stubExecutor.CheckActionAvailable(ctx, req, resource)+}++// dispatchHookExecutor runs one hook once the dispatch is durably admitted,+// immediately before the executor would act.+type dispatchHookExecutor struct {+ *stubExecutor+ onDispatch func()+}++func (e *dispatchHookExecutor) ExecuteAction(ctx context.Context, record unified.ActionAuditRecord) (*unified.ExecutionResult, error) {+ if hook := e.onDispatch; hook != nil {+ e.onDispatch = nil+ hook()+ }+ return e.stubExecutor.ExecuteAction(ctx, record)+}++func approveForExecution(t *testing.T, service *Service, actionID string) {+ t.Helper()+ if _, err := service.Decide(context.Background(), "default", actionID, testActionDecision(t, service, "default", actionID, unified.ActionApprovalRecord{+ Actor: "[email protected]", Method: unified.MethodAPI, Outcome: unified.OutcomeApproved,+ })); err != nil {+ t.Fatalf("Decide: %v", err)+ }+}++func saveRemediationLock(store unified.ResourceStore) error {+ return store.SetResourceOperatorState(unified.ResourceOperatorState{CanonicalID: "vm:42", NeverAutoRemediate: true})+}++// A lock saved while a human-approved action is between its readiness check and+// its admission commit must not be missed: the save either lands before the+// readiness check (the dispatch is refused) or queues behind the admission (the+// dispatch is already committed). What it may never do is acknowledge to the+// operator before the admission commits and still let the dispatch through.+func runExecuteAdmissionLinearizesWithOperatorLockSave(t *testing.T, newStore func(t *testing.T) unified.ResourceStore) {+ now := time.Now().UTC()+ var seq atomic.Int64+ store := &admissionOrderStore{ResourceStore: newStore(t), seq: &seq}+ executor := &windowExecutor{stubExecutor: &stubExecutor{result: &unified.ExecutionResult{Success: true, Output: "restarted"}}}+ service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), executor)++ // Both actions are approved up front: once the lock is saved, Decide itself+ // refuses to record an approval for the locked resource.+ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))+ if err != nil {+ t.Fatalf("Plan: %v", err)+ }+ approveForExecution(t, service, plan.ActionID)+ second, err := service.Plan(context.Background(), "default", unified.ActionRequest{+ RequestID: "req-2", ResourceID: "vm:42", CapabilityName: "restart",+ Params: map[string]any{"mode": "graceful"}, Reason: "Second restart", RequestedBy: "agent:test",+ }, testActionActor("requester", "default"))+ if err != nil {+ t.Fatalf("second Plan: %v", err)+ }+ approveForExecution(t, service, second.ActionID)++ var ackSeq atomic.Int64+ acked := make(chan struct{})+ entered := make(chan struct{})+ saveErr := make(chan error, 1)+ executor.window = func() {+ // Deterministic half: a policy writer must be excluded while Execute sits+ // between the lock read and the admission commit.+ coordinator := service.admissionCoordinator()+ if coordinator.mu.TryLock() {+ coordinator.mu.Unlock()+ t.Error("a policy writer could enter between the readiness check and the admission commit: Execute is not holding the admission coordinator")+ }+ // Concurrent half: a real writer gets its chance to finish inside the+ // window. A lifecycle holding the coordinator keeps it queued, so the+ // bounded wait only elapses on the correct path.+ go func() {+ err := service.WithPolicyMutation(func() error {+ close(entered)+ return saveRemediationLock(store)+ })+ ackSeq.Store(seq.Add(1))+ saveErr <- err+ close(acked)+ }()+ select {+ case <-acked:+ case <-entered:+ waitFor(t, acked, "lock save to acknowledge")+ case <-time.After(100 * time.Millisecond):+ }+ }++ completed, execErr := service.Execute(context.Background(), "default", plan.ActionID, testActionActor("[email protected]", "default"), "approved restart")+ waitFor(t, acked, "lock save to acknowledge after Execute")+ if err := <-saveErr; err != nil {+ t.Fatalf("lock save: %v", err)+ }++ if executor.calls > 0 && ackSeq.Load() < store.admission.Load() {+ t.Fatalf("operator lock save was acknowledged (seq %d) before the dispatch was admitted (seq %d), yet the action still reached the executor (calls=%d, state=%q, err=%v)",+ ackSeq.Load(), store.admission.Load(), executor.calls, completed.State, execErr)+ }+ if execErr != nil || completed.State != unified.ActionStateCompleted || executor.calls != 1 {+ t.Fatalf("admitted dispatch must complete once the save queues behind it: state=%q calls=%d err=%v", completed.State, executor.calls, execErr)+ }++ // The queued save is durable and refuses every later dispatch, including an+ // action that was approved before the lock existed.+ refused, err := service.Execute(context.Background(), "default", second.ActionID, testActionActor("[email protected]", "default"), "")+ if !errors.Is(err, unified.ErrResourceRemediationLocked) || refused.State != unified.ActionStateFailed || executor.calls != 1 {+ t.Fatalf("later dispatch after the lock save: state=%q calls=%d err=%v, want refused with ErrResourceRemediationLocked", refused.State, executor.calls, err)+ }+}++func waitFor(t *testing.T, done <-chan struct{}, what string) {+ t.Helper()+ select {+ case <-done:+ case <-time.After(10 * time.Second):+ t.Fatalf("timed out waiting for %s", what)+ }+}++func TestExecuteAdmissionLinearizesWithOperatorLockSaveMemoryStore(t *testing.T) {+ runExecuteAdmissionLinearizesWithOperatorLockSave(t, func(*testing.T) unified.ResourceStore { return unified.NewMemoryStore() })+}++func TestExecuteAdmissionLinearizesWithOperatorLockSaveSQLite(t *testing.T) {+ runExecuteAdmissionLinearizesWithOperatorLockSave(t, func(t *testing.T) unified.ResourceStore {+ store, err := unified.NewSQLiteResourceStore(t.TempDir(), "default")+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = store.Close() })+ return store+ })+}++// Admission is the linearization point: a lock saved after the dispatch is+// durably admitted never recalls it, and the save must not wait for the+// dispatch, so Execute may not hold the admission coordinator across the+// executor call.+func TestExecuteAdmittedDispatchIsNotRecalledByLaterOperatorLock(t *testing.T) {+ now := time.Now().UTC()+ store := unified.NewMemoryStore()+ executor := &dispatchHookExecutor{stubExecutor: &stubExecutor{result: &unified.ExecutionResult{Success: true, Output: "restarted"}}}+ service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), executor)++ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))+ if err != nil {+ t.Fatalf("Plan: %v", err)+ }+ approveForExecution(t, service, plan.ActionID)++ saved := make(chan error, 1)+ executor.onDispatch = func() {+ go func() {+ saved <- service.WithPolicyMutation(func() error { return saveRemediationLock(store) })+ }()+ select {+ case err := <-saved:+ if err != nil {+ t.Errorf("lock save during dispatch: %v", err)+ }+ case <-time.After(5 * time.Second):+ t.Error("lock save blocked behind an admitted dispatch: Execute must release the admission coordinator before calling the executor")+ }+ }++ completed, err := service.Execute(context.Background(), "default", plan.ActionID, testActionActor("[email protected]", "default"), "approved restart")+ if err != nil || completed.State != unified.ActionStateCompleted || executor.calls != 1 {+ t.Fatalf("admitted dispatch: state=%q calls=%d err=%v, want completed exactly once", completed.State, executor.calls, err)+ }+ if state, found, getErr := store.GetResourceOperatorState("vm:42"); getErr != nil || !found || !state.BlocksRemediation() {+ t.Fatalf("lock save during dispatch must persist: found=%v state=%#v err=%v", found, state, getErr)+ }+}++// The no-recall boundary also covers an attempt that was admitted but never+// sent (a crash between admission and send): restart recovery drives the+// committed authority and does not re-read the operator lock. Recalling known+// unsent work would need an atomic pre-send terminalization the stores do not+// have, so this pins the contract instead of leaving it implicit.+func TestRecoverExecutingActionsDoesNotRecallQueuedAttemptAfterOperatorLock(t *testing.T) {+ now := time.Now().UTC()+ store := unified.NewMemoryStore()+ executor := &stubExecutor{result: &unified.ExecutionResult{Success: true, Output: "restarted"}}+ service := serviceForStore(t, store, testResource(now, unified.ApprovalNone), executor)++ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))+ if err != nil {+ t.Fatalf("Plan: %v", err)+ }+ record, found, err := store.GetActionAudit(plan.ActionID)+ if err != nil || !found {+ t.Fatalf("GetActionAudit: found=%v err=%v", found, err)+ }+ started, event, err := unified.BeginActionExecution(record, "operator", now)+ if err != nil {+ t.Fatal(err)+ }+ attempt, err := unified.NewActionDispatchAttempt(started.ID, now)+ if err != nil {+ t.Fatal(err)+ }+ if err := store.RecordActionExecutionAdmission(started, event, attempt); err != nil {+ t.Fatal(err)+ }+ if queued, ok, getErr := store.GetActionDispatchAttempt(started.ID); getErr != nil || !ok || queued.State != unified.ActionDispatchQueued {+ t.Fatalf("setup attempt = %#v found=%v err=%v, want queued", queued, ok, getErr)+ }++ if err := service.WithPolicyMutation(func() error { return saveRemediationLock(store) }); err != nil {+ t.Fatalf("lock save: %v", err)+ }+ recovered, err := service.RecoverExecutingActions(context.Background(), "default", "system:recovery", 10)+ if err != nil || len(recovered) != 1 || recovered[0].State != unified.ActionStateCompleted || executor.calls != 1 {+ t.Fatalf("recovered=%#v calls=%d err=%v, want the admitted attempt dispatched once", recovered, executor.calls, err)+ }+}+func TestExecuteRefusesDriftedPlan(t *testing.T) {now := time.Now().UTC()env := newServiceEnv(t, testResource(now, unified.ApprovalNone))