Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Record bounded unattended recovery comparison
rcourtman151f095a2c63
diff --git a/docs/release-control/v6/internal/status.json b/docs/release-control/v6/internal/status.jsonindex 7a948956e2..a8746bdf88 100644--- a/docs/release-control/v6/internal/status.json+++ b/docs/release-control/v6/internal/status.json@@ -10068,7 +10068,7 @@},{"id": "patrol-assistant-customer-outcome-qualification",- "summary": "The executable redesign plan, product contract, honest baseline and exact qualification receipts are in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Model-owned investigation and the named healthy, unhealthy, dependency, missing-access, storage-capacity, approved and rejected local matrix landed through core PR1960 and enterprise PR23. The shared incident-history continuation landed through PR1973 as 783571bb35c42c4813415d4981673f65c069ee0f, preserving canonical provenance, risk, bounded history, operator notes and the linked Assistant journey. Final worker hooks, affected race suites, all eight hosted Playwright shards, frontend, API and remaining CI tests pass. All 28 reported checks pass after one controlled benchmark repeat. Its first failure and loaded-worker comparison remain recorded, as do the earlier API timeouts. Final source-bound local browser proof covers /alerts at 1440, 900, 768, 767 and 390 widths. Five funded history turns cost US$0.36895475 estimated, with three Astra passes and two Gemini failures. The restored local Gemini route remains unqualified for its unsupported action-absence claims. Passing Astra history explanations do not qualify Gemini or transfer the earlier lab matrix between models. Production-wide readiness requires independent volunteered customer environments and known-condition outcome review beyond this maintainer homelab. Adoption and reported resolutions do not establish population useful-diagnosis, false-alarm, missed-problem or latency rates, which remain unknown. Storage-capacity proof does not qualify backup restore, unattended autonomy or every compatibility filter. Preserve the subscription-provider refusal without retry or bypass. Keep this gap and candidate open while these evidence limits remain. Unattended qualification also remains open after the 2026-10-16 restart-reconciliation work in Pulse PR2989 and enterprise PR121. A clean 36-minute Pro run retained an empty provider response followed by a 20-second final-summary timeout. The narrative fallback was completed as needs_attention and excluded later reinvestigation while the target remained unhealthy. This provider-failure-to-terminal-outcome path remains unresolved. Passing restart readback and unresolved-verification follow-up proofs does not qualify the complete unattended workflow.",+ "summary": "The executable redesign plan, product contract, honest baseline and exact qualification receipts are in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Model-owned investigation and the named healthy, unhealthy, dependency, missing-access, storage-capacity, approved and rejected local matrix landed through core PR1960 and enterprise PR23. The shared incident-history continuation landed through PR1973 as 783571bb35c42c4813415d4981673f65c069ee0f, preserving canonical provenance, risk, bounded history, operator notes and the linked Assistant journey. Final worker hooks, affected race suites, all eight hosted Playwright shards, frontend, API and remaining CI tests pass. All 28 reported checks pass after one controlled benchmark repeat. Its first failure and loaded-worker comparison remain recorded, as do the earlier API timeouts. Final source-bound local browser proof covers /alerts at 1440, 900, 768, 767 and 390 widths. Five funded history turns cost US$0.36895475 estimated, with three Astra passes and two Gemini failures. The restored local Gemini route remains unqualified for its unsupported action-absence claims. Passing Astra history explanations do not qualify Gemini or transfer the earlier lab matrix between models. Production-wide readiness requires independent volunteered customer environments and known-condition outcome review beyond this maintainer homelab. Adoption and reported resolutions do not establish population useful-diagnosis, false-alarm, missed-problem or latency rates, which remain unknown. Storage-capacity proof does not qualify backup restore, unattended autonomy or every compatibility filter. Preserve the subscription-provider refusal without retry or bypass. Keep this gap and candidate open while these evidence limits remain. The earlier 2026-10-16 unattended provider failure is retained as the pre-repair result. Pulse PR2994 and enterprise PR123 repair incomplete investigation outcomes: empty or failed final narratives preserve evidence and return failure, clear stale prior conclusions, and permit the next scheduled assessment within existing admission and attempt limits. Causal before/after tests cover empty output, summary errors and caller deadlines, persisted eligibility, retained evidence and blocked proposal progression. A fresh frozen 36-minute Opus 5.5 trial per arm recovered all three injected target failures in both Pro and free Pulse plus Claude, including after the planned controller restart; both left the healthy control untouched and the persistent dependency unresolved without restart. Observed Pro recovery delays were 672.734, 504.897 and 457.347 seconds; Claude delays were 514.012, 397.035 and 332.276 seconds. Provider-reported list-equivalent usage was US$5.5860168 and US$1.2533858 respectively, with complete receipts; actual subscription cash cost was not measured. The live Pro trial did not naturally reproduce an empty response or summary timeout. This single fixed-order, restart-only, development-licensing trial does not establish repeatability, a general comparative advantage, permanent repair, production licence onboarding or cross-environment readiness. The external setup reused a scoped bridge, timer and journal and does not measure minimum DIY effort; free unified action APIs remain available. Claude also retained an unsupported no-alerts claim despite critical container-health alerts in recorded Pulse state. The broader gap and candidate remain planned.","owner": "project-owner","status": "planned","recorded_at": "2026-09-05",@@ -10110,11 +10110,21 @@"path": "internal/ai/chat/agentic_final.go","kind": "file"},+ {+ "repo": "pulse",+ "path": "internal/ai/chat/agentic_final_failure_test.go",+ "kind": "file"+ },{"repo": "pulse","path": "internal/ai/findings.go","kind": "file"},+ {+ "repo": "pulse",+ "path": "internal/ai/findings_provider_failure_test.go",+ "kind": "file"+ },{"repo": "pulse","path": "internal/ai/qualification",@@ -10129,6 +10139,11 @@"repo": "pulse-enterprise","path": "internal/investigation/orchestrator.go","kind": "file"+ },+ {+ "repo": "pulse-enterprise",+ "path": "internal/investigation/orchestrator_test.go",+ "kind": "file"}]}diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.mdindex 0a75d2a053..4bad35d0e9 100644--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@ -8101,6 +8101,17 @@ Proofs: `internal/ai/chat/agentic_final_failure_test.go`,`TestProviderFailureClearsPriorConclusionAndRetainsEvidence` regression.These causal tests do not themselves qualify model-owned unattended recovery.+### Bounded unattended provider follow-up rerun (2026-10-16)++- Pulse PR2994 and enterprise PR123 retain incomplete investigations as failures, preserve their evidence, clear prior outcomes before execution, and leave the next scheduled assessment eligible within the existing policy and attempt limit. New tests fail on the original code and pass after the repair; full affected suites and the private investigation race suite pass. Both source PRs have passing CI.+- Frozen proof `provider-followup-20261016` ran Pro and free Pulse plus Claude for 36 minutes each with the same Opus 5.5 model, goal, ten-minute cadence, target faults at 120/840/1500 seconds, persistent dependency fault at 120 seconds, and controller restart at 1440 seconds. No operator diagnosis, approval, remediation or configuration change occurred inside either scored window. The independent observer retained 208 samples per arm, with no transport loss or source/helper/binary identity mismatch.+- Both arms recovered all three distinct target outages with HTTP 200 and a new process boot, including after controller restart. Both left the healthy control untouched and the dependency unresolved without restart. Pro recovery delays from injection were 672.734/504.897/457.347 seconds; Claude delays were 514.012/397.035/332.276 seconds, sampled approximately every ten seconds. Pro retained three succeeded action receipts with confirmed health. All four external scheduled turns completed and retained their journal.+- Provider-reported list-equivalent usage was US$5.5860168 for Pro and US$1.2533858 for Claude. All 38 Pro CLI invocations and four external turns retained completion receipts. These counts describe different architectural units; actual subscription cash cost and steady-state operating cost were not measured.+- The live Pro run did not naturally encounter an empty response or summary timeout. Controlled causal tests establish those error and eligibility boundaries. The live result establishes this bounded workflow only: fixed order, one trial per arm, restart-only authority and development-mode licensing. Production licence onboarding, repeatability, permanent root-cause repair and wider customer readiness remain unqualified. Keep the customer-outcome gap and candidate planned.+- Community used shipped monitoring MCP/APIs, a monitoring-only agent/read token and the retained scoped Docker bridge, timer and journal. Core free unified action APIs also exist; this experiment does not establish minimum DIY effort or make manual action receipts and guardrails paid-only. Pro restarted the Pulse process; the external arm restarted its scheduler while monitoring continued.+- The external journal repeatedly claimed Pulse had raised no alerts, while retained state included critical container-health alerts. Zero fleet findings did not establish zero alerts. The Pro late history collector also stopped on an unrelated finding ID containing a slash; retained sessions, final records and independent observations succeeded, and a separate read-only post-cutoff supplement retained both workload investigation records. Preserve that collector error with the evidence.+- Final-source browser proof covered `/patrol` > Activity > Finding options and history at 1440x1000 and 390x844: failed empty response, timed-out summary, genuine human attention, retained error/evidence scrolling, thread disclosure, enabled retry focus, close/focus return and reload persistence. This qualification entry changes evidence metadata only; runtime and rendered source remain the tested bytes. Scoped tokens were revoked, owned containers removed, agent state archived and task listeners closed.+### Proxmox guest lifecycle Patrol detector floor`internal/ai/findings_proxmox_lifecycle.go` owns the deterministic production