Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

fix(patrol): restore collected investigation evidence

rcourtman223c2d244995
diff --git a/docs/release-control/v6/internal/status.json b/docs/release-control/v6/internal/status.json
index 36f3fce00c..07cf8bbc7d 100644
--- a/docs/release-control/v6/internal/status.json
+++ b/docs/release-control/v6/internal/status.json
@@ -9376,7 +9376,7 @@
"lane_followups": [
{
"id": "architecture-post-rc-canonicalization",
- "summary": "Track the remaining post-RC architecture cleanup that still belongs to L6 after canonicalizing Connected infrastructure and the frontend state contract by construction, including retirement of newly legacy or clearly obsolete old-way internal paths unless a boundary-only exception is explicitly governed.",
+ "summary": "Track the remaining post-RC architecture cleanup that still belongs to L6 after canonicalizing Connected infrastructure and the frontend state contract by construction, including retirement of newly legacy or clearly obsolete old-way internal paths unless a boundary-only exception is explicitly governed. The investigation backup inventory overview still filters legacy guest identifiers (including a rune conversion for PVE VM IDs), so canonical resource queries can return empty while recovery points exist. Move that overview onto canonical recovery-point reads; backup-task failure reads and storage history parity do not close this separate inventory gap.",
"owner": "project-owner",
"status": "planned",
"recorded_at": "2026-03-13",
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 4c2b955baa..af0c3c5b55 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -15,6 +15,15 @@
## Purpose
+### Collected backup evidence does not require command-agent access
+
+Guest investigation manifests include the existing read-only storage tool for
+collected backup attempts. The org-scoped recovery reader preserves task error
+text independently of PVE command clients and connected agents. Missing command
+access remains an execution boundary; reading a failure or a successful task
+does not grant a lifecycle action or prove a restorable artifact. Manifest,
+persisted-task and API/tool parity regressions cover this separation.
+
### Patrol maintenance coverage adds no agent authority
The Patrol operator-state projection in `internal/api/router.go` now reads the
diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md
index 021037491f..cd9d8078e8 100644
--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md
+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md
@@ -25,6 +25,30 @@ that same result. Successful reads retain their content and execution provenance
## Purpose
+### Investigations retain collected backup and storage history evidence
+
+Guest investigation manifests include the existing read-only storage tool so
+backup failures do not require a host command connection. `backup_tasks` queries
+the tenant's canonical recovery store without misclassifying execution records
+as backup artifacts, supports canonical subject IDs and preserves failed-task
+error text. Task success alone never proves that a restore artifact exists.
+The API and tool adapter share `Manager.ListPointsForOrg`, including the
+canonical graph source in demo mode, rather than presenting fixture evidence
+through HTTP alone while tools consult an empty store.
+`TestInvestigationReadsCanonicalBackupFailureWithoutAgent` uses mapped, persisted
+task evidence and checks subject and tenant isolation.
+
+Storage performance reads resolve the canonical registry metrics target and use
+the monitor-owned history selector. Responses contain usage percentages with
+timestamps and source provenance, including `mock_synthetic`, rather than
+invented zero CPU/memory observations. Retained read errors remain errors and an
+empty requested window cannot be filled with older chart samples.
+`TestStoragePerformanceReadsRetainedHistoryAcrossRestart`,
+`TestStorageInvestigationHistoryKeepsRequestedWindowAndStoreErrors` and
+`TestInvestigationStorageHistoryUsesCommunityMonitoringSource` cover retained
+storage history, measured zero, failure, window bounds and API/tool parity.
+These are read-path controls, not proof of real remediation or release readiness.
+
### Pro Patrol defaults to Ask first until the operator chooses
`AIConfig.PatrolAutonomyLevel` stays empty until an operator saves a Patrol
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 373e84de22..413503bc1d 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -20,6 +20,16 @@
## Purpose
+### Recovery point reads stay coherent between HTTP and investigations
+
+The points endpoint and investigation adapter use the same org-scoped manager
+reader. Demo composition reads the canonical fixture graph through the recovery
+query helpers; ordinary tenants read their persisted store. Moving the existing
+in-memory filters into recovery preserves point, series and facet query semantics
+and pagination bounds. The API/tool parity regression checks that a collected
+backup failure remains readable without a PVE client or command agent. No route,
+entitlement, recovery mutation or action approval is added.
+
### Patrol setup panel reuses existing AI endpoints
The Patrol page setup panel adds no endpoint or payload field. It calls
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 656f78f2c0..01b410ac4e 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -17,6 +17,16 @@
## Purpose
+### Storage investigation history shares the chart source boundary
+
+`ReadStorageMetricHistory` and `GetStorageMetricsForChart` use the same monitor
+selector for memory, retained and generated storage history. Analytical reads
+retain the requested window and propagate retained-store errors. Chart-only
+lookback remains presentation behavior and cannot supply older observations as
+current investigation history. Generated demo samples carry `mock_synthetic`;
+they are never evidence of a real storage growth rate. The requested-window and
+store-error regression is `internal/monitoring/storage_history_test.go`.
+
### Cluster endpoint membership belongs to its client
`NewClusterClient` snapshots the supplied endpoint slice before registering
diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json
index 75b7ff1f62..c022d1318b 100644
--- a/docs/release-control/v6/internal/subsystems/registry.json
+++ b/docs/release-control/v6/internal/subsystems/registry.json
@@ -2245,6 +2245,7 @@
"internal/agentcapabilities/invocation_test.go",
"internal/agentcapabilities/manifest_test.go",
"internal/agentcapabilities/mcp_test.go",
+ "internal/agentcapabilities/patrol_scope_tools_test.go",
"internal/agentcapabilities/schema_test.go",
"internal/agentcapabilities/types_test.go",
"internal/api/agent_capabilities_test.go",
@@ -2909,6 +2910,7 @@
"internal/agentcapabilities/invocation_test.go",
"internal/agentcapabilities/manifest_test.go",
"internal/agentcapabilities/mcp_test.go",
+ "internal/agentcapabilities/patrol_scope_tools_test.go",
"internal/agentcapabilities/schema_test.go",
"internal/agentcapabilities/types_test.go",
"internal/api/agent_capabilities_test.go",
@@ -3132,6 +3134,7 @@
"exact_files": [
"frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts",
"frontend-modern/src/types/recovery.ts",
+ "internal/api/investigation_evidence_parity_test.go",
"internal/api/recovery_handlers_test.go",
"internal/api/route_inventory_test.go"
]
@@ -3245,6 +3248,7 @@
"internal/api/docker_container_action_executor_test.go",
"internal/api/docker_container_action_result_test.go",
"internal/api/host_agent_removal_lifecycle_integration_test.go",
+ "internal/api/investigation_evidence_parity_test.go",
"internal/api/issue1640_readiness_gate_test.go",
"internal/api/issue1640_readiness_transport_test.go",
"internal/api/metadata_handlers_test.go",
@@ -6224,7 +6228,8 @@
"internal/api/truenas_disk_history_integration_test.go",
"internal/monitoring/disk_drawer_native_history_test.go",
"internal/monitoring/monitor_metrics_chart_batch_bench_test.go",
- "internal/monitoring/monitor_metrics_slo_test.go"
+ "internal/monitoring/monitor_metrics_slo_test.go",
+ "internal/monitoring/storage_history_test.go"
]
},
{
diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md
index 64699f86d2..93d8deaea5 100644
--- a/docs/release-control/v6/internal/subsystems/security-privacy.md
+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md
@@ -18,6 +18,18 @@
## Purpose
+### Investigation evidence retains tenant and runtime-source boundaries
+
+Recovery HTTP and tool reads share `Manager.ListPointsForOrg`: a nonblank tenant
+is required, cancellation and read failures remain errors, and ordinary reads
+use that tenant's cached store. The optional demo source follows the current
+fixture graph and never merges persisted tenant rows into demo evidence. The
+real-to-demo-to-real regression checks both consumers and retained store data;
+the persisted task regression excludes another tenant and another guest with
+the same provider VM ID. Historical storage reads preserve source labels and
+requested time bounds. These read paths add no credential, execution or approval
+authority.
+
### Patrol maintenance coverage widens no authority
Patrol inherits only maintenance from an ancestor window scoped
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 0668621750..6670db1851 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -21,6 +21,19 @@
## Purpose
+### Recovery APIs and investigations share the org-scoped point reader
+
+`Manager.ListPointsForOrg` selects the composed recovery evidence source for
+both the HTTP points API and investigation tools. Real installations retain
+tenant-scoped SQLite reads, including errors; demo composition projects the
+current canonical fixture graph through shared recovery filters and pagination.
+It does not seed a second store or grant recovery execution. Source selection is
+bound before consumers share the manager and follows runtime demo-mode changes
+without exposing persisted rows through demo reads. Manager boundary tests preserve tenant
+validation, cancellation and read failures; the API/tool parity regression in
+`internal/api/investigation_evidence_parity_test.go` reads actual graph-owned
+failure text without a PVE command connection.
+
### Patrol maintenance coverage adds no recovery execution
Patrol's operator-state projection now projects the active recurring
diff --git a/internal/agentcapabilities/patrol_scope_tools.go b/internal/agentcapabilities/patrol_scope_tools.go
index 6f8fadd49a..2479a3ffc4 100644
--- a/internal/agentcapabilities/patrol_scope_tools.go
+++ b/internal/agentcapabilities/patrol_scope_tools.go
@@ -75,7 +75,12 @@ func patrolEvidenceToolNamesForResourceTypes(resourceTypes []string, includeDeep
add(PulsePMGToolName)
case "network-endpoint", "endpoint":
// pulse_query and pulse_read already cover endpoint evidence.
- case "agent", "host", "truenas", "node", "vm", "system-container", "container", "service":
+ case "vm", "system-container", "container":
+ add(PulseMetricsToolName)
+ // Guest findings can concern backup attempts and restore points.
+ // Those are collected evidence, independent of command access.
+ add(PulseStorageToolName)
+ case "agent", "host", "truenas", "node", "service":
add(PulseMetricsToolName)
default:
return nil, false
diff --git a/internal/agentcapabilities/patrol_scope_tools_test.go b/internal/agentcapabilities/patrol_scope_tools_test.go
index 80f26dfbe4..35b782172c 100644
--- a/internal/agentcapabilities/patrol_scope_tools_test.go
+++ b/internal/agentcapabilities/patrol_scope_tools_test.go
@@ -62,3 +62,19 @@ func TestPatrolScopedToolProjectionComposesProfileOwnedTools(t *testing.T) {
t.Fatalf("detection surface gained investigation proposal authority: %v", detection)
}
}
+
+func TestGuestInvestigationsCanReadCollectedBackupEvidence(t *testing.T) {
+ for _, kind := range []string{"vm", "system-container"} {
+ t.Run(kind, func(t *testing.T) {
+ names, ok := PatrolInvestigationToolNamesForResourceTypes([]string{kind})
+ if !ok || !slices.Contains(names, PulseStorageToolName) {
+ t.Fatalf("guest investigation lacks backup evidence: %v", names)
+ }
+ for _, name := range names {
+ if !IsPatrolInfrastructureEvidenceToolName(name) && name != PatrolActionCapabilitiesToolName && name != PatrolProposeActionToolName {
+ t.Fatalf("read projection gained action authority: %s", name)
+ }
+ }
+ })
+ }
+}
diff --git a/internal/ai/chat/service_filter_test.go b/internal/ai/chat/service_filter_test.go
index cf0d18ce8d..3f68c10c3e 100644
--- a/internal/ai/chat/service_filter_test.go
+++ b/internal/ai/chat/service_filter_test.go
@@ -54,6 +54,31 @@ func TestRestrictInvestigationProviderToolsForResourceType(t *testing.T) {
}
}
+func TestGuestInvestigationProjectionRetainsBackupTool(t *testing.T) {
+ providerTools := []providers.Tool{
+ {Name: agentcapabilities.PulseQueryToolName},
+ {Name: agentcapabilities.PulseMetricsToolName},
+ {Name: agentcapabilities.PulseStorageToolName},
+ {Name: agentcapabilities.PulseDockerToolName},
+ {Name: agentcapabilities.PatrolActionCapabilitiesToolName},
+ {Name: agentcapabilities.PatrolProposeActionToolName},
+ }
+ filtered, err := restrictInvestigationProviderToolsForResourceType(providerTools, "vm")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var found bool
+ for _, tool := range filtered {
+ found = found || tool.Name == agentcapabilities.PulseStorageToolName
+ if tool.Name == agentcapabilities.PulseDockerToolName {
+ t.Fatal("guest backup evidence should not expose unrelated platform tools")
+ }
+ }
+ if !found {
+ t.Fatal("investigation filtered out collected backup evidence")
+ }
+}
+
func TestRestrictInvestigationProviderToolsAllowsUnavailableOptionalDeepRead(t *testing.T) {
providerTools := []providers.Tool{
{Name: agentcapabilities.PulseQueryToolName},
diff --git a/internal/ai/tools/adapters.go b/internal/ai/tools/adapters.go
index 8f132ec4f8..ffe74cf027 100644
--- a/internal/ai/tools/adapters.go
+++ b/internal/ai/tools/adapters.go
@@ -205,11 +205,7 @@ func (a *RecoveryPointsToolAdapter) ListPoints(ctx context.Context, opts recover
ctx, cancel := context.WithTimeout(ctx, a.timeout)
defer cancel()
- store, err := a.manager.StoreForOrg(a.orgID)
- if err != nil {
- return nil, 0, err
- }
- return store.ListPoints(ctx, opts)
+ return a.manager.ListPointsForOrg(ctx, a.orgID, opts)
}
// DiskHealthToolAdapter adapts unified read-state hosts to DiskHealthProvider.
@@ -235,8 +231,8 @@ func (a *DiskHealthToolAdapter) GetHosts() []*unifiedresources.HostView {
// RawMetricPoint represents a single metric value at a point in time
type RawMetricPoint struct {
- Value float64
- Timestamp time.Time
+ Value float64 `json:"value"`
+ Timestamp time.Time `json:"timestamp"`
}
// MetricsSource provides access to historical metrics data
diff --git a/internal/ai/tools/investigation_evidence_test.go b/internal/ai/tools/investigation_evidence_test.go
new file mode 100644
index 0000000000..c78cb5b7b4
--- /dev/null
+++ b/internal/ai/tools/investigation_evidence_test.go
@@ -0,0 +1,106 @@
+package tools
+
+import (
+ "context"
+ "encoding/json"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"
+ recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
+ proxmoxmapper "github.com/rcourtman/pulse-go-rewrite/internal/recovery/mapper/proxmox"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
+ "github.com/stretchr/testify/require"
+)
+
+func TestInvestigationReadsCanonicalBackupFailureWithoutAgent(t *testing.T) {
+ ctx := context.Background()
+ manager := recoverymanager.New(config.NewMultiTenantPersistence(t.TempDir()))
+ store, err := manager.StoreForOrg("default")
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = store.Close() })
+ at := time.Now().Add(-time.Hour).UTC()
+ // Use the real mapper and persisted query, so a stale KindBackup filter
+ // cannot pass by asking an in-memory stub to ignore query constraints.
+ points := proxmoxmapper.FromPVEBackupTasks([]models.BackupTask{{
+ ID: "failed-attempt", VMID: 101, Node: "pve1", Instance: "west", Type: "qemu",
+ Status: "failed", StartTime: at, EndTime: at.Add(time.Minute),
+ Error: "backup write failed: repository connection timed out",
+ }}, nil)
+ require.Len(t, points, 1)
+ require.Equal(t, recovery.KindOther, points[0].Kind)
+ points[0].SubjectResourceID = "vm-canonical"
+ other := points[0]
+ other.ID = "pve-task:another-guest"
+ other.SubjectResourceID = "vm-other-cluster"
+ artifact := points[0]
+ artifact.ID = "pve-backup:artifact"
+ artifact.Kind = recovery.KindBackup
+ artifact.Outcome = recovery.OutcomeSuccess
+ require.NoError(t, store.UpsertPoints(ctx, []recovery.RecoveryPoint{points[0], other, artifact}))
+ foreign, err := manager.StoreForOrg("another-tenant")
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = foreign.Close() })
+ foreignPoint := points[0]
+ foreignPoint.ID = "pve-task:foreign"
+ require.NoError(t, foreign.UpsertPoints(ctx, []recovery.RecoveryPoint{foreignPoint}))
+ executor := NewPulseToolExecutor(ExecutorConfig{
+ RecoveryPointsProvider: NewRecoveryPointsToolAdapter(manager, "default"),
+ })
+ for _, filter := range []string{"", "error", "failed"} {
+ result, err := executor.executeListBackupTasks(ctx, map[string]interface{}{
+ "resource_id": "vm-canonical", "status": filter,
+ })
+ require.NoError(t, err)
+ require.False(t, result.IsError, result.Content)
+ var response BackupTasksListResponse
+ require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &response))
+ require.Len(t, response.Tasks, 1)
+ require.Equal(t, "failed-attempt", response.Tasks[0].ID)
+ require.Equal(t, points[0].Details["error"], response.Tasks[0].Error)
+ require.Equal(t, at.Unix(), response.Tasks[0].StartTime.Unix())
+ }
+}
+
+func TestStoragePerformanceReadsRetainedHistoryAcrossRestart(t *testing.T) {
+ cfg := metrics.DefaultConfig(filepath.Join(t.TempDir(), "metrics"))
+ cfg.RetentionRaw = 8 * 24 * time.Hour
+ store, err := metrics.NewStore(cfg)
+ require.NoError(t, err)
+ at := time.Now().Add(-6 * time.Hour).Truncate(time.Second)
+ store.Write("storage", "native-pool", "usage", 0, at)
+ store.Write("storage", "native-pool", "usage", 83, at.Add(time.Hour))
+ require.NoError(t, store.Close())
+ store, err = metrics.NewStore(cfg)
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = store.Close() })
+ adapter := NewMetricsHistoryToolAdapter(&fakeMetricsSource{}, &fakeReadState{}, store)
+ resource := unifiedresources.Resource{ID: "storage-canonical", Type: unifiedresources.ResourceTypeStorage, Name: "pool", MetricsTarget: &unifiedresources.MetricsTarget{ResourceType: "storage", ResourceID: "native-pool"}}
+ executor := NewPulseToolExecutor(ExecutorConfig{MetricsHistory: adapter, UnifiedResourceProvider: &stubUnifiedResourceProvider{resources: []unifiedresources.Resource{resource}}})
+ result, err := executor.executeGetMetrics(context.Background(), map[string]interface{}{"resource_id": resource.ID, "resource_type": "storage", "period": "7d"})
+ require.NoError(t, err)
+ require.False(t, result.IsError, result.Content)
+ var response struct {
+ ResourceID string `json:"resource_id"`
+ Metric string `json:"metric"`
+ Source string `json:"source"`
+ Points []map[string]interface{} `json:"points"`
+ }
+ require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &response))
+ require.Equal(t, resource.ID, response.ResourceID)
+ require.Equal(t, "usage", response.Metric)
+ require.Equal(t, "store", response.Source)
+ require.Len(t, response.Points, 2)
+ require.Equal(t, 0.0, response.Points[0]["value"])
+ require.Equal(t, 83.0, response.Points[1]["value"])
+ require.NotContains(t, response.Points[0], "cpu")
+ require.NotContains(t, response.Points[0], "memory")
+ require.NoError(t, store.Close())
+ result, err = executor.executeGetMetrics(context.Background(), map[string]interface{}{"resource_id": resource.ID, "period": "7d"})
+ require.NoError(t, err)
+ require.True(t, result.IsError, "store error must not become empty successful history")
+}
diff --git a/internal/ai/tools/resource_metrics.go b/internal/ai/tools/resource_metrics.go
index 6efb8d9c64..fe5a4e0905 100644
--- a/internal/ai/tools/resource_metrics.go
+++ b/internal/ai/tools/resource_metrics.go
@@ -34,6 +34,7 @@ func (e *PulseToolExecutor) resolvePerformanceResource(reference, kind string) (
var matches []unifiedresources.Resource
for _, resourceType := range []unifiedresources.ResourceType{
unifiedresources.ResourceTypeAgent, unifiedresources.ResourceTypeVM, unifiedresources.ResourceTypeSystemContainer,
+ unifiedresources.ResourceTypeStorage, unifiedresources.ResourceTypeCeph,
} {
if kind != "" && canonicalQueryResourceType(kind) != string(resourceType) {
continue
diff --git a/internal/ai/tools/storage_metrics_history.go b/internal/ai/tools/storage_metrics_history.go
new file mode 100644
index 0000000000..4d37b91aef
--- /dev/null
+++ b/internal/ai/tools/storage_metrics_history.go
@@ -0,0 +1,77 @@
+package tools
+
+import (
+ "fmt"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+)
+
+// storageMetricHistorySource exposes the monitor-owned history selection used
+// by charts. The source label distinguishes generated fixtures from observations.
+type storageMetricHistorySource interface {
+ GetStorageMetricHistory(string, time.Duration) (map[string][]RawMetricPoint, string, error)
+}
+
+func (a *MetricsHistoryToolAdapter) GetStorageMetricHistory(target unifiedresources.MetricsTarget, period time.Duration) ([]RawMetricPoint, string, error) {
+ if source, ok := a.metricsSource.(storageMetricHistorySource); ok {
+ series, provenance, err := source.GetStorageMetricHistory(target.ResourceID, period)
+ return series["usage"], provenance, err
+ }
+ // Compatibility for adapters without the monitor chart boundary. Keep the
+ // requested retained window and propagate store errors, just like guest reads.
+ if a.retainedMetrics == nil {
+ return nil, "unavailable", nil
+ }
+ end := time.Now()
+ points, err := a.retainedMetrics.Query("storage", target.ResourceID, "usage", end.Add(-period), end, 0)
+ if err != nil {
+ return nil, "", err
+ }
+ result := make([]RawMetricPoint, 0, len(points))
+ for _, point := range points {
+ result = append(result, RawMetricPoint{Timestamp: point.Timestamp, Value: point.Value})
+ }
+ return result, "store", nil
+}
+
+func (e *PulseToolExecutor) storageMetricsHistory(resourceID string, target unifiedresources.MetricsTarget, period string, duration time.Duration) (CallToolResult, error) {
+ provider, ok := e.metricsHistory.(interface {
+ GetStorageMetricHistory(unifiedresources.MetricsTarget, time.Duration) ([]RawMetricPoint, string, error)
+ })
+ if !ok {
+ return NewErrorResult(fmt.Errorf("storage metrics history provider is not available for resource %s", resourceID)), nil
+ }
+ points, source, err := provider.GetStorageMetricHistory(target, duration)
+ if err != nil {
+ return NewErrorResult(err), nil
+ }
+ originalCount := len(points)
+ if len(points) > maxMetricPoints {
+ values := make([]MetricPoint, len(points))
+ for i, point := range points {
+ values[i] = MetricPoint{Timestamp: point.Timestamp, Disk: point.Value}
+ }
+ values = downsampleMetrics(values, maxMetricPoints)
+ points = make([]RawMetricPoint, len(values))
+ for i, value := range values {
+ points[i] = RawMetricPoint{Timestamp: value.Timestamp, Value: value.Disk}
+ }
+ }
+ if points == nil {
+ points = []RawMetricPoint{}
+ }
+ // A storage series has no CPU/memory measurements. Do not manufacture zero
+ // observations by projecting it through the guest performance point shape.
+ return NewJSONResult(struct {
+ ResourceID string `json:"resource_id"`
+ ResourceType string `json:"resource_type"`
+ Metric string `json:"metric"`
+ Unit string `json:"unit"`
+ Period string `json:"period"`
+ Source string `json:"source"`
+ Points []RawMetricPoint `json:"points"`
+ Downsampled bool `json:"downsampled"`
+ OriginalCount int `json:"original_count"`
+ }{resourceID, "storage", "usage", "percent", period, source, points, originalCount > len(points), originalCount}), nil
+}
diff --git a/internal/ai/tools/tools_metrics.go b/internal/ai/tools/tools_metrics.go
index ba8c082534..f3f720d5cb 100644
--- a/internal/ai/tools/tools_metrics.go
+++ b/internal/ai/tools/tools_metrics.go
@@ -20,7 +20,7 @@ func (e *PulseToolExecutor) registerMetricsTools() {
Description: `Get performance metrics, baselines, and sensor data.
Types:
-- performance: Historical CPU/memory/disk metrics over 24h or 7d
+- performance: Historical CPU/memory/disk metrics, or storage usage percentages, over 24h or 7d. Accepts the canonical resource_id from pulse_query. Storage responses retain the history source; mock_synthetic is demo data, not observed infrastructure history.
- temperatures: CPU, disk, and sensor temperatures from hosts
- network: Network interface statistics (rx/tx bytes, speed)
- diskio: Disk I/O statistics (read/write bytes, ops)
@@ -51,7 +51,7 @@ Examples:
},
"resource_type": {
Type: "string",
- Description: "Filter by resource type: agent, vm, system-container (performance). The node alias is also accepted. Baselines accept vm, system-container, node.",
+ Description: "Filter by resource type: agent, vm, system-container, storage, ceph (performance). The node alias is also accepted. Baselines accept vm, system-container, node.",
},
"host": {
Type: "string",
@@ -143,9 +143,9 @@ func (e *PulseToolExecutor) executeGetMetrics(_ context.Context, args map[string
return NewErrorResult(fmt.Errorf("invalid resource_type: %s. Use agent, vm, or system-container (node is an alias)", resourceType)), nil
}
resourceType = canonicalMetricsResourceType(resourceType)
- validTypes := map[string]bool{"agent": true, "vm": true, "system-container": true, "node": true}
+ validTypes := map[string]bool{"agent": true, "vm": true, "system-container": true, "node": true, "storage": true, "ceph": true}
if !validTypes[resourceType] {
- return NewErrorResult(fmt.Errorf("invalid resource_type: %s. Use agent, vm, or system-container (node is an alias)", resourceType)), nil
+ return NewErrorResult(fmt.Errorf("invalid resource_type: %s. Use agent, vm, system-container, storage, or ceph (node is an alias)", resourceType)), nil
}
}
@@ -190,6 +190,9 @@ func (e *PulseToolExecutor) executeGetMetrics(_ context.Context, args map[string
metricsID = target.ResourceID
metricsTarget = target
}
+ if metricsTarget != nil && metricsTarget.ResourceType == "storage" {
+ return e.storageMetricsHistory(response.ResourceID, *metricsTarget, period, duration)
+ }
var metrics []MetricPoint
var err error
if retained, ok := e.metricsHistory.(interface {
diff --git a/internal/ai/tools/tools_storage.go b/internal/ai/tools/tools_storage.go
index d7953950e0..93662a5238 100644
--- a/internal/ai/tools/tools_storage.go
+++ b/internal/ai/tools/tools_storage.go
@@ -19,7 +19,7 @@ func (e *PulseToolExecutor) registerStorageTools() {
e.registry.registerBuiltin(RegisteredTool{
Definition: Tool{
Name: agentcapabilities.PulseStorageToolName,
- Description: `Query storage pools, backups, snapshots, Ceph, replication, RAID, and disk health. Use the "type" parameter to select what to query.`,
+ Description: `Query storage pools, backups, snapshots, Ceph, replication, RAID, and disk health. Use the "type" parameter to select what to query. backup_tasks returns collected backup attempt status, timestamps and failure error text without requiring host command access. Task success alone does not prove a restorable backup exists.`,
InputSchema: InputSchema{
Type: "object",
Properties: map[string]PropertySchema{
@@ -34,7 +34,7 @@ func (e *PulseToolExecutor) registerStorageTools() {
},
"resource_id": {
Type: "string",
- Description: "Filter by VM/container ID (for backups, snapshots, resource_disks)",
+ Description: "Filter by resource ID (canonical subject ID for backup_tasks; VM/container ID for backups, snapshots, resource_disks)",
},
"guest_id": {
Type: "string",
@@ -780,6 +780,20 @@ func recoveryPointCanonicalTaskError(p recovery.RecoveryPoint) string {
return ""
}
+func backupTaskMatchesStatus(filter, status string, outcome recovery.Outcome) bool {
+ if filter == "" || strings.EqualFold(filter, status) {
+ return true
+ }
+ switch strings.ToLower(strings.TrimSpace(filter)) {
+ case "ok", "success":
+ return outcome == recovery.OutcomeSuccess
+ case "error", "failed":
+ return outcome == recovery.OutcomeFailed
+ default:
+ return false
+ }
+}
+
func (e *PulseToolExecutor) executeListSnapshots(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {
if e.backupProvider == nil && e.recoveryPointsProvider == nil {
return NewTextResult("State provider not available."), nil
@@ -1104,6 +1118,7 @@ func (e *PulseToolExecutor) executeListBackupTasks(ctx context.Context, args map
instanceFilter, _ := args["instance"].(string)
guestIDFilter, _ := args["guest_id"].(string)
+ resourceIDFilter, _ := args["resource_id"].(string)
statusFilter, _ := args["status"].(string)
limit := intArg(args, "limit", 50)
@@ -1140,9 +1155,11 @@ func (e *PulseToolExecutor) executeListBackupTasks(ctx context.Context, args map
for page := 1; page <= maxPages; page++ {
points, _, err := e.recoveryPointsProvider.ListPoints(ctx, recovery.ListPointsOptions{
Provider: recovery.ProviderProxmoxPVE,
- Kind: recovery.KindBackup,
- Page: page,
- Limit: pageLimit,
+ // Task rows are execution evidence (KindOther), not backup
+ // artifacts. The prefix below also accepts older stored task kinds.
+ SubjectResourceID: resourceIDFilter,
+ Page: page,
+ Limit: pageLimit,
})
if err != nil {
return NewErrorResult(err), nil
@@ -1155,6 +1172,9 @@ func (e *PulseToolExecutor) executeListBackupTasks(ctx context.Context, args map
if !strings.HasPrefix(strings.TrimSpace(p.ID), "pve-task:") {
continue
}
+ if resourceIDFilter != "" && p.SubjectResourceID != resourceIDFilter {
+ continue
+ }
totalCount++
instance := recoveryPointCanonicalInstance(p)
@@ -1171,7 +1191,7 @@ func (e *PulseToolExecutor) executeListBackupTasks(ctx context.Context, args map
if guestIDFilter != "" && strconv.Itoa(vmid) != guestIDFilter {
continue
}
- if statusFilter != "" && !strings.EqualFold(status, statusFilter) {
+ if !backupTaskMatchesStatus(statusFilter, status, p.Outcome) {
continue
}
@@ -1223,6 +1243,9 @@ func (e *PulseToolExecutor) executeListBackupTasks(ctx context.Context, args map
if !strings.HasPrefix(strings.TrimSpace(p.ID), "pve-task:") {
continue
}
+ if resourceIDFilter != "" && p.SubjectResourceID != resourceIDFilter {
+ continue
+ }
instance := recoveryPointCanonicalInstance(p)
node := recoveryPointCanonicalNode(p)
@@ -1238,7 +1261,7 @@ func (e *PulseToolExecutor) executeListBackupTasks(ctx context.Context, args map
if guestIDFilter != "" && strconv.Itoa(vmid) != guestIDFilter {
continue
}
- if statusFilter != "" && !strings.EqualFold(status, statusFilter) {
+ if !backupTaskMatchesStatus(statusFilter, status, p.Outcome) {
continue
}
diff --git a/internal/api/investigation_evidence_parity_test.go b/internal/api/investigation_evidence_parity_test.go
new file mode 100644
index 0000000000..f710abb04b
--- /dev/null
+++ b/internal/api/investigation_evidence_parity_test.go
@@ -0,0 +1,146 @@
+package api
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "net/url"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/ai/tools"
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"
+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"
+ "github.com/rcourtman/pulse-go-rewrite/internal/testutil"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/stretchr/testify/require"
+)
+
+func TestInvestigationRecoverySourceFollowsRuntimeMode(t *testing.T) {
+ testutil.SetMockMode(t, false)
+ manager := newRecoveryReadManager(config.NewMultiTenantPersistence(t.TempDir()))
+ store, err := manager.StoreForOrg("default")
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = store.Close() })
+ require.NoError(t, store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{{
+ ID: "persisted-marker", Provider: recovery.ProviderProxmoxPVE, Kind: recovery.KindOther, Mode: recovery.ModeRemote,
+ Outcome: recovery.OutcomeFailed, SubjectResourceID: "vm-persisted",
+ }}))
+ adapter := tools.NewRecoveryPointsToolAdapter(manager, "default")
+ for _, demo := range []bool{false, true, false} {
+ require.NoError(t, mock.SetEnabled(demo))
+ points, _, err := adapter.ListPoints(context.Background(), recovery.ListPointsOptions{SubjectResourceID: "vm-persisted"})
+ require.NoError(t, err)
+ recorder := httptest.NewRecorder()
+ NewRecoveryHandlers(manager).HandleListPoints(recorder, httptest.NewRequest(http.MethodGet,
+ "/api/recovery/points?itemResourceId=vm-persisted", nil))
+ require.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String())
+ if demo {
+ require.Empty(t, points, "demo reads must not leak persisted tenant evidence")
+ require.NotContains(t, recorder.Body.String(), "persisted-marker")
+ graphPoints, _, err := adapter.ListPoints(context.Background(), recovery.ListPointsOptions{})
+ require.NoError(t, err)
+ require.NotEmpty(t, graphPoints, "a manager composed before demo was enabled must see its current graph")
+ } else {
+ require.Len(t, points, 1)
+ require.Equal(t, "persisted-marker", points[0].ID)
+ require.Contains(t, recorder.Body.String(), "persisted-marker")
+ }
+ }
+}
+
+func TestInvestigationBackupTasksUseCommunityRecoverySource(t *testing.T) {
+ testutil.SetMockMode(t, true)
+ // Exercise the same composition used by Router, without a PVE client or
+ // a seeded second database. The previous HTTP-only fixture branch failed
+ // here even though isolated tests of persisted tools passed.
+ manager := newRecoveryReadManager(nil)
+ var selected recovery.RecoveryPoint
+ for _, point := range mock.CurrentFixtureGraph().RecoveryPoints() {
+ if point.Provider == recovery.ProviderProxmoxPVE && strings.HasPrefix(point.ID, "pve-task:") && point.Outcome == recovery.OutcomeFailed && point.Details["error"] != "" {
+ selected = point
+ break
+ }
+ }
+ require.NotEmpty(t, selected.ID, "fixture must contain collected task failure text")
+ recorder := httptest.NewRecorder()
+ NewRecoveryHandlers(manager).HandleListPoints(recorder, httptest.NewRequest(http.MethodGet,
+ "/api/recovery/points?platform=proxmox-pve&itemResourceId="+url.QueryEscape(selected.SubjectResourceID), nil))
+ require.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String())
+ require.Contains(t, recorder.Body.String(), selected.Details["error"])
+ executor := tools.NewPulseToolExecutor(tools.ExecutorConfig{RecoveryPointsProvider: tools.NewRecoveryPointsToolAdapter(manager, "default")})
+ result, err := executor.ExecuteTool(context.Background(), "pulse_storage", map[string]interface{}{
+ "type": "backup_tasks", "resource_id": selected.SubjectResourceID, "status": "error",
+ })
+ require.NoError(t, err)
+ require.False(t, result.IsError, result.Content)
+ var response struct {
+ Tasks []struct {
+ ID string `json:"id"`
+ Error string `json:"error"`
+ } `json:"tasks"`
+ }
+ require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &response))
+ require.NotEmpty(t, response.Tasks)
+ for _, task := range response.Tasks {
+ require.Equal(t, strings.TrimPrefix(selected.ID, "pve-task:"), task.ID)
+ require.Equal(t, selected.Details["error"], task.Error)
+ }
+}
+
+type investigationHistoryResources struct{ resource unifiedresources.Resource }
+
+func (p investigationHistoryResources) GetByType(kind unifiedresources.ResourceType) []unifiedresources.Resource {
+ if p.resource.Type == kind {
+ return []unifiedresources.Resource{p.resource}
+ }
+ return nil
+}
+
+func TestInvestigationStorageHistoryUsesCommunityMonitoringSource(t *testing.T) {
+ testutil.SetMockMode(t, false)
+ monitor, state, history := newTestMonitor(t)
+ syncTestResourceStore(t, monitor, state)
+ at := time.Now().UTC().Truncate(time.Second)
+ history.AddStorageMetric("native-pool", "usage", 0, at.Add(-30*time.Minute))
+ history.AddStorageMetric("native-pool", "usage", 83, at.Add(-10*time.Minute))
+ router := &Router{monitor: monitor}
+ recorder := httptest.NewRecorder()
+ router.handleMetricsHistory(recorder, httptest.NewRequest(http.MethodGet,
+ "/api/metrics-store/history?resourceType=storage&resourceId=native-pool&metric=usage&range=7d", nil))
+ require.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String())
+ var apiHistory struct {
+ Source string `json:"source"`
+ Points []struct {
+ Value float64 `json:"value"`
+ } `json:"points"`
+ }
+ require.NoError(t, json.Unmarshal(recorder.Body.Bytes(), &apiHistory))
+ adapter := tools.NewMetricsHistoryToolAdapter(&metricsSourceWrapper{history: history, monitor: monitor}, monitor.GetUnifiedReadState())
+ require.NotNil(t, adapter)
+ resource := unifiedresources.Resource{ID: "storage-canonical", Name: "pool", Type: unifiedresources.ResourceTypeStorage,
+ MetricsTarget: &unifiedresources.MetricsTarget{ResourceType: "storage", ResourceID: "native-pool"}}
+ executor := tools.NewPulseToolExecutor(tools.ExecutorConfig{MetricsHistory: adapter,
+ UnifiedResourceProvider: investigationHistoryResources{resource: resource}})
+ result, err := executor.ExecuteTool(context.Background(), "pulse_metrics", map[string]interface{}{
+ "type": "performance", "resource_id": resource.ID, "period": "7d",
+ })
+ require.NoError(t, err)
+ require.False(t, result.IsError, result.Content)
+ var toolHistory struct {
+ ResourceID string `json:"resource_id"`
+ Source string `json:"source"`
+ Points []tools.RawMetricPoint `json:"points"`
+ }
+ require.NoError(t, json.Unmarshal([]byte(result.Content[0].Text), &toolHistory))
+ require.Equal(t, resource.ID, toolHistory.ResourceID)
+ require.Equal(t, apiHistory.Source, toolHistory.Source)
+ require.Len(t, toolHistory.Points, 2)
+ require.Len(t, apiHistory.Points, len(toolHistory.Points))
+ for i, point := range toolHistory.Points {
+ require.Equal(t, apiHistory.Points[i].Value, point.Value)
+ }
+}
diff --git a/internal/api/recovery_handlers.go b/internal/api/recovery_handlers.go
index 7a74016cb1..1b834a7620 100644
--- a/internal/api/recovery_handlers.go
+++ b/internal/api/recovery_handlers.go
@@ -10,6 +10,7 @@ import (
"strings"
"time"
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
@@ -26,11 +27,20 @@ type RecoveryHandlers struct {
}
func NewRecoveryHandlers(manager *recoverymanager.Manager) *RecoveryHandlers {
+ if manager == nil {
+ manager = newRecoveryReadManager(nil)
+ }
return &RecoveryHandlers{
manager: manager,
}
}
+// Compose recovery reads once so HTTP and investigation consumers see the same
+// evidence. Demo mode uses the current canonical graph; real tenants use SQLite.
+func newRecoveryReadManager(persistence *config.MultiTenantPersistence) *recoverymanager.Manager {
+ return recoverymanager.NewWithPointSource(persistence, mock.RecoveryPointsSource{})
+}
+
func (h *RecoveryHandlers) storeForOrg(orgID string) (*recoverystore.Store, error) {
if h == nil || h.manager == nil {
return nil, fmt.Errorf("recovery manager is not configured")
@@ -228,31 +238,10 @@ func (h *RecoveryHandlers) HandleListPoints(w http.ResponseWriter, r *http.Reque
Limit: limit,
}
- var (
- points []recovery.RecoveryPoint
- total int
- )
-
- if mock.IsMockEnabled() {
- all := mock.CurrentFixtureGraph().RecoveryPoints()
- filtered := filterRecoveryPoints(all, opts)
- total = len(filtered)
- points = paginateRecoveryPoints(filtered, opts.Page, opts.Limit)
- } else {
- orgID := GetOrgID(r.Context())
- store, err := h.storeForOrg(orgID)
- if err != nil {
- http.Error(w, sanitizeErrorForClient(err, "Internal server error"), http.StatusInternalServerError)
- return
- }
-
- p, t, err := store.ListPoints(r.Context(), opts)
- if err != nil {
- http.Error(w, sanitizeErrorForClient(err, "Internal server error"), http.StatusInternalServerError)
- return
- }
- points = p
- total = t
+ points, total, err := h.manager.ListPointsForOrg(r.Context(), GetOrgID(r.Context()), opts)
+ if err != nil {
+ http.Error(w, sanitizeErrorForClient(err, "Internal server error"), http.StatusInternalServerError)
+ return
}
// Ensure normalized display data is present for UIs (works for both real and mock paths).
@@ -413,139 +402,7 @@ func (h *RecoveryHandlers) HandleListFacets(w http.ResponseWriter, r *http.Reque
}
func filterRecoveryPoints(all []recovery.RecoveryPoint, opts recovery.ListPointsOptions) []recovery.RecoveryPoint {
- if len(all) == 0 {
- return nil
- }
-
- provider := strings.TrimSpace(string(opts.Provider))
- kind := strings.TrimSpace(string(opts.Kind))
- mode := strings.TrimSpace(string(opts.Mode))
- outcome := strings.TrimSpace(string(opts.Outcome))
- subjectRID := strings.TrimSpace(opts.SubjectResourceID)
- rollupID := strings.TrimSpace(opts.RollupID)
- cluster := strings.TrimSpace(opts.ClusterLabel)
- node := strings.TrimSpace(opts.NodeHostLabel)
- namespace := strings.TrimSpace(opts.NamespaceLabel)
- itemType := recovery.NormalizeRecoveryItemType(opts.ItemType)
- q := strings.ToLower(strings.TrimSpace(opts.Query))
- workloadOnly := opts.WorkloadOnly
- verification := strings.ToLower(strings.TrimSpace(opts.Verification))
- if rollupID != "" && !strings.HasPrefix(rollupID, "res:") && !strings.HasPrefix(rollupID, "ext:") {
- rollupID = "res:" + rollupID
- }
-
- out := make([]recovery.RecoveryPoint, 0, len(all))
- for _, p := range all {
- // Ensure display/index data exists for filtering parity with sqlite store.
- if p.Display == nil {
- idx := recovery.DeriveIndex(p)
- p.Display = idx.ToDisplay()
- }
- disp := p.Display
-
- if provider != "" && strings.TrimSpace(string(p.Provider)) != provider {
- continue
- }
- if kind != "" && strings.TrimSpace(string(p.Kind)) != kind {
- continue
- }
- if mode != "" && strings.TrimSpace(string(p.Mode)) != mode {
- continue
- }
- if outcome != "" && strings.TrimSpace(string(p.Outcome)) != outcome {
- continue
- }
- if rollupID != "" {
- if strings.TrimSpace(recovery.SubjectKeyForPoint(p)) != rollupID {
- continue
- }
- } else if subjectRID != "" && strings.TrimSpace(p.SubjectResourceID) != subjectRID {
- continue
- }
- if opts.From != nil && !opts.From.IsZero() {
- // Match store semantics: completed_at_ms IS NULL OR completed_at_ms >= from
- if p.CompletedAt != nil && !p.CompletedAt.IsZero() && p.CompletedAt.Before(opts.From.UTC()) {
- continue
- }
- }
- if opts.To != nil && !opts.To.IsZero() {
- // Match store semantics: completed_at_ms IS NULL OR completed_at_ms <= to
- if p.CompletedAt != nil && !p.CompletedAt.IsZero() && p.CompletedAt.After(opts.To.UTC()) {
- continue
- }
- }
-
- if cluster != "" && strings.TrimSpace(getDisplayClusterLabel(p.Display)) != cluster {
- continue
- }
- if node != "" && strings.TrimSpace(getDisplayNodeHostLabel(p.Display)) != node {
- continue
- }
- if namespace != "" && strings.TrimSpace(getDisplayNamespaceLabel(p.Display)) != namespace {
- continue
- }
- if itemType != "" && strings.TrimSpace(getDisplayItemType(p.Display)) != itemType {
- continue
- }
- if workloadOnly && !(disp != nil && disp.IsWorkload) {
- continue
- }
- if verification != "" {
- v := p.Verified
- switch verification {
- case "verified":
- if v == nil || !*v {
- continue
- }
- case "unverified":
- if v == nil || *v {
- continue
- }
- case "unknown":
- if v != nil {
- continue
- }
- }
- }
- if q != "" {
- // Best-effort match across normalized display fields.
- var subjectLabel, subjectType, itemTypeLabel, clusterLabel, nodeLabel, nsLabel, entityID, repoLabel, detailSummary string
- if disp != nil {
- subjectLabel = disp.SubjectLabel
- subjectType = disp.SubjectType
- itemTypeLabel = disp.ItemType
- clusterLabel = disp.ClusterLabel
- nodeLabel = disp.NodeHostLabel
- nsLabel = disp.NamespaceLabel
- entityID = disp.EntityIDLabel
- repoLabel = disp.RepositoryLabel
- detailSummary = disp.DetailsSummary
- }
-
- hay := strings.ToLower(strings.Join([]string{
- strings.TrimSpace(p.ID),
- strings.TrimSpace(string(p.Provider)),
- strings.TrimSpace(string(p.Kind)),
- strings.TrimSpace(string(p.Mode)),
- strings.TrimSpace(string(p.Outcome)),
- strings.TrimSpace(subjectLabel),
- strings.TrimSpace(subjectType),
- strings.TrimSpace(itemTypeLabel),
- strings.TrimSpace(clusterLabel),
- strings.TrimSpace(nodeLabel),
- strings.TrimSpace(nsLabel),
- strings.TrimSpace(entityID),
- strings.TrimSpace(repoLabel),
- strings.TrimSpace(detailSummary),
- }, " "))
- if !strings.Contains(hay, q) {
- continue
- }
- }
-
- out = append(out, p)
- }
- return out
+ return recovery.FilterPoints(all, opts)
}
func getDisplayClusterLabel(d *recovery.RecoveryPointDisplay) string {
@@ -605,25 +462,6 @@ func normalizeRecoveryLimit(limit int) int {
return limit
}
-func paginateRecoveryPoints(filtered []recovery.RecoveryPoint, page int, limit int) []recovery.RecoveryPoint {
- if len(filtered) == 0 {
- return []recovery.RecoveryPoint{}
- }
-
- normalizedLimit := normalizeRecoveryLimit(limit)
- normalizedPage := normalizeRecoveryPage(page)
-
- offset := (normalizedPage - 1) * normalizedLimit
- if offset >= len(filtered) {
- return []recovery.RecoveryPoint{}
- }
- end := offset + normalizedLimit
- if end > len(filtered) {
- end = len(filtered)
- }
- return filtered[offset:end]
-}
-
func (h *RecoveryHandlers) HandleListRollups(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
diff --git a/internal/api/router.go b/internal/api/router.go
index 376add53be..09c7ce818b 100644
--- a/internal/api/router.go
+++ b/internal/api/router.go
@@ -49,7 +49,6 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
- recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
"github.com/rcourtman/pulse-go-rewrite/internal/relay"
"github.com/rcourtman/pulse-go-rewrite/internal/telemetry"
"github.com/rcourtman/pulse-go-rewrite/internal/truenas"
@@ -453,7 +452,7 @@ func (r *Router) setupRoutes() {
return service
}),
)
- recoveryManager := recoverymanager.New(r.multiTenant)
+ recoveryManager := newRecoveryReadManager(r.multiTenant)
r.recoveryHandlers = NewRecoveryHandlers(recoveryManager)
r.attentionHandlers = NewAttentionHandlers(r.configHandlers.Monitor, recoveryManager)
if r.mtMonitor != nil {
@@ -3275,7 +3274,7 @@ func (r *Router) wireAIChatDependenciesForService(ctx context.Context, service A
if monitor != nil {
if metricsHistory := monitor.GetMetricsHistory(); metricsHistory != nil {
metricsAdapter := tools.NewMetricsHistoryToolAdapter(
- &metricsSourceWrapper{history: metricsHistory},
+ &metricsSourceWrapper{history: metricsHistory, monitor: monitor},
monitor.GetUnifiedReadState(),
monitor.GetMetricsStore(),
)
@@ -3523,6 +3522,22 @@ func (w *eventCorrelatorProviderWrapper) GetCorrelationsForResource(resourceID s
// metricsSourceWrapper wraps monitoring.MetricsHistory to implement tools.MetricsSource
type metricsSourceWrapper struct {
history *monitoring.MetricsHistory
+ monitor *monitoring.Monitor
+}
+
+func (w *metricsSourceWrapper) GetStorageMetricHistory(storageID string, duration time.Duration) (map[string][]tools.RawMetricPoint, string, error) {
+ if w.monitor == nil {
+ return nil, "unavailable", fmt.Errorf("storage metrics monitor is not available")
+ }
+ series, source, err := w.monitor.ReadStorageMetricHistory(storageID, duration)
+ if err != nil {
+ return nil, "", err
+ }
+ result := make(map[string][]tools.RawMetricPoint, len(series))
+ for metric, points := range series {
+ result[metric] = convertMetricPoints(points)
+ }
+ return result, source, nil
}
func (w *metricsSourceWrapper) GetGuestMetrics(guestID string, metricType string, duration time.Duration) []tools.RawMetricPoint {
diff --git a/internal/mock/recovery_points.go b/internal/mock/recovery_points.go
index e44407e8cc..a241b1dfee 100644
--- a/internal/mock/recovery_points.go
+++ b/internal/mock/recovery_points.go
@@ -1,6 +1,7 @@
package mock
import (
+ "context"
"crypto/sha256"
"encoding/hex"
"fmt"
@@ -16,6 +17,21 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
+// RecoveryPointsSource projects the canonical demo graph through the same
+// filters and pagination for API and investigation reads. It does not seed or
+// consult a second store, and cannot supply evidence outside demo mode.
+type RecoveryPointsSource struct{}
+
+func (RecoveryPointsSource) Active() bool { return IsMockEnabled() }
+
+func (RecoveryPointsSource) ListPoints(ctx context.Context, _ string, opts recovery.ListPointsOptions) ([]recovery.RecoveryPoint, int, error) {
+ if err := ctx.Err(); err != nil {
+ return nil, 0, err
+ }
+ points := recovery.FilterPoints(CurrentFixtureGraph().RecoveryPoints(), opts)
+ return recovery.PaginatePoints(points, opts.Page, opts.Limit), len(points), nil
+}
+
func (g FixtureGraph) RecoveryPoints() []recovery.RecoveryPoint {
resources, _ := g.UnifiedResourceSnapshot()
return generateMockRecoveryPoints(g.State, g.PlatformFixtures, resources)
diff --git a/internal/mock/recovery_points_test.go b/internal/mock/recovery_points_test.go
index 06364a2091..bf9f346f28 100644
--- a/internal/mock/recovery_points_test.go
+++ b/internal/mock/recovery_points_test.go
@@ -1,6 +1,7 @@
package mock
import (
+ "context"
"strings"
"testing"
"time"
@@ -10,6 +11,47 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
+func TestRecoveryPointsSourceUsesCurrentGraphAndCanonicalSubject(t *testing.T) {
+ previous := IsMockEnabled()
+ mustSetEnabled(t, true)
+ t.Cleanup(func() { mustSetEnabled(t, previous) })
+ var expected []recovery.RecoveryPoint
+ var subjectID string
+ for _, point := range CurrentFixtureGraph().RecoveryPoints() {
+ if point.Provider == recovery.ProviderProxmoxPVE && point.Outcome == recovery.OutcomeFailed && strings.HasPrefix(point.ID, "pve-task:") {
+ subjectID = point.SubjectResourceID
+ break
+ }
+ }
+ if subjectID == "" {
+ t.Fatal("fixture has no failed guest backup")
+ }
+ for _, point := range CurrentFixtureGraph().RecoveryPoints() {
+ if point.Provider == recovery.ProviderProxmoxPVE && point.Outcome == recovery.OutcomeFailed && point.SubjectResourceID == subjectID {
+ expected = append(expected, point)
+ }
+ }
+ source := RecoveryPointsSource{}
+ opts := recovery.ListPointsOptions{Provider: recovery.ProviderProxmoxPVE, Outcome: recovery.OutcomeFailed, SubjectResourceID: subjectID, Page: 1, Limit: 1}
+ points, total, err := source.ListPoints(context.Background(), "default", opts)
+ if err != nil || total != len(expected) || len(points) != 1 || points[0].ID != expected[0].ID {
+ t.Fatalf("canonical page = %v, total %d, error %v; expected %v", points, total, err, expected)
+ }
+ opts.Page = int(^uint(0) >> 1)
+ points, total, err = source.ListPoints(context.Background(), "default", opts)
+ if err != nil || len(points) != 0 || total != len(expected) {
+ t.Fatalf("out-of-range page = %v, total %d, error %v", points, total, err)
+ }
+ // A composed reader must consult the current graph, not preserve old demo
+ // evidence after demo mode is disabled.
+ mustSetEnabled(t, false)
+ opts.Page = 1
+ points, total, err = source.ListPoints(context.Background(), "default", opts)
+ if err != nil || len(points) != 0 || total != 0 {
+ t.Fatalf("disabled demo leaked old evidence: %v, %d, %v", points, total, err)
+ }
+}
+
func TestCurrentFixtureGraphReturnsDefensiveCopies(t *testing.T) {
previous := IsMockEnabled()
mustSetEnabled(t, true)
diff --git a/internal/monitoring/monitor_metrics.go b/internal/monitoring/monitor_metrics.go
index 1d4d4d767b..17b450a80e 100644
--- a/internal/monitoring/monitor_metrics.go
+++ b/internal/monitoring/monitor_metrics.go
@@ -222,27 +222,53 @@ func (m *Monitor) GetDiskMetricsForChart(resourceID string, metricType string, d
// GetStorageMetricsForChart returns storage metrics, falling back to SQLite +
// LTTB for longer ranges.
func (m *Monitor) GetStorageMetricsForChart(storageID string, duration time.Duration) map[string][]MetricPoint {
+ series, _, _ := m.readStorageMetricHistory(storageID, duration, true)
+ return series
+}
+
+// ReadStorageMetricHistory uses the same storage history selection as charts,
+// but never fills an empty requested window with older observations. Analytical
+// consumers must see store errors and synthetic provenance rather than treating
+// an empty, failed or generated read as current operational evidence.
+func (m *Monitor) ReadStorageMetricHistory(storageID string, duration time.Duration) (map[string][]MetricPoint, string, error) {
+ return m.readStorageMetricHistory(storageID, duration, false)
+}
+
+func (m *Monitor) readStorageMetricHistory(storageID string, duration time.Duration, allowGapFill bool) (map[string][]MetricPoint, string, error) {
inMemoryResult := map[string][]MetricPoint{}
if m != nil && m.metricsHistory != nil {
inMemoryResult = m.metricsHistory.GetAllStorageMetrics(storageID, duration)
}
+ if m == nil {
+ return inMemoryResult, "unavailable", nil
+ }
if mock.IsMockEnabled() {
- return m.mockStorageMetricsForChartCached(storageID, duration, inMemoryResult, m.GetUnifiedReadStateOrSnapshot)
+ return m.mockStorageMetricsForChartCached(storageID, duration, inMemoryResult, m.GetUnifiedReadStateOrSnapshot), "mock_synthetic", nil
}
if m.metricsStore == nil {
- return inMemoryResult
+ return inMemoryResult, "memory", nil
}
- if hasSufficientChartMapCoverageForMetrics(inMemoryResult, duration, storageChartMetricTypes) {
- return inMemoryResult
+ if allowGapFill && hasSufficientChartMapCoverageForMetrics(inMemoryResult, duration, storageChartMetricTypes) {
+ return inMemoryResult, "memory", nil
}
// inMemoryResult is a fresh map from GetAllStorageMetrics; merge mutates
// in place.
best := inMemoryResult
- if converted, ok := m.queryStoreMetricMapWithGapFill("storage", storageID, duration); ok {
+ hadMemorySamples := monitorMetricMapPointCount(inMemoryResult) > 0
+ converted, ok, err := m.queryStoreMetricHistory("storage", storageID, duration, allowGapFill)
+ if err != nil {
+ return best, "", err
+ }
+ if ok {
best = mergeMetricHistory(best, converted, duration)
+ source := "store"
+ if hadMemorySamples {
+ source = "store+memory"
+ }
+ return best, source, nil
}
- return best
+ return best, "memory", nil
}
// DiskChartEntry holds temperature chart data and display metadata for a
@@ -834,23 +860,28 @@ func convertSeriesAndDownsample(points []metrics.MetricPoint, target int) []Metr
}
func (m *Monitor) queryStoreMetricMapWithGapFill(resourceType, resourceID string, duration time.Duration) (map[string][]MetricPoint, bool) {
+ result, ok, _ := m.queryStoreMetricHistory(resourceType, resourceID, duration, true)
+ return result, ok
+}
+
+func (m *Monitor) queryStoreMetricHistory(resourceType, resourceID string, duration time.Duration, allowGapFill bool) (map[string][]MetricPoint, bool, error) {
if m == nil || m.metricsStore == nil {
- return nil, false
+ return nil, false, nil
}
end := time.Now()
start := end.Add(-duration)
- query := func(from time.Time) (map[string][]MetricPoint, bool) {
+ query := func(from time.Time) (map[string][]MetricPoint, bool, error) {
sqlResult, err := m.metricsStore.QueryAll(resourceType, resourceID, from, end, 0)
if err != nil || len(sqlResult) == 0 {
- return nil, false
+ return nil, false, err
}
- return convertAndDownsample(sqlResult, chartDownsampleTarget), true
+ return convertAndDownsample(sqlResult, chartDownsampleTarget), true, nil
}
- if result, ok := query(start); ok {
- return result, true
+ if result, ok, err := query(start); ok || err != nil || !allowGapFill {
+ return result, ok, err
}
return query(end.Add(-chartGapFillLookbackWindow(duration)))
diff --git a/internal/monitoring/storage_history_test.go b/internal/monitoring/storage_history_test.go
new file mode 100644
index 0000000000..4e73f867ad
--- /dev/null
+++ b/internal/monitoring/storage_history_test.go
@@ -0,0 +1,33 @@
+package monitoring
+
+import (
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+)
+
+func TestStorageInvestigationHistoryKeepsRequestedWindowAndStoreErrors(t *testing.T) {
+ monitor := newChartFallbackTestMonitor(t)
+ now := time.Now().UTC().Truncate(time.Second)
+ writeRawMetricBatch(t, monitor.metricsStore, "storage", "pool", "usage", []MetricPoint{
+ {Timestamp: now.Add(-3 * time.Hour), Value: 44},
+ })
+ // Sparkline gap filling remains available, but an investigation asking for
+ // the last hour must not receive a three-hour-old sample as that history.
+ charts := monitor.GetStorageMetricsForChart("pool", time.Hour)
+ require.NotEmpty(t, charts["usage"])
+ series, _, err := monitor.ReadStorageMetricHistory("pool", time.Hour)
+ require.NoError(t, err)
+ require.Empty(t, series["usage"])
+ series, source, err := monitor.ReadStorageMetricHistory("pool", 7*24*time.Hour)
+ require.NoError(t, err)
+ require.Equal(t, "store", source)
+ require.Len(t, series["usage"], 1)
+ require.Equal(t, 44.0, series["usage"][0].Value)
+ // Even with an available memory sample, a failed retained read is explicit.
+ monitor.metricsHistory.AddStorageMetric("pool", "usage", 45, now)
+ require.NoError(t, monitor.metricsStore.Close())
+ _, _, err = monitor.ReadStorageMetricHistory("pool", time.Hour)
+ require.Error(t, err)
+}
diff --git a/internal/recovery/manager/manager.go b/internal/recovery/manager/manager.go
index 66e0d1a834..1eb5733fa8 100644
--- a/internal/recovery/manager/manager.go
+++ b/internal/recovery/manager/manager.go
@@ -1,12 +1,14 @@
package manager
import (
+ "context"
"fmt"
"path/filepath"
"strings"
"sync"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"
recoverystore "github.com/rcourtman/pulse-go-rewrite/internal/recovery/store"
)
@@ -14,18 +16,55 @@ import (
// This avoids opening multiple sqlite connections to the same file across the process.
type Manager struct {
multiTenant *config.MultiTenantPersistence
+ pointSource PointsSource
mu sync.Mutex
stores map[string]*recoverystore.Store
}
+// PointsSource supplies read-only recovery evidence from an alternative runtime
+// source, such as the canonical demo graph. It never replaces persisted writes.
+type PointsSource interface {
+ Active() bool
+ ListPoints(context.Context, string, recovery.ListPointsOptions) ([]recovery.RecoveryPoint, int, error)
+}
+
func New(mtp *config.MultiTenantPersistence) *Manager {
+ return NewWithPointSource(mtp, nil)
+}
+
+// NewWithPointSource binds the read source once, before runtime consumers share
+// the manager. Inactive sources defer to the manager's same org-scoped store.
+func NewWithPointSource(mtp *config.MultiTenantPersistence, source PointsSource) *Manager {
return &Manager{
multiTenant: mtp,
+ pointSource: source,
stores: make(map[string]*recoverystore.Store),
}
}
+// ListPointsForOrg is the shared API/tool boundary for recovery point reads.
+func (m *Manager) ListPointsForOrg(ctx context.Context, orgID string, opts recovery.ListPointsOptions) ([]recovery.RecoveryPoint, int, error) {
+ if m == nil {
+ return nil, 0, fmt.Errorf("recovery manager is not configured")
+ }
+ orgID = strings.TrimSpace(orgID)
+ if orgID == "" {
+ return nil, 0, fmt.Errorf("org ID is required")
+ }
+ if err := ctx.Err(); err != nil {
+ return nil, 0, err
+ }
+ if m.pointSource != nil && m.pointSource.Active() {
+ return m.pointSource.ListPoints(ctx, orgID, opts)
+ }
+ store, err := m.StoreForOrg(orgID)
+ if err != nil {
+ return nil, 0, err
+ }
+ return store.ListPoints(ctx, opts)
+}
+
func (m *Manager) StoreForOrg(orgID string) (*recoverystore.Store, error) {
orgID = strings.TrimSpace(orgID)
if orgID == "" {
diff --git a/internal/recovery/manager/manager_test.go b/internal/recovery/manager/manager_test.go
index 26a049f7f3..5a90f4fc63 100644
--- a/internal/recovery/manager/manager_test.go
+++ b/internal/recovery/manager/manager_test.go
@@ -1,9 +1,58 @@
package manager
import (
+ "context"
+ "errors"
"testing"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"
)
+type testPointsSource struct {
+ active bool
+ orgID string
+ err error
+ called bool
+}
+
+func (s *testPointsSource) Active() bool { return s.active }
+
+func (s *testPointsSource) ListPoints(_ context.Context, orgID string, _ recovery.ListPointsOptions) ([]recovery.RecoveryPoint, int, error) {
+ s.called = true
+ s.orgID = orgID
+ return []recovery.RecoveryPoint{{ID: "source-point"}}, 1, s.err
+}
+
+func TestListPointsForOrgPreservesSourceBoundary(t *testing.T) {
+ source := &testPointsSource{active: true}
+ m := NewWithPointSource(nil, source)
+ if _, _, err := m.ListPointsForOrg(context.Background(), " ", recovery.ListPointsOptions{}); err == nil || source.called {
+ t.Fatal("missing tenant must fail before reading any source")
+ }
+ points, total, err := m.ListPointsForOrg(context.Background(), " tenant-a ", recovery.ListPointsOptions{})
+ if err != nil || source.orgID != "tenant-a" || total != 1 || len(points) != 1 || points[0].ID != "source-point" {
+ t.Fatalf("read = %v, %d, %v; org = %q", points, total, err, source.orgID)
+ }
+ want := errors.New("source unavailable")
+ source.err = want
+ if _, _, err := m.ListPointsForOrg(context.Background(), "tenant-a", recovery.ListPointsOptions{}); !errors.Is(err, want) {
+ t.Fatalf("source error was lost: %v", err)
+ }
+ source.called = false
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ if _, _, err := m.ListPointsForOrg(ctx, "tenant-a", recovery.ListPointsOptions{}); !errors.Is(err, context.Canceled) || source.called {
+ t.Fatal("cancelled read must not call the source")
+ }
+ if _, _, err := New(nil).ListPointsForOrg(context.Background(), "tenant-a", recovery.ListPointsOptions{}); err == nil {
+ t.Fatal("unconfigured persisted source must not become an empty success")
+ }
+ source.active = false
+ if _, _, err := m.ListPointsForOrg(context.Background(), "tenant-a", recovery.ListPointsOptions{}); err == nil || source.called {
+ t.Fatal("inactive source must defer to the persisted boundary")
+ }
+}
+
func TestNew(t *testing.T) {
// Test that New returns a non-nil Manager
m := New(nil)
diff --git a/internal/recovery/points_query.go b/internal/recovery/points_query.go
new file mode 100644
index 0000000000..806a41cce2
--- /dev/null
+++ b/internal/recovery/points_query.go
@@ -0,0 +1,195 @@
+package recovery
+
+import "strings"
+
+// FilterPoints applies the canonical recovery query to graph-owned in-memory evidence.
+// Persisted reads use the equivalent indexed store query.
+func FilterPoints(all []RecoveryPoint, opts ListPointsOptions) []RecoveryPoint {
+ if len(all) == 0 {
+ return nil
+ }
+
+ provider := strings.TrimSpace(string(opts.Provider))
+ kind := strings.TrimSpace(string(opts.Kind))
+ mode := strings.TrimSpace(string(opts.Mode))
+ outcome := strings.TrimSpace(string(opts.Outcome))
+ subjectRID := strings.TrimSpace(opts.SubjectResourceID)
+ rollupID := strings.TrimSpace(opts.RollupID)
+ cluster := strings.TrimSpace(opts.ClusterLabel)
+ node := strings.TrimSpace(opts.NodeHostLabel)
+ namespace := strings.TrimSpace(opts.NamespaceLabel)
+ itemType := NormalizeRecoveryItemType(opts.ItemType)
+ q := strings.ToLower(strings.TrimSpace(opts.Query))
+ workloadOnly := opts.WorkloadOnly
+ verification := strings.ToLower(strings.TrimSpace(opts.Verification))
+ if rollupID != "" && !strings.HasPrefix(rollupID, "res:") && !strings.HasPrefix(rollupID, "ext:") {
+ rollupID = "res:" + rollupID
+ }
+
+ out := make([]RecoveryPoint, 0, len(all))
+ for _, p := range all {
+ // Ensure display/index data exists for filtering parity with sqlite store.
+ if p.Display == nil {
+ idx := DeriveIndex(p)
+ p.Display = idx.ToDisplay()
+ }
+ disp := p.Display
+
+ if provider != "" && strings.TrimSpace(string(p.Provider)) != provider {
+ continue
+ }
+ if kind != "" && strings.TrimSpace(string(p.Kind)) != kind {
+ continue
+ }
+ if mode != "" && strings.TrimSpace(string(p.Mode)) != mode {
+ continue
+ }
+ if outcome != "" && strings.TrimSpace(string(p.Outcome)) != outcome {
+ continue
+ }
+ if rollupID != "" {
+ if strings.TrimSpace(SubjectKeyForPoint(p)) != rollupID {
+ continue
+ }
+ } else if subjectRID != "" && strings.TrimSpace(p.SubjectResourceID) != subjectRID {
+ continue
+ }
+ if opts.From != nil && !opts.From.IsZero() {
+ // Match store semantics: completed_at_ms IS NULL OR completed_at_ms >= from
+ if p.CompletedAt != nil && !p.CompletedAt.IsZero() && p.CompletedAt.Before(opts.From.UTC()) {
+ continue
+ }
+ }
+ if opts.To != nil && !opts.To.IsZero() {
+ // Match store semantics: completed_at_ms IS NULL OR completed_at_ms <= to
+ if p.CompletedAt != nil && !p.CompletedAt.IsZero() && p.CompletedAt.After(opts.To.UTC()) {
+ continue
+ }
+ }
+
+ if cluster != "" && strings.TrimSpace(getDisplayClusterLabel(p.Display)) != cluster {
+ continue
+ }
+ if node != "" && strings.TrimSpace(getDisplayNodeHostLabel(p.Display)) != node {
+ continue
+ }
+ if namespace != "" && strings.TrimSpace(getDisplayNamespaceLabel(p.Display)) != namespace {
+ continue
+ }
+ if itemType != "" && strings.TrimSpace(getDisplayItemType(p.Display)) != itemType {
+ continue
+ }
+ if workloadOnly && !(disp != nil && disp.IsWorkload) {
+ continue
+ }
+ if verification != "" {
+ v := p.Verified
+ switch verification {
+ case "verified":
+ if v == nil || !*v {
+ continue
+ }
+ case "unverified":
+ if v == nil || *v {
+ continue
+ }
+ case "unknown":
+ if v != nil {
+ continue
+ }
+ }
+ }
+ if q != "" {
+ // Best-effort match across normalized display fields.
+ var subjectLabel, subjectType, itemTypeLabel, clusterLabel, nodeLabel, nsLabel, entityID, repoLabel, detailSummary string
+ if disp != nil {
+ subjectLabel = disp.SubjectLabel
+ subjectType = disp.SubjectType
+ itemTypeLabel = disp.ItemType
+ clusterLabel = disp.ClusterLabel
+ nodeLabel = disp.NodeHostLabel
+ nsLabel = disp.NamespaceLabel
+ entityID = disp.EntityIDLabel
+ repoLabel = disp.RepositoryLabel
+ detailSummary = disp.DetailsSummary
+ }
+
+ hay := strings.ToLower(strings.Join([]string{
+ strings.TrimSpace(p.ID),
+ strings.TrimSpace(string(p.Provider)),
+ strings.TrimSpace(string(p.Kind)),
+ strings.TrimSpace(string(p.Mode)),
+ strings.TrimSpace(string(p.Outcome)),
+ strings.TrimSpace(subjectLabel),
+ strings.TrimSpace(subjectType),
+ strings.TrimSpace(itemTypeLabel),
+ strings.TrimSpace(clusterLabel),
+ strings.TrimSpace(nodeLabel),
+ strings.TrimSpace(nsLabel),
+ strings.TrimSpace(entityID),
+ strings.TrimSpace(repoLabel),
+ strings.TrimSpace(detailSummary),
+ }, " "))
+ if !strings.Contains(hay, q) {
+ continue
+ }
+ }
+
+ out = append(out, p)
+ }
+ return out
+}
+
+func getDisplayClusterLabel(d *RecoveryPointDisplay) string {
+ if d == nil {
+ return ""
+ }
+ return d.ClusterLabel
+}
+
+func getDisplayNodeHostLabel(d *RecoveryPointDisplay) string {
+ if d == nil {
+ return ""
+ }
+ return d.NodeHostLabel
+}
+
+func getDisplayNamespaceLabel(d *RecoveryPointDisplay) string {
+ if d == nil {
+ return ""
+ }
+ return d.NamespaceLabel
+}
+
+func getDisplayItemType(d *RecoveryPointDisplay) string {
+ if d == nil {
+ return ""
+ }
+ if v := NormalizeRecoveryItemType(d.ItemType); v != "" {
+ return v
+ }
+ return NormalizeRecoveryItemType(d.SubjectType)
+}
+
+// PaginatePoints applies the same bounds as the persisted recovery store.
+func PaginatePoints(points []RecoveryPoint, page, limit int) []RecoveryPoint {
+ if page <= 0 {
+ page = 1
+ }
+ if limit <= 0 {
+ limit = DefaultListPageLimit
+ }
+ if limit > MaxListPageLimit {
+ limit = MaxListPageLimit
+ }
+ // Compare before multiplying so a caller-supplied page cannot overflow.
+ if len(points) == 0 || page > (len(points)-1)/limit+1 {
+ return []RecoveryPoint{}
+ }
+ offset := (page - 1) * limit
+ end := offset + limit
+ if end > len(points) {
+ end = len(points)
+ }
+ return points[offset:end]
+}
diff --git a/scripts/release_control/canonical_completion_guard_test.py b/scripts/release_control/canonical_completion_guard_test.py
index ab859a4dd2..6968062118 100644
--- a/scripts/release_control/canonical_completion_guard_test.py
+++ b/scripts/release_control/canonical_completion_guard_test.py
@@ -1379,6 +1379,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"internal/api/docker_container_action_executor_test.go",
"internal/api/docker_container_action_result_test.go",
"internal/api/host_agent_removal_lifecycle_integration_test.go",
+ "internal/api/investigation_evidence_parity_test.go",
"internal/api/issue1640_readiness_gate_test.go",
"internal/api/issue1640_readiness_transport_test.go",
"internal/api/metadata_handlers_test.go",
@@ -2841,6 +2842,7 @@ None yet.
"internal/monitoring/disk_drawer_native_history_test.go",
"internal/monitoring/monitor_metrics_chart_batch_bench_test.go",
"internal/monitoring/monitor_metrics_slo_test.go",
+ "internal/monitoring/storage_history_test.go",
],
}
],