Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Collect LXC filesystems without pmxcfs or LXC socket IPC
rcourtman6b1fc6eaaff4
diff --git a/docs/AGENT_SECURITY.md b/docs/AGENT_SECURITY.mdindex 20b5bf4cff..cd298ca6a6 100644--- a/docs/AGENT_SECURITY.md+++ b/docs/AGENT_SECURITY.md@@ -279,7 +279,7 @@ or generic command path.| Proxmox VE/PBS/PMG inventory, status, storage, and ordinary metrics | API-only connection with a narrowly scoped token; no host agent | No collector, helper, or runner authority is required for this data | Supported independently of the safe host-agent profile; it does not prove host-local SMART, LXC filesystem, or action parity | `agent-lifecycle`: keep API permissions and returned telemetry covered by provider tests || Standard Linux systemd host telemetry and collector update | Unprivileged `pulse-agent` plus the root-owned typed helper | Core `/proc`, filesystem, network, RAID, and hwmon telemetry stays in the collector; signed updates cross the fixed helper protocol and commit only after an authoritative report | **Qualified on disposable Ubuntu 24.04.4 arm64 at committed main `22fd662fb794f63efb9d3ca2158de73c4e07e1b8`** across the schema-v6 twenty-scenario run. It covers install, migration, explicit/automatic rollback, helper protocol and resource boundaries, effective-unit override rejection, reporting continuity, authoritative executable-digest commit, watchdog rollback, helper-restart recovery, and last-known-good restoration. It remains opt-in pending exact-RC reproduction, representative provider/appliance parity, and external review | `deployment-installability` and `security-privacy`: reproduce the same helper transaction from trusted release artifacts on representative hosts and accept the external boundary review || Linux SMART telemetry | `smart.snapshot` through the no-network helper; no caller-selected device or arguments | Implemented, unqualified on representative physical disks. Helper failure omits/degrades SMART only; the collector does not retry as root | Does not yet justify SMART parity or a default change | `agent-lifecycle`: record live SATA, SAS/controller, USB bridge, and NVMe evidence, including standby, permission failure, timeout, and partial-data cases |-| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper using fixed bounded `pct` operations | Implemented, unqualified on a representative PVE node. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record live running/stopped LXC, mount, timeout, output-bound, and helper-loss behavior on supported PVE versions |+| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper. Configs are read through the pmxcfs mount, running guests and their init processes come from cgroup v2 and `/proc`, and usage comes from the shared filesystem prober, which pins the guest's init, re-checks its identity, and resolves each mount inside the guest root (openat2, or a symlink-refusing component walk where the helper's seccomp filter blocks openat2); the caller selects no VMID, path, or command. `pct` and `lxc-info` cannot work inside the helper, because pmxcfs and the LXC monitor listen on abstract Unix sockets that belong to the host network namespace (#2511); they remain the fallback whenever discovery cannot establish a guest's state, on cgroup v1 hosts, for root and `--grant-pct` installs, and per container when the prober read fails | Implemented. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties, with running unprivileged (rootfs plus a mount point) and privileged guests and a stopped one; readings matched `pct df`. This is one PVE version, with no timeout, output-bound, helper-loss, or live nested-namespace evidence. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record timeout, output-bound, and helper-loss behavior across supported PVE versions || Rootful Docker or Podman inventory | No direct collector access to a root-equivalent daemon socket | Implemented as a typed-helper summary-only fallback. Migration preserves container ID/name/image/state/status/creation inventory and marks the report `typed-helper-summary`; stats, secondary inventories, update checks, and actions remain unavailable | The standalone rootless packet exercises live same-family rootful Docker and Podman helper fallback plus summary-inventory semantic parity during socket loss. It does not qualify the complete standalone rootful fresh-install, migration, restart, helper-loss/recovery, and bounds matrix. Full rootful parity remains an explicit default blocker; the legacy/root profile is not safe-profile evidence | `agent-lifecycle`: record the remaining standalone lifecycle matrix on representative rootful Docker and Podman and decide explicitly whether reduced telemetry is sufficient || Collector-owned rootless Docker or Podman | Direct access only to one usable runtime socket owned by the `pulse-agent` UID whose daemon also attests rootless mode | Qualified locally for the exact source-bound monitoring path. Discovery is deferred until the collector account exists and accepts exactly one readable/writable, non-symlink socket owned by that UID across Docker and Podman. Cross-user, root-owned, remote, unreadable, non-rootless-daemon, or ambiguous endpoints are rejected. An installer-owned exact pin survives daemon loss and update; the running collector can move between direct rootless monitoring and rootful typed-helper summary without regaining action authority | **Qualified on distinct disposable Ubuntu 24.04/systemd 255 arm64 hosts at commit `60041ad9e60c282c892f944e04f777b874991a5d`.** All eleven canonical scenarios passed for Docker and all eleven for Podman. Receipt SHA-256: `7a116d63ab0cd1560482165055f0a8c9158ce0a8333a27bd9e8256582a52dfb5`; local attestation SHA-256: `566279ecd7d7dfa89c92f24243e9fcd5ae3e295d4284bd559d4be42f1e83b3b5`. This is local opt-in artifact-bound self-attestation, not published-RC provenance, action/update qualification, external review, or authorization for a default change | `deployment-installability`: reproduce the exact packet from published release artifacts; separately qualify any desired rootless image-update/action surface; retain the opt-in default until the remaining provider, appliance, and review conditions pass || Separate runner package update and package-cache cleanup | Root-owned `pulse-agent-runner`, host-bound action credential, typed request, postcondition, and durable receipt | The schema-v6 committed-main systemd receipt records a real verified apt-cache mutation, stale-fingerprint refusal, replay, nonce-bound readiness, effective-unit override rejection, exact credential rotation, bodyless self-revocation, and collector/helper continuity. A separate production Router regression exercises HTTPS issuance, WSS admission, encrypted token persistence, failed-rotation rollback, exact socket invalidation, two server restarts, old-secret rejection, and durable self-revoke | Qualified for the exercised systemd fixture paths and focused production Router lifecycle. Focused installer tests also cover atomic pending cancellation and fail-closed credential retention. The evidence does not cover every runner operation, representative providers, or an exact release candidate | `agent-lifecycle` and `api-contracts`: reproduce the combined systemd and production Router path from trusted RC artifacts with representative package-update success/failure/cancellation evidence |diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 911aa56513..b51b2d57dc 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -572,11 +572,32 @@ standby evidence. Enumeration and each device probe have independent boundeddeadlines so one slow disk cannot consume the complete SMART pass, whilecommand errors retain bounded stderr for compatibility diagnosis.On a PVE node, the Unified Agent may also report the mounted filesystem-capacity of running LXCs. The collector uses the fixed local `pct list` command-to admit only node-local running guests, reads each admitted guest's main-`/etc/pve/lxc/<vmid>.conf` mount declarations, resolves its init PID through-the fixed `lxc-info -n <vmid> -p` shape, and probes usage through the Linux-`/proc/<pid>/root` namespace. A configured path that resolves to the same+capacity of running LXCs. On a cgroup v2 node the collector admits node-local+guests from the `/etc/pve/lxc/*.conf` listing and admits only those with a live+init process: a PID in `/sys/fs/cgroup/lxc/<vmid>`'s tree that is PID 1 of the+namespace directly below the host's (`NSpid` exactly `<pid> 1`) and whose own+cgroup is inside `lxc/<vmid>`. It names each by its configured hostname or+`CT<vmid>`, as `pct list` does, reads its main mount declarations, and reads+usage through the shared `internal/filesystemprobe` observer, which pins the+process, re-checks that identity, resolves each mount inside the guest root+and holds one bounded in-flight probe per guest across the process. No command+runs on this path. `pct list`, `pct df` and `lxc-info` reach pmxcfs and the+LXC monitor through abstract Unix sockets scoped to the host network+namespace, so inside the `PrivateNetwork` typed helper they always fail, and+least-privilege installs using that helper reported `provider_unavailable`+on every report (#2511). Discovery hands the whole node back to `pct list`+whenever it cannot establish a guest's state (no cgroup v2, an unreadable+listing, config, cgroup or process, a search limit, or cancellation); only a+guest without a cgroup counts as stopped. The fixed `lxc-info -n <vmid> -p`+PID lookup remains for that path, and `pct df` remains the per-container+fallback, including through the `--grant-pct` wrapper.+`TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets` drives the collector+the helper's provider runs with every command failing and requires a complete+inventory; `TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct`+pins the fallbacks. On a disposable PVE 9.2.18 node, under the helper unit's+sandbox properties, it returned unprivileged (rootfs plus mount point) and+privileged guests matching `pct df` and omitted a stopped one; a nested PID+namespace was not reproduced there and is covered by tests only. A configured path that resolves to the samedevice as its parent is not a live mount and must be omitted instead ofborrowing the parent's capacity. Least-privilege installs that cannot read theconfig, resolve the PID, or traverse `/proc` may fall back to the fixeddiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 5ee9db4dcc..f02a9c2224 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -844,6 +844,27 @@ tests, container collection boundary tests, ingestion regression and theopt-in owned tmpfs collector fixture. Installed-agent and model qualificationare tracked separately in PATROL_ASSISTANT_CUSTOMER_JOURNEY.md.+The same observer reads Proxmox LXC guests for the host agent and its typed+helper (#2511). An `lxc` request names the decimal VMID; the pinned process+must sit in that guest's exact `lxc/<vmid>` cgroup and be its init (`NSpid` is+exactly `<pid> 1`, so the PID 1 of a namespace nested inside the guest, such+as Docker in LXC, cannot stand in for it), checked before and after the read.+With `MountsOnly`, a configured mountpoint sharing its parent's device reports+`ErrNotMounted` instead of the parent's counters. Docker and Podman identity+rules are unchanged. Confined resolution prefers openat2 with `RESOLVE_IN_ROOT`;+when the kernel or a systemd seccomp filter answers ENOSYS (the typed helper's+`RestrictSUIDSGID=true` blocks openat2 because its flags cannot be inspected),+the clean path is walked component by component with `O_NOFOLLOW` and any+symbolic link is refused, so neither path resolves outside the guest root.+Docker and Podman reads take the same fallback, so a collector under such a+filter now gets confined counters instead of unavailable evidence.+`TestComponentWalkRefusesSymlinksAndStaysBeneathRoot`,+`TestMountsOnlyDetectsUnmountedPathsWithinRoot`,+`TestNonInitProcessCannotSupplyProxmoxGuestCapacity` and+`TestProxmoxLXCIdentityIsExact` pin these rules; the Linux tests also passed+as root on a disposable PVE 9.2.18 node, where the helper-sandboxed collector+returned readings matching `pct df` through the component walk.+Docker mount collection preserves both native `Mounts` records and entriesreported only in `HostConfig.Tmpfs`. Existing reported destinations remainauthoritative. Additional tmpfs destinations are ordered deterministically,diff --git a/frontend-modern/public/docs/AGENT_SECURITY.md b/frontend-modern/public/docs/AGENT_SECURITY.mdindex 20b5bf4cff..cd298ca6a6 100644--- a/frontend-modern/public/docs/AGENT_SECURITY.md+++ b/frontend-modern/public/docs/AGENT_SECURITY.md@@ -279,7 +279,7 @@ or generic command path.| Proxmox VE/PBS/PMG inventory, status, storage, and ordinary metrics | API-only connection with a narrowly scoped token; no host agent | No collector, helper, or runner authority is required for this data | Supported independently of the safe host-agent profile; it does not prove host-local SMART, LXC filesystem, or action parity | `agent-lifecycle`: keep API permissions and returned telemetry covered by provider tests || Standard Linux systemd host telemetry and collector update | Unprivileged `pulse-agent` plus the root-owned typed helper | Core `/proc`, filesystem, network, RAID, and hwmon telemetry stays in the collector; signed updates cross the fixed helper protocol and commit only after an authoritative report | **Qualified on disposable Ubuntu 24.04.4 arm64 at committed main `22fd662fb794f63efb9d3ca2158de73c4e07e1b8`** across the schema-v6 twenty-scenario run. It covers install, migration, explicit/automatic rollback, helper protocol and resource boundaries, effective-unit override rejection, reporting continuity, authoritative executable-digest commit, watchdog rollback, helper-restart recovery, and last-known-good restoration. It remains opt-in pending exact-RC reproduction, representative provider/appliance parity, and external review | `deployment-installability` and `security-privacy`: reproduce the same helper transaction from trusted release artifacts on representative hosts and accept the external boundary review || Linux SMART telemetry | `smart.snapshot` through the no-network helper; no caller-selected device or arguments | Implemented, unqualified on representative physical disks. Helper failure omits/degrades SMART only; the collector does not retry as root | Does not yet justify SMART parity or a default change | `agent-lifecycle`: record live SATA, SAS/controller, USB bridge, and NVMe evidence, including standby, permission failure, timeout, and partial-data cases |-| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper using fixed bounded `pct` operations | Implemented, unqualified on a representative PVE node. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record live running/stopped LXC, mount, timeout, output-bound, and helper-loss behavior on supported PVE versions |+| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper. Configs are read through the pmxcfs mount, running guests and their init processes come from cgroup v2 and `/proc`, and usage comes from the shared filesystem prober, which pins the guest's init, re-checks its identity, and resolves each mount inside the guest root (openat2, or a symlink-refusing component walk where the helper's seccomp filter blocks openat2); the caller selects no VMID, path, or command. `pct` and `lxc-info` cannot work inside the helper, because pmxcfs and the LXC monitor listen on abstract Unix sockets that belong to the host network namespace (#2511); they remain the fallback whenever discovery cannot establish a guest's state, on cgroup v1 hosts, for root and `--grant-pct` installs, and per container when the prober read fails | Implemented. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties, with running unprivileged (rootfs plus a mount point) and privileged guests and a stopped one; readings matched `pct df`. This is one PVE version, with no timeout, output-bound, helper-loss, or live nested-namespace evidence. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record timeout, output-bound, and helper-loss behavior across supported PVE versions || Rootful Docker or Podman inventory | No direct collector access to a root-equivalent daemon socket | Implemented as a typed-helper summary-only fallback. Migration preserves container ID/name/image/state/status/creation inventory and marks the report `typed-helper-summary`; stats, secondary inventories, update checks, and actions remain unavailable | The standalone rootless packet exercises live same-family rootful Docker and Podman helper fallback plus summary-inventory semantic parity during socket loss. It does not qualify the complete standalone rootful fresh-install, migration, restart, helper-loss/recovery, and bounds matrix. Full rootful parity remains an explicit default blocker; the legacy/root profile is not safe-profile evidence | `agent-lifecycle`: record the remaining standalone lifecycle matrix on representative rootful Docker and Podman and decide explicitly whether reduced telemetry is sufficient || Collector-owned rootless Docker or Podman | Direct access only to one usable runtime socket owned by the `pulse-agent` UID whose daemon also attests rootless mode | Qualified locally for the exact source-bound monitoring path. Discovery is deferred until the collector account exists and accepts exactly one readable/writable, non-symlink socket owned by that UID across Docker and Podman. Cross-user, root-owned, remote, unreadable, non-rootless-daemon, or ambiguous endpoints are rejected. An installer-owned exact pin survives daemon loss and update; the running collector can move between direct rootless monitoring and rootful typed-helper summary without regaining action authority | **Qualified on distinct disposable Ubuntu 24.04/systemd 255 arm64 hosts at commit `60041ad9e60c282c892f944e04f777b874991a5d`.** All eleven canonical scenarios passed for Docker and all eleven for Podman. Receipt SHA-256: `7a116d63ab0cd1560482165055f0a8c9158ce0a8333a27bd9e8256582a52dfb5`; local attestation SHA-256: `566279ecd7d7dfa89c92f24243e9fcd5ae3e295d4284bd559d4be42f1e83b3b5`. This is local opt-in artifact-bound self-attestation, not published-RC provenance, action/update qualification, external review, or authorization for a default change | `deployment-installability`: reproduce the exact packet from published release artifacts; separately qualify any desired rootless image-update/action surface; retain the opt-in default until the remaining provider, appliance, and review conditions pass || Separate runner package update and package-cache cleanup | Root-owned `pulse-agent-runner`, host-bound action credential, typed request, postcondition, and durable receipt | The schema-v6 committed-main systemd receipt records a real verified apt-cache mutation, stale-fingerprint refusal, replay, nonce-bound readiness, effective-unit override rejection, exact credential rotation, bodyless self-revocation, and collector/helper continuity. A separate production Router regression exercises HTTPS issuance, WSS admission, encrypted token persistence, failed-rotation rollback, exact socket invalidation, two server restarts, old-secret rejection, and durable self-revoke | Qualified for the exercised systemd fixture paths and focused production Router lifecycle. Focused installer tests also cover atomic pending cancellation and fail-closed credential retention. The evidence does not cover every runner operation, representative providers, or an exact release candidate | `agent-lifecycle` and `api-contracts`: reproduce the combined systemd and production Router path from trusted RC artifacts with representative package-update success/failure/cancellation evidence |diff --git a/internal/filesystemprobe/observer.go b/internal/filesystemprobe/observer.goindex a4aa8587d0..7d255f0737 100644--- a/internal/filesystemprobe/observer.go+++ b/internal/filesystemprobe/observer.go@@ -7,6 +7,7 @@ import ("errors""fmt""path"+ "strconv""strings""sync""time"@@ -24,8 +25,24 @@ const (type ContainerRequest struct {PID intContainerID string+ // Runtime is docker or podman (ContainerID is the 64-hex ID) or lxc, a+ // Proxmox container (ContainerID is the decimal VMID and PID must be the+ // container's init process).Runtime stringMountpoints []string+ // MountsOnly reports a mountpoint that shares its parent's device as+ // ErrNotMounted instead of returning the parent filesystem's counters. A+ // Proxmox config can declare a mount point the running guest never mounted.+ MountsOnly bool+}++// ErrNotMounted marks an observation whose path is not a distinct mounted+// filesystem inside the container. Callers test it with IsNotMounted.+var ErrNotMounted = errors.New("path is not a mounted filesystem in the container")++// IsNotMounted reports whether an observation was rejected as ErrNotMounted.+func IsNotMounted(observation filesystem.Observation) bool {+ return observation.Error == ErrNotMounted.Error()}// Observer bounds potentially uninterruptible kernel reads. A timed-out probe@@ -101,12 +118,18 @@ func validateRequest(req ContainerRequest) error {if req.PID <= 0 {return errors.New("container process is unavailable")}- if len(req.ContainerID) != 64 || strings.IndexFunc(req.ContainerID, func(r rune) bool {- return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'f')- }) >= 0 {- return errors.New("filesystem observation requires an exact container ID")- }- if req.Runtime != "docker" && req.Runtime != "podman" {+ switch req.Runtime {+ case "docker", "podman":+ if len(req.ContainerID) != 64 || strings.IndexFunc(req.ContainerID, func(r rune) bool {+ return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'f')+ }) >= 0 {+ return errors.New("filesystem observation requires an exact container ID")+ }+ case "lxc":+ if !validLXCVMID(req.ContainerID) {+ return errors.New("filesystem observation requires an exact Proxmox VMID")+ }+ default:return errors.New("unsupported container runtime")}if len(req.Mountpoints) == 0 || len(req.Mountpoints) > maxMountpoints {@@ -122,9 +145,53 @@ func validateRequest(req ContainerRequest) error {return nil}+func validLXCVMID(id string) bool {+ vmid, err := strconv.Atoi(id)+ return err == nil && vmid >= 100 && vmid <= 999999999 && strconv.Itoa(vmid) == id+}++// CgroupMatches reports whether a /proc/<pid>/cgroup body places the process+// inside the given container, with the same exact-identity rules the observer+// applies to its pinned process.+func CgroupMatches(raw, runtime, containerID string) bool {+ return cgroupMatches(raw, ContainerRequest{Runtime: runtime, ContainerID: containerID})+}++// IsNamespaceInit reports whether a /proc/<pid>/status body describes pid as+// PID 1 of a PID namespace one level below the reader's: NSpid is exactly+// "<pid> 1". A process that is PID 1 of a namespace nested inside the+// container (Docker inside an LXC guest) lists three or more PIDs and is not+// the container's init.+func IsNamespaceInit(status string, pid int) bool {+ for _, line := range strings.Split(status, "\n") {+ rest, ok := strings.CutPrefix(line, "NSpid:")+ if !ok {+ continue+ }+ fields := strings.Fields(rest)+ return len(fields) == 2 && fields[0] == strconv.Itoa(pid) && fields[1] == "1"+ }+ return false+}+// cgroupMatches accepts an exact runtime-owned cgroup component, never a// substring, container name, short ID, or unrelated process with the same PID.func cgroupMatches(raw string, req ContainerRequest) bool {+ if req.Runtime == "lxc" {+ // pve-container places a guest under lxc/<vmid> (payload in ns/).+ // lxc.monitor/<vmid> and other VMIDs sharing a prefix never match.+ for _, line := range strings.Split(raw, "\n") {+ parts := strings.SplitN(line, ":", 3)+ if len(parts) != 3 {+ continue+ }+ components := strings.Split(parts[2], "/")+ if len(components) >= 3 && components[0] == "" && components[1] == "lxc" && components[2] == req.ContainerID {+ return true+ }+ }+ return false+ }scope := "docker-" + req.ContainerID + ".scope"if req.Runtime == "podman" {scope = "libpod-" + req.ContainerID + ".scope"diff --git a/internal/filesystemprobe/observer_linux.go b/internal/filesystemprobe/observer_linux.goindex 2e2b33ee44..63006ff4d1 100644--- a/internal/filesystemprobe/observer_linux.go+++ b/internal/filesystemprobe/observer_linux.go@@ -9,7 +9,9 @@ import ("io""math""os"+ "path""strconv"+ "strings""time""github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"@@ -25,7 +27,7 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.Oreturn nil, fmt.Errorf("open container process: %w", err)}defer unix.Close(proc)- if err := verifyCgroup(proc, req); err != nil {+ if err := verifyIdentity(proc, req); err != nil {return nil, err}root, err := unix.Openat(proc, "root", unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)@@ -33,7 +35,7 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.Oreturn nil, fmt.Errorf("open container filesystem namespace: %w", err)}defer unix.Close(root)- if err := verifyCgroup(proc, req); err != nil {+ if err := verifyIdentity(proc, req); err != nil {return nil, err}observations := make([]filesystem.Observation, 0, len(req.Mountpoints))@@ -42,6 +44,14 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.Oreturn nil, err}observation := filesystem.Observation{Mountpoint: mount, Source: Source}+ if req.MountsOnly {+ if mounted, mountErr := mountedWithinRoot(root, mount); mountErr == nil && !mounted {+ observation.ObservedAt = time.Now().UTC()+ observation.Error = ErrNotMounted.Error()+ observations = append(observations, observation)+ continue+ }+ }observation.Type, observation.Usage, err = observePath(root, mount)observation.ObservedAt = time.Now().UTC()if err != nil {@@ -49,12 +59,70 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O}observations = append(observations, observation)}- if err := verifyCgroup(proc, req); err != nil {+ if err := verifyIdentity(proc, req); err != nil {return nil, err}return observations, nil}+func verifyIdentity(proc int, req ContainerRequest) error {+ if err := verifyCgroup(proc, req); err != nil {+ return err+ }+ if req.Runtime == "lxc" {+ return verifyNamespaceInit(proc, req.PID)+ }+ return nil+}++// verifyNamespaceInit requires the pinned process to be the guest's init, so a+// process of a namespace nested inside the guest cannot supply its own root.+func verifyNamespaceInit(proc, pid int) error {+ fd, err := unix.Openat(proc, "status", unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)+ if err != nil {+ return fmt.Errorf("read container process status: %w", err)+ }+ file := os.NewFile(uintptr(fd), "status")+ defer file.Close()+ raw, err := io.ReadAll(io.LimitReader(file, 65537))+ if err != nil || len(raw) > 65536 {+ return errors.New("container process status is unavailable")+ }+ if !IsNamespaceInit(string(raw), pid) {+ return errors.New("process is not the container's init")+ }+ return nil+}++// mountedWithinRoot reports whether mount is a different filesystem from its+// parent directory, both resolved within the resource root.+func mountedWithinRoot(root int, mount string) (bool, error) {+ if mount == "/" {+ return true, nil+ }+ device := func(target string) (uint64, error) {+ fd, err := openInRoot(root, target)+ if err != nil {+ return 0, err+ }+ defer unix.Close(fd)+ var stat unix.Stat_t+ if err := unix.Fstat(fd, &stat); err != nil {+ return 0, err+ }+ return uint64(stat.Dev), nil+ }+ own, err := device(mount)+ if err != nil {+ return false, err+ }+ parent, err := device(path.Dir(mount))+ if err != nil {+ return false, err+ }+ return own != parent, nil+}+func verifyCgroup(proc int, req ContainerRequest) error {fd, err := unix.Openat(proc, "cgroup", unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)if err != nil {@@ -72,13 +140,57 @@ func verifyCgroup(proc int, req ContainerRequest) error {return nil}-func observePath(root int, mount string) (string, *filesystem.Usage, error) {- // Absolute symlinks stay within the resource root. Magic links cannot escape- // it. Unsupported kernels fail rather than falling back to unsafe resolution.+// openInRoot opens mount beneath root without ever leaving it. With openat2,+// absolute symlinks stay within the resource root and magic links cannot+// escape it. systemd answers openat2 with ENOSYS under RestrictSUIDSGID,+// because seccomp cannot inspect its flags, and the typed helper sets that, so+// there the clean path is walked one component at a time and any symbolic link+// is refused rather than resolved. Neither path falls back to unconfined+// resolution.+func openInRoot(root int, mount string) (int, error) {fd, err := unix.Openat2(root, mount, &unix.OpenHow{Flags: unix.O_PATH | unix.O_CLOEXEC,Resolve: unix.RESOLVE_IN_ROOT | unix.RESOLVE_NO_MAGICLINKS,})+ if !errors.Is(err, unix.ENOSYS) {+ return fd, err+ }+ return openBeneathWithoutSymlinks(root, mount)+}++func openBeneathWithoutSymlinks(root int, mount string) (int, error) {+ if !path.IsAbs(mount) || path.Clean(mount) != mount {+ return -1, errors.New("mount path must be clean and absolute")+ }+ current, err := unix.Dup(root)+ if err != nil {+ return -1, err+ }+ for _, component := range strings.Split(strings.TrimPrefix(mount, "/"), "/") {+ if component == "" {+ continue+ }+ next, err := unix.Openat(current, component, unix.O_PATH|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)+ unix.Close(current)+ if err != nil {+ return -1, err+ }+ var stat unix.Stat_t+ if err := unix.Fstat(next, &stat); err != nil {+ unix.Close(next)+ return -1, err+ }+ if stat.Mode&unix.S_IFMT == unix.S_IFLNK {+ unix.Close(next)+ return -1, errors.New("mount path crosses a symbolic link")+ }+ current = next+ }+ return current, nil+}++func observePath(root int, mount string) (string, *filesystem.Usage, error) {+ fd, err := openInRoot(root, mount)if err != nil {return "", nil, fmt.Errorf("open mount within resource namespace: %w", err)}diff --git a/internal/filesystemprobe/observer_linux_test.go b/internal/filesystemprobe/observer_linux_test.goindex 2cfb979393..31e9f52be8 100644--- a/internal/filesystemprobe/observer_linux_test.go+++ b/internal/filesystemprobe/observer_linux_test.go@@ -81,3 +81,68 @@ func TestWrongProcessCannotSupplyContainerCapacity(t *testing.T) {t.Fatalf("unrelated process supplied capacity: %+v %v", got, err)}}++// A configured mountpoint that shares its parent's device is not mounted in+// the guest; reporting it would copy the parent filesystem's counters.+func TestMountsOnlyDetectsUnmountedPathsWithinRoot(t *testing.T) {+ dir := t.TempDir()+ if err := os.Mkdir(filepath.Join(dir, "data"), 0700); err != nil {+ t.Fatal(err)+ }+ fd, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)+ if err != nil {+ t.Fatal(err)+ }+ defer unix.Close(fd)+ if mounted, err := mountedWithinRoot(fd, "/"); err != nil || !mounted {+ t.Fatalf("root reported unmounted: %v %v", mounted, err)+ }+ if mounted, err := mountedWithinRoot(fd, "/data"); err != nil || mounted {+ t.Fatalf("plain directory reported as a mount: %v %v", mounted, err)+ }+ if _, err := mountedWithinRoot(fd, "/missing"); err == nil {+ t.Fatal("missing path reported a mount state")+ }+}++// The test process is not PID 1 of a namespace, so it cannot supply a Proxmox+// guest's filesystems even if a request names it.+func TestNonInitProcessCannotSupplyProxmoxGuestCapacity(t *testing.T) {+ r := ContainerRequest{PID: os.Getpid(), ContainerID: "126", Runtime: "lxc", Mountpoints: []string{"/"}, MountsOnly: true}+ if got, err := observeContainer(context.Background(), r); err == nil || got != nil {+ t.Fatalf("non-init process supplied capacity: %+v %v", got, err)+ }+}++// Inside the typed helper systemd blocks openat2 (RestrictSUIDSGID), so mounts+// are resolved component by component. That walk must stay beneath the root+// by refusing every symbolic link, including ones that would stay inside.+func TestComponentWalkRefusesSymlinksAndStaysBeneathRoot(t *testing.T) {+ dir := t.TempDir()+ if err := os.MkdirAll(filepath.Join(dir, "srv", "data"), 0700); err != nil {+ t.Fatal(err)+ }+ for name, target := range map[string]string{"inside-link": "/srv", "outside": "/etc", "relative-up": "../../.."} {+ if err := os.Symlink(target, filepath.Join(dir, name)); err != nil {+ t.Fatal(err)+ }+ }+ root, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)+ if err != nil {+ t.Fatal(err)+ }+ defer unix.Close(root)+ for _, mount := range []string{"/", "/srv", "/srv/data"} {+ fd, err := openBeneathWithoutSymlinks(root, mount)+ if err != nil {+ t.Fatalf("plain path %s refused: %v", mount, err)+ }+ unix.Close(fd)+ }+ for _, mount := range []string{"/inside-link", "/inside-link/data", "/outside", "/outside/passwd", "/relative-up", "/srv/../etc", "srv", "/proc/self/root"} {+ if fd, err := openBeneathWithoutSymlinks(root, mount); err == nil {+ unix.Close(fd)+ t.Fatalf("%s resolved through the component walk", mount)+ }+ }+}diff --git a/internal/filesystemprobe/observer_test.go b/internal/filesystemprobe/observer_test.goindex 45b606e0f1..21efb6ba6f 100644--- a/internal/filesystemprobe/observer_test.go+++ b/internal/filesystemprobe/observer_test.go@@ -79,3 +79,57 @@ func TestObserverRejectsInvalidCoordinatesBeforeNativeRead(t *testing.T) {t.Fatal(err)}}++// A Proxmox guest is identified by its exact lxc/<vmid> cgroup and its init+// process. Neither another VMID with the same prefix, LXC's monitor cgroup,+// nor the PID 1 of a namespace nested inside the guest (Docker in LXC) may+// stand in for it.+func TestProxmoxLXCIdentityIsExact(t *testing.T) {+ for raw, want := range map[string]bool{+ "0::/lxc/126/ns": true,+ "0::/lxc/126/ns/init.scope": true,+ "0::/lxc/126": true,+ "0::/lxc/1260/ns": false,+ "0::/lxc/12/ns": false,+ "0::/lxc.monitor/126": false,+ "0::/system.slice/lxc/126/ns": false,+ "0::/docker/" + strings.Repeat("a", 64): false,+ } {+ if got := CgroupMatches(raw, "lxc", "126"); got != want {+ t.Errorf("CgroupMatches(%q) = %v, want %v", raw, got, want)+ }+ }+ for status, want := range map[string]bool{+ "Name:\tinit\nNSpid:\t3019253\t1\n": true,+ "NSpid:\t3019253\t413\n": false,+ "NSpid:\t3019253\t420\t1\n": false, // nested namespace init+ "NSpid:\t1\n": false, // the host's own init+ "NSpid:\t999\t1\n": false, // another process's line+ "Name:\tinit\n": false,+ } {+ if got := IsNamespaceInit(status, 3019253); got != want {+ t.Errorf("IsNamespaceInit(%q) = %v, want %v", status, got, want)+ }+ }+}++func TestObserverRejectsInexactProxmoxVMIDs(t *testing.T) {+ o := &Observer{probe: func(context.Context, ContainerRequest) ([]filesystem.Observation, error) {+ t.Error("invalid coordinates reached native read")+ return nil, nil+ }}+ for _, id := range []string{"99", "0126", "126a", "-126", "1000000000", ""} {+ r := ContainerRequest{PID: 42, ContainerID: id, Runtime: "lxc", Mountpoints: []string{"/"}}+ if _, err := o.Observe(context.Background(), r); err == nil {+ t.Fatalf("accepted VMID %q", id)+ }+ }+ accepted := false+ o.probe = func(context.Context, ContainerRequest) ([]filesystem.Observation, error) {+ accepted = true+ return nil, nil+ }+ if _, err := o.Observe(context.Background(), ContainerRequest{PID: 42, ContainerID: "126", Runtime: "lxc", Mountpoints: []string{"/"}}); err != nil || !accepted {+ t.Fatalf("exact VMID rejected: %v", err)+ }+}diff --git a/internal/hostagent/mock_collector_test.go b/internal/hostagent/mock_collector_test.goindex 7d6e4ddc26..d58936c4fe 100644--- a/internal/hostagent/mock_collector_test.go+++ b/internal/hostagent/mock_collector_test.go@@ -7,8 +7,10 @@ import ("os""time"+ "github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe""github.com/rcourtman/pulse-go-rewrite/internal/hostmetrics""github.com/rcourtman/pulse-go-rewrite/internal/sensors"+ "github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"gohost "github.com/shirou/gopsutil/v4/host")@@ -44,15 +46,27 @@ type mockCollector struct {name string,arg ...string,) (string, error)- lookPathFn func(file string) (string, error)- filesystemUsageFn func(path string) (hostFilesystemUsage, error)+ lookPathFn func(file string) (string, error)+ observeLXCFn func(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error)+ readDirFn func(name string) ([]os.DirEntry, error)}-func (m *mockCollector) FilesystemUsage(path string) (hostFilesystemUsage, error) {- if m.filesystemUsageFn != nil {- return m.filesystemUsageFn(path)+// ReadDir satisfies proxmoxLXCDirLister. Without readDirFn it reports the+// directory missing, so collectors keep their pct path unless a test opts in.+func (m *mockCollector) ReadDir(name string) ([]os.DirEntry, error) {+ if m.readDirFn != nil {+ return m.readDirFn(name)}- return hostFilesystemUsage{}, os.ErrNotExist+ return nil, os.ErrNotExist+}++// ObserveProxmoxLXC satisfies proxmoxLXCObserver. Without observeLXCFn the+// probe fails, as it does off Linux, so collectors fall back to pct df.+func (m *mockCollector) ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {+ if m.observeLXCFn != nil {+ return m.observeLXCFn(ctx, req)+ }+ return nil, os.ErrNotExist}func (m *mockCollector) HostInfo(ctx context.Context) (*gohost.InfoStat, error) {diff --git a/internal/hostagent/privilege_helper_client_test.go b/internal/hostagent/privilege_helper_client_test.goindex 1b81a6e806..0798cf2fc1 100644--- a/internal/hostagent/privilege_helper_client_test.go+++ b/internal/hostagent/privilege_helper_client_test.go@@ -412,3 +412,44 @@ func requirePrivilegeHelperModuleStatus(t *testing.T, statuses []agentshost.Modut.Fatalf("typed privilege helper module missing from %+v", statuses)return agentshost.ModuleStatus{}}++// The typed helper runs with PrivateNetwork=true, where pct and lxc-info cannot+// reach pmxcfs or the LXC monitor: their abstract Unix sockets belong to the+// host network namespace (#2511). This drives the collector the helper's+// provider runs, with every command failing as it does there: it must still+// return a complete inventory from configs, cgroups and /proc alone. A+// degraded result is what the provider turns into provider_unavailable.+func TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets(t *testing.T) {+ h := newFakeProxmoxLXCHost()+ h.config(126, "hostname: qual2511\nrootfs: delly2-lvm:vm-126-disk-0,size=1G\nmp0: delly2-lvm:vm-126-disk-1,mp=/srv/data,size=1G\n")+ h.cgroup("126/ns", 3019253)+ h.proc(3019253, "3019253\t1", "/lxc/126/ns")+ h.config(127, "hostname: qual2511-priv\nrootfs: delly2-lvm:vm-127-disk-0,size=1G\n")+ h.cgroup("127/ns", 3021000)+ h.proc(3021000, "3021000\t1", "/lxc/127/ns")+ h.config(128, "hostname: stopped\nrootfs: delly2-lvm:vm-128-disk-0,size=1G\n")+ // Counters observed on a real PVE 9.2 node for the same layout.+ h.observations[3019253] = map[string]lxcObservation{+ "/": lxcUsage(1020702720, 12021760, 938217472),+ "/srv/data": lxcUsage(1020702720, 290816, 949948416),+ }+ h.observations[3021000] = map[string]lxcObservation{"/": lxcUsage(1020702720, 12017664, 938221568)}++ collector, commands := h.collector(t)+ agent := &Agent{logger: zerolog.Nop(), collector: collector}+ result := agent.collectProxmoxLXCFilesystemsResult(context.Background())++ if !result.Applicable || result.Degraded || result.FailedContainers != 0 {+ t.Fatalf("collection = applicable %v, degraded %v, failed %d; want a complete inventory", result.Applicable, result.Degraded, result.FailedContainers)+ }+ if *commands != 0 {+ t.Fatalf("collection ran %d pct/lxc-info commands; none can work inside the helper", *commands)+ }+ got := result.Inventory.Containers+ if len(got) != 2 || got[0].VMID != 126 || got[0].Name != "qual2511" || got[1].VMID != 127 || got[1].Name != "qual2511-priv" {+ t.Fatalf("containers = %+v, want the two running ones by their pct names", got)+ }+ if len(got[0].Disks) != 2 || got[0].Disks[1].Mountpoint != "/srv/data" || got[0].Disks[1].UsedBytes != 290816 {+ t.Fatalf("126 disks = %+v, want rootfs and the mp0 mount with its own usage", got[0].Disks)+ }+}diff --git a/internal/hostagent/proxmox_lxc_filesystems.go b/internal/hostagent/proxmox_lxc_filesystems.goindex a1e3eaa112..e1a6c0d7f3 100644--- a/internal/hostagent/proxmox_lxc_filesystems.go+++ b/internal/hostagent/proxmox_lxc_filesystems.go@@ -15,6 +15,7 @@ import ("time""unicode/utf8"+ "github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host""github.com/rs/zerolog")@@ -39,12 +40,15 @@ const (proxmoxLXCCollectionTimeout = 20 * time.SecondproxmoxLXCMaxContainers = 128proxmoxLXCMaxDisks = 257- proxmoxLXCMaxListOutputBytes = 64 * 1024- proxmoxLXCMaxDFOutputBytes = 64 * 1024- proxmoxLXCMaxConfigBytes = 64 * 1024- proxmoxLXCMaxPIDOutputBytes = 4 * 1024- proxmoxLXCMaxNameBytes = 256- proxmoxLXCMaxLabelBytes = 512+ // proxmoxLXCMaxObservedMounts is the shared filesystem prober's per-request+ // mountpoint limit.+ proxmoxLXCMaxObservedMounts = 128+ proxmoxLXCMaxListOutputBytes = 64 * 1024+ proxmoxLXCMaxDFOutputBytes = 64 * 1024+ proxmoxLXCMaxConfigBytes = 64 * 1024+ proxmoxLXCMaxPIDOutputBytes = 4 * 1024+ proxmoxLXCMaxNameBytes = 256+ proxmoxLXCMaxLabelBytes = 512)// proxmoxLXCConfigDir is a pmxcfs symlink to the local node's container@@ -54,6 +58,9 @@ const proxmoxLXCConfigDir = "/etc/pve/lxc"type proxmoxLXCRunningContainer struct {VMID intName string+ // PID is the container's init process when discovery found it without+ // lxc-info; 0 means unknown.+ PID int}// ProxmoxLXCFilesystemCollectionResult describes applicability separately from@@ -68,21 +75,6 @@ type ProxmoxLXCFilesystemCollectionResult struct {FailedContainers int}-// hostFilesystemUsage carries statfs-derived usage for one path plus the-// st_dev identity needed to tell a real mount from its parent filesystem.-type hostFilesystemUsage struct {- TotalBytes int64- UsedBytes int64- AvailBytes int64- Device uint64-}--// filesystemUsageProber is an optional collector capability. It exists on-// Linux builds of the default collector; mocks opt in for tests.-type filesystemUsageProber interface {- FilesystemUsage(path string) (hostFilesystemUsage, error)-}-type proxmoxLXCConfigMount struct {Key stringVolume string@@ -142,30 +134,38 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL}result.Applicable = true+ observer, _ := a.collector.(proxmoxLXCObserver)listCtx, cancelList := context.WithTimeout(ctx, proxmoxLXCQueryTimeout)- listOutput, err := collectCommandOutputLimited(- listCtx,- a.collector,- proxmoxLXCMaxListOutputBytes,- pctPath,- "list",- )- cancelList()- if err != nil {- a.logger.Debug().Err(err).Msg("Failed to list local Proxmox LXC containers")- result.Degraded = true- return result- }- containers, err := parseProxmoxLXCRunningContainers(listOutput)- if err != nil {- a.logger.Debug().Err(err).Msg("Failed to parse local Proxmox LXC container list")- result.Degraded = true- return result+ containers, discovered := []proxmoxLXCRunningContainer(nil), false+ if observer != nil {+ containers, discovered = a.discoverRunningProxmoxLXCContainers(listCtx)+ }+ if !discovered {+ listOutput, listErr := collectCommandOutputLimited(+ listCtx,+ a.collector,+ proxmoxLXCMaxListOutputBytes,+ pctPath,+ "list",+ )+ if listErr != nil {+ cancelList()+ a.logger.Debug().Err(listErr).Msg("Failed to list local Proxmox LXC containers")+ result.Degraded = true+ return result+ }+ containers, err = parseProxmoxLXCRunningContainers(listOutput)+ if err != nil {+ cancelList()+ a.logger.Debug().Err(err).Msg("Failed to parse local Proxmox LXC container list")+ result.Degraded = true+ return result+ }}+ cancelList()- prober, _ := a.collector.(filesystemUsageProber)lxcInfoPath := ""- if prober != nil {+ if observer != nil && !discovered {if resolved, lookErr := a.collector.LookPath("lxc-info"); lookErr == nil {lxcInfoPath = resolved}@@ -184,7 +184,7 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxLcontinue}containerCtx, cancelContainer := context.WithTimeout(collectionCtx, proxmoxLXCContainerQueryTimeout)- disks, collectionErr := a.collectProxmoxLXCContainerDisks(containerCtx, prober, lxcInfoPath, pctPath, container.VMID)+ disks, collectionErr := a.collectProxmoxLXCContainerDisks(containerCtx, observer, lxcInfoPath, pctPath, container)cancelContainer()if collectionErr != nil {result.FailedContainers++@@ -209,13 +209,14 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxLfunc (a *Agent) collectProxmoxLXCContainerDisks(ctx context.Context,- prober filesystemUsageProber,+ observer proxmoxLXCObserver,lxcInfoPath string,pctPath string,- vmid int,+ container proxmoxLXCRunningContainer,) ([]agentshost.Disk, error) {- if prober != nil && lxcInfoPath != "" {- disks, fastErr := a.collectProxmoxLXCContainerDisksFast(ctx, prober, lxcInfoPath, vmid)+ vmid := container.VMID+ if observer != nil && (container.PID > 0 || lxcInfoPath != "") {+ disks, fastErr := a.collectProxmoxLXCContainerDisksFast(ctx, observer, lxcInfoPath, container)if fastErr == nil {return disks, nil}@@ -259,10 +260,11 @@ func (a *Agent) collectProxmoxLXCContainerDisks(// numbers — so it is dropped rather than invented.func (a *Agent) collectProxmoxLXCContainerDisksFast(ctx context.Context,- prober filesystemUsageProber,+ observer proxmoxLXCObserver,lxcInfoPath string,- vmid int,+ container proxmoxLXCRunningContainer,) ([]agentshost.Disk, error) {+ vmid := container.VMIDraw, err := a.collector.ReadFile(proxmoxLXCConfigPath(vmid))if err != nil {return nil, fmt.Errorf("read container config: %w", err)@@ -275,59 +277,76 @@ func (a *Agent) collectProxmoxLXCContainerDisksFast(return nil, errors.New("no filesystem mounts in container config")}- pidOutput, err := collectCommandOutputLimited(- ctx,- a.collector,- proxmoxLXCMaxPIDOutputBytes,- lxcInfoPath,- "-n",- strconv.Itoa(vmid),- "-p",- )- if err != nil {- return nil, fmt.Errorf("resolve container pid: %w", err)+ pid := container.PID+ if pid <= 0 {+ pidOutput, pidErr := collectCommandOutputLimited(+ ctx,+ a.collector,+ proxmoxLXCMaxPIDOutputBytes,+ lxcInfoPath,+ "-n",+ strconv.Itoa(vmid),+ "-p",+ )+ if pidErr != nil {+ return nil, fmt.Errorf("resolve container pid: %w", pidErr)+ }+ pid, err = parseLXCInfoPID(pidOutput)+ if err != nil {+ return nil, err+ }+ }++ // The shared prober pins /proc/<pid>, requires it to be this guest's init+ // in lxc/<vmid>, resolves every mount inside the guest's root (an absolute+ // symlink cannot reach the host), and bounds a read stuck on a hung guest+ // filesystem without starting another for the same guest.+ byPath := make(map[string]proxmoxLXCConfigMount, len(mounts))+ mountpoints := make([]string, 0, len(mounts))+ for _, mount := range mounts {+ if _, seen := byPath[mount.Path]; seen || len(mountpoints) == proxmoxLXCMaxObservedMounts {+ continue+ }+ byPath[mount.Path] = mount+ mountpoints = append(mountpoints, mount.Path)}- pid, err := parseLXCInfoPID(pidOutput)+ observations, err := observer.ObserveProxmoxLXC(ctx, filesystemprobe.ContainerRequest{+ PID: pid,+ ContainerID: strconv.Itoa(vmid),+ Runtime: "lxc",+ Mountpoints: mountpoints,+ MountsOnly: true,+ })if err != nil {- return nil, err+ return nil, fmt.Errorf("observe container filesystems: %w", err)}- procRoot := fmt.Sprintf("/proc/%d/root", pid)- result := make([]agentshost.Disk, 0, len(mounts))- for _, mount := range mounts {- if ctx.Err() != nil {- return nil, ctx.Err()+ result := make([]agentshost.Disk, 0, len(observations))+ for _, observation := range observations {+ mount, known := byPath[observation.Mountpoint]+ if !known {+ return nil, fmt.Errorf("observation for unrequested mount %q", observation.Mountpoint)+ }+ if filesystemprobe.IsNotMounted(observation) {+ // Declared in the config but not mounted in the running guest; its+ // path would report the parent filesystem's numbers.+ continue}- target := path.Join(procRoot, mount.Path)- usage, usageErr := prober.FilesystemUsage(target)- if usageErr != nil {- return nil, fmt.Errorf("probe %s filesystem usage: %w", mount.Key, usageErr)+ if observation.Error != "" || observation.Usage == nil {+ return nil, fmt.Errorf("probe %s filesystem usage: %s", mount.Key, observation.Error)}- if usage.TotalBytes <= 0 {+ usage := observation.Usage+ if usage.CapacityBytes == 0 || usage.CapacityBytes > math.MaxInt64 {return nil, fmt.Errorf("probe %s filesystem usage: invalid total bytes", mount.Key)}- if mount.Path != "/" {- parent, parentErr := prober.FilesystemUsage(path.Dir(target))- if parentErr != nil {- return nil, fmt.Errorf("probe %s parent filesystem usage: %w", mount.Key, parentErr)- }- if parent.Device == usage.Device {- continue- }- }- used := usage.UsedBytes+ total := int64(usage.CapacityBytes)+ used := total - int64(usage.FreeBytes)if used < 0 {used = 0}- if used > usage.TotalBytes {- used = usage.TotalBytes- }- avail := usage.AvailBytes- if avail < 0 {- avail = 0- }- if avail > usage.TotalBytes {- avail = usage.TotalBytes+ avail := int64(usage.AvailableBytes)+ if avail > total {+ avail = total}usagePct := 0.0if denominator := used + avail; denominator > 0 {@@ -340,14 +359,11 @@ func (a *Agent) collectProxmoxLXCContainerDisksFast(Device: mount.Volume,Mountpoint: mount.Path,Type: mount.Key,- TotalBytes: usage.TotalBytes,+ TotalBytes: total,UsedBytes: used,FreeBytes: avail,Usage: usagePct,})- if len(result) == proxmoxLXCMaxDisks {- break- }}if len(result) == 0 {return nil, errors.New("no mounted container filesystems resolved")diff --git a/internal/hostagent/proxmox_lxc_filesystems_linux.go b/internal/hostagent/proxmox_lxc_filesystems_linux.godeleted file mode 100644index 52c9664c25..0000000000--- a/internal/hostagent/proxmox_lxc_filesystems_linux.go+++ /dev/null@@ -1,41 +0,0 @@-//go:build linux--package hostagent--import (- "fmt"- "syscall"-)--// FilesystemUsage satisfies filesystemUsageProber on Linux. Statfs through a-// /proc/<pid>/root path traverses into that process's mount namespace, so the-// numbers match what df reports inside the container.-func (c *defaultCollector) FilesystemUsage(path string) (hostFilesystemUsage, error) {- var fs syscall.Statfs_t- if err := syscall.Statfs(path, &fs); err != nil {- return hostFilesystemUsage{}, fmt.Errorf("statfs %s: %w", path, err)- }- var st syscall.Stat_t- if err := syscall.Stat(path, &st); err != nil {- return hostFilesystemUsage{}, fmt.Errorf("stat %s: %w", path, err)- }- blockSize := int64(fs.Frsize)- if blockSize <= 0 {- blockSize = int64(fs.Bsize)- }- if blockSize <= 0 {- return hostFilesystemUsage{}, fmt.Errorf("statfs %s: invalid block size", path)- }- total := int64(fs.Blocks) * blockSize- free := int64(fs.Bfree) * blockSize- used := total - free- if used < 0 {- used = 0- }- return hostFilesystemUsage{- TotalBytes: total,- UsedBytes: used,- AvailBytes: int64(fs.Bavail) * blockSize,- Device: uint64(st.Dev),- }, nil-}diff --git a/internal/hostagent/proxmox_lxc_filesystems_test.go b/internal/hostagent/proxmox_lxc_filesystems_test.goindex 706ea29df2..4592d49e10 100644--- a/internal/hostagent/proxmox_lxc_filesystems_test.go+++ b/internal/hostagent/proxmox_lxc_filesystems_test.go@@ -4,11 +4,16 @@ import ("context""errors""fmt"+ "io/fs""os"+ "path"+ "strconv""strings""testing""time"+ "github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"+ "github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem""github.com/rs/zerolog")@@ -153,16 +158,15 @@ mp2: tank:snap-only,mp=/snap-only,size=5G`,"/etc/pve/lxc/101.conf": "rootfs: local-lvm:vm-101-disk-0,size=4G\n",}- usageByPath := map[string]hostFilesystemUsage{- "/proc/4242/root": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},- "/proc/4242/root/srv": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},- "/proc/4242/root/srv/data": {TotalBytes: 1 << 40, UsedBytes: 512 << 30, AvailBytes: 512 << 30, Device: 12},- // mp1 is configured but not mounted in the running container, so the- // path resolves to the rootfs device and must be dropped, not reported- // with the parent's numbers.- "/proc/4242/root/mnt": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},- "/proc/4242/root/mnt/missing": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},- "/proc/5252/root": {TotalBytes: 4 << 30, UsedBytes: 1 << 30, AvailBytes: 3 << 30, Device: 21},+ observations := map[int]map[string]lxcObservation{+ 4242: {+ "/": lxcUsage(8<<30, 2<<30, 6<<30),+ "/srv/data": lxcUsage(1<<40, 512<<30, 512<<30),+ // mp1 is configured but not mounted in the running container, so+ // the prober rejects it instead of reporting the parent's numbers.+ "/mnt/missing": {notMounted: true},+ },+ 5252: {"/": lxcUsage(4<<30, 1<<30, 3<<30)},}collector := &mockCollector{goos: "linux",@@ -182,13 +186,7 @@ mp2: tank:snap-only,mp=/snap-only,size=5G}return []byte(config), nil},- filesystemUsageFn: func(path string) (hostFilesystemUsage, error) {- usage, ok := usageByPath[path]- if !ok {- t.Fatalf("unexpected statfs path %q", path)- }- return usage, nil- },+ observeLXCFn: fakeLXCObserve(t, observations),commandCombinedOutputLimitedFn: func(_ context.Context,_ int,@@ -267,12 +265,9 @@ func TestCollectProxmoxLXCFilesystemsFallsBackToPctDFPerContainer(t *testing.T)// back to pct df.return nil, os.ErrPermission},- filesystemUsageFn: func(path string) (hostFilesystemUsage, error) {- if path != "/proc/4242/root" {- t.Fatalf("unexpected statfs path %q", path)- }- return hostFilesystemUsage{TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11}, nil- },+ observeLXCFn: fakeLXCObserve(t, map[int]map[string]lxcObservation{+ 4242: {"/": lxcUsage(8<<30, 2<<30, 6<<30)},+ }),commandCombinedOutputLimitedFn: func(_ context.Context,_ int,@@ -345,17 +340,12 @@ mp0 tank:subvol-100-disk-0 1.0T 512.0G 512.0G 50.0 /srv/datareadFileFn: func(string) ([]byte, error) {return []byte("rootfs: local-lvm:vm-100-disk-0,size=8G\nmp0: tank:subvol-100-disk-0,mp=/srv/data,size=1T\n"), nil},- filesystemUsageFn: func(target string) (hostFilesystemUsage, error) {- switch target {- case "/proc/4242/root":- return hostFilesystemUsage{TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11}, nil- case "/proc/4242/root/srv/data":- return hostFilesystemUsage{}, errors.New("statfs access failed")- default:- t.Fatalf("unexpected statfs path %q", target)- return hostFilesystemUsage{}, nil- }- },+ observeLXCFn: fakeLXCObserve(t, map[int]map[string]lxcObservation{+ 4242: {+ "/": lxcUsage(8<<30, 2<<30, 6<<30),+ "/srv/data": {err: "read filesystem counters: permission denied"},+ },+ }),commandCombinedOutputLimitedFn: func(_ context.Context, _ int, name string, args ...string) (string, error) {joined := strings.Join(args, " ")switch {@@ -469,3 +459,281 @@ mp5 pool:hot 10.0G 1.0G 9.0G 101.0 /hott.Fatal("expected oversized pct df error")}}++type fakeLXCDirEntry struct {+ name string+ dir bool+}++func (e fakeLXCDirEntry) Name() string { return e.name }+func (e fakeLXCDirEntry) IsDir() bool { return e.dir }+func (e fakeLXCDirEntry) Type() fs.FileMode {+ if e.dir {+ return fs.ModeDir+ }+ return 0+}+func (e fakeLXCDirEntry) Info() (fs.FileInfo, error) { return nil, fs.ErrNotExist }++// fakeProxmoxLXCHost models what socket-free discovery reads on a cgroup v2+// Proxmox node. The layout follows a real PVE 9.2 node: configs under+// /etc/pve/lxc, the payload under /sys/fs/cgroup/lxc/<vmid>/ns, and the+// container init identified by an NSpid ending in 1.+type fakeProxmoxLXCHost struct {+ files map[string]string+ dirs map[string]map[string]bool+ observations map[int]map[string]lxcObservation+}++func newFakeProxmoxLXCHost() *fakeProxmoxLXCHost {+ h := &fakeProxmoxLXCHost{+ files: map[string]string{proxmoxLXCCgroupV2Marker: "cpuset cpu io memory pids\n"},+ dirs: map[string]map[string]bool{},+ observations: map[int]map[string]lxcObservation{},+ }+ h.dir(proxmoxLXCConfigDir)+ return h+}++func (h *fakeProxmoxLXCHost) dir(name string) {+ if _, ok := h.dirs[name]; !ok {+ h.dirs[name] = map[string]bool{}+ }+}++func (h *fakeProxmoxLXCHost) config(vmid int, content string) {+ h.dirs[proxmoxLXCConfigDir][strconv.Itoa(vmid)+".conf"] = false+ h.files[proxmoxLXCConfigPath(vmid)] = content+}++// cgroup registers a cgroup directory (relative to lxc/) and its processes.+func (h *fakeProxmoxLXCHost) cgroup(rel string, pids ...int) {+ full := path.Join(proxmoxLXCCgroupRoot, rel)+ for child := full; child != proxmoxLXCCgroupRoot; child = path.Dir(child) {+ h.dir(child)+ // Every cgroup directory has a cgroup.procs, empty when its+ // processes live in child cgroups.+ if _, ok := h.files[path.Join(child, "cgroup.procs")]; !ok {+ h.files[path.Join(child, "cgroup.procs")] = ""+ }+ parent := path.Dir(child)+ h.dir(parent)+ h.dirs[parent][path.Base(child)] = true+ }+ fields := make([]string, 0, len(pids))+ for _, pid := range pids {+ fields = append(fields, strconv.Itoa(pid))+ }+ h.files[path.Join(full, "cgroup.procs")] = strings.Join(fields, "\n") + "\n"+}++func (h *fakeProxmoxLXCHost) proc(pid int, nspid, cgroup string) {+ h.files["/proc/"+strconv.Itoa(pid)+"/status"] = "Name:\tinit\nNSpid:\t" + nspid + "\n"+ h.files["/proc/"+strconv.Itoa(pid)+"/cgroup"] = "0::" + cgroup + "\n"+}++// collector serves the fake host. pct and lxc-info exist but every call fails+// the way they fail inside the PrivateNetwork helper.+func (h *fakeProxmoxLXCHost) collector(t *testing.T) (*mockCollector, *int) {+ t.Helper()+ commands := 0+ return &mockCollector{+ goos: "linux",+ lookPathFn: func(file string) (string, error) {+ switch file {+ case "pct":+ return "/usr/sbin/pct", nil+ case "lxc-info":+ return "/usr/bin/lxc-info", nil+ }+ return "", os.ErrNotExist+ },+ statFn: func(name string) (os.FileInfo, error) {+ if _, ok := h.files[name]; ok {+ return nil, nil+ }+ if _, ok := h.dirs[name]; ok {+ return nil, nil+ }+ return nil, os.ErrNotExist+ },+ readFileFn: func(name string) ([]byte, error) {+ content, ok := h.files[name]+ if !ok {+ return nil, os.ErrNotExist+ }+ return []byte(content), nil+ },+ readDirFn: func(name string) ([]os.DirEntry, error) {+ children, ok := h.dirs[name]+ if !ok {+ return nil, os.ErrNotExist+ }+ entries := make([]os.DirEntry, 0, len(children))+ for child, isDir := range children {+ entries = append(entries, fakeLXCDirEntry{name: child, dir: isDir})+ }+ return entries, nil+ },+ observeLXCFn: fakeLXCObserve(t, h.observations),+ commandCombinedOutputLimitedFn: func(_ context.Context, _ int, name string, args ...string) (string, error) {+ commands+++ return "ipcc_send_rec[1] failed: Connection refused", errors.New("exit status 2")+ },+ }, &commands+}++func TestProxmoxLXCConfigNameMatchesPctList(t *testing.T) {+ cases := []struct {+ config string+ want string+ }{+ {"#managed\narch: amd64\nhostname: web\nrootfs: x,size=1G\n", "web"},+ {"arch: amd64\nrootfs: x,size=1G\n", "CT126"},+ {"hostname: \n", "CT126"},+ {"arch: amd64\n\n[snap]\nhostname: old\n", "CT126"},+ }+ for _, tc := range cases {+ if got := proxmoxLXCConfigName(tc.config, 126); got != tc.want {+ t.Errorf("proxmoxLXCConfigName(%q) = %q, want %q", tc.config, got, tc.want)+ }+ }+}++func TestDiscoverRunningProxmoxLXCContainersFindsInitWithoutSockets(t *testing.T) {+ h := newFakeProxmoxLXCHost()+ // Alpine guest running Docker: a nested container's PID 1 shares the+ // guest's cgroup listing and comes first, but is not the guest's init.+ h.config(126, "hostname: qual2511\nrootfs: lvm:vm-126-disk-0,size=1G\n")+ h.cgroup("126")+ h.cgroup("126/ns", 3019900, 3019673, 3019253)+ h.cgroup("126/ns/openrc.syslog", 3019638)+ h.proc(3019900, "3019900\t420\t1", "/lxc/126/ns")+ h.proc(3019673, "3019673\t413", "/lxc/126/ns")+ h.proc(3019253, "3019253\t1", "/lxc/126/ns")+ // systemd guest: init in ns/init.scope, services elsewhere.+ h.config(127, "rootfs: lvm:vm-127-disk-0,size=1G\n")+ h.cgroup("127/ns/system.slice/cron.service", 4100)+ h.cgroup("127/ns/init.scope", 4000)+ h.proc(4100, "4100\t88", "/lxc/127/ns/system.slice/cron.service")+ h.proc(4000, "4000\t1", "/lxc/127/ns/init.scope")+ // Stopped: config only, no cgroup.+ h.config(128, "hostname: stopped\nrootfs: lvm:vm-128-disk-0,size=1G\n")++ collector, commands := h.collector(t)+ agent := &Agent{logger: zerolog.Nop(), collector: collector}+ got, ok := agent.discoverRunningProxmoxLXCContainers(context.Background())+ if !ok {+ t.Fatal("discovery was not applicable on a cgroup v2 Proxmox host")+ }+ want := []proxmoxLXCRunningContainer{+ {VMID: 126, Name: "qual2511", PID: 3019253},+ {VMID: 127, Name: "CT127", PID: 4000},+ }+ if fmt.Sprint(got) != fmt.Sprint(want) {+ t.Fatalf("discovered %+v, want %+v", got, want)+ }+ if *commands != 0 {+ t.Fatalf("discovery ran %d commands, want none", *commands)+ }+}++// Discovery must never turn a running guest it could not read into a smaller,+// healthy-looking inventory. Whenever a guest's state is not established it+// hands the whole node back to pct, which works for root installs and is+// reported as degraded inside the helper.+func TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct(t *testing.T) {+ running := func() *fakeProxmoxLXCHost {+ h := newFakeProxmoxLXCHost()+ h.config(126, "hostname: web\n")+ h.cgroup("126/ns", 10)+ h.proc(10, "10\t1", "/lxc/126/ns")+ return h+ }+ cases := map[string]func(h *fakeProxmoxLXCHost) context.Context{+ "cgroup v1 host": func(h *fakeProxmoxLXCHost) context.Context {+ delete(h.files, proxmoxLXCCgroupV2Marker)+ return context.Background()+ },+ "unreadable config of a running guest": func(h *fakeProxmoxLXCHost) context.Context {+ delete(h.files, proxmoxLXCConfigPath(126))+ return context.Background()+ },+ "guest cgroup without an init (a reused PID from another guest)": func(h *fakeProxmoxLXCHost) context.Context {+ h.proc(10, "10\t1", "/lxc/130/ns")+ return context.Background()+ },+ "unreadable cgroup.procs": func(h *fakeProxmoxLXCHost) context.Context {+ delete(h.files, path.Join(proxmoxLXCCgroupRoot, "126/ns/cgroup.procs"))+ return context.Background()+ },+ "guest-created cgroup tree beyond the search limit": func(h *fakeProxmoxLXCHost) context.Context {+ h.proc(10, "10\t2", "/lxc/126/ns")+ for i := 0; i < proxmoxLXCMaxCgroupDirs; i++ {+ h.cgroup("126/ns/c" + strconv.Itoa(i))+ }+ return context.Background()+ },+ "cancelled collection": func(*fakeProxmoxLXCHost) context.Context {+ ctx, cancel := context.WithCancel(context.Background())+ cancel()+ return ctx+ },+ }+ for name, mutate := range cases {+ t.Run(name, func(t *testing.T) {+ h := running()+ ctx := mutate(h)+ collector, _ := h.collector(t)+ got, ok := (&Agent{logger: zerolog.Nop(), collector: collector}).discoverRunningProxmoxLXCContainers(ctx)+ if ok {+ t.Fatalf("discovery reported %+v as complete", got)+ }+ })+ }+}++// lxcObservation is one mount's answer from the fake filesystem prober.+type lxcObservation struct {+ usage *filesystem.Usage+ notMounted bool+ err string+}++func lxcUsage(total, used, avail uint64) lxcObservation {+ return lxcObservation{usage: &filesystem.Usage{CapacityBytes: total, FreeBytes: total - used, AvailableBytes: avail}}+}++// fakeLXCObserve answers prober requests from per-PID mount tables. A PID the+// table does not know is not the guest's init, as the real prober would find.+func fakeLXCObserve(t *testing.T, byPID map[int]map[string]lxcObservation) func(context.Context, filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {+ t.Helper()+ return func(_ context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {+ if req.Runtime != "lxc" || !req.MountsOnly || req.ContainerID == "" {+ t.Errorf("prober request = %+v, want an exact lxc request with MountsOnly", req)+ }+ mounts, ok := byPID[req.PID]+ if !ok {+ return nil, errors.New("process is not the container's init")+ }+ out := make([]filesystem.Observation, 0, len(req.Mountpoints))+ for _, mountpoint := range req.Mountpoints {+ answer, known := mounts[mountpoint]+ if !known {+ t.Errorf("unexpected mount %q for pid %d", mountpoint, req.PID)+ continue+ }+ observation := filesystem.Observation{Mountpoint: mountpoint, Source: filesystemprobe.Source}+ switch {+ case answer.notMounted:+ observation.Error = filesystemprobe.ErrNotMounted.Error()+ case answer.err != "":+ observation.Error = answer.err+ default:+ observation.Usage = answer.usage+ }+ out = append(out, observation)+ }+ return out, nil+ }+}diff --git a/internal/hostagent/proxmox_lxc_running.go b/internal/hostagent/proxmox_lxc_running.gonew file mode 100644index 0000000000..8a33988788--- /dev/null+++ b/internal/hostagent/proxmox_lxc_running.go@@ -0,0 +1,243 @@+package hostagent++import (+ "context"+ "errors"+ "os"+ "path"+ "sort"+ "strconv"+ "strings"++ "github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"+ "github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"+)++// Socket-free discovery of the node's running containers (#2511).+//+// pct list, pct df and lxc-info reach pmxcfs and the LXC monitor through+// abstract Unix sockets, which belong to a network namespace. The typed helper+// runs with PrivateNetwork=true, so every one of them fails there. Container+// configs are ordinary reads through the pmxcfs FUSE mount, and a running+// container's init process is visible in its cgroup and in /proc, so the helper+// can find what it needs without leaving its sandbox. Usage is then read by+// the shared filesystem prober, which pins the process, re-checks its identity+// and resolves every path inside the container's root.++const (+ // proxmoxLXCCgroupRoot is where pve-container places each container's+ // cgroup (lxc/<vmid>, with the payload under ns/) on a cgroup v2 host.+ proxmoxLXCCgroupRoot = "/sys/fs/cgroup/lxc"+ proxmoxLXCCgroupV2Marker = "/sys/fs/cgroup/cgroup.controllers"+ proxmoxLXCMaxCgroupDirs = 64+ proxmoxLXCMaxCgroupPIDs = 4096+ proxmoxLXCMaxCgroupProcsBytes = 64 * 1024+ proxmoxLXCMaxProcStatusBytes = 64 * 1024+ proxmoxLXCMaxProcCgroupBytes = 64 * 1024+ proxmoxLXCMaxConfigEntries = 4096+)++// proxmoxLXCDirLister is an optional collector capability, like+// proxmoxLXCObserver: the default collector lists directories and test+// collectors opt in.+type proxmoxLXCDirLister interface {+ ReadDir(name string) ([]os.DirEntry, error)+}++// proxmoxLXCObserver reads a running container's filesystem counters through+// its pinned init process.+type proxmoxLXCObserver interface {+ ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error)+}++// proxmoxLXCFilesystemObserver is shared by every collection in the process.+// A probe stuck on a guest's hung filesystem keeps its slot here, so the+// typed helper never stacks a second probe on that guest.+var proxmoxLXCFilesystemObserver filesystemprobe.Observer++func (c *defaultCollector) ReadDir(name string) ([]os.DirEntry, error) {+ return os.ReadDir(name)+}++func (c *defaultCollector) ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {+ return proxmoxLXCFilesystemObserver.Observe(ctx, req)+}++type proxmoxLXCInitState int++const (+ proxmoxLXCStopped proxmoxLXCInitState = iota+ proxmoxLXCRunning+ proxmoxLXCUnknown+)++// discoverRunningProxmoxLXCContainers lists this node's running containers+// with their init PIDs. ok is false when the host cannot be read this way or+// any container's state cannot be established (no directory listing, no+// cgroup v2 hierarchy, an unreadable config or cgroup, a search limit, or+// cancellation), so the caller uses pct instead of reporting an inventory that+// silently misses a running guest. Only a container with no cgroup at all is+// stopped and left out, as pct list leaves it out.+func (a *Agent) discoverRunningProxmoxLXCContainers(ctx context.Context) ([]proxmoxLXCRunningContainer, bool) {+ lister, isLister := a.collector.(proxmoxLXCDirLister)+ if !isLister {+ return nil, false+ }+ if _, err := a.collector.Stat(proxmoxLXCCgroupV2Marker); err != nil {+ return nil, false+ }+ entries, err := lister.ReadDir(proxmoxLXCConfigDir)+ if err != nil {+ a.logger.Debug().Err(err).Msg("Cannot list Proxmox LXC configs; using pct")+ return nil, false+ }+ if len(entries) > proxmoxLXCMaxConfigEntries {+ return nil, false+ }++ vmids := make([]int, 0, len(entries))+ for _, entry := range entries {+ name := entry.Name()+ if !strings.HasSuffix(name, ".conf") {+ continue+ }+ vmid, convErr := strconv.Atoi(strings.TrimSuffix(name, ".conf"))+ if convErr != nil || vmid < 100 || vmid > 999999999 {+ continue+ }+ vmids = append(vmids, vmid)+ }+ sort.Ints(vmids)++ running := make([]proxmoxLXCRunningContainer, 0, len(vmids))+ for _, vmid := range vmids {+ if ctx.Err() != nil {+ return nil, false+ }+ pid, state := a.proxmoxLXCInitPID(ctx, lister, vmid)+ switch state {+ case proxmoxLXCStopped:+ continue+ case proxmoxLXCUnknown:+ a.logger.Debug().Int("vmid", vmid).Msg("Cannot establish a Proxmox LXC init process; using pct")+ return nil, false+ }+ raw, readErr := a.collector.ReadFile(proxmoxLXCConfigPath(vmid))+ if readErr != nil || len(raw) > proxmoxLXCMaxConfigBytes {+ // Without its config a running container has no trustworthy name or+ // mounts; leave the whole node to pct rather than guess.+ a.logger.Debug().Err(readErr).Int("vmid", vmid).Msg("Cannot read a running Proxmox LXC config; using pct")+ return nil, false+ }+ running = append(running, proxmoxLXCRunningContainer{VMID: vmid, Name: proxmoxLXCConfigName(string(raw), vmid), PID: pid})+ if len(running) == proxmoxLXCMaxContainers {+ break+ }+ }+ return running, true+}++// proxmoxLXCInitPID finds the container's init process: the process in the+// container's cgroup tree that is PID 1 of the namespace directly below the+// host's and whose own cgroup is inside lxc/<vmid>. The filesystem prober+// re-checks both on the pinned process before reading anything.+func (a *Agent) proxmoxLXCInitPID(ctx context.Context, lister proxmoxLXCDirLister, vmid int) (int, proxmoxLXCInitState) {+ root := path.Join(proxmoxLXCCgroupRoot, strconv.Itoa(vmid))+ if _, err := lister.ReadDir(root); err != nil {+ if errors.Is(err, os.ErrNotExist) {+ return 0, proxmoxLXCStopped+ }+ return 0, proxmoxLXCUnknown+ }++ queue := []string{root}+ queued := 1+ checked := 0+ for len(queue) > 0 {+ if ctx.Err() != nil {+ return 0, proxmoxLXCUnknown+ }+ dir := queue[0]+ queue = queue[1:]++ raw, err := a.collector.ReadFile(path.Join(dir, "cgroup.procs"))+ if err != nil || len(raw) > proxmoxLXCMaxCgroupProcsBytes {+ return 0, proxmoxLXCUnknown+ }+ for _, field := range strings.Fields(string(raw)) {+ if checked == proxmoxLXCMaxCgroupPIDs {+ return 0, proxmoxLXCUnknown+ }+ checked+++ pid, convErr := strconv.Atoi(field)+ if convErr != nil || pid <= 0 {+ continue+ }+ if a.isProxmoxLXCInit(pid, vmid) {+ return pid, proxmoxLXCRunning+ }+ }++ entries, err := lister.ReadDir(dir)+ if err != nil {+ return 0, proxmoxLXCUnknown+ }+ children := make([]string, 0, len(entries))+ for _, entry := range entries {+ if entry.IsDir() {+ children = append(children, entry.Name())+ }+ }+ // systemd guests run init in init.scope; look there first.+ sort.SliceStable(children, func(i, j int) bool {+ return children[i] == "init.scope" && children[j] != "init.scope"+ })+ for _, child := range children {+ if queued == proxmoxLXCMaxCgroupDirs {+ // A guest can create cgroups in its delegated subtree; give up+ // rather than walk an unbounded tree.+ return 0, proxmoxLXCUnknown+ }+ queued+++ queue = append(queue, path.Join(dir, child))+ }+ }+ // A cgroup without an init is a guest starting or stopping.+ return 0, proxmoxLXCUnknown+}++func (a *Agent) isProxmoxLXCInit(pid, vmid int) bool {+ status, err := a.collector.ReadFile("/proc/" + strconv.Itoa(pid) + "/status")+ if err != nil || len(status) > proxmoxLXCMaxProcStatusBytes {+ return false+ }+ if !filesystemprobe.IsNamespaceInit(string(status), pid) {+ return false+ }+ cgroup, err := a.collector.ReadFile("/proc/" + strconv.Itoa(pid) + "/cgroup")+ if err != nil || len(cgroup) > proxmoxLXCMaxProcCgroupBytes {+ return false+ }+ return filesystemprobe.CgroupMatches(string(cgroup), "lxc", strconv.Itoa(vmid))+}++// proxmoxLXCConfigName is the name pct list and the Proxmox API report: the+// configured hostname, or CT<vmid> when none is set. Pulse matches agent+// readings to API containers by this name.+func proxmoxLXCConfigName(config string, vmid int) string {+ for _, rawLine := range strings.Split(config, "\n") {+ line := strings.TrimSpace(rawLine)+ if strings.HasPrefix(line, "[") {+ break+ }+ value, ok := strings.CutPrefix(line, "hostname:")+ if !ok {+ continue+ }+ if name := strings.TrimSpace(value); name != "" && safeProxmoxLXCText(name, proxmoxLXCMaxNameBytes) {+ return name+ }+ break+ }+ return "CT" + strconv.Itoa(vmid)+}