Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

feat(worker_node): enhance monitoring setup and documentation for worker node

Dmitry Ngf6fe867a994e
diff --git a/examples/guides/worker_node.md b/examples/guides/worker_node.md
index 0828a476..1d8b5177 100644
--- a/examples/guides/worker_node.md
+++ b/examples/guides/worker_node.md
@@ -62,11 +62,11 @@ Set the private IP address that will be used throughout this setup, the metrics
```bash
export PRIVATE_IP=192.168.10.10 # Replace with your worker node IP
-export METRICS_IP=127.0.0.1 # Bind address for Docker metrics ports 9323 / 9324
+export METRICS_IP=127.0.0.1 # Bind address for metrics ports 8080 / 9100 / 9323 / 9324
export GUIDE_URL=https://raw.githubusercontent.com/vxcontrol/pentagi/main/examples/guides/worker_node
```
-`METRICS_IP` defaults to `127.0.0.1` so Prometheus metrics stay local to the worker node. Set it to `${PRIVATE_IP}` only if a remote scraper must reach ports `9323` / `9324`.
+`METRICS_IP` defaults to `127.0.0.1` so Prometheus metrics (Docker daemon, dind, cAdvisor, node-exporter) stay local to the worker node. Set it to `${PRIVATE_IP}` only if a remote scraper must reach ports `8080`, `9100`, `9323` / `9324`.
## Install Docker on Both Nodes
@@ -554,6 +554,131 @@ SCRAPER_PRIVATE_URL=https://${SCRAPER_USERNAME}:${SCRAPER_PASSWORD}@${PRIVATE_IP
> Nested containers may use `--network host` and can therefore reach `${PRIVATE_IP}:9443`. Basic Auth is the gate — keep the password strong and do not expose 9443 beyond the main PentAGI node.
+## Deploy Monitoring Exporters on Worker Node (Host Docker)
+
+Run cAdvisor (per-container metrics) and node-exporter (host-level metrics) on the **worker node's host Docker** (outside dind), bound to `${METRICS_IP}` — the same bind address already used for the Docker daemon and dind metrics endpoints (`9323`/`9324`). PentAGI's bundled [observability stack](mdc:observability/otel/config.yml) scrapes both under the `docker-container-collector` and `otel-collector` job names; on a distributed setup point those Prometheus targets at `${METRICS_IP}:8080` and `${METRICS_IP}:9100` instead of the in-network container names.
+
+Both exporters are read-only introspection tools with no PentAGI-specific credentials, so — unlike the scraper — they need no secrets step.
+
+### Create and Install `run-cadvisor`
+
+```bash
+sudo tee /usr/local/bin/run-cadvisor > /dev/null << EOF
+#!/bin/bash
+# Launch the cAdvisor container-metrics exporter on the worker host Docker daemon.
+# Usage: run-cadvisor
+
+set -e
+
+CONTAINER_NAME=cadvisor
+IMAGE=ghcr.io/google/cadvisor:0.56.2
+LISTEN_IP=${METRICS_IP}
+LISTEN_PORT=8080
+RESTART_POLICY=unless-stopped
+LOG_MAX_SIZE=50m
+LOG_MAX_FILE=7
+
+if docker ps --format '{{.Names}}' | grep -q "^\${CONTAINER_NAME}\$"; then
+ echo "container \${CONTAINER_NAME} is already running"
+ echo "to recreate, first remove: docker rm -f \${CONTAINER_NAME}"
+ exit 0
+fi
+
+docker rm "\${CONTAINER_NAME}" >/dev/null 2>&1 || true
+
+docker run -d \\
+ --name "\${CONTAINER_NAME}" \\
+ --hostname cadvisor \\
+ --restart "\${RESTART_POLICY}" \\
+ -p "\${LISTEN_IP}:\${LISTEN_PORT}:8080" \\
+ -v /:/rootfs:ro \\
+ -v /var/run:/var/run:rw \\
+ -v /sys:/sys:ro \\
+ -v /var/lib/docker/:/var/lib/docker:ro \\
+ --log-opt "max-size=\${LOG_MAX_SIZE}" \\
+ --log-opt "max-file=\${LOG_MAX_FILE}" \\
+ "\${IMAGE}" \\
+ --store_container_labels=false \\
+ --docker_only=true \\
+ --disable_root_cgroup_stats=true
+
+echo "cadvisor metrics listening on http://\${LISTEN_IP}:\${LISTEN_PORT}/metrics"
+EOF
+
+sudo chmod +x /usr/local/bin/run-cadvisor
+```
+
+> cAdvisor needs read-write access to `/var/run` to reach `docker.sock` for container metadata — the same trust level the host Docker daemon itself already extends to anything on this node. Keep `${METRICS_IP}` at the default `127.0.0.1` unless a remote scraper needs it.
+
+### Create and Install `run-node-exporter`
+
+```bash
+sudo tee /usr/local/bin/run-node-exporter > /dev/null << EOF
+#!/bin/bash
+# Launch the Prometheus node-exporter host-metrics exporter on the worker host Docker daemon.
+# Usage: run-node-exporter
+
+set -e
+
+CONTAINER_NAME=node_exporter
+IMAGE=prom/node-exporter:v1.5.0
+LISTEN_IP=${METRICS_IP}
+LISTEN_PORT=9100
+TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
+RESTART_POLICY=unless-stopped
+LOG_MAX_SIZE=50m
+LOG_MAX_FILE=7
+
+if docker ps --format '{{.Names}}' | grep -q "^\${CONTAINER_NAME}\$"; then
+ echo "container \${CONTAINER_NAME} is already running"
+ echo "to recreate, first remove: docker rm -f \${CONTAINER_NAME}"
+ exit 0
+fi
+
+sudo mkdir -p "\${TEXTFILE_DIR}"
+docker rm "\${CONTAINER_NAME}" >/dev/null 2>&1 || true
+
+docker run -d \\
+ --name "\${CONTAINER_NAME}" \\
+ --hostname node-exporter \\
+ --restart "\${RESTART_POLICY}" \\
+ -p "\${LISTEN_IP}:\${LISTEN_PORT}:9100" \\
+ -v /proc:/host/proc:ro \\
+ -v /sys:/host/sys:ro \\
+ -v /:/rootfs:ro \\
+ -v "\${TEXTFILE_DIR}:/textfile_collector:ro" \\
+ --log-opt "max-size=\${LOG_MAX_SIZE}" \\
+ --log-opt "max-file=\${LOG_MAX_FILE}" \\
+ "\${IMAGE}" \\
+ --path.procfs=/host/proc \\
+ --path.sysfs=/host/sys \\
+ --path.rootfs=/rootfs \\
+ --collector.filesystem.mount-points-exclude \\
+ '^/(dev|proc|sys|run|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)' \\
+ --collector.filesystem.fs-types-exclude \\
+ '^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|efivarfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|pstore|ramfs|rpc_pipefs|securityfs|squashfs|sysfs|tmpfs|tracefs)$' \\
+ --collector.textfile.directory=/textfile_collector
+
+echo "node-exporter metrics listening on http://\${LISTEN_IP}:\${LISTEN_PORT}/metrics"
+EOF
+
+sudo chmod +x /usr/local/bin/run-node-exporter
+```
+
+> The compose reference this is derived from sets `deploy: mode: global`, a Swarm-only directive with no equivalent for a plain `docker run` container — one instance per worker node is already what this script produces.
+
+### Start and Verify
+
+```bash
+sudo run-cadvisor
+sudo run-node-exporter
+
+curl -s "http://${METRICS_IP}:8080/metrics" | head -5
+curl -s "http://${METRICS_IP}:9100/metrics" | head -5
+```
+
+Both endpoints should return Prometheus-formatted `# HELP` / `# TYPE` lines immediately — neither exporter has the cold-start delay the scraper's browser backend has.
+
## Security & Firewall Configuration
### Required Port Access
@@ -567,10 +692,12 @@ The worker node exposes the following services:
| 9443 | `${PRIVATE_IP}` | Scraper (Browserless) | HTTPS browser scraper for PentAGI browser tool |
| 9323 | `${METRICS_IP}` | Host Docker Metrics | Prometheus metrics endpoint for host Docker |
| 9324 | `${METRICS_IP}` | dind Metrics | Prometheus metrics endpoint for dind |
+| 8080 | `${METRICS_IP}` | cAdvisor Metrics | Prometheus per-container metrics (`run-cadvisor`) |
+| 9100 | `${METRICS_IP}` | node-exporter Metrics | Prometheus host-level metrics (`run-node-exporter`) |
With the default `METRICS_IP=127.0.0.1`, metrics are reachable only from the worker node itself. Set `METRICS_IP=${PRIVATE_IP}` if a remote metrics scraper must reach them.
-**Metrics Integration:** The metrics ports (9323, 9324) can be configured in PentAGI's `observability/otel/config.yml` under the `docker-engine-collector` job name for monitoring integration.
+**Metrics Integration:** The metrics ports (9323, 9324, 8080, 9100) can be configured in PentAGI's `observability/otel/config.yml` under the `docker-engine-collector`, `docker-container-collector` and `otel-collector` job names for monitoring integration — point their `targets` at `${METRICS_IP}:<port>` instead of the in-network container names when the worker node is remote.
### OOB Attack Port Range
@@ -578,11 +705,11 @@ Each worker container (`pentagi-terminal-N`) dynamically allocates **2 ports** f
**Firewall Requirements:**
- **Inbound**: Allow access to ports 2376, 3376, 9443 on `${PRIVATE_IP}` from the main PentAGI node
-- **Inbound**: If `METRICS_IP=${PRIVATE_IP}`, also allow 9323, 9324 on `${PRIVATE_IP}` from the metrics scraper host; with the default `127.0.0.1` no remote metrics access is needed
+- **Inbound**: If `METRICS_IP=${PRIVATE_IP}`, also allow 9323, 9324, 8080, 9100 on `${PRIVATE_IP}` from the metrics scraper host; with the default `127.0.0.1` no remote metrics access is needed
- **Inbound**: Allow access to port range 28000-30000 from target networks being tested
- Configure perimeter firewall to permit OOB traffic from target networks to worker node
-Both Docker APIs and the scraper must be reachable **only** from the main node — nested containers may use `--network host` and can therefore probe every port on this node.
+Both Docker APIs and the scraper must be reachable **only** from the main node — nested containers may use `--network host` and can therefore probe every port on this node. The same applies to cAdvisor and node-exporter: cAdvisor mounts `docker.sock`, so exposing `8080` beyond the metrics scraper host would hand any reachable client the same access level as the host Docker daemon.
## Transfer Certificates to Main Node