Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

perf+feat: quick wins from the optimization milestone

karloks20056c7823a2a2e7
diff --git a/README.md b/README.md
index e8281d3..b0fc2c1 100644
--- a/README.md
+++ b/README.md
@@ -443,6 +443,7 @@ stops firing.
| Possible memory leak (steady climb over 12 hours) | 10 MB/h and 20% growth | warning |
| Docker unreachable (after it worked before) | 3 check-ins in a row | critical |
| Docker slow to list containers | over 2000 ms, 3 check-ins in a row | warning |
+| Collector can't keep up with its check interval | gathering takes over 80% of the interval, 3 check-ins in a row | warning |
| New listener reachable from other machines | stays open 60 minutes | warning |
| Firewall configuration changed | stays open 60 minutes | info |
| Firewall protection reduced (ufw turned off, rules removed, a file gone) | stays open 60 minutes | warning |
@@ -480,7 +481,7 @@ the same number of check-ins.
Admins can change every threshold, the number of check-ins in a row, how long
event incidents (reboots, OOM kills, container exits, removals and image changes)
-stay open, the slow-Docker limit, the memory-leak window and sensitivity, and
+stay open, the slow-Docker and slow-collection limits, the memory-leak window and sensitivity, and
**per-mount disk thresholds** under
**Admin → Alerts**. A per-mount override replaces the default disk thresholds for
that mount path on every server. Changes apply from each server's next check-in
@@ -653,7 +654,7 @@ from the backend's response to each report.
- **Collector privileges.** In the Compose installs the collector has no Docker
socket, drops all capabilities and has a read-only filesystem. Only the Docker
- proxy mounts the socket; it has no network, answers three read-only requests and
+ proxy mounts the socket; it has no network, answers four read-only requests and
strips environment variables, labels, mounts and error text from what it returns.
Everything a collector reports about containers is validated again by the backend
(types, ranges, lengths, counts) before it is stored.
diff --git a/SECURITY.md b/SECURITY.md
index d2755d9..50ceade 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -8,7 +8,7 @@ This page says what each component can reach, so you can decide what to run.
| Component | Runs as | Reaches | Notes |
|---|---|---|---|
| Collector | container | host `/proc`, the host filesystem (read-only), the Docker proxy, the backend over HTTP | No Docker socket in the compose installs. |
-| Docker proxy | container, uid 0, no capabilities | the Docker socket; a Unix socket shared only with the collector; no network | Allows three read-only requests. See below. |
+| Docker proxy | container, uid 0, no capabilities | the Docker socket; a Unix socket shared only with the collector; no network | Allows four read-only requests. See below. |
| System helper (optional) | container, uid 0 with **one** capability | kernel log, `/var/log/wtmp` (read-only), one output volume | Off by default. Details below. |
| Collector gate (server stack) | nginx container, no capabilities | the backend; publishes one port on your VPN address | The only way to reach collector endpoints. See below. |
| Backend / frontend / database | containers | each other | Only the frontend port is published. |
@@ -51,8 +51,10 @@ A Docker socket mount is root on the host: the `:ro` flag only protects the sock
API behind it, and anything that can reach the API can start a privileged container. So in the
compose files the collector **does not mount the socket**. A separate `docker-proxy` container does:
-- It answers exactly three requests: `GET /containers/json?all=1`, `GET /containers/<id>/json` and
- `GET /containers/<id>/stats?stream=false`. Everything else (other endpoints, other methods, extra
+- It answers exactly four requests: `GET /containers/json?all=1`, `GET /containers/<id>/json`,
+ `GET /containers/<id>/stats?stream=false` and the same stats request with `&one-shot=true` (which makes
+ Docker answer at once instead of waiting a second for a second CPU reading; the collector works out CPU %
+ from the change since its previous check-in). Everything else (other endpoints, other methods, extra
query parameters, encoded or padded paths) gets a 403 and is logged. Matching is literal against the
whole request target, with no decoding.
- It cuts the answers down to a fixed list of fields. For container inspect that is the ID, name,
diff --git a/TESTS.md b/TESTS.md
index 37176c5..f7ee386 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -4,7 +4,7 @@ Vexion ships with an automated test suite. This page explains how to run it, wha
covers (and what it doesn't), and what every single test is for, so you can decide
whether a failure matters before you upgrade or change your setup.
-- **628 test cases** from 366 test functions (some functions run once per input).
+- **676 test cases** from 384 test functions (some functions run once per input).
- **About 15 to 20 seconds** to run. No Docker, PostgreSQL, Redis, network or root access needed.
- **Safe to run next to a live install.** The tests use a throwaway in-memory database
and fakes; they never touch your running containers or your data. (Two collector
@@ -127,11 +127,11 @@ docker compose logs --tail 20 collector # "Sent host metrics and N contai
| [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. |
| [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. |
| [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |
-| [`test_container_detail.py`](#test_container_detailpy) | 5 | Per-container details and parsing of Docker's stats. |
+| [`test_container_detail.py`](#test_container_detailpy) | 18 | Per-container details and parsing of Docker's stats. |
| [`test_conversations.py`](#test_conversationspy) | 1 | Deleting one assistant conversation leaves the others alone. |
-| [`test_container_checks.py`](#test_container_checkspy) | 111 | Docker health, container exit codes, inventory and image changes, memory-leak trends, and the validation of everything a collector sends about containers. |
+| [`test_container_checks.py`](#test_container_checkspy) | 126 | Docker health, container exit codes, inventory and image changes, memory-leak trends, and the validation of everything a collector sends about containers. |
| [`test_dashboard_host.py`](#test_dashboard_hostpy) | 8 | The "this machine" logic, history charts, container list and incident counts. |
-| [`test_docker_proxy.py`](#test_docker_proxypy) | 91 | The read-only Docker proxy: what it allows, refuses and hides. |
+| [`test_docker_proxy.py`](#test_docker_proxypy) | 111 | The read-only Docker proxy: what it allows, refuses and hides. |
| [`test_enrollment.py`](#test_enrollmentpy) | 10 | Enrollment keys, collector credentials and offline-server detection. |
| [`test_health_checks.py`](#test_health_checkspy) | 4 | The assistant's fixed health checks and the critical-only webhook. |
| [`test_network_checks.py`](#test_network_checkspy) | 94 | The host's real network totals, listening-port inventory, optional internet check, firewall fingerprints, and the validation of all of it. |
@@ -155,6 +155,7 @@ out of their own install.
|---|---|
| `test_codes_match_the_rfc_6238_reference_vectors` *(6 inputs)* | The one-time code generator gives the exact answers published in the TOTP standard (RFC 6238), so it will match any authenticator app. |
| `test_a_code_is_accepted_in_its_own_period_and_one_either_side_but_not_further` | A code works in its own 30-second period and one either side (clock drift), not further away. |
+| `test_a_code_cannot_be_replayed` | A code that just signed you in is refused if used again in its window; the next period's code works. |
| `test_a_code_cannot_be_used_twice_or_after_a_newer_one` | A code that was already used, or an older one, is refused. |
| `test_malformed_codes_are_rejected` *(12 inputs)* | Empty, short, long, non-numeric, huge and non-string codes are refused without errors. |
| `test_spaces_and_a_hyphen_in_a_typed_code_are_ignored` | `123 456` and `123-456` are accepted as `123456`. |
@@ -323,6 +324,14 @@ Accounts, passwords, first-run protection and the cross-site check.
| `test_container_detail_returns_latest_and_chronological_history` | A container's detail view returns its latest numbers and a history sorted oldest to newest. |
| `test_container_detail_is_visible_to_regular_users` | Regular users can open container details (they are read-only). |
| `test_container_detail_returns_404_for_unknown_server` | Asking for an unknown server gives a clean 404. |
+| `test_the_quick_one_shot_request_is_tried_first` | Stats are fetched with the fast one-shot request first. |
+| `test_cpu_is_the_change_since_the_previous_check_in_and_the_first_has_none` | CPU % is worked out from the change since the last check-in; the first check-in shows no figure rather than a false 0. |
+| `test_containers_do_not_share_cpu_history` | Each container's CPU is measured against its own earlier reading. |
+| `test_a_counter_that_goes_backwards_skips_that_sample_and_recovers` | A container restart (counters reset) skips one sample, then measures from the new baseline. |
+| `test_the_normal_request_is_used_when_one_shot_is_refused_and_remembered` | With an older proxy that refuses one-shot, the collector falls back to the normal request and stops retrying the quick one. |
+| `test_a_container_that_vanished_does_not_make_one_shot_look_unsupported` | A container removed mid-collection doesn't switch the quick request off. |
+| `test_hostile_cpu_counters_give_no_figure_and_no_crash` *(6 inputs)* | Text, null, boolean, list, object and NaN counters give no CPU figure and no error. |
+| `test_cpu_state_for_containers_that_are_gone_is_dropped` | Remembered readings for containers that no longer exist are discarded. |
| `test_container_usage_parses_docker_stats` | Docker's raw statistics are converted correctly into CPU %, memory used and percentage (ignoring reclaimable cache), network and disk totals, and process count. |
| `test_format_ports_lists_published_bindings` | Published ports are shown as `8080->80/tcp`, once even if both IPv4 and IPv6 are bound, and unpublished ports are skipped. |
@@ -394,6 +403,14 @@ validation of container data, because everything a collector reports is untruste
| `test_the_api_schema_already_refuses_container_entries_that_are_not_objects` | The API itself rejects container lists with non-object entries. |
| `test_hostile_container_records_never_break_ingest_or_other_data` | Odd records are cleaned or dropped, and the host's own numbers in the same check-in are still recorded. |
| `test_no_more_than_500_containers_are_accepted_per_check_in` | At most 500 containers are stored per check-in. |
+| `test_a_collector_too_slow_for_its_interval_warns_when_sustained_and_clears` | Gathering data over 80% of the check interval for 3 check-ins in a row opens a warning, and it clears when collection is fast again. |
+| `test_collection_within_the_limit_never_warns` | Collection just under the limit never warns. |
+| `test_one_slow_check_in_among_fast_ones_does_not_warn` | A single slow check-in isn't enough. |
+| `test_old_collectors_that_do_not_send_the_duration_never_trigger_it` | Collectors that don't report the duration (or skip it sometimes) never trigger the warning. |
+| `test_the_limit_follows_the_servers_check_interval` | The limit is a share of that server's own interval, so a 15 s interval is stricter than a 60 s one. |
+| `test_the_limit_is_an_editable_validated_threshold` | The 80% default, its 1–100 bounds and rejection of text, booleans and null. |
+| `test_a_hostile_collection_time_is_dropped_not_stored` *(8 inputs)* | Text, negative, absurd, boolean, fractional, null, list and NaN durations are discarded. |
+| `test_a_sane_collection_time_is_stored` | A normal duration is saved with the check-in. |
| `test_the_longest_container_name_still_fits_every_incident_key` | Even the longest allowed name keeps every incident key within the database limit, so a crafted name cannot make check-ins fail. |
**Inventory changes and image drift**
@@ -448,12 +465,14 @@ tested hardest (91 cases). If any of these fail, do not run the proxy until it i
| Test | What it checks |
|---|---|
-| `test_the_three_read_requests_are_allowed` *(5 inputs)* | The exact requests the collector needs (list containers, inspect one, one stats snapshot) are allowed. |
+| `test_the_four_read_requests_are_allowed` *(6 inputs)* | The exact requests the collector needs (list containers, inspect one, one stats snapshot, and the quick one-shot stats form) are allowed. |
| `test_only_get_is_allowed` *(8 inputs)* | Every method except GET (POST, PUT, DELETE, PATCH, HEAD, OPTIONS, lowercase `get`, CONNECT) is refused. |
| `test_everything_else_is_refused` *(63 inputs)* | Everything outside the allow-list is refused: other endpoints (`/info`, images, exec, start, kill, export, archive, logs), extra or changed query strings, path traversal, encoded or padded paths, injected line breaks, odd container IDs. |
| `test_inspect_output_drops_environment_and_everything_unneeded` | The container description is cut down to the needed fields (including exit code, out-of-memory flag, finish time and image ID); environment variables (often secrets), mounts, labels, host configuration and Docker's free-text error message never leave the proxy. |
| `test_list_and_stats_output_are_cut_down` | The same filtering applies to the container list and stats. |
| `test_unexpected_upstream_shapes_do_not_crash_the_filters` *(7 inputs)* | Strange or malformed answers from Docker don't crash the proxy. |
+| `test_the_one_shot_request_is_passed_on_and_answers_are_cut_down` | The quick stats request reaches Docker unchanged and the answer is cut down to the usual fields. |
+| `test_cpu_comes_from_the_change_between_two_collections_through_the_proxy` | Through the real proxy the collector uses only the quick request and remembers the CPU counters for next time. |
| `test_collector_works_through_the_proxy_and_never_sees_secrets` | End to end: the real collector gets correct container data through the proxy and no secret reaches it. |
| `test_refused_requests_never_reach_the_docker_daemon` | Attack requests are rejected with 403 and Docker is never contacted. |
| `test_upstream_failures_become_a_generic_502` | If Docker errors or is down, the caller gets a plain "Upstream error" with no details. |
diff --git a/src/agent/collector.py b/src/agent/collector.py
index 3a4a596..e290763 100644
--- a/src/agent/collector.py
+++ b/src/agent/collector.py
@@ -17,6 +17,7 @@ import ctypes.util
import hashlib
import http.client
import json
+import math
import os
import platform
import re
@@ -631,6 +632,8 @@ def collect_containers(socket_path: str | None = None, info: dict[str, Any] | No
if info is not None:
info["latency_ms"] = int((time.monotonic() - listing_started) * 1000)
result = []
+ for stale in set(_cpu_previous) - {item.get("Id") for item in containers}:
+ del _cpu_previous[stale]
for item in containers:
container_id = item.get("Id", "")
if not container_id:
@@ -698,21 +701,67 @@ def _format_ports(ports: dict[str, Any] | None) -> str | None:
return ", ".join(sorted(set(mappings)))[:512] or None
+# Cumulative CPU counters (container, whole system) from the previous check-in, per container. The one-shot stats
+# call returns at once but carries no earlier reading, so CPU % is the change since the last check-in. The first
+# check-in after a start has no figure for a container (unknown, not zero).
+_cpu_previous: dict[str, tuple[float, float]] = {}
+# Set once the proxy or daemon refuses the one-shot request while the normal one works (an older proxy image).
+_one_shot_unsupported = False
+
+
+def _fetch_stats(socket_path: str, container_id: str) -> Any:
+ """One container's stats: the quick one-shot request, falling back to the normal one that waits about a second."""
+ global _one_shot_unsupported
+ base = f"/containers/{container_id}/stats?stream=false"
+ tried_one_shot = not _one_shot_unsupported
+ if tried_one_shot:
+ try:
+ return _docker_request(socket_path, f"{base}&one-shot=true")
+ except (OSError, ValueError):
+ pass
+ stats = _docker_request(socket_path, base)
+ if tried_one_shot:
+ _one_shot_unsupported = True
+ return stats
+
+
+def _number(value: Any) -> float | None:
+ return float(value) if isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) else None
+
+
+def _cpu_percent(container_id: str, stats: dict[str, Any]) -> float | None:
+ cpu = stats.get("cpu_stats") or {}
+ previous = stats.get("precpu_stats") or {}
+ total = _number((cpu.get("cpu_usage") or {}).get("total_usage"))
+ system = _number(cpu.get("system_cpu_usage"))
+ before_total = _number((previous.get("cpu_usage") or {}).get("total_usage"))
+ before_system = _number(previous.get("system_cpu_usage"))
+ if total is not None and system is not None:
+ remembered = _cpu_previous.get(container_id)
+ _cpu_previous[container_id] = (total, system)
+ if not before_system:
+ before_total, before_system = remembered if remembered else (None, None)
+ if None in (total, system, before_total, before_system):
+ return None
+ cpu_delta, system_delta = total - before_total, system - before_system
+ # A counter that went backwards means the container restarted; skip that sample.
+ if cpu_delta <= 0 or system_delta <= 0:
+ return None
+ # cgroup v2 omits percpu_usage, so prefer online_cpus like `docker stats` does.
+ cores = cpu.get("online_cpus") or len((cpu.get("cpu_usage") or {}).get("percpu_usage") or []) or 1
+ return round(cpu_delta / system_delta * cores * 100, 2)
+
+
def _container_usage(socket_path: str, container_id: str) -> dict[str, Any]:
try:
- stats = _docker_request(socket_path, f"/containers/{container_id}/stats?stream=false")
+ stats = _fetch_stats(socket_path, container_id)
except (OSError, ValueError):
return {}
usage: dict[str, Any] = {}
try:
- cpu = stats.get("cpu_stats") or {}
- previous = stats.get("precpu_stats") or {}
- cpu_delta = (cpu.get("cpu_usage") or {}).get("total_usage", 0) - (previous.get("cpu_usage") or {}).get("total_usage", 0)
- system_delta = cpu.get("system_cpu_usage", 0) - previous.get("system_cpu_usage", 0)
- # cgroup v2 omits percpu_usage, so prefer online_cpus like `docker stats` does.
- cores = cpu.get("online_cpus") or len((cpu.get("cpu_usage") or {}).get("percpu_usage") or []) or 1
- if cpu_delta > 0 and system_delta > 0:
- usage["cpu_pct"] = round(cpu_delta / system_delta * cores * 100, 2)
+ cpu_pct = _cpu_percent(container_id, stats)
+ if cpu_pct is not None:
+ usage["cpu_pct"] = cpu_pct
memory = stats.get("memory_stats") or {}
# Page cache is reclaimable; subtract it the same way `docker stats` does.
@@ -891,6 +940,8 @@ def run_collector() -> None:
server, host = collect_host()
containers, docker = collect_docker()
host.update(docker)
+ # How long gathering took, so the dashboard can warn when it no longer fits in the check interval.
+ host["collect_duration_ms"] = int((time.monotonic() - started) * 1000)
response = send_payload(api_url, server_id, token, {"server": server, "host": host, "containers": containers})
interval = max(15, int(response.get("interval_seconds", interval)))
print(f"Sent host metrics and {len(containers)} container records; next check in {interval}s", flush=True)
diff --git a/src/agent/docker_proxy.py b/src/agent/docker_proxy.py
index 1725d15..8137b89 100644
--- a/src/agent/docker_proxy.py
+++ b/src/agent/docker_proxy.py
@@ -2,11 +2,14 @@
The Docker socket is all-or-nothing: whatever can talk to it can start a privileged container,
which is root on the host. Only this proxy mounts the socket. It listens on a Unix socket in a
-volume that only the collector shares (no network port at all), and answers exactly three kinds of request:
+volume that only the collector shares (no network port at all), and answers exactly four kinds of request:
GET /containers/json?all=1
GET /containers/<id>/json
GET /containers/<id>/stats?stream=false
+ GET /containers/<id>/stats?stream=false&one-shot=true
+
+The last one returns immediately instead of waiting a second for Docker to take a second CPU reading.
Everything else is refused, and the answers are cut down to the fields the collector uses: the
inspect output is where a container's environment variables (often secrets) live, and they never
@@ -36,7 +39,7 @@ _ID = r"[0-9a-f]{12,64}"
# Matched against the whole request target, query string included.
_LIST = re.compile(r"/containers/json(\?all=1)?")
_INSPECT = re.compile(rf"/containers/{_ID}/json")
-_STATS = re.compile(rf"/containers/{_ID}/stats\?stream=false")
+_STATS = re.compile(rf"/containers/{_ID}/stats\?stream=false(&one-shot=true)?")
def _pick(source: Any, keys: tuple[str, ...]) -> dict[str, Any]:
diff --git a/src/backend/main.py b/src/backend/main.py
index 78d61f1..6aba15f 100644
--- a/src/backend/main.py
+++ b/src/backend/main.py
@@ -221,7 +221,7 @@ def startup_event():
("boot_id", "VARCHAR(64)"), ("oom_kill_total", "INTEGER"), ("kernel_tainted", "INTEGER"),
("ntp_synced", "BOOLEAN"), ("reboot_required", "BOOLEAN"), ("last_shutdown", "VARCHAR(16)"),
("docker_status", "VARCHAR(16)"), ("docker_reason", "VARCHAR(16)"), ("docker_latency_ms", "INTEGER"),
- ("internet_status", "VARCHAR(16)"), ("internet_latency_ms", "INTEGER"),
+ ("internet_status", "VARCHAR(16)"), ("internet_latency_ms", "INTEGER"), ("collect_duration_ms", "INTEGER"),
):
connection.execute(text(f"ALTER TABLE host_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}"))
for column, column_type in (("exit_code", "INTEGER"), ("oom_killed", "BOOLEAN"), ("finished_at", "TIMESTAMP WITH TIME ZONE"), ("image_id", "VARCHAR(80)")):
@@ -1433,6 +1433,8 @@ def _clean_system_fields(fields: dict) -> None:
fields["docker_reason"] = None
if "docker_latency_ms" in fields:
fields["docker_latency_ms"] = _whole_number(fields["docker_latency_ms"], 0, 600_000)
+ if "collect_duration_ms" in fields:
+ fields["collect_duration_ms"] = _whole_number(fields["collect_duration_ms"], 0, 3_600_000)
if "internet_status" in fields and fields["internet_status"] not in INTERNET_STATUSES:
fields["internet_status"] = None
if "internet_latency_ms" in fields:
diff --git a/src/backend/models.py b/src/backend/models.py
index 95052d7..0c150ea 100644
--- a/src/backend/models.py
+++ b/src/backend/models.py
@@ -204,6 +204,8 @@ class HostMetric(Base):
# Whether the host can reach the internet (ok | degraded | dns_failed | down); None when the check is off.
internet_status: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
internet_latency_ms: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
+ # How long the collector took to gather this check-in (milliseconds); None from collectors that don't say.
+ collect_duration_ms: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
class SystemEvent(Base):
diff --git a/src/backend/rules.py b/src/backend/rules.py
index f3f43b7..dc76b40 100644
--- a/src/backend/rules.py
+++ b/src/backend/rules.py
@@ -31,6 +31,8 @@ DEFAULT_THRESHOLDS: dict[str, float | int] = {
"event_window_minutes": 60,
# Listing containers slower than this (sustained) means the Docker daemon is struggling.
"docker_slow_ms": 2000,
+ # Collecting a check-in taking more than this share of the server's check interval (sustained) means it is falling behind.
+ "collect_slow_pct": 80,
# A container whose memory climbs steadily is flagged as a possible leak; see memory_trend_rules.
"leak_window_hours": 12, "leak_min_mb_per_hour": 10, "leak_min_growth_pct": 20,
}
@@ -288,6 +290,11 @@ def system_rules(db, server, thresholds: dict, now: datetime, active_keys: froze
firing[key] = (CRITICAL, f"Docker is unreachable on {host}", f"Container data has stopped updating because {DOCKER_REASONS.get(latest.docker_reason, DOCKER_REASONS['error'])}. Check that the Docker daemon (and the docker-proxy container) is running.")
elif len(samples) >= sustained and all(sample.docker_status == "ok" and sample.docker_latency_ms is not None and sample.docker_latency_ms > thresholds["docker_slow_ms"] for sample in samples):
firing[f"{SYSTEM_RULE_PREFIX}docker_slow"] = (WARNING, f"Docker is responding slowly on {host}", f"Listing containers took {latest.docker_latency_ms} ms (limit {thresholds['docker_slow_ms']} ms) for several check-ins in a row. The daemon may be overloaded or the host starved of resources.")
+ budget_ms = server.check_interval_seconds * 1000 * thresholds["collect_slow_pct"] / 100
+ if len(samples) >= sustained and all(sample.collect_duration_ms is not None and sample.collect_duration_ms > budget_ms for sample in samples):
+ firing[f"{SYSTEM_RULE_PREFIX}collect_slow"] = (
+ WARNING, f"The collector on {host} can't keep up with its check interval",
+ f"Gathering data took {latest.collect_duration_ms / 1000:.1f} s of the {server.check_interval_seconds} s check interval (limit {thresholds['collect_slow_pct']:g}%) for several check-ins in a row, so data arrives late. Raise the check interval or reduce the number of containers on this machine.")
if latest.reboot_required:
firing[f"{SYSTEM_RULE_PREFIX}reboot_required"] = (INFO, f"{host} needs a reboot", "Installed updates (often a new kernel) only take effect after a restart.")
return firing
diff --git a/src/frontend/src/App.jsx b/src/frontend/src/App.jsx
index f929159..bbd9018 100644
--- a/src/frontend/src/App.jsx
+++ b/src/frontend/src/App.jsx
@@ -895,10 +895,23 @@ const refresh = async () => {
// Inicijalno učitavanje
refresh()
- const timer = setInterval(refresh, LIVE_GRAPH_REFRESH_INTERVAL_SECONDS * 1000)
+ // No polling while the tab is hidden; refresh straight away when it comes back.
+ let timer = null
+ const start = () => { if (timer === null) timer = setInterval(refresh, LIVE_GRAPH_REFRESH_INTERVAL_SECONDS * 1000) }
+ const stop = () => { if (timer !== null) { clearInterval(timer); timer = null } }
+ const onVisibilityChange = () => {
+ if (document.hidden) { stop(); return }
+ refresh()
+ start()
+ }
+ if (!document.hidden) start()
+ document.addEventListener('visibilitychange', onVisibilityChange)
// Obavezno čišćenje timera pri unmountu
- return () => clearInterval(timer)
+ return () => {
+ stop()
+ document.removeEventListener('visibilitychange', onVisibilityChange)
+ }
}, [user?.id]) // <-- Stabilna instanca, okida se samo pri promjeni ID-a
const servers = overview.servers || []
@@ -2059,6 +2072,10 @@ function IncidentThresholdsSection() {
<span><span className="text-[#bcbcbc]">Slow Docker</span><span className="block text-xs text-[#7f8589]">Milliseconds to list containers before Docker counts as slow (sustained)</span></span>
<ThresholdInput label="Slow Docker threshold in milliseconds" value={values?.docker_slow_ms} step={100} onChange={set('docker_slow_ms')} disabled={loading} />
</label>
+ <label className="flex flex-col gap-3 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3 text-sm sm:flex-row sm:items-center sm:justify-between sm:gap-4 sm:py-2.5">
+ <span><span className="text-[#bcbcbc]">Slow collection</span><span className="block text-xs text-[#7f8589]">Warning when gathering data takes more than this percent of the check interval (sustained)</span></span>
+ <ThresholdInput label="Slow collection threshold in percent of the check interval" value={values?.collect_slow_pct} step={5} onChange={set('collect_slow_pct')} disabled={loading} />
+ </label>
<div className="flex flex-col gap-3 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3 text-sm sm:flex-row sm:items-center sm:justify-between sm:gap-4 sm:py-2.5">
<span className="shrink-0"><span className="text-[#bcbcbc]">Possible memory leak</span><span className="block text-xs text-[#7f8589]">Warning when a container's memory climbs steadily</span></span>
<span className="flex flex-wrap items-center gap-2 text-xs text-[#9aa0a6]">
diff --git a/tests/test_container_checks.py b/tests/test_container_checks.py
index ae535c0..880b4e4 100644
--- a/tests/test_container_checks.py
+++ b/tests/test_container_checks.py
@@ -607,3 +607,65 @@ def test_the_longest_container_name_still_fits_every_incident_key(db):
keys = list(_active(db))
assert keys and max(len(key) for key in keys) < 255
+
+
+# --- collector keeping up with its check interval (#44) ---------------------------------------------
+
+def test_a_collector_too_slow_for_its_interval_warns_when_sustained_and_clears(db):
+ for _ in range(2):
+ _ingest(db, collect_duration_ms=55_000) # 60 s interval, limit 80% = 48 s
+ assert "system:collect_slow" not in _active(db)
+ _ingest(db, collect_duration_ms=55_000)
+ alert = _active(db)["system:collect_slow"]
+ assert alert.severity == alerts.WARNING and "55.0 s" in alert.description and "60 s" in alert.description
+
+ _ingest(db, collect_duration_ms=900)
+ assert "system:collect_slow" not in _active(db)
+
+
+def test_collection_within_the_limit_never_warns(db):
+ for _ in range(6):
+ _ingest(db, collect_duration_ms=47_999)
+ assert "system:collect_slow" not in _active(db)
+
+
+def test_one_slow_check_in_among_fast_ones_does_not_warn(db):
+ for duration in (900, 55_000, 900, 55_000, 900):
+ _ingest(db, collect_duration_ms=duration)
+ assert "system:collect_slow" not in _active(db)
+
+
+def test_old_collectors_that_do_not_send_the_duration_never_trigger_it(db):
+ for _ in range(6):
+ _ingest(db)
+ assert "system:collect_slow" not in _active(db)
+ for duration in (55_000, None, 55_000):
+ _ingest(db, collect_duration_ms=duration)
+ assert "system:collect_slow" not in _active(db)
+
+
+def test_the_limit_follows_the_servers_check_interval(db):
+ db.get(Server, 5).check_interval_seconds = 15
+ db.commit()
+ for _ in range(3):
+ _ingest(db, collect_duration_ms=13_000) # fine at 60 s, too slow at 15 s
+ assert "system:collect_slow" in _active(db)
+
+
+def test_the_limit_is_an_editable_validated_threshold():
+ assert rules.DEFAULT_THRESHOLDS["collect_slow_pct"] == 80
+ for bad in (0, 101, -5, "80", True, None):
+ with pytest.raises(ValueError):
+ rules.validate_thresholds({"collect_slow_pct": bad}, rules.DEFAULT_THRESHOLDS)
+ assert rules.validate_thresholds({"collect_slow_pct": 60}, rules.DEFAULT_THRESHOLDS)["collect_slow_pct"] == 60
+
+
[email protected]("bad", ["12", -1, 10**12, True, 1.5, None, [1], float("nan")])
+def test_a_hostile_collection_time_is_dropped_not_stored(db, bad):
+ _ingest(db, collect_duration_ms=bad)
+ assert db.query(HostMetric).order_by(HostMetric.id.desc()).first().collect_duration_ms is None
+
+
+def test_a_sane_collection_time_is_stored(db):
+ _ingest(db, collect_duration_ms=1234)
+ assert db.query(HostMetric).order_by(HostMetric.id.desc()).first().collect_duration_ms == 1234
diff --git a/tests/test_container_detail.py b/tests/test_container_detail.py
index 9e7d562..c509046 100644
--- a/tests/test_container_detail.py
+++ b/tests/test_container_detail.py
@@ -14,6 +14,20 @@ from src.backend import main
from src.backend.models import Base, ContainerMetric, Server, User
[email protected](autouse=True)
+def _fresh_collector_state():
+ collector._cpu_previous.clear()
+ collector._one_shot_unsupported = False
+ yield
+ collector._cpu_previous.clear()
+ collector._one_shot_unsupported = False
+
+
+def _one_shot(total, system, cpus=4, **extra):
+ """Docker's one-shot answer: current counters and no earlier reading."""
+ return {"cpu_stats": {"cpu_usage": {"total_usage": total}, "system_cpu_usage": system, "online_cpus": cpus}, "precpu_stats": {}, **extra}
+
+
def _session():
engine = create_engine("sqlite://")
Base.metadata.create_all(engine)
@@ -79,6 +93,108 @@ def test_container_usage_parses_docker_stats(monkeypatch):
assert usage["pids"] == 7
+def test_the_quick_one_shot_request_is_tried_first(monkeypatch):
+ asked = []
+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: asked.append(endpoint) or _one_shot(100, 1000))
+
+ collector._container_usage("/sock", "abc")
+
+ assert asked == ["/containers/abc/stats?stream=false&one-shot=true"]
+
+
+def test_cpu_is_the_change_since_the_previous_check_in_and_the_first_has_none(monkeypatch):
+ readings = iter([_one_shot(100, 1000), _one_shot(300, 1400), _one_shot(300, 1800)])
+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: next(readings))
+
+ first = collector._container_usage("/sock", "abc")
+ second = collector._container_usage("/sock", "abc")
+ idle = collector._container_usage("/sock", "abc")
+
+ assert "cpu_pct" not in first # unknown, not zero
+ assert second["cpu_pct"] == 200.0 # 200 of 400 CPU time across 4 cores
+ assert "cpu_pct" not in idle # no CPU used at all gives no figure, as before
+
+
+def test_containers_do_not_share_cpu_history(monkeypatch):
+ readings = {"a": iter([_one_shot(100, 1000), _one_shot(200, 1400)]), "b": iter([_one_shot(5000, 1000), _one_shot(5040, 1400)])}
+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: next(readings[endpoint.split("/")[2]]))
+
+ for name in ("a", "b"):
+ collector._container_usage("/sock", name)
+
+ assert collector._container_usage("/sock", "a")["cpu_pct"] == 100.0
+ assert collector._container_usage("/sock", "b")["cpu_pct"] == 40.0
+
+
+def test_a_counter_that_goes_backwards_skips_that_sample_and_recovers(monkeypatch):
+ readings = iter([_one_shot(900, 5000), _one_shot(50, 5400), _one_shot(150, 5800)])
+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: next(readings))
+
+ collector._container_usage("/sock", "abc")
+ restarted = collector._container_usage("/sock", "abc")
+ after = collector._container_usage("/sock", "abc")
+
+ assert "cpu_pct" not in restarted # the container restarted
+ assert after["cpu_pct"] == 100.0 # measured from the new baseline
+
+
+def test_the_normal_request_is_used_when_one_shot_is_refused_and_remembered(monkeypatch):
+ classic = {"cpu_stats": {"cpu_usage": {"total_usage": 300}, "system_cpu_usage": 1000, "online_cpus": 4}, "precpu_stats": {"cpu_usage": {"total_usage": 100}, "system_cpu_usage": 600}}
+ asked = []
+
+ def fake(socket_path, endpoint):
+ asked.append(endpoint)
+ if "one-shot" in endpoint:
+ raise OSError("Docker API returned HTTP 403")
+ return classic
+
+ monkeypatch.setattr(collector, "_docker_request", fake)
+
+ assert collector._container_usage("/sock", "abc")["cpu_pct"] == 200.0
+ assert collector._one_shot_unsupported
+ asked.clear()
+ assert collector._container_usage("/sock", "abc")["cpu_pct"] == 200.0
+ assert asked == ["/containers/abc/stats?stream=false"] # no wasted one-shot attempt next time
+
+
+def test_a_container_that_vanished_does_not_make_one_shot_look_unsupported(monkeypatch):
+ def gone(socket_path, endpoint):
+ raise OSError("Docker API returned HTTP 404")
+
+ monkeypatch.setattr(collector, "_docker_request", gone)
+
+ assert collector._container_usage("/sock", "abc") == {}
+ assert not collector._one_shot_unsupported
+
+
[email protected]("bad", ["12", None, True, [1], {"a": 1}, float("nan")])
+def test_hostile_cpu_counters_give_no_figure_and_no_crash(monkeypatch, bad):
+ readings = iter([_one_shot(100, 1000), _one_shot(bad, 1400), _one_shot(200, bad), _one_shot(300, 1800)])
+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: next(readings))
+
+ results = [collector._container_usage("/sock", "abc") for _ in range(3)]
+
+ assert all("cpu_pct" not in usage for usage in results[:2])
+
+
+def test_cpu_state_for_containers_that_are_gone_is_dropped(monkeypatch):
+ collector._cpu_previous.update({"gone" + "0" * 8: (1, 1), "kept": (2, 2)})
+
+ def fake(socket_path, endpoint):
+ if endpoint.startswith("/containers/json"):
+ return [{"Id": "kept", "Names": ["/kept"], "Image": "x", "State": "running"}]
+ if endpoint.endswith("/json"):
+ return {"Id": "kept", "Name": "/kept", "State": {"Status": "running"}, "Config": {"Image": "x"}}
+ return _one_shot(10, 100)
+
+ monkeypatch.setattr(collector, "_docker_request", fake)
+ monkeypatch.setattr(collector.Path, "exists", lambda self: True)
+
+ collector.collect_containers("/sock")
+
+ assert list(collector._cpu_previous) == ["kept"]
+
+
def test_format_ports_lists_published_bindings():
ports = {"80/tcp": [{"HostIp": "0.0.0.0", "HostPort": "8080"}, {"HostIp": "::", "HostPort": "8080"}], "443/tcp": None}
assert collector._format_ports(ports) == "8080->80/tcp"
diff --git a/tests/test_docker_proxy.py b/tests/test_docker_proxy.py
index 5ea57c2..d47591f 100644
--- a/tests/test_docker_proxy.py
+++ b/tests/test_docker_proxy.py
@@ -24,12 +24,26 @@ STATS = {"cpu_stats": {"cpu_usage": {"total_usage": 10}, "system_cpu_usage": 100
"blkio_stats": {"io_service_bytes_recursive": []}, "pids_stats": {"current": 3}, "name": "/web", "id": CID}
+# What Docker returns for one-shot: the same readings but no earlier one to compare with.
+ONE_SHOT_STATS = {**STATS, "precpu_stats": {}}
+
+
[email protected](autouse=True)
+def _fresh_collector_state():
+ collector._cpu_previous.clear()
+ collector._one_shot_unsupported = False
+ yield
+ collector._cpu_previous.clear()
+ collector._one_shot_unsupported = False
+
+
# --- routing: the allow-list ---------------------------------------------------------------
@pytest.mark.parametrize("target", [
"/containers/json", "/containers/json?all=1", f"/containers/{CID}/json", f"/containers/{CID}/stats?stream=false", f"/containers/{CID[:12]}/json",
+ f"/containers/{CID}/stats?stream=false&one-shot=true",
])
-def test_the_three_read_requests_are_allowed(target):
+def test_the_four_read_requests_are_allowed(target):
assert proxy.route("GET", target) is not None
@@ -47,6 +61,13 @@ def test_only_get_is_allowed(method):
# query abuse
"/containers/json?all=1&filters=x", "/containers/json?all=0", "/containers/json?all=1&all=1", "/containers/json?size=1", "/containers/json?",
f"/containers/{CID}/stats", f"/containers/{CID}/stats?stream=true", f"/containers/{CID}/stats?stream=false&one-shot=1", f"/containers/{CID}/json?size=1",
+ # one-shot: exactly one spelling, in exactly one order
+ f"/containers/{CID}/stats?stream=false&one-shot=false", f"/containers/{CID}/stats?stream=false&one-shot=", f"/containers/{CID}/stats?stream=false&one-shot",
+ f"/containers/{CID}/stats?one-shot=true&stream=false", f"/containers/{CID}/stats?one-shot=true", f"/containers/{CID}/stats?stream=true&one-shot=true",
+ f"/containers/{CID}/stats?stream=false&one-shot=true&one-shot=true", f"/containers/{CID}/stats?stream=false&one-shot=true&stream=true",
+ f"/containers/{CID}/stats?stream=false&one-shot=True", f"/containers/{CID}/stats?stream=false&one-shot=TRUE", f"/containers/{CID}/stats?stream=false&One-Shot=true",
+ f"/containers/{CID}/stats?stream=false&one-shot=true&", f"/containers/{CID}/stats?stream=false&one-shot=true&x=1", f"/containers/{CID}/stats?stream=false&&one-shot=true",
+ f"/containers/{CID}/stats?stream=false&one-shot=true#f", f"/containers/{CID}/stats?stream=false&one-shot=true ", f"/containers/{CID}/stats?stream=false&one-shot=%74rue",
# traversal, encoding, padding
f"/containers/{CID}/../../images/json", "/containers/../images/json", f"/containers/{CID}/json/..", "//containers/json", "/containers//json",
"/containers/json/", f"/containers/{CID}/json/", "/containers/%6Ason", "/containers/json%00", f"/containers/{CID}%2fjson", "/%2e%2e/containers/json",
@@ -90,7 +111,8 @@ class _FakeDocker(BaseHTTPRequestHandler):
def do_GET(self):
_FakeDocker.seen.append(self.path)
- routes = {"/containers/json?all=1": LISTING, f"/containers/{CID}/json": INSPECT, f"/containers/{CID}/stats?stream=false": STATS}
+ routes = {"/containers/json?all=1": LISTING, f"/containers/{CID}/json": INSPECT, f"/containers/{CID}/stats?stream=false": STATS,
+ f"/containers/{CID}/stats?stream=false&one-shot=true": ONE_SHOT_STATS}
body = json.dumps(routes.get(self.path)).encode() if self.path in routes else b"nope"
self.send_response(200 if self.path in routes else 404)
self.send_header("Content-Length", str(len(body)))
@@ -136,10 +158,31 @@ def test_collector_works_through_the_proxy_and_never_sees_secrets(stack):
assert SECRET not in json.dumps(container)
+def test_the_one_shot_request_is_passed_on_and_answers_are_cut_down(stack):
+ port, _ = stack
+
+ status, body = _raw(port, "GET", f"/containers/{CID}/stats?stream=false&one-shot=true")
+
+ assert status == 200 and _FakeDocker.seen == [f"/containers/{CID}/stats?stream=false&one-shot=true"]
+ assert set(json.loads(body)) == {"cpu_stats", "precpu_stats", "memory_stats", "networks", "blkio_stats", "pids_stats"}
+
+
+def test_cpu_comes_from_the_change_between_two_collections_through_the_proxy(stack):
+ port, _ = stack
+
+ [first] = collector.collect_containers(port)
+ [second] = collector.collect_containers(port)
+
+ assert "cpu_pct" not in first and "cpu_pct" not in second # the fake daemon reports identical counters twice
+ assert collector._cpu_previous == {CID: (10, 100)} and not collector._one_shot_unsupported
+ assert f"/containers/{CID}/stats?stream=false" not in _FakeDocker.seen # only the one-shot form was used
+
+
def test_refused_requests_never_reach_the_docker_daemon(stack):
port, _ = stack
attacks = [("POST", "/containers/create"), ("DELETE", f"/containers/{CID}"), ("GET", f"/containers/{CID}/archive?path=/etc"), ("GET", "/containers/../info"),
- ("GET", f"/containers/{CID}/json?size=1"), ("POST", f"/containers/{CID}/start"), ("GET", "/containers/%6Ason")]
+ ("GET", f"/containers/{CID}/json?size=1"),
+ ("GET", f"/containers/{CID}/stats?one-shot=true&stream=false"), ("GET", f"/containers/{CID}/stats?stream=false&one-shot=false"), ("POST", f"/containers/{CID}/start"), ("GET", "/containers/%6Ason")]
for method, path in attacks:
assert _raw(port, method, path)[0] == 403, (method, path)