Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
perf+feat: quick wins from the optimization milestone
karloks20056c7823a2a2e7
diff --git a/README.md b/README.mdindex e8281d3..b0fc2c1 100644--- a/README.md+++ b/README.md@@ -443,6 +443,7 @@ stops firing.| Possible memory leak (steady climb over 12 hours) | 10 MB/h and 20% growth | warning || Docker unreachable (after it worked before) | 3 check-ins in a row | critical || Docker slow to list containers | over 2000 ms, 3 check-ins in a row | warning |+| Collector can't keep up with its check interval | gathering takes over 80% of the interval, 3 check-ins in a row | warning || New listener reachable from other machines | stays open 60 minutes | warning || Firewall configuration changed | stays open 60 minutes | info || Firewall protection reduced (ufw turned off, rules removed, a file gone) | stays open 60 minutes | warning |@@ -480,7 +481,7 @@ the same number of check-ins.Admins can change every threshold, the number of check-ins in a row, how longevent incidents (reboots, OOM kills, container exits, removals and image changes)-stay open, the slow-Docker limit, the memory-leak window and sensitivity, and+stay open, the slow-Docker and slow-collection limits, the memory-leak window and sensitivity, and**per-mount disk thresholds** under**Admin → Alerts**. A per-mount override replaces the default disk thresholds forthat mount path on every server. Changes apply from each server's next check-in@@ -653,7 +654,7 @@ from the backend's response to each report.- **Collector privileges.** In the Compose installs the collector has no Dockersocket, drops all capabilities and has a read-only filesystem. Only the Docker- proxy mounts the socket; it has no network, answers three read-only requests and+ proxy mounts the socket; it has no network, answers four read-only requests andstrips environment variables, labels, mounts and error text from what it returns.Everything a collector reports about containers is validated again by the backend(types, ranges, lengths, counts) before it is stored.diff --git a/SECURITY.md b/SECURITY.mdindex d2755d9..50ceade 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -8,7 +8,7 @@ This page says what each component can reach, so you can decide what to run.| Component | Runs as | Reaches | Notes ||---|---|---|---|| Collector | container | host `/proc`, the host filesystem (read-only), the Docker proxy, the backend over HTTP | No Docker socket in the compose installs. |-| Docker proxy | container, uid 0, no capabilities | the Docker socket; a Unix socket shared only with the collector; no network | Allows three read-only requests. See below. |+| Docker proxy | container, uid 0, no capabilities | the Docker socket; a Unix socket shared only with the collector; no network | Allows four read-only requests. See below. || System helper (optional) | container, uid 0 with **one** capability | kernel log, `/var/log/wtmp` (read-only), one output volume | Off by default. Details below. || Collector gate (server stack) | nginx container, no capabilities | the backend; publishes one port on your VPN address | The only way to reach collector endpoints. See below. || Backend / frontend / database | containers | each other | Only the frontend port is published. |@@ -51,8 +51,10 @@ A Docker socket mount is root on the host: the `:ro` flag only protects the sockAPI behind it, and anything that can reach the API can start a privileged container. So in thecompose files the collector **does not mount the socket**. A separate `docker-proxy` container does:-- It answers exactly three requests: `GET /containers/json?all=1`, `GET /containers/<id>/json` and- `GET /containers/<id>/stats?stream=false`. Everything else (other endpoints, other methods, extra+- It answers exactly four requests: `GET /containers/json?all=1`, `GET /containers/<id>/json`,+ `GET /containers/<id>/stats?stream=false` and the same stats request with `&one-shot=true` (which makes+ Docker answer at once instead of waiting a second for a second CPU reading; the collector works out CPU %+ from the change since its previous check-in). Everything else (other endpoints, other methods, extraquery parameters, encoded or padded paths) gets a 403 and is logged. Matching is literal against thewhole request target, with no decoding.- It cuts the answers down to a fixed list of fields. For container inspect that is the ID, name,diff --git a/TESTS.md b/TESTS.mdindex 37176c5..f7ee386 100644--- a/TESTS.md+++ b/TESTS.md@@ -4,7 +4,7 @@ Vexion ships with an automated test suite. This page explains how to run it, whacovers (and what it doesn't), and what every single test is for, so you can decidewhether a failure matters before you upgrade or change your setup.-- **628 test cases** from 366 test functions (some functions run once per input).+- **676 test cases** from 384 test functions (some functions run once per input).- **About 15 to 20 seconds** to run. No Docker, PostgreSQL, Redis, network or root access needed.- **Safe to run next to a live install.** The tests use a throwaway in-memory databaseand fakes; they never touch your running containers or your data. (Two collector@@ -127,11 +127,11 @@ docker compose logs --tail 20 collector # "Sent host metrics and N contai| [`test_alerts.py`](#test_alertspy) | 10 | The incident engine: one incident per problem, resolve, container rules. || [`test_auth_backend.py`](#test_auth_backendpy) | 50 | Accounts, passwords, the first-account setup token, and the cross-site request check. || [`test_collector_mounts.py`](#test_collector_mountspy) | 1 | Which filesystems the collector reports. |-| [`test_container_detail.py`](#test_container_detailpy) | 5 | Per-container details and parsing of Docker's stats. |+| [`test_container_detail.py`](#test_container_detailpy) | 18 | Per-container details and parsing of Docker's stats. || [`test_conversations.py`](#test_conversationspy) | 1 | Deleting one assistant conversation leaves the others alone. |-| [`test_container_checks.py`](#test_container_checkspy) | 111 | Docker health, container exit codes, inventory and image changes, memory-leak trends, and the validation of everything a collector sends about containers. |+| [`test_container_checks.py`](#test_container_checkspy) | 126 | Docker health, container exit codes, inventory and image changes, memory-leak trends, and the validation of everything a collector sends about containers. || [`test_dashboard_host.py`](#test_dashboard_hostpy) | 8 | The "this machine" logic, history charts, container list and incident counts. |-| [`test_docker_proxy.py`](#test_docker_proxypy) | 91 | The read-only Docker proxy: what it allows, refuses and hides. |+| [`test_docker_proxy.py`](#test_docker_proxypy) | 111 | The read-only Docker proxy: what it allows, refuses and hides. || [`test_enrollment.py`](#test_enrollmentpy) | 10 | Enrollment keys, collector credentials and offline-server detection. || [`test_health_checks.py`](#test_health_checkspy) | 4 | The assistant's fixed health checks and the critical-only webhook. || [`test_network_checks.py`](#test_network_checkspy) | 94 | The host's real network totals, listening-port inventory, optional internet check, firewall fingerprints, and the validation of all of it. |@@ -155,6 +155,7 @@ out of their own install.|---|---|| `test_codes_match_the_rfc_6238_reference_vectors` *(6 inputs)* | The one-time code generator gives the exact answers published in the TOTP standard (RFC 6238), so it will match any authenticator app. || `test_a_code_is_accepted_in_its_own_period_and_one_either_side_but_not_further` | A code works in its own 30-second period and one either side (clock drift), not further away. |+| `test_a_code_cannot_be_replayed` | A code that just signed you in is refused if used again in its window; the next period's code works. || `test_a_code_cannot_be_used_twice_or_after_a_newer_one` | A code that was already used, or an older one, is refused. || `test_malformed_codes_are_rejected` *(12 inputs)* | Empty, short, long, non-numeric, huge and non-string codes are refused without errors. || `test_spaces_and_a_hyphen_in_a_typed_code_are_ignored` | `123 456` and `123-456` are accepted as `123456`. |@@ -323,6 +324,14 @@ Accounts, passwords, first-run protection and the cross-site check.| `test_container_detail_returns_latest_and_chronological_history` | A container's detail view returns its latest numbers and a history sorted oldest to newest. || `test_container_detail_is_visible_to_regular_users` | Regular users can open container details (they are read-only). || `test_container_detail_returns_404_for_unknown_server` | Asking for an unknown server gives a clean 404. |+| `test_the_quick_one_shot_request_is_tried_first` | Stats are fetched with the fast one-shot request first. |+| `test_cpu_is_the_change_since_the_previous_check_in_and_the_first_has_none` | CPU % is worked out from the change since the last check-in; the first check-in shows no figure rather than a false 0. |+| `test_containers_do_not_share_cpu_history` | Each container's CPU is measured against its own earlier reading. |+| `test_a_counter_that_goes_backwards_skips_that_sample_and_recovers` | A container restart (counters reset) skips one sample, then measures from the new baseline. |+| `test_the_normal_request_is_used_when_one_shot_is_refused_and_remembered` | With an older proxy that refuses one-shot, the collector falls back to the normal request and stops retrying the quick one. |+| `test_a_container_that_vanished_does_not_make_one_shot_look_unsupported` | A container removed mid-collection doesn't switch the quick request off. |+| `test_hostile_cpu_counters_give_no_figure_and_no_crash` *(6 inputs)* | Text, null, boolean, list, object and NaN counters give no CPU figure and no error. |+| `test_cpu_state_for_containers_that_are_gone_is_dropped` | Remembered readings for containers that no longer exist are discarded. || `test_container_usage_parses_docker_stats` | Docker's raw statistics are converted correctly into CPU %, memory used and percentage (ignoring reclaimable cache), network and disk totals, and process count. || `test_format_ports_lists_published_bindings` | Published ports are shown as `8080->80/tcp`, once even if both IPv4 and IPv6 are bound, and unpublished ports are skipped. |@@ -394,6 +403,14 @@ validation of container data, because everything a collector reports is untruste| `test_the_api_schema_already_refuses_container_entries_that_are_not_objects` | The API itself rejects container lists with non-object entries. || `test_hostile_container_records_never_break_ingest_or_other_data` | Odd records are cleaned or dropped, and the host's own numbers in the same check-in are still recorded. || `test_no_more_than_500_containers_are_accepted_per_check_in` | At most 500 containers are stored per check-in. |+| `test_a_collector_too_slow_for_its_interval_warns_when_sustained_and_clears` | Gathering data over 80% of the check interval for 3 check-ins in a row opens a warning, and it clears when collection is fast again. |+| `test_collection_within_the_limit_never_warns` | Collection just under the limit never warns. |+| `test_one_slow_check_in_among_fast_ones_does_not_warn` | A single slow check-in isn't enough. |+| `test_old_collectors_that_do_not_send_the_duration_never_trigger_it` | Collectors that don't report the duration (or skip it sometimes) never trigger the warning. |+| `test_the_limit_follows_the_servers_check_interval` | The limit is a share of that server's own interval, so a 15 s interval is stricter than a 60 s one. |+| `test_the_limit_is_an_editable_validated_threshold` | The 80% default, its 1–100 bounds and rejection of text, booleans and null. |+| `test_a_hostile_collection_time_is_dropped_not_stored` *(8 inputs)* | Text, negative, absurd, boolean, fractional, null, list and NaN durations are discarded. |+| `test_a_sane_collection_time_is_stored` | A normal duration is saved with the check-in. || `test_the_longest_container_name_still_fits_every_incident_key` | Even the longest allowed name keeps every incident key within the database limit, so a crafted name cannot make check-ins fail. |**Inventory changes and image drift**@@ -448,12 +465,14 @@ tested hardest (91 cases). If any of these fail, do not run the proxy until it i| Test | What it checks ||---|---|-| `test_the_three_read_requests_are_allowed` *(5 inputs)* | The exact requests the collector needs (list containers, inspect one, one stats snapshot) are allowed. |+| `test_the_four_read_requests_are_allowed` *(6 inputs)* | The exact requests the collector needs (list containers, inspect one, one stats snapshot, and the quick one-shot stats form) are allowed. || `test_only_get_is_allowed` *(8 inputs)* | Every method except GET (POST, PUT, DELETE, PATCH, HEAD, OPTIONS, lowercase `get`, CONNECT) is refused. || `test_everything_else_is_refused` *(63 inputs)* | Everything outside the allow-list is refused: other endpoints (`/info`, images, exec, start, kill, export, archive, logs), extra or changed query strings, path traversal, encoded or padded paths, injected line breaks, odd container IDs. || `test_inspect_output_drops_environment_and_everything_unneeded` | The container description is cut down to the needed fields (including exit code, out-of-memory flag, finish time and image ID); environment variables (often secrets), mounts, labels, host configuration and Docker's free-text error message never leave the proxy. || `test_list_and_stats_output_are_cut_down` | The same filtering applies to the container list and stats. || `test_unexpected_upstream_shapes_do_not_crash_the_filters` *(7 inputs)* | Strange or malformed answers from Docker don't crash the proxy. |+| `test_the_one_shot_request_is_passed_on_and_answers_are_cut_down` | The quick stats request reaches Docker unchanged and the answer is cut down to the usual fields. |+| `test_cpu_comes_from_the_change_between_two_collections_through_the_proxy` | Through the real proxy the collector uses only the quick request and remembers the CPU counters for next time. || `test_collector_works_through_the_proxy_and_never_sees_secrets` | End to end: the real collector gets correct container data through the proxy and no secret reaches it. || `test_refused_requests_never_reach_the_docker_daemon` | Attack requests are rejected with 403 and Docker is never contacted. || `test_upstream_failures_become_a_generic_502` | If Docker errors or is down, the caller gets a plain "Upstream error" with no details. |diff --git a/src/agent/collector.py b/src/agent/collector.pyindex 3a4a596..e290763 100644--- a/src/agent/collector.py+++ b/src/agent/collector.py@@ -17,6 +17,7 @@ import ctypes.utilimport hashlibimport http.clientimport json+import mathimport osimport platformimport re@@ -631,6 +632,8 @@ def collect_containers(socket_path: str | None = None, info: dict[str, Any] | Noif info is not None:info["latency_ms"] = int((time.monotonic() - listing_started) * 1000)result = []+ for stale in set(_cpu_previous) - {item.get("Id") for item in containers}:+ del _cpu_previous[stale]for item in containers:container_id = item.get("Id", "")if not container_id:@@ -698,21 +701,67 @@ def _format_ports(ports: dict[str, Any] | None) -> str | None:return ", ".join(sorted(set(mappings)))[:512] or None+# Cumulative CPU counters (container, whole system) from the previous check-in, per container. The one-shot stats+# call returns at once but carries no earlier reading, so CPU % is the change since the last check-in. The first+# check-in after a start has no figure for a container (unknown, not zero).+_cpu_previous: dict[str, tuple[float, float]] = {}+# Set once the proxy or daemon refuses the one-shot request while the normal one works (an older proxy image).+_one_shot_unsupported = False+++def _fetch_stats(socket_path: str, container_id: str) -> Any:+ """One container's stats: the quick one-shot request, falling back to the normal one that waits about a second."""+ global _one_shot_unsupported+ base = f"/containers/{container_id}/stats?stream=false"+ tried_one_shot = not _one_shot_unsupported+ if tried_one_shot:+ try:+ return _docker_request(socket_path, f"{base}&one-shot=true")+ except (OSError, ValueError):+ pass+ stats = _docker_request(socket_path, base)+ if tried_one_shot:+ _one_shot_unsupported = True+ return stats+++def _number(value: Any) -> float | None:+ return float(value) if isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) else None+++def _cpu_percent(container_id: str, stats: dict[str, Any]) -> float | None:+ cpu = stats.get("cpu_stats") or {}+ previous = stats.get("precpu_stats") or {}+ total = _number((cpu.get("cpu_usage") or {}).get("total_usage"))+ system = _number(cpu.get("system_cpu_usage"))+ before_total = _number((previous.get("cpu_usage") or {}).get("total_usage"))+ before_system = _number(previous.get("system_cpu_usage"))+ if total is not None and system is not None:+ remembered = _cpu_previous.get(container_id)+ _cpu_previous[container_id] = (total, system)+ if not before_system:+ before_total, before_system = remembered if remembered else (None, None)+ if None in (total, system, before_total, before_system):+ return None+ cpu_delta, system_delta = total - before_total, system - before_system+ # A counter that went backwards means the container restarted; skip that sample.+ if cpu_delta <= 0 or system_delta <= 0:+ return None+ # cgroup v2 omits percpu_usage, so prefer online_cpus like `docker stats` does.+ cores = cpu.get("online_cpus") or len((cpu.get("cpu_usage") or {}).get("percpu_usage") or []) or 1+ return round(cpu_delta / system_delta * cores * 100, 2)++def _container_usage(socket_path: str, container_id: str) -> dict[str, Any]:try:- stats = _docker_request(socket_path, f"/containers/{container_id}/stats?stream=false")+ stats = _fetch_stats(socket_path, container_id)except (OSError, ValueError):return {}usage: dict[str, Any] = {}try:- cpu = stats.get("cpu_stats") or {}- previous = stats.get("precpu_stats") or {}- cpu_delta = (cpu.get("cpu_usage") or {}).get("total_usage", 0) - (previous.get("cpu_usage") or {}).get("total_usage", 0)- system_delta = cpu.get("system_cpu_usage", 0) - previous.get("system_cpu_usage", 0)- # cgroup v2 omits percpu_usage, so prefer online_cpus like `docker stats` does.- cores = cpu.get("online_cpus") or len((cpu.get("cpu_usage") or {}).get("percpu_usage") or []) or 1- if cpu_delta > 0 and system_delta > 0:- usage["cpu_pct"] = round(cpu_delta / system_delta * cores * 100, 2)+ cpu_pct = _cpu_percent(container_id, stats)+ if cpu_pct is not None:+ usage["cpu_pct"] = cpu_pctmemory = stats.get("memory_stats") or {}# Page cache is reclaimable; subtract it the same way `docker stats` does.@@ -891,6 +940,8 @@ def run_collector() -> None:server, host = collect_host()containers, docker = collect_docker()host.update(docker)+ # How long gathering took, so the dashboard can warn when it no longer fits in the check interval.+ host["collect_duration_ms"] = int((time.monotonic() - started) * 1000)response = send_payload(api_url, server_id, token, {"server": server, "host": host, "containers": containers})interval = max(15, int(response.get("interval_seconds", interval)))print(f"Sent host metrics and {len(containers)} container records; next check in {interval}s", flush=True)diff --git a/src/agent/docker_proxy.py b/src/agent/docker_proxy.pyindex 1725d15..8137b89 100644--- a/src/agent/docker_proxy.py+++ b/src/agent/docker_proxy.py@@ -2,11 +2,14 @@The Docker socket is all-or-nothing: whatever can talk to it can start a privileged container,which is root on the host. Only this proxy mounts the socket. It listens on a Unix socket in a-volume that only the collector shares (no network port at all), and answers exactly three kinds of request:+volume that only the collector shares (no network port at all), and answers exactly four kinds of request:GET /containers/json?all=1GET /containers/<id>/jsonGET /containers/<id>/stats?stream=false+ GET /containers/<id>/stats?stream=false&one-shot=true++The last one returns immediately instead of waiting a second for Docker to take a second CPU reading.Everything else is refused, and the answers are cut down to the fields the collector uses: theinspect output is where a container's environment variables (often secrets) live, and they never@@ -36,7 +39,7 @@ _ID = r"[0-9a-f]{12,64}"# Matched against the whole request target, query string included._LIST = re.compile(r"/containers/json(\?all=1)?")_INSPECT = re.compile(rf"/containers/{_ID}/json")-_STATS = re.compile(rf"/containers/{_ID}/stats\?stream=false")+_STATS = re.compile(rf"/containers/{_ID}/stats\?stream=false(&one-shot=true)?")def _pick(source: Any, keys: tuple[str, ...]) -> dict[str, Any]:diff --git a/src/backend/main.py b/src/backend/main.pyindex 78d61f1..6aba15f 100644--- a/src/backend/main.py+++ b/src/backend/main.py@@ -221,7 +221,7 @@ def startup_event():("boot_id", "VARCHAR(64)"), ("oom_kill_total", "INTEGER"), ("kernel_tainted", "INTEGER"),("ntp_synced", "BOOLEAN"), ("reboot_required", "BOOLEAN"), ("last_shutdown", "VARCHAR(16)"),("docker_status", "VARCHAR(16)"), ("docker_reason", "VARCHAR(16)"), ("docker_latency_ms", "INTEGER"),- ("internet_status", "VARCHAR(16)"), ("internet_latency_ms", "INTEGER"),+ ("internet_status", "VARCHAR(16)"), ("internet_latency_ms", "INTEGER"), ("collect_duration_ms", "INTEGER"),):connection.execute(text(f"ALTER TABLE host_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}"))for column, column_type in (("exit_code", "INTEGER"), ("oom_killed", "BOOLEAN"), ("finished_at", "TIMESTAMP WITH TIME ZONE"), ("image_id", "VARCHAR(80)")):@@ -1433,6 +1433,8 @@ def _clean_system_fields(fields: dict) -> None:fields["docker_reason"] = Noneif "docker_latency_ms" in fields:fields["docker_latency_ms"] = _whole_number(fields["docker_latency_ms"], 0, 600_000)+ if "collect_duration_ms" in fields:+ fields["collect_duration_ms"] = _whole_number(fields["collect_duration_ms"], 0, 3_600_000)if "internet_status" in fields and fields["internet_status"] not in INTERNET_STATUSES:fields["internet_status"] = Noneif "internet_latency_ms" in fields:diff --git a/src/backend/models.py b/src/backend/models.pyindex 95052d7..0c150ea 100644--- a/src/backend/models.py+++ b/src/backend/models.py@@ -204,6 +204,8 @@ class HostMetric(Base):# Whether the host can reach the internet (ok | degraded | dns_failed | down); None when the check is off.internet_status: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)internet_latency_ms: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)+ # How long the collector took to gather this check-in (milliseconds); None from collectors that don't say.+ collect_duration_ms: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)class SystemEvent(Base):diff --git a/src/backend/rules.py b/src/backend/rules.pyindex f3f43b7..dc76b40 100644--- a/src/backend/rules.py+++ b/src/backend/rules.py@@ -31,6 +31,8 @@ DEFAULT_THRESHOLDS: dict[str, float | int] = {"event_window_minutes": 60,# Listing containers slower than this (sustained) means the Docker daemon is struggling."docker_slow_ms": 2000,+ # Collecting a check-in taking more than this share of the server's check interval (sustained) means it is falling behind.+ "collect_slow_pct": 80,# A container whose memory climbs steadily is flagged as a possible leak; see memory_trend_rules."leak_window_hours": 12, "leak_min_mb_per_hour": 10, "leak_min_growth_pct": 20,}@@ -288,6 +290,11 @@ def system_rules(db, server, thresholds: dict, now: datetime, active_keys: frozefiring[key] = (CRITICAL, f"Docker is unreachable on {host}", f"Container data has stopped updating because {DOCKER_REASONS.get(latest.docker_reason, DOCKER_REASONS['error'])}. Check that the Docker daemon (and the docker-proxy container) is running.")elif len(samples) >= sustained and all(sample.docker_status == "ok" and sample.docker_latency_ms is not None and sample.docker_latency_ms > thresholds["docker_slow_ms"] for sample in samples):firing[f"{SYSTEM_RULE_PREFIX}docker_slow"] = (WARNING, f"Docker is responding slowly on {host}", f"Listing containers took {latest.docker_latency_ms} ms (limit {thresholds['docker_slow_ms']} ms) for several check-ins in a row. The daemon may be overloaded or the host starved of resources.")+ budget_ms = server.check_interval_seconds * 1000 * thresholds["collect_slow_pct"] / 100+ if len(samples) >= sustained and all(sample.collect_duration_ms is not None and sample.collect_duration_ms > budget_ms for sample in samples):+ firing[f"{SYSTEM_RULE_PREFIX}collect_slow"] = (+ WARNING, f"The collector on {host} can't keep up with its check interval",+ f"Gathering data took {latest.collect_duration_ms / 1000:.1f} s of the {server.check_interval_seconds} s check interval (limit {thresholds['collect_slow_pct']:g}%) for several check-ins in a row, so data arrives late. Raise the check interval or reduce the number of containers on this machine.")if latest.reboot_required:firing[f"{SYSTEM_RULE_PREFIX}reboot_required"] = (INFO, f"{host} needs a reboot", "Installed updates (often a new kernel) only take effect after a restart.")return firingdiff --git a/src/frontend/src/App.jsx b/src/frontend/src/App.jsxindex f929159..bbd9018 100644--- a/src/frontend/src/App.jsx+++ b/src/frontend/src/App.jsx@@ -895,10 +895,23 @@ const refresh = async () => {// Inicijalno učitavanjerefresh()- const timer = setInterval(refresh, LIVE_GRAPH_REFRESH_INTERVAL_SECONDS * 1000)+ // No polling while the tab is hidden; refresh straight away when it comes back.+ let timer = null+ const start = () => { if (timer === null) timer = setInterval(refresh, LIVE_GRAPH_REFRESH_INTERVAL_SECONDS * 1000) }+ const stop = () => { if (timer !== null) { clearInterval(timer); timer = null } }+ const onVisibilityChange = () => {+ if (document.hidden) { stop(); return }+ refresh()+ start()+ }+ if (!document.hidden) start()+ document.addEventListener('visibilitychange', onVisibilityChange)// Obavezno čišćenje timera pri unmountu- return () => clearInterval(timer)+ return () => {+ stop()+ document.removeEventListener('visibilitychange', onVisibilityChange)+ }}, [user?.id]) // <-- Stabilna instanca, okida se samo pri promjeni ID-aconst servers = overview.servers || []@@ -2059,6 +2072,10 @@ function IncidentThresholdsSection() {<span><span className="text-[#bcbcbc]">Slow Docker</span><span className="block text-xs text-[#7f8589]">Milliseconds to list containers before Docker counts as slow (sustained)</span></span><ThresholdInput label="Slow Docker threshold in milliseconds" value={values?.docker_slow_ms} step={100} onChange={set('docker_slow_ms')} disabled={loading} /></label>+ <label className="flex flex-col gap-3 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3 text-sm sm:flex-row sm:items-center sm:justify-between sm:gap-4 sm:py-2.5">+ <span><span className="text-[#bcbcbc]">Slow collection</span><span className="block text-xs text-[#7f8589]">Warning when gathering data takes more than this percent of the check interval (sustained)</span></span>+ <ThresholdInput label="Slow collection threshold in percent of the check interval" value={values?.collect_slow_pct} step={5} onChange={set('collect_slow_pct')} disabled={loading} />+ </label><div className="flex flex-col gap-3 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3 text-sm sm:flex-row sm:items-center sm:justify-between sm:gap-4 sm:py-2.5"><span className="shrink-0"><span className="text-[#bcbcbc]">Possible memory leak</span><span className="block text-xs text-[#7f8589]">Warning when a container's memory climbs steadily</span></span><span className="flex flex-wrap items-center gap-2 text-xs text-[#9aa0a6]">diff --git a/tests/test_container_checks.py b/tests/test_container_checks.pyindex ae535c0..880b4e4 100644--- a/tests/test_container_checks.py+++ b/tests/test_container_checks.py@@ -607,3 +607,65 @@ def test_the_longest_container_name_still_fits_every_incident_key(db):keys = list(_active(db))assert keys and max(len(key) for key in keys) < 255+++# --- collector keeping up with its check interval (#44) ---------------------------------------------++def test_a_collector_too_slow_for_its_interval_warns_when_sustained_and_clears(db):+ for _ in range(2):+ _ingest(db, collect_duration_ms=55_000) # 60 s interval, limit 80% = 48 s+ assert "system:collect_slow" not in _active(db)+ _ingest(db, collect_duration_ms=55_000)+ alert = _active(db)["system:collect_slow"]+ assert alert.severity == alerts.WARNING and "55.0 s" in alert.description and "60 s" in alert.description++ _ingest(db, collect_duration_ms=900)+ assert "system:collect_slow" not in _active(db)+++def test_collection_within_the_limit_never_warns(db):+ for _ in range(6):+ _ingest(db, collect_duration_ms=47_999)+ assert "system:collect_slow" not in _active(db)+++def test_one_slow_check_in_among_fast_ones_does_not_warn(db):+ for duration in (900, 55_000, 900, 55_000, 900):+ _ingest(db, collect_duration_ms=duration)+ assert "system:collect_slow" not in _active(db)+++def test_old_collectors_that_do_not_send_the_duration_never_trigger_it(db):+ for _ in range(6):+ _ingest(db)+ assert "system:collect_slow" not in _active(db)+ for duration in (55_000, None, 55_000):+ _ingest(db, collect_duration_ms=duration)+ assert "system:collect_slow" not in _active(db)+++def test_the_limit_follows_the_servers_check_interval(db):+ db.get(Server, 5).check_interval_seconds = 15+ db.commit()+ for _ in range(3):+ _ingest(db, collect_duration_ms=13_000) # fine at 60 s, too slow at 15 s+ assert "system:collect_slow" in _active(db)+++def test_the_limit_is_an_editable_validated_threshold():+ assert rules.DEFAULT_THRESHOLDS["collect_slow_pct"] == 80+ for bad in (0, 101, -5, "80", True, None):+ with pytest.raises(ValueError):+ rules.validate_thresholds({"collect_slow_pct": bad}, rules.DEFAULT_THRESHOLDS)+ assert rules.validate_thresholds({"collect_slow_pct": 60}, rules.DEFAULT_THRESHOLDS)["collect_slow_pct"] == 60++[email protected]("bad", ["12", -1, 10**12, True, 1.5, None, [1], float("nan")])+def test_a_hostile_collection_time_is_dropped_not_stored(db, bad):+ _ingest(db, collect_duration_ms=bad)+ assert db.query(HostMetric).order_by(HostMetric.id.desc()).first().collect_duration_ms is None+++def test_a_sane_collection_time_is_stored(db):+ _ingest(db, collect_duration_ms=1234)+ assert db.query(HostMetric).order_by(HostMetric.id.desc()).first().collect_duration_ms == 1234diff --git a/tests/test_container_detail.py b/tests/test_container_detail.pyindex 9e7d562..c509046 100644--- a/tests/test_container_detail.py+++ b/tests/test_container_detail.py@@ -14,6 +14,20 @@ from src.backend import mainfrom src.backend.models import Base, ContainerMetric, Server, User[email protected](autouse=True)+def _fresh_collector_state():+ collector._cpu_previous.clear()+ collector._one_shot_unsupported = False+ yield+ collector._cpu_previous.clear()+ collector._one_shot_unsupported = False+++def _one_shot(total, system, cpus=4, **extra):+ """Docker's one-shot answer: current counters and no earlier reading."""+ return {"cpu_stats": {"cpu_usage": {"total_usage": total}, "system_cpu_usage": system, "online_cpus": cpus}, "precpu_stats": {}, **extra}++def _session():engine = create_engine("sqlite://")Base.metadata.create_all(engine)@@ -79,6 +93,108 @@ def test_container_usage_parses_docker_stats(monkeypatch):assert usage["pids"] == 7+def test_the_quick_one_shot_request_is_tried_first(monkeypatch):+ asked = []+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: asked.append(endpoint) or _one_shot(100, 1000))++ collector._container_usage("/sock", "abc")++ assert asked == ["/containers/abc/stats?stream=false&one-shot=true"]+++def test_cpu_is_the_change_since_the_previous_check_in_and_the_first_has_none(monkeypatch):+ readings = iter([_one_shot(100, 1000), _one_shot(300, 1400), _one_shot(300, 1800)])+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: next(readings))++ first = collector._container_usage("/sock", "abc")+ second = collector._container_usage("/sock", "abc")+ idle = collector._container_usage("/sock", "abc")++ assert "cpu_pct" not in first # unknown, not zero+ assert second["cpu_pct"] == 200.0 # 200 of 400 CPU time across 4 cores+ assert "cpu_pct" not in idle # no CPU used at all gives no figure, as before+++def test_containers_do_not_share_cpu_history(monkeypatch):+ readings = {"a": iter([_one_shot(100, 1000), _one_shot(200, 1400)]), "b": iter([_one_shot(5000, 1000), _one_shot(5040, 1400)])}+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: next(readings[endpoint.split("/")[2]]))++ for name in ("a", "b"):+ collector._container_usage("/sock", name)++ assert collector._container_usage("/sock", "a")["cpu_pct"] == 100.0+ assert collector._container_usage("/sock", "b")["cpu_pct"] == 40.0+++def test_a_counter_that_goes_backwards_skips_that_sample_and_recovers(monkeypatch):+ readings = iter([_one_shot(900, 5000), _one_shot(50, 5400), _one_shot(150, 5800)])+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: next(readings))++ collector._container_usage("/sock", "abc")+ restarted = collector._container_usage("/sock", "abc")+ after = collector._container_usage("/sock", "abc")++ assert "cpu_pct" not in restarted # the container restarted+ assert after["cpu_pct"] == 100.0 # measured from the new baseline+++def test_the_normal_request_is_used_when_one_shot_is_refused_and_remembered(monkeypatch):+ classic = {"cpu_stats": {"cpu_usage": {"total_usage": 300}, "system_cpu_usage": 1000, "online_cpus": 4}, "precpu_stats": {"cpu_usage": {"total_usage": 100}, "system_cpu_usage": 600}}+ asked = []++ def fake(socket_path, endpoint):+ asked.append(endpoint)+ if "one-shot" in endpoint:+ raise OSError("Docker API returned HTTP 403")+ return classic++ monkeypatch.setattr(collector, "_docker_request", fake)++ assert collector._container_usage("/sock", "abc")["cpu_pct"] == 200.0+ assert collector._one_shot_unsupported+ asked.clear()+ assert collector._container_usage("/sock", "abc")["cpu_pct"] == 200.0+ assert asked == ["/containers/abc/stats?stream=false"] # no wasted one-shot attempt next time+++def test_a_container_that_vanished_does_not_make_one_shot_look_unsupported(monkeypatch):+ def gone(socket_path, endpoint):+ raise OSError("Docker API returned HTTP 404")++ monkeypatch.setattr(collector, "_docker_request", gone)++ assert collector._container_usage("/sock", "abc") == {}+ assert not collector._one_shot_unsupported++[email protected]("bad", ["12", None, True, [1], {"a": 1}, float("nan")])+def test_hostile_cpu_counters_give_no_figure_and_no_crash(monkeypatch, bad):+ readings = iter([_one_shot(100, 1000), _one_shot(bad, 1400), _one_shot(200, bad), _one_shot(300, 1800)])+ monkeypatch.setattr(collector, "_docker_request", lambda socket_path, endpoint: next(readings))++ results = [collector._container_usage("/sock", "abc") for _ in range(3)]++ assert all("cpu_pct" not in usage for usage in results[:2])+++def test_cpu_state_for_containers_that_are_gone_is_dropped(monkeypatch):+ collector._cpu_previous.update({"gone" + "0" * 8: (1, 1), "kept": (2, 2)})++ def fake(socket_path, endpoint):+ if endpoint.startswith("/containers/json"):+ return [{"Id": "kept", "Names": ["/kept"], "Image": "x", "State": "running"}]+ if endpoint.endswith("/json"):+ return {"Id": "kept", "Name": "/kept", "State": {"Status": "running"}, "Config": {"Image": "x"}}+ return _one_shot(10, 100)++ monkeypatch.setattr(collector, "_docker_request", fake)+ monkeypatch.setattr(collector.Path, "exists", lambda self: True)++ collector.collect_containers("/sock")++ assert list(collector._cpu_previous) == ["kept"]++def test_format_ports_lists_published_bindings():ports = {"80/tcp": [{"HostIp": "0.0.0.0", "HostPort": "8080"}, {"HostIp": "::", "HostPort": "8080"}], "443/tcp": None}assert collector._format_ports(ports) == "8080->80/tcp"diff --git a/tests/test_docker_proxy.py b/tests/test_docker_proxy.pyindex 5ea57c2..d47591f 100644--- a/tests/test_docker_proxy.py+++ b/tests/test_docker_proxy.py@@ -24,12 +24,26 @@ STATS = {"cpu_stats": {"cpu_usage": {"total_usage": 10}, "system_cpu_usage": 100"blkio_stats": {"io_service_bytes_recursive": []}, "pids_stats": {"current": 3}, "name": "/web", "id": CID}+# What Docker returns for one-shot: the same readings but no earlier one to compare with.+ONE_SHOT_STATS = {**STATS, "precpu_stats": {}}++[email protected](autouse=True)+def _fresh_collector_state():+ collector._cpu_previous.clear()+ collector._one_shot_unsupported = False+ yield+ collector._cpu_previous.clear()+ collector._one_shot_unsupported = False++# --- routing: the allow-list ---------------------------------------------------------------@pytest.mark.parametrize("target", ["/containers/json", "/containers/json?all=1", f"/containers/{CID}/json", f"/containers/{CID}/stats?stream=false", f"/containers/{CID[:12]}/json",+ f"/containers/{CID}/stats?stream=false&one-shot=true",])-def test_the_three_read_requests_are_allowed(target):+def test_the_four_read_requests_are_allowed(target):assert proxy.route("GET", target) is not None@@ -47,6 +61,13 @@ def test_only_get_is_allowed(method):# query abuse"/containers/json?all=1&filters=x", "/containers/json?all=0", "/containers/json?all=1&all=1", "/containers/json?size=1", "/containers/json?",f"/containers/{CID}/stats", f"/containers/{CID}/stats?stream=true", f"/containers/{CID}/stats?stream=false&one-shot=1", f"/containers/{CID}/json?size=1",+ # one-shot: exactly one spelling, in exactly one order+ f"/containers/{CID}/stats?stream=false&one-shot=false", f"/containers/{CID}/stats?stream=false&one-shot=", f"/containers/{CID}/stats?stream=false&one-shot",+ f"/containers/{CID}/stats?one-shot=true&stream=false", f"/containers/{CID}/stats?one-shot=true", f"/containers/{CID}/stats?stream=true&one-shot=true",+ f"/containers/{CID}/stats?stream=false&one-shot=true&one-shot=true", f"/containers/{CID}/stats?stream=false&one-shot=true&stream=true",+ f"/containers/{CID}/stats?stream=false&one-shot=True", f"/containers/{CID}/stats?stream=false&one-shot=TRUE", f"/containers/{CID}/stats?stream=false&One-Shot=true",+ f"/containers/{CID}/stats?stream=false&one-shot=true&", f"/containers/{CID}/stats?stream=false&one-shot=true&x=1", f"/containers/{CID}/stats?stream=false&&one-shot=true",+ f"/containers/{CID}/stats?stream=false&one-shot=true#f", f"/containers/{CID}/stats?stream=false&one-shot=true ", f"/containers/{CID}/stats?stream=false&one-shot=%74rue",# traversal, encoding, paddingf"/containers/{CID}/../../images/json", "/containers/../images/json", f"/containers/{CID}/json/..", "//containers/json", "/containers//json","/containers/json/", f"/containers/{CID}/json/", "/containers/%6Ason", "/containers/json%00", f"/containers/{CID}%2fjson", "/%2e%2e/containers/json",@@ -90,7 +111,8 @@ class _FakeDocker(BaseHTTPRequestHandler):def do_GET(self):_FakeDocker.seen.append(self.path)- routes = {"/containers/json?all=1": LISTING, f"/containers/{CID}/json": INSPECT, f"/containers/{CID}/stats?stream=false": STATS}+ routes = {"/containers/json?all=1": LISTING, f"/containers/{CID}/json": INSPECT, f"/containers/{CID}/stats?stream=false": STATS,+ f"/containers/{CID}/stats?stream=false&one-shot=true": ONE_SHOT_STATS}body = json.dumps(routes.get(self.path)).encode() if self.path in routes else b"nope"self.send_response(200 if self.path in routes else 404)self.send_header("Content-Length", str(len(body)))@@ -136,10 +158,31 @@ def test_collector_works_through_the_proxy_and_never_sees_secrets(stack):assert SECRET not in json.dumps(container)+def test_the_one_shot_request_is_passed_on_and_answers_are_cut_down(stack):+ port, _ = stack++ status, body = _raw(port, "GET", f"/containers/{CID}/stats?stream=false&one-shot=true")++ assert status == 200 and _FakeDocker.seen == [f"/containers/{CID}/stats?stream=false&one-shot=true"]+ assert set(json.loads(body)) == {"cpu_stats", "precpu_stats", "memory_stats", "networks", "blkio_stats", "pids_stats"}+++def test_cpu_comes_from_the_change_between_two_collections_through_the_proxy(stack):+ port, _ = stack++ [first] = collector.collect_containers(port)+ [second] = collector.collect_containers(port)++ assert "cpu_pct" not in first and "cpu_pct" not in second # the fake daemon reports identical counters twice+ assert collector._cpu_previous == {CID: (10, 100)} and not collector._one_shot_unsupported+ assert f"/containers/{CID}/stats?stream=false" not in _FakeDocker.seen # only the one-shot form was used++def test_refused_requests_never_reach_the_docker_daemon(stack):port, _ = stackattacks = [("POST", "/containers/create"), ("DELETE", f"/containers/{CID}"), ("GET", f"/containers/{CID}/archive?path=/etc"), ("GET", "/containers/../info"),- ("GET", f"/containers/{CID}/json?size=1"), ("POST", f"/containers/{CID}/start"), ("GET", "/containers/%6Ason")]+ ("GET", f"/containers/{CID}/json?size=1"),+ ("GET", f"/containers/{CID}/stats?one-shot=true&stream=false"), ("GET", f"/containers/{CID}/stats?stream=false&one-shot=false"), ("POST", f"/containers/{CID}/start"), ("GET", "/containers/%6Ason")]for method, path in attacks:assert _raw(port, method, path)[0] == 403, (method, path)