Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat: Statistics page: fleet view, container cards, histograms, heatmap, storage forecast
karloks20052fa8a13e722e
diff --git a/README.md b/README.mdindex b0fc2c1..e722451 100644--- a/README.md+++ b/README.md@@ -419,6 +419,42 @@ lists always start closed.<ui - Fleet section with every card collapsed to one line (status dot, name, This machine and incident badges, CPU/RAM/fullest disk) and the Collapse all / Expand all button>+## Statistics++**Statistics** in the navigation shows how one server behaved over time. Pick a server and a+range (1 hour, 6 hours, 24 hours, 7 days, 30 days, 90 days, 1 year or all time) and you get+CPU, memory, swap, load and root disk charts. Each chart is the average as a line with a+lighter band up to the highest value, so a short spike stays visible even in a yearly view.+Hover or touch a point for the exact time, average, highest value and number of check-ins, or+use **View as table**. Gaps (a server that was offline) are drawn as gaps. A note under the+cards says what each point is (for example "one per hour, from hourly summaries") and when+the history starts. The selection is kept in the address, so a view can be bookmarked, and+the range is remembered per browser. The short ranges refresh every 30 seconds, except+while the tab is hidden.++Under the charts, **Containers** shows every container of that server as a small card (status+dot, health, restarts, CPU, memory against its limit), with search and sorting (problems first+by default). Select a card for that container's CPU and memory over the chosen range, with its+memory limit drawn as a dashed line, how much of the time it was not running, and how many+times it restarted in the range. A container only reports CPU and memory while it runs.++Further down a server's page: how its CPU and memory use were **distributed** (the share of the+time in each 10% band), **when it is busy** (average CPU or memory by hour of the day and weekday+over the last 30 days, in your time zone), **storage** (a chart per filesystem with an estimate+such as "full in about 3 days" when there is a clear growth trend, and none otherwise), and+**reliability** (how long Docker takes to list containers, and how long the collector takes).++Choose **All servers** for the whole fleet: containers by state and open incidents by severity+(donuts), incidents opened over time by severity (stacked columns), what the incidents were+about, servers by disk usage, the containers using the most memory and CPU right now, and each+server's availability. Availability is the share of expected check-ins that arrived; it is+approximate because it assumes each server's current check interval. Every chart can be shown+as a table.++Short ranges come from the stored check-ins, longer ones from the summaries described under+[Database schema](#database-schema), so a year of history costs about as much to load as an+hour. CSV/JSON export of the charts is planned (Milestone 8 on GitHub).+## Alerts and incidentsRules run on every collector check-in. Each problem opens at most one incident@@ -758,7 +794,8 @@ pip install -r requirements-dev.txt # the app's requirements plus pytestpytest```-No Docker, PostgreSQL or Redis is needed; the suite takes about 15 seconds. See+No Docker, PostgreSQL or Redis is needed; the suite takes about a minute and a half. The chart+helpers have their own tests that need only Node 20 or newer: `cd src/frontend && node --test`. See[TESTS.md](TESTS.md) for how the tests work, what they don't cover, and what everytest is for.@@ -777,6 +814,21 @@ recovery codes (the encrypted two-factor secret lives on the `users` row), and pthresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialisesthe `users` table on a fresh volume.+**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-in+samples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or container+per hour, day, week (Monday to Sunday) and month, all in UTC, with the average and maximum of each metric.+Each tier is built from the finer one when its period ends, a period without check-ins gets no row, and the+purge never deletes samples that have not been summarised yet. Nothing displays these yet; they are the+foundation for the long-range graphs and statistics.++The metrics tables carry only the indexes their queries use: `time` (for the hourly purge) and+`(server_id, time)`, or `(server_id, container_id, time)` for containers. Older databases are+migrated at start: the new indexes are created and the redundant single-column ones are dropped,+which can be repeated safely. Creating an index briefly blocks writes to that table (a few+seconds for a table of millions of rows); on a very large install, create the indexes yourself+first with `CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (<columns>)` using the names+in `src/backend/models.py`, and the start-up migration will skip them.+## Project layout```text@@ -804,6 +856,9 @@ src/backend/ API: auth, monitoring, enrollment, rate limits, modelsalerts.py alert engine: one incident per rule, auto-resolverules.py alert rules and default thresholds+ rollups.py summarises check-ins into hourly/daily/weekly/monthly rollups+ history.py range-aware history behind the Statistics page+ stats.py fleet statistics: incidents over time, availability, busy-hours heatmaptotp.py one-time codes and recovery codes (standard library only)secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new keyreset_2fa.py command-line two-factor reset for a locked-out account@@ -813,7 +868,9 @@ src/system_helper.py optional kernel-log / wtmp helper (standard library only)health_checks.py read-only checks available to the assistantservice.py, prompts.py Gemini assistant- frontend/ React + Tailwind dashboard (Vite)+ frontend/ React + Tailwind dashboard (Vite); charts.jsx and chartMath.js are the+ charts (distributions.jsx: columns, donut, bars, heatmap), StatisticsPage.jsx,+ FleetStatistics.jsx, ServerInsights.jsx and ContainerStatistics.jsx the pagetests/ pytest suite```@@ -832,7 +889,10 @@ All paths are relative to `/api` when going through the frontend/nginx.| `GET /auth/sessions`, `POST /auth/sessions/revoke-others`, `DELETE /auth/sessions/{id}` | session | List and end your own sessions. || `GET /auth/config` | – | Whether registration is open and whether the first account still needs a setup token. || `GET /monitoring/overview` | session | Servers (latest metrics, filesystems, containers from the latest check-in, open incident count), pending servers (admins), active incidents and disk thresholds. |-| `GET /monitoring/servers/{id}/history` | session | CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). |+| `GET /monitoring/servers/{id}/history` | session | A server's metrics over time. With `?range=` (`1h`, `6h`, `24h`, `7d`, `30d`, `90d`, `1y`, `all`) it returns at most about 500 points from the right level of detail, each with average and maximum of the requested `metrics` (`cpu,ram,swap,load,disk,inodes,docker_ms,collect_ms`). Without `range`: CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). |+| `GET /stats/fleet` | session | Incidents opened per bucket and severity, by kind, open now, mean time to resolve, and per-server availability for `?range=`. Cached for a minute. |+| `GET /stats/servers/{id}/heatmap` | session | Average `?metric=cpu\|ram` by hour of the day and weekday over the last 30 days, for `?offset=` minutes from UTC. |+| `GET /monitoring/servers/{id}/mounts/history`, `GET /monitoring/servers/{id}/containers/{container_id}/history` | session | The same for one filesystem (`?mount=/data`) or one container. || `GET /monitoring/servers/{id}/containers/{container_id}` | session | One container's latest stats and history (`?limit=`, max 500). || `POST /monitoring/servers` | admin | Add a server manually; returns its collector token. || `PATCH /monitoring/servers/{id}/interval` | admin | Set check interval (15–86400 s). |diff --git a/TESTS.md b/TESTS.mdindex f7ee386..a6d3bbd 100644--- a/TESTS.md+++ b/TESTS.md@@ -4,8 +4,8 @@ Vexion ships with an automated test suite. This page explains how to run it, whacovers (and what it doesn't), and what every single test is for, so you can decidewhether a failure matters before you upgrade or change your setup.-- **676 test cases** from 384 test functions (some functions run once per input).-- **About 15 to 20 seconds** to run. No Docker, PostgreSQL, Redis, network or root access needed.+- **798 test cases** from 469 test functions, plus 16 small tests of the chart helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).+- **About a minute and a half** to run. No Docker, PostgreSQL, Redis, network or root access needed.- **Safe to run next to a live install.** The tests use a throwaway in-memory databaseand fakes; they never touch your running containers or your data. (Two collectortests read this machine's `/proc` and root filesystem, read-only.)@@ -86,7 +86,11 @@ Being clear about the limits matters, especially for a security-focused project:- **The web interface** (React) has no automated tests. Build errors show up when yourun `docker compose up -d --build`; look at the dashboard after upgrading. Thisincludes the Account page and the two-factor prompts: the endpoints behind them are- tested, the screens are not.+ tested, the screens are not. The Account page, the sign-in code step and the Statistics+ page (including container cards, the all-servers view, histograms, the heatmap and storage+ charts) were checked by hand in a real browser (headless Chromium against a running stack,+ desktop and phone width). Only the plain helper functions behind the charts have automated+ tests.- **Your deployment**: Compose files, Traefik labels, nginx configuration, your VPN,firewall rules and the collector gate are not exercised by the suite. They werechecked by hand against a running stack when written, not by an automated test.@@ -134,9 +138,13 @@ docker compose logs --tail 20 collector # "Sent host metrics and N contai| [`test_docker_proxy.py`](#test_docker_proxypy) | 111 | The read-only Docker proxy: what it allows, refuses and hides. || [`test_enrollment.py`](#test_enrollmentpy) | 10 | Enrollment keys, collector credentials and offline-server detection. || [`test_health_checks.py`](#test_health_checkspy) | 4 | The assistant's fixed health checks and the critical-only webhook. |+| [`test_history.py`](#test_historypy) | 36 | The history API behind long-range charts: which level of detail answers each range, point limits, gaps, and agreement with the raw samples. |+| [`test_indexes.py`](#test_indexespy) | 9 | The database indexes on the metrics tables: only the ones queries use, none duplicated. || [`test_network_checks.py`](#test_network_checkspy) | 94 | The host's real network totals, listening-port inventory, optional internet check, firewall fingerprints, and the validation of all of it. || [`test_pending_collectors.py`](#test_pending_collectorspy) | 28 | Adding servers: approval, join tokens, strict mode, re-pairing safety. |+| [`test_rollups.py`](#test_rollupspy) | 31 | Summarised history: hourly, daily, weekly and monthly rollups, catching up, gaps, and purging only what has been summarised. || [`test_rules.py`](#test_rulespy) | 20 | Alert thresholds: sustained breaches, per-mount disks, settings validation. |+| [`test_stats.py`](#test_statspy) | 46 | Fleet statistics (incidents over time, availability) and the busy-hours heatmap: grouping, ranges, caching and validation. || [`test_system_checks.py`](#test_system_checkspy) | 17 | Reboot, OOM, kernel-fault, clock and pending-reboot detection, and the System row's event list. || [`test_system_helper.py`](#test_system_helperpy) | 35 | The optional system helper: kernel-log parsing, `wtmp`, crash detection, hostile input. |@@ -504,6 +512,49 @@ tested hardest (91 cases). If any of these fail, do not run the proxy until it i| `test_unknown_check_is_rejected` | Asking for any check that isn't one of the fixed ones raises an error. || `test_daily_checkup_alerts_only_for_critical_findings` | The daily check-up notifies the webhook only when something is critical. |+### test_history.py++The history endpoints that charts will use for any range from the last hour to all time. The key promise is+that the answer is the same whether or not the background summarising has caught up.++| Test | What it checks |+|---|---|+| `test_the_last_hour_is_one_point_per_check_in_with_its_average_and_maximum` | The last hour is one point per check-in (both ends included), each with average and maximum, oldest first, with UTC times. |+| `test_a_week_of_samples_is_grouped_into_at_most_five_hundred_points_without_losing_anything` | A week is grouped into at most 500 points; the check-in count, the overall average and the highest maximum are unchanged by grouping. |+| `test_grouping_keeps_the_exact_average_and_maximum_of_each_group` | Three check-ins grouped into one five-minute point give their exact average and maximum. |+| `test_a_gap_stays_a_gap` | Two bursts of data hours apart have no points between them. |+| `test_a_metric_that_was_never_reported_is_null_not_zero` | A metric the collector never sent comes back as null, not 0. |+| `test_a_server_without_data_gives_an_empty_series` | A server with no data gives an empty series for every range, without errors. |+| `test_thirty_days_use_hourly_summaries_and_match_the_samples` | 30 days is answered from hourly summaries whose figures equal the samples they summarise. |+| `test_the_hour_that_has_not_been_summarised_yet_is_still_in_the_answer` | The newest, not yet summarised hour is built from the samples so the right edge is current. |+| `test_the_answer_does_not_depend_on_how_far_the_rollup_job_has_got` *(4 inputs)* | For 30 days, 90 days, 1 year and all time, the answer is identical with no summaries, part-way through catching up, and fully caught up. |+| `test_every_sample_is_counted_exactly_once_in_the_all_time_view` | All time is monthly and every check-in is counted exactly once. |+| `test_weekly_points_start_on_mondays_and_the_current_week_is_partial` | Weekly points start on Mondays and the unfinished week is included. |+| `test_a_long_series_is_thinned_to_the_point_limit` | 720 hourly rows are merged into at most 500 points without losing check-ins. |+| `test_history_starts_at_the_first_check_in_not_at_a_bucket_boundary` | "History starts" is the first check-in, not the midnight or hour boundary of a summary, even after summarising. |+| `test_when_old_samples_are_gone_the_oldest_summary_that_reaches_back_decides` | Once old samples are purged, the oldest hourly (then daily) summary that reaches further back gives the start. |+| `test_a_series_with_no_data_has_no_start` | A series without data has no start. |+| `test_a_container_series_has_cpu_memory_limit_and_down_counts` | A container series has CPU, memory, its limit and restarts (limit and restarts only with a maximum), and counts the check-ins when it wasn't running. |+| `test_a_mount_series_is_picked_by_path` | A filesystem is chosen by its path, and other mounts are excluded. |+| `test_container_summaries_agree_with_the_samples` *(2 inputs)* | Container history is identical before and after summarising. |+| `test_the_endpoint_returns_the_series_for_a_range` | The server endpoint returns the requested metrics for a range. |+| `test_without_a_range_the_original_answer_is_unchanged` | Without a range, the dashboard's original answer is unchanged. |+| `test_unknown_ranges_and_metrics_are_refused` *(9 inputs)* | Unknown ranges, empty, doubled, oversized or odd metric lists are refused (422). |+| `test_unknown_servers_mounts_and_containers_are_not_found` | Unknown servers, mounts and containers give 404; a relative mount path, a missing mount or an oversized id are refused. |+| `test_the_container_and_mount_endpoints_return_their_series` | The container and mount endpoints return their series with names. |+| `test_signing_in_is_required` | All the history endpoints refuse requests without a session. |++### test_indexes.py++The indexes on the metrics tables. The live migration is checked by hand against PostgreSQL+(these tests use SQLite and check what the models declare).++| Test | What it checks |+|---|---|+| `test_the_metrics_tables_carry_exactly_the_indexes_they_need` | Each metrics table has only the planned indexes: `time` for the purge, `(server_id, time)` (or `(server_id, container_id, time)` for containers) for reads. |+| `test_deleting_a_server_and_purging_old_rows_can_both_use_an_index` *(4 inputs)* | Every metrics table keeps an index starting with `server_id` (deleting a server cascades through it) and one on `time` alone (the hourly purge). |+| `test_a_primary_key_is_never_indexed_a_second_time` *(4 inputs)* | No table gets a redundant index on its own `id`. |+### test_network_checks.pyNetwork checks: the host's real traffic totals, which ports it listens on, whether it can@@ -648,6 +699,46 @@ from machines or people you didn't invite.| `test_a_bad_token_is_refused_even_in_open_mode` | Even in open mode, a token that is presented but wrong is refused. || `test_install_info_reports_the_configured_addresses` | **Connect a server** receives the configured VPN address and image names, and whether strict mode is on. |+### test_rollups.py++The background job that summarises check-in samples into hourly, daily, weekly (Monday to Sunday) and+monthly rows, all in UTC. If these fail, long-range history could be wrong, duplicated, or deleted before+it was kept.++| Test | What it checks |+|---|---|+| `test_buckets_start_on_utc_boundaries_and_weeks_on_monday` | Hours, days and months start on the boundary and a week starts on Monday (Sunday 23:59 belongs to the week before the next Monday). |+| `test_a_time_with_another_offset_is_converted_to_utc_first` | A time with another offset (or none) is converted to UTC before choosing its bucket. |+| `test_the_following_bucket_handles_month_ends_and_leap_years` | The next month after December is January, February in a leap year ends correctly, and weeks cross new year. |+| `test_an_hour_is_summarised_with_average_maximum_and_count` | An hour's row has the correct check-in count, average and maximum, and a metric that was never reported stays unknown (not zero). |+| `test_a_metric_missing_from_some_samples_is_averaged_over_the_ones_that_have_it` | A metric missing from some check-ins is averaged over those that have it. |+| `test_a_history_of_nothing_but_unknowns_still_counts_check_ins_and_does_not_break` | Check-ins that report nothing still produce rows with unknown averages and don't break the job. |+| `test_only_completed_buckets_are_written_after_a_short_grace_period` | An hour is written only after it ended plus a five-minute grace, so check-ins in flight are not missed. |+| `test_running_twice_changes_nothing` | A second run with nothing new builds nothing and leaves every row as it was. |+| `test_rebuilding_a_bucket_replaces_it_instead_of_duplicating_it` | Rebuilding a bucket (for example after losing the watermark) replaces its rows and includes late samples. |+| `test_servers_mounts_and_containers_are_summarised_separately` | Each server, mount and container gets its own rows. |+| `test_down_samples_count_the_check_ins_where_a_container_was_not_running` | A container's row records how many check-ins it was not running. |+| `test_a_day_is_the_samples_weighted_average_of_its_hours_and_keeps_the_maximum` | A day is the weighted average of its hours (one reading of 100 and three of 0 average 25, not 50) and keeps the highest maximum. |+| `test_each_average_remembers_how_many_check_ins_reported_it` | Every averaged metric records how many check-ins reported it. |+| `test_a_container_that_was_stopped_part_of_the_time_is_averaged_over_the_time_it_ran` | A container stopped for part of the day isn't averaged as if it were idle: its average is over the time it ran. |+| `test_hours_that_never_reported_a_metric_do_not_pull_a_day_toward_zero` | Hours without a value for a metric don't drag a day's average toward zero. |+| `test_a_week_runs_monday_to_sunday_and_is_written_only_when_sunday_has_ended` | A week covers Monday to Sunday, excludes the next Monday, and is written only once Sunday is over. |+| `test_a_month_is_built_from_its_days_including_a_leap_february` | A month is built from its days, including the 29 days of a leap February. |+| `test_data_either_side_of_new_year_lands_in_different_months_and_weeks` | New Year's Eve and New Year's Day land in different months but in the same week. |+| `test_every_tier_is_built_for_mounts_and_containers_too` | Filesystems and containers get all four tiers, not just servers. |+| `test_a_gap_stays_a_gap` | Hours, days or servers with no data get no row; nothing is filled in. |+| `test_the_first_run_summarises_the_history_that_already_exists` | On upgrade, existing samples are summarised, and only complete weeks are written. |+| `test_new_samples_are_picked_up_by_the_next_run` | Later runs continue from where the last one stopped. |+| `test_after_downtime_the_job_catches_up_a_chunk_at_a_time` | After downtime the job catches up in limited chunks until nothing is left. |+| `test_a_long_empty_stretch_cannot_stall_the_job` | A long stretch with no data at all still moves the job forward. |+| `test_the_watermark_is_stored_per_tier_and_a_damaged_one_is_ignored` | The progress marker is saved per tier, and a damaged one makes the job start over harmlessly. |+| `test_without_any_summaries_the_purge_behaves_as_before` | With no summaries yet, retention alone decides what the purge deletes (unchanged behaviour). |+| `test_samples_that_were_not_summarised_yet_are_never_purged` | The purge never deletes a sample newer than the last summarised hour, even when retention says it is old enough. |+| `test_maintenance_summarises_before_it_purges` | Maintenance summarises first, then purges, so history is kept before it is deleted. |+| `test_a_failing_rollup_job_stops_the_purge` | If summarising fails, nothing is purged that run. |+| `test_between_purges_maintenance_only_summarises` | Between hourly purges, maintenance only summarises. |+| `test_deleting_a_server_deletes_its_summaries_but_not_another_servers` | Deleting a server removes its summaries and leaves other servers' alone. |+### test_rules.pyThe alert thresholds, and the settings that control them.@@ -675,6 +766,40 @@ The alert thresholds, and the settings that control them.| `test_corrupt_mount_thresholds_are_ignored` | Damaged per-mount settings are ignored. || `test_overview_shows_latest_mounts_and_disk_thresholds` | The dashboard shows the latest filesystems and the thresholds used to colour them. |+### test_stats.py++The all-servers statistics and the busy-hours heatmap behind the Statistics page.++| Test | What it checks |+|---|---|+| `test_rule_keys_become_readable_groups` *(15 inputs)* | Alert rule keys become readable groups ("Disk space", "Server offline"); unknown, empty or odd keys become "Other". |+| `test_long_or_odd_rule_keys_never_leak_through` | A huge or unknown rule key is never shown as is. |+| `test_incidents_are_counted_per_day_and_severity_for_a_week` | Incidents opened are counted per day and severity, and per kind of problem. |+| `test_the_bucket_size_follows_the_range` | Hours up to a day, days up to a month, weeks up to a year and months beyond; weeks start on Monday. |+| `test_incidents_older_than_the_range_are_left_out` | Only incidents that opened inside the range are counted. |+| `test_mean_time_to_resolve_uses_only_resolved_incidents` | The average time to resolve counts resolved incidents only. |+| `test_open_incidents_are_counted_by_severity_whatever_the_range` | Incidents open right now are counted by severity, however old they are. |+| `test_an_install_without_incidents_gives_empty_answers` | No incidents gives empty, well-formed answers for every range. |+| `test_an_unknown_severity_is_counted_as_info_not_dropped` | An unexpected severity is counted as info instead of being lost. |+| `test_the_amount_of_data_read_is_capped_and_says_so` | Reading is capped and the answer says when it was cut. |+| `test_a_server_that_always_reported_is_fully_available` | A server that reported every time is 100% available; one that never reported is unknown, not 0%. |+| `test_a_server_that_was_silent_for_half_the_time_is_half_available` | A long silence lowers availability accordingly. |+| `test_a_new_server_is_not_blamed_for_the_time_before_it_existed` | A new server is judged from its first check-in. |+| `test_availability_is_exact_even_after_the_history_has_been_summarised` | Availability stays correct once history is summarised (a first version counted from midnight). |+| `test_availability_never_exceeds_one_hundred_percent` | Reporting more often than expected still caps at 100%. |+| `test_availability_works_for_every_range_without_data` | Every range works for servers without data. |+| `test_the_endpoint_returns_incidents_and_availability` | The endpoint returns both parts. |+| `test_answers_are_cached_for_a_minute_per_range_only` | Answers are cached for a minute, one entry per range. |+| `test_a_broken_cache_does_not_break_the_page` | If Redis is down the page still works. |+| `test_unknown_ranges_are_refused_and_never_become_cache_keys` *(5 inputs)* | Odd ranges are refused (422) and leave nothing in the cache. |+| `test_signing_in_is_required` | The fleet endpoint needs a session. |+| `test_the_heatmap_puts_each_hour_in_its_weekday_and_hour_cell` | Each hour lands in the right weekday and hour cell, averaged. |+| `test_the_heatmap_follows_the_viewers_time_zone` | The viewer's time zone moves cells across midnight correctly. |+| `test_the_heatmap_ignores_samples_older_than_thirty_days_and_unknowns` | Samples older than 30 days and missing readings are left out. |+| `test_the_heatmap_is_the_same_before_and_after_summarising` | The heatmap is identical before and after the history is summarised. |+| `test_the_heatmap_endpoint_validates_everything_and_caches_per_server_metric_and_offset` | Unknown metric, offset out of range or unknown server are refused and cache nothing new. |+| `test_the_heatmap_endpoint_needs_a_session` | The heatmap endpoint needs a session. |+### test_system_checks.pyReboots, OOM kills, kernel faults, clock sync and pending reboots, using only what the@@ -732,6 +857,24 @@ log lines, user names), so most of these tests feed it hostile input.| `test_helper_arriving_after_the_reboot_was_noticed_updates_the_event_and_alert` | If the helper finishes after the reboot was first noticed, the event and incident are updated to the right answer. || `test_invalid_last_shutdown_values_are_dropped` | An unexpected value for "last shutdown" is discarded. |+## Frontend helper tests++The chart helpers (axis ticks, gaps, histograms, the hour-by-weekday grid, donut slices, the+"full in about N days" estimate, container and fleet rankings) are plain JavaScript with no+dependencies, so they have tests that run with Node's built-in runner:++```bash+cd src/frontend && node --test # Node 20 or newer; nothing to install+```++| File | Tests | What they protect |+|---|---|---|+| `tests/chartMath.test.js` | 9 | Histogram bins and weights, the hour-by-weekday grid, donut fractions, the disk-full estimate (clear trends only), axis ticks, gaps staying gaps, weighted summaries, time labels and forecast wording. |+| `tests/fleetLogic.test.js` | 7 | Container states, top-N rankings, disk ranking and colours, online detection, availability rows, readable durations, and incident columns that fill quiet days with zeros. |++The React components themselves (the page layout, tooltips, selection) have no automated tests;+they were checked by hand in a real browser.+## Adding a test1. Put it in the file for the area it belongs to, or a new `tests/test_<area>.py`diff --git a/src/backend/history.py b/src/backend/history.pynew file mode 100644index 0000000..ef19774--- /dev/null+++ b/src/backend/history.py@@ -0,0 +1,242 @@+"""Range-aware history for charts: the last hour to all time, from the right level of detail.++A request names a range (``1h`` ... ``all``). Short ranges are read from the check-in samples, longer ones from the+hourly, weekly or monthly summaries (rollups.py). Whatever the source, the answer is a series of at most about+``MAX_POINTS`` points, each with the average and maximum of every requested metric, so a year costs about as much+as an hour. The newest period, which the rollup job has not closed yet, is built from the finer data below it, so+the right-hand edge is always current. Periods without data have no point: a gap stays a gap, and the response+says how long one point's period is (``bucket_seconds``) so a chart can tell a gap from a neighbour.+"""+from __future__ import annotations++from dataclasses import dataclass+from datetime import datetime, timedelta, timezone+from typing import Any++from sqlalchemy import func, select++from . import rollups+from .models import ContainerMetric, ContainerRollup, HostMetric, HostRollup, MountMetric, MountRollup++DETAILED = "detailed"+RANGES: dict[str, timedelta | None] = {+ "1h": timedelta(hours=1), "6h": timedelta(hours=6), "24h": timedelta(hours=24), "7d": timedelta(days=7),+ "30d": timedelta(days=30), "90d": timedelta(days=90), "1y": timedelta(days=365), "all": None,+}+# Which level of detail answers each range.+SOURCE = {"1h": DETAILED, "6h": DETAILED, "24h": DETAILED, "7d": DETAILED,+ "30d": rollups.HOUR, "90d": rollups.WEEK, "1y": rollups.WEEK, "all": rollups.MONTH}+MAX_POINTS = 500+NATURAL_SECONDS = {rollups.HOUR: 3600, rollups.DAY: 86400, rollups.WEEK: 7 * 86400, rollups.MONTH: 31 * 86400}+# Bucket sizes a chart can be given; the smallest that keeps a range within MAX_POINTS is used.+NICE_STEPS = (15, 30, 60, 120, 300, 600, 900, 1800, 3600, 7200, 14400, 21600, 43200, 86400, 7 * 86400)+# The tiers a partly finished period of each tier is assembled from, coarsest first.+FINER_CHAIN = {rollups.HOUR: (), rollups.DAY: (rollups.HOUR,), rollups.WEEK: (rollups.DAY, rollups.HOUR), rollups.MONTH: (rollups.DAY, rollups.HOUR)}+++@dataclass(frozen=True)+class Metric:+ """How one charted value is read from a raw row and from a rollup row. Either side may be absent."""++ raw: str+ avg: str | None = None # rollup column holding the average (its count column is n_<name>)+ max: str | None = None+++def _m(raw, name, *, avg=True, maximum=True):+ return Metric(raw, f"avg_{name}" if avg else None, f"max_{name}" if maximum else None)+++HOST_METRICS = {+ "cpu": _m("cpu_utilization_pct", "cpu"), "load": _m("load_1m", "load"), "ram": _m("ram_pct", "ram"), "swap": _m("swap_pct", "swap"),+ "disk": _m("disk_usage_pct", "disk"), "inodes": _m("disk_inodes_pct", "inodes"),+ "docker_ms": _m("docker_latency_ms", "docker_ms"), "collect_ms": _m("collect_duration_ms", "collect_ms"),+}+MOUNT_METRICS = {+ "used_pct": _m("used_pct", "used_pct"), "used_gb": _m("used_gb", "used_gb", maximum=False), "total_gb": _m("total_gb", "total_gb", avg=False),+ "inodes_pct": _m("inodes_pct", "inodes_pct"),+}+CONTAINER_METRICS = {+ "cpu": _m("cpu_pct", "cpu"), "memory_mb": _m("memory_used_mb", "memory_mb"), "memory_pct": _m("memory_pct", "memory_pct"),+ "memory_limit_mb": _m("memory_limit_mb", "memory_limit_mb", avg=False), "restarts": _m("restart_count", "restarts", avg=False),+}+DEFAULT_HOST_METRICS = ("cpu", "ram", "swap", "load", "disk")+++@dataclass(frozen=True)+class Series:+ raw: Any+ rollup: Any+ metrics: dict[str, Metric]+ has_down: bool = False+++HOST = Series(HostMetric, HostRollup, HOST_METRICS)+MOUNT = Series(MountMetric, MountRollup, MOUNT_METRICS)+CONTAINER = Series(ContainerMetric, ContainerRollup, CONTAINER_METRICS, has_down=True)+++class Agg:+ """Sum, count and maximum of each metric over some check-ins; adding two of them gives the exact combined figures."""++ __slots__ = ("samples", "down", "values")++ def __init__(self, samples: int = 0, down: int = 0):+ self.samples, self.down, self.values = samples, down, {}++ def add_value(self, name: str, total: float | None, count: int, maximum: float | None) -> None:+ current = self.values.setdefault(name, [0.0, 0, None])+ if total is not None and count:+ current[0] += total+ current[1] += count+ if maximum is not None and (current[2] is None or maximum > current[2]):+ current[2] = maximum++ def merge(self, other: "Agg") -> None:+ self.samples += other.samples+ self.down += other.down+ for name, (total, count, maximum) in other.values.items():+ self.add_value(name, total, count, maximum)+++def _raw_agg(spec: Series, wanted, row) -> Agg:+ agg = Agg(samples=1)+ for index, name in enumerate(wanted):+ value = row[index + 1]+ if value is not None:+ agg.add_value(name, float(value), 1, float(value))+ if spec.has_down:+ agg.down = 0 if row[-1] == "running" else 1+ return agg+++def _rollup_agg(spec: Series, wanted, row) -> Agg:+ agg = Agg(samples=row.samples, down=getattr(row, "down_samples", 0) if spec.has_down else 0)+ for name in wanted:+ metric = spec.metrics[name]+ average = getattr(row, metric.avg) if metric.avg else None+ count = getattr(row, "n_" + metric.avg[len("avg_"):]) if metric.avg else 0+ maximum = getattr(row, metric.max) if metric.max else None+ agg.add_value(name, average * count if average is not None and count else None, count if average is not None else 0, maximum)+ return agg+++def _point(spec: Series, wanted, start: datetime, agg: Agg) -> dict:+ point: dict[str, Any] = {"time": start.isoformat(), "samples": agg.samples}+ if spec.has_down:+ point["down"] = agg.down+ for name in wanted:+ total, count, maximum = agg.values.get(name, (0.0, 0, None))+ metric = spec.metrics[name]+ value: dict[str, float] = {}+ if metric.avg and count:+ value["avg"] = total / count+ if metric.max and maximum is not None:+ value["max"] = maximum+ point[name] = value or None+ return point+++def nice_step(seconds: float) -> int:+ return next((step for step in NICE_STEPS if step >= seconds), NICE_STEPS[-1])+++def _where(spec: Series, kind: str, server_id: int, key: str | None):+ """Filters that pick one server's series (and one mount or container) from a raw or rollup table."""+ table = spec.raw if kind == "raw" else spec.rollup+ conditions = [table.server_id == server_id]+ if spec is MOUNT:+ conditions.append(table.mount == key)+ elif spec is CONTAINER:+ conditions.append(table.container_id == key)+ return conditions+++def _raw_items(db, spec: Series, wanted, server_id, key, since: datetime, until: datetime):+ columns = [getattr(spec.raw, spec.metrics[name].raw) for name in wanted]+ if spec.has_down:+ columns.append(spec.raw.status)+ query = select(spec.raw.time, *columns).where(*_where(spec, "raw", server_id, key), spec.raw.time >= since, spec.raw.time < until).order_by(spec.raw.time)+ return [(rollups._utc(row[0]), _raw_agg(spec, wanted, row)) for row in db.execute(query)]+++def _rollup_items(db, spec: Series, wanted, server_id, key, tier: str, since: datetime, until: datetime):+ if until <= since:+ return []+ query = select(spec.rollup).where(*_where(spec, "rollup", server_id, key), spec.rollup.tier == tier,+ spec.rollup.bucket_start >= since, spec.rollup.bucket_start < until).order_by(spec.rollup.bucket_start)+ return [(rollups._utc(row.bucket_start), _rollup_agg(spec, wanted, row)) for row in db.scalars(query)]+++def data_since(db, spec: Series, server_id: int, key: str | None) -> datetime | None:+ """The earliest moment this series has data: the first sample if it is still stored, else the start of the+ oldest summary that reaches further back than anything finer.++ A summary's bucket starts on a boundary (midnight, Monday), which can be long before the first check-in, so a+ coarser tier only counts when its oldest bucket ends before the finer data begins; otherwise the finer data+ already says exactly when the history starts.+ """+ first = db.execute(select(func.min(spec.raw.time)).where(*_where(spec, "raw", server_id, key))).scalar()+ since = rollups._utc(first) if first else None+ for tier in (rollups.HOUR, rollups.DAY, rollups.WEEK, rollups.MONTH):+ oldest = db.execute(select(func.min(spec.rollup.bucket_start)).where(*_where(spec, "rollup", server_id, key), spec.rollup.tier == tier)).scalar()+ if oldest is None:+ continue+ oldest = rollups._utc(oldest)+ if since is None or rollups.following(tier, oldest) <= since:+ since = oldest+ return since+++def _group(items, bucket_of):+ grouped: dict[datetime, Agg] = {}+ for start, agg in items:+ grouped.setdefault(bucket_of(start), Agg()).merge(agg)+ return grouped+++def _tiered_items(db, spec, wanted, server_id, key, tier: str, window_start: datetime, now: datetime):+ """One Agg per bucket of ``tier``: closed buckets from the rollups, the newest ones from the finer data."""+ first = rollups.floor(tier, window_start)+ done = rollups._read_watermark(db, tier)+ closed_until = min(done, now) if done and done > first else first+ items = _rollup_items(db, spec, wanted, server_id, key, tier, first, closed_until)+ grouped = {start: agg for start, agg in items}+ tail: list = []+ cursor = closed_until+ for finer in FINER_CHAIN[tier]:+ finer_done = rollups._read_watermark(db, finer)+ if finer_done and finer_done > cursor:+ tail += _rollup_items(db, spec, wanted, server_id, key, finer, cursor, min(finer_done, now))+ cursor = min(finer_done, now)+ tail += _raw_items(db, spec, wanted, server_id, key, cursor, now + timedelta(seconds=1))+ for start, agg in _group(tail, lambda moment: rollups.floor(tier, moment)).items():+ grouped.setdefault(start, Agg()).merge(agg)+ return sorted(grouped.items()), NATURAL_SECONDS[tier]+++def history(db, spec: Series, server_id: int, key: str | None, range_name: str, wanted, interval_seconds: int, now: datetime | None = None) -> dict:+ """The chart series for one server (and mount or container) over ``range_name``."""+ now = rollups._utc(now or datetime.now(timezone.utc))+ since = data_since(db, spec, server_id, key)+ length = RANGES[range_name]+ window_start = max(now - length, since) if length and since else (since or now - (length or timedelta(0)))+ source = SOURCE[range_name]+ if since is None:+ items, natural = [], NATURAL_SECONDS.get(source, interval_seconds)+ elif source == DETAILED:+ items, natural = _raw_items(db, spec, wanted, server_id, key, window_start, now + timedelta(seconds=1)), interval_seconds+ else:+ items, natural = _tiered_items(db, spec, wanted, server_id, key, source, window_start, now)+ span = max((now - window_start).total_seconds(), 1)+ step = max(nice_step(max(span / MAX_POINTS, interval_seconds if source == DETAILED else 0)), natural if source != DETAILED else 0)+ if source == DETAILED or step > natural:+ epoch = lambda moment: datetime.fromtimestamp(int(moment.timestamp()) // step * step, tz=timezone.utc) # noqa: E731+ items = sorted(_group(items, epoch).items())+ else:+ step = natural+ return {+ "range": range_name, "source": source, "bucket_seconds": step,+ "start": window_start.isoformat(), "end": now.isoformat(), "data_since": since.isoformat() if since else None,+ "metrics": list(wanted), "points": [_point(spec, wanted, start, agg) for start, agg in items],+ }diff --git a/src/backend/main.py b/src/backend/main.pyindex 6aba15f..34c9fcf 100644--- a/src/backend/main.py+++ b/src/backend/main.py@@ -10,25 +10,26 @@ import threadingimport timefrom pathlib import Pathfrom datetime import datetime, timedelta, timezone-from typing import Annotated+from typing import Annotated, Literalfrom urllib.parse import urlsplitimport redisfrom fastapi import Cookie, Depends, FastAPI, Header, HTTPException, Query, Request, Response, status+from fastapi import Path as PathParamfrom fastapi.middleware.cors import CORSMiddlewarefrom fastapi.responses import JSONResponse, PlainTextResponsefrom pydantic import BaseModel, EmailStr, Fieldfrom sqlalchemy import case, func, inspect, textfrom .alerts import CONTAINER_RULE_PREFIX, CRITICAL, SERVER_OFFLINE_RULE, SEVERITIES, active_rule_keys, raise_alert, resolve_alerts, sync_alerts-from . import secrets_box, totp+from . import history as history_service, rollups, secrets_box, stats, totpfrom .auth import (STAGE_ENROLLMENT, STAGE_FULL, count_pending_attempt, create_pending_two_factor, create_session, delete_pending_two_factor, delete_session, get_recent_logs,get_session, get_user_from_session, hash_password, issue_recovery_codes, list_user_sessions, read_pending_two_factor, recovery_codes_left, record_audit_log,register_user, revoke_other_sessions, revoke_user_session, validate_password, verify_credentials, verify_password, verify_second_factor,)from .db import SessionLocal, check_database, get_db-from .models import AppSetting, AuditLog, Base, ChatMessage, ContainerMetric, Conversation, EnrollmentKey, FirewallState, HostMetric, IncidentAlert, JoinToken, ListenerState, MountMetric, PendingCollector, RecoveryCode, Server, SystemEvent, User+from .models import AppSetting, AuditLog, Base, ChatMessage, ContainerMetric, ContainerRollup, Conversation, EnrollmentKey, FirewallState, HostMetric, HostRollup, IncidentAlert, JoinToken, ListenerState, MountMetric, MountRollup, PendingCollector, RecoveryCode, Server, SystemEvent, Userfrom .db import enginefrom .config import (@@ -229,6 +230,22 @@ def startup_event():connection.execute(text("ALTER TABLE system_events ADD COLUMN IF NOT EXISTS event_key VARCHAR(128)"))connection.execute(text("ALTER TABLE pending_collectors ADD COLUMN IF NOT EXISTS join_token_id INTEGER REFERENCES join_tokens(id) ON DELETE SET NULL"))connection.execute(text("CREATE INDEX IF NOT EXISTS ix_system_events_event_key ON system_events (event_key)"))+ # Composite indexes that match how the metrics tables are read, then the single-column ones they make+ # redundant (and the pointless indexes on each table's own primary key). Safe to repeat. Creating an index+ # briefly blocks writes to that table; on a very large table create them first by hand with CONCURRENTLY.+ for table, name, columns in (+ ("host_metrics", "ix_host_metrics_server_time", "server_id, time"),+ ("mount_metrics", "ix_mount_metrics_server_time", "server_id, time"),+ ("system_events", "ix_system_events_server_time", "server_id, time"),+ ("container_metrics", "ix_container_metrics_server_container_time", "server_id, container_id, time"),+ ):+ connection.execute(text(f"CREATE INDEX IF NOT EXISTS {name} ON {table} ({columns})"))+ for redundant in (+ "ix_host_metrics_id", "ix_host_metrics_server_id", "ix_mount_metrics_id", "ix_mount_metrics_server_id",+ "ix_system_events_id", "ix_system_events_server_id",+ "ix_container_metrics_id", "ix_container_metrics_server_id", "ix_container_metrics_container_id",+ ):+ connection.execute(text(f"DROP INDEX IF EXISTS {redundant}"))for column, column_type in (("totp_secret_enc", "TEXT"), ("totp_enabled", "BOOLEAN NOT NULL DEFAULT FALSE"), ("totp_last_step", "INTEGER"), ("password_changed_at", "TIMESTAMP WITH TIME ZONE")):connection.execute(text(f"ALTER TABLE users ADD COLUMN IF NOT EXISTS {column} {column_type}"))connection.execute(text("ALTER TABLE incident_alerts ADD COLUMN IF NOT EXISTS rule_key VARCHAR(255)"))@@ -724,6 +741,10 @@ def _admin_settings(db) -> dict:def purge_old_metrics(db, now: datetime | None = None) -> int:"""Delete host and container samples older than the retention period."""cutoff = (now or datetime.now(timezone.utc)) - timedelta(days=get_metrics_retention_days(db))+ # Check-in samples that haven't been summarised yet are never deleted, whatever the retention says.+ summarised_until = rollups.hour_watermark(db)+ if summarised_until is not None:+ cutoff = min(cutoff, summarised_until)deleted = db.query(HostMetric).filter(HostMetric.time < cutoff).delete(synchronize_session=False)deleted += db.query(ContainerMetric).filter(ContainerMetric.time < cutoff).delete(synchronize_session=False)deleted += db.query(MountMetric).filter(MountMetric.time < cutoff).delete(synchronize_session=False)@@ -914,17 +935,132 @@ def latest_containers(db, server_id: int, host_time: datetime | None) -> list[CoHISTORY_MAX_MINUTES = 24 * 60+RangeName = Literal["1h", "6h", "24h", "7d", "30d", "90d", "1y", "all"]+++def _wanted_metrics(raw: str | None, allowed: dict, default: tuple[str, ...]) -> tuple[str, ...]:+ """The metrics a history request asked for, from a comma-separated list; only known names are accepted."""+ if raw is None:+ return default+ names = tuple(dict.fromkeys(name.strip() for name in raw.split(",")))+ if not names or len(raw) > 200 or any(name not in allowed for name in names):+ raise HTTPException(status_code=422, detail=f"metrics must be a comma-separated list of: {', '.join(allowed)}.")+ return names+++def _history_response(db, spec, server: Server, key: str | None, range_name: str, wanted: tuple[str, ...]) -> dict:+ result = history_service.history(db, spec, server.id, key, range_name, wanted, server.check_interval_seconds)+ return {"server_id": server.id, **result}++@app.get("/monitoring/servers/{server_id}/history")-def server_history(server_id: int, minutes: int = 30, session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):- """CPU, memory and root disk usage of one server over the last ``minutes``, oldest first."""+def server_history(+ server_id: int, minutes: int = 30, range_name: Annotated[RangeName | None, Query(alias="range")] = None, metrics: str | None = None,+ session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db),+):+ """A server's metrics over time.++ With ``range`` (1h to all): at most about 500 points from the right level of detail, each with the average and+ maximum of the requested ``metrics``. Without it: the original answer for the last ``minutes`` (CPU, memory and+ root disk, oldest first), which the dashboard's live chart uses.+ """_current_user(session_id, db)- if not db.query(Server).filter(Server.id == server_id).first():+ server = db.query(Server).filter(Server.id == server_id).first()+ if not server:raise HTTPException(status_code=404, detail="Server not found.")+ if range_name is not None:+ return _history_response(db, history_service.HOST, server, None, range_name, _wanted_metrics(metrics, history_service.HOST_METRICS, history_service.DEFAULT_HOST_METRICS))since = datetime.now(timezone.utc) - timedelta(minutes=max(1, min(minutes, HISTORY_MAX_MINUTES)))rows = db.query(HostMetric.time, HostMetric.cpu_utilization_pct, HostMetric.ram_pct, HostMetric.disk_usage_pct).filter(HostMetric.server_id == server_id, HostMetric.time >= since).order_by(HostMetric.time).all()return {"server_id": server_id, "samples": [{"time": row.time.isoformat(), "cpu": row.cpu_utilization_pct, "ram": row.ram_pct, "disk": row.disk_usage_pct} for row in rows]}+STATS_CACHE_SECONDS = 60++[email protected]("/stats/fleet")+def fleet_statistics(range_name: Annotated[RangeName, Query(alias="range")] = "24h", session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):+ """Incidents over time and per-server availability for the Statistics page's all-servers view."""+ _current_user(session_id, db)+ # Keyed only by the validated range, so a request can't create an unbounded number of cache entries.+ cache_key = f"homelab:stats:fleet:{range_name}"+ try:+ cached = redis_client.get(cache_key)+ if cached:+ return json.loads(cached)+ except Exception: # a cache problem must never break the page+ cached = None+ result = stats.fleet_stats(db, range_name)+ try:+ redis_client.set(cache_key, json.dumps(result), ex=STATS_CACHE_SECONDS)+ except Exception:+ pass+ return result++[email protected]("/stats/servers/{server_id}/heatmap")+def server_heatmap(+ server_id: int, metric: Literal["cpu", "ram"] = "cpu", offset: Annotated[int, Query(ge=-stats.MAX_OFFSET_MINUTES, le=stats.MAX_OFFSET_MINUTES)] = 0,+ session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db),+):+ """When a server is busy: its average per hour of the day and weekday over the last 30 days."""+ _current_user(session_id, db)+ server = db.query(Server).filter(Server.id == server_id).first()+ if not server:+ raise HTTPException(status_code=404, detail="Server not found.")+ # Only existing servers, known metrics and a bounded offset get this far, so the cache can't grow without limit.+ cache_key = f"homelab:stats:heatmap:{server.id}:{metric}:{offset}"+ try:+ cached = redis_client.get(cache_key)+ if cached:+ return json.loads(cached)+ except Exception:+ pass+ result = stats.hour_weekday_heatmap(db, server, metric, offset)+ try:+ redis_client.set(cache_key, json.dumps(result), ex=STATS_CACHE_SECONDS)+ except Exception:+ pass+ return result++[email protected]("/monitoring/servers/{server_id}/mounts/history")+def mount_history(+ server_id: int, mount: Annotated[str, Query(min_length=1, max_length=255)], range_name: Annotated[RangeName, Query(alias="range")] = "24h", metrics: str | None = None,+ session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db),+):+ """One filesystem's usage over time (``mount`` is its path, for example ``/data``)."""+ _current_user(session_id, db)+ server = db.query(Server).filter(Server.id == server_id).first()+ if not server or not mount.startswith("/"):+ raise HTTPException(status_code=404, detail="Server or mount not found.")+ wanted = _wanted_metrics(metrics, history_service.MOUNT_METRICS, ("used_pct", "used_gb", "total_gb"))+ result = _history_response(db, history_service.MOUNT, server, mount, range_name, wanted)+ if result["data_since"] is None:+ raise HTTPException(status_code=404, detail="Server or mount not found.")+ return {**result, "mount": mount}++[email protected]("/monitoring/servers/{server_id}/containers/{container_id}/history")+def container_history(+ server_id: int, container_id: Annotated[str, PathParam(min_length=1, max_length=128)], range_name: Annotated[RangeName, Query(alias="range")] = "24h", metrics: str | None = None,+ session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db),+):+ """One container's CPU and memory over time."""+ _current_user(session_id, db)+ server = db.query(Server).filter(Server.id == server_id).first()+ if not server:+ raise HTTPException(status_code=404, detail="Server or container not found.")+ wanted = _wanted_metrics(metrics, history_service.CONTAINER_METRICS, ("cpu", "memory_mb", "memory_pct", "memory_limit_mb"))+ result = _history_response(db, history_service.CONTAINER, server, container_id, range_name, wanted)+ if result["data_since"] is None:+ raise HTTPException(status_code=404, detail="Server or container not found.")+ name = db.query(ContainerMetric.name).filter(ContainerMetric.server_id == server.id, ContainerMetric.container_id == container_id).order_by(ContainerMetric.time.desc()).limit(1).scalar()+ if name is None:+ name = db.query(ContainerRollup.name).filter(ContainerRollup.server_id == server.id, ContainerRollup.container_id == container_id).order_by(ContainerRollup.bucket_start.desc()).limit(1).scalar()+ return {**result, "container_id": container_id, "name": name}++CONTAINER_DETAIL_FIELDS = ("status", "healthcheck_status", "restart_count", "cpu_pct", "memory_used_mb", "memory_limit_mb", "memory_pct","net_rx_mb", "net_tx_mb", "block_read_mb", "block_write_mb", "pids",@@ -1233,6 +1369,24 @@ def analyze_memory_trends(db, now: datetime | None = None) -> None:db.commit()+def metrics_maintenance(db, purge_due: bool) -> bool:+ """Summarise newly completed periods, then (when due) purge and analyse. True when the purge ran.++ Summaries come first and the purge is skipped if they fail, so history is never deleted before it was kept.+ """+ try:+ rollups.run_rollups(db)+ except Exception as exc:+ db.rollback()+ print(f"Rollups failed: {type(exc).__name__}: {exc}", flush=True)+ return False+ if not purge_due:+ return False+ purge_old_metrics(db)+ analyze_memory_trends(db)+ return True++def _offline_watch_loop() -> None:last_metrics_purge: float | None = Nonewhile True:@@ -1241,9 +1395,8 @@ def _offline_watch_loop() -> None:try:mark_offline_servers(db)purge_stale_pending(db)- if last_metrics_purge is None or time.monotonic() - last_metrics_purge >= METRICS_PURGE_INTERVAL_SECONDS:- purge_old_metrics(db)- analyze_memory_trends(db)+ purge_due = last_metrics_purge is None or time.monotonic() - last_metrics_purge >= METRICS_PURGE_INTERVAL_SECONDS+ if metrics_maintenance(db, purge_due):last_metrics_purge = time.monotonic()except Exception as exc: # keep the watcher alive through transient DB errorsprint(f"Offline watch failed: {type(exc).__name__}: {exc}", flush=True)@@ -1270,6 +1423,8 @@ def delete_monitored_server(server_id: int, session_id: Annotated[str | None, Coif not server:raise HTTPException(status_code=404, detail="Server not found.")+ for rollup in (HostRollup, MountRollup, ContainerRollup):+ db.query(rollup).filter(rollup.server_id == server.id).delete(synchronize_session=False)db.query(HostMetric).filter(HostMetric.server_id == server.id).delete(synchronize_session=False)db.query(ContainerMetric).filter(ContainerMetric.server_id == server.id).delete(synchronize_session=False)db.query(MountMetric).filter(MountMetric.server_id == server.id).delete(synchronize_session=False)diff --git a/src/backend/models.py b/src/backend/models.pyindex 0c150ea..50a44b0 100644--- a/src/backend/models.py+++ b/src/backend/models.py@@ -1,7 +1,7 @@from datetime import datetime, timezonefrom typing import Optional-from sqlalchemy import Boolean, DateTime, Float, ForeignKey, Integer, String, Text, UniqueConstraint+from sqlalchemy import Boolean, DateTime, Float, ForeignKey, Index, Integer, String, Text, UniqueConstraintfrom sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column@@ -172,8 +172,11 @@ class EnrollmentKey(Base):class HostMetric(Base):__tablename__ = "host_metrics"- id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)- server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)+ # Reads are "this server, newest first", so (server_id, time) serves them; time alone serves the purge.+ __table_args__ = (Index("ix_host_metrics_server_time", "server_id", "time"),)++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)time: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)cpu_utilization_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)load_1m: Mapped[Optional[float]] = mapped_column(Float, nullable=True)@@ -213,8 +216,10 @@ class SystemEvent(Base):__tablename__ = "system_events"- id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)- server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)+ __table_args__ = (Index("ix_system_events_server_time", "server_id", "time"),)++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)time: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)kind: Mapped[str] = mapped_column(String(32), nullable=False) # reboot | oom_kill | kernel_fault | kernel_error# Identifies the occurrence so a snapshot that repeats it doesn't record it twice.@@ -227,8 +232,10 @@ class MountMetric(Base):__tablename__ = "mount_metrics"- id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)- server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)+ __table_args__ = (Index("ix_mount_metrics_server_time", "server_id", "time"),)++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)time: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)mount: Mapped[str] = mapped_column(String(255), nullable=False)fs_type: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)@@ -271,9 +278,13 @@ class FirewallState(Base):class ContainerMetric(Base):__tablename__ = "container_metrics"- id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)- server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)- container_id: Mapped[str] = mapped_column(String(128), index=True, nullable=False)+ # One container's history is "this server, this container, newest first". time alone serves the purge and the+ # "containers at one check-in" lookup, where an exact timestamp already identifies the check-in.+ __table_args__ = (Index("ix_container_metrics_server_container_time", "server_id", "container_id", "time"),)++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)+ container_id: Mapped[str] = mapped_column(String(128), nullable=False)name: Mapped[str] = mapped_column(String(255), nullable=False)image: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)status: Mapped[str] = mapped_column(String(64), nullable=False)@@ -311,3 +322,97 @@ class IncidentAlert(Base):rule_key: Mapped[Optional[str]] = mapped_column(String(255), index=True, nullable=True)created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)resolved_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)+++class HostRollup(Base):+ """A summary of one server's check-ins over an hour, day, week or month (UTC). See rollups.py.++ Every metric has an average and a maximum. ``samples`` is how many check-ins went into the row; ``n_<metric>`` is+ how many of them reported that metric, which is the weight when a coarser tier averages this one. A metric the+ collector never reported is None.+ """++ __tablename__ = "host_rollups"+ # The unique key serves one server's history; (tier, bucket_start) serves building and purging a tier.+ __table_args__ = (UniqueConstraint("server_id", "tier", "bucket_start"), Index("ix_host_rollups_tier_bucket", "tier", "bucket_start"))++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)+ tier: Mapped[str] = mapped_column(String(8), nullable=False) # hour | day | week | month+ bucket_start: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)+ samples: Mapped[int] = mapped_column(Integer, nullable=False)+ avg_cpu: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_cpu: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_cpu: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_load: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_load: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_load: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_ram: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_ram: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_ram: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_swap: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_swap: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_swap: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_disk: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_disk: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_disk: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_inodes: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_inodes: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_inodes: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_docker_ms: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_docker_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_docker_ms: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_collect_ms: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_collect_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_collect_ms: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+++class MountRollup(Base):+ """A summary of one filesystem's usage over an hour, day, week or month (UTC)."""++ __tablename__ = "mount_rollups"+ __table_args__ = (UniqueConstraint("server_id", "mount", "tier", "bucket_start"), Index("ix_mount_rollups_tier_bucket", "tier", "bucket_start"))++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)+ mount: Mapped[str] = mapped_column(String(255), nullable=False)+ tier: Mapped[str] = mapped_column(String(8), nullable=False)+ bucket_start: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)+ samples: Mapped[int] = mapped_column(Integer, nullable=False)+ avg_used_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_used_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_used_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_used_gb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_used_gb: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_total_gb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_inodes_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_inodes_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_inodes_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+++class ContainerRollup(Base):+ """A summary of one container over an hour, day, week or month (UTC)."""++ __tablename__ = "container_rollups"+ __table_args__ = (UniqueConstraint("server_id", "container_id", "tier", "bucket_start"), Index("ix_container_rollups_tier_bucket", "tier", "bucket_start"))++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)+ container_id: Mapped[str] = mapped_column(String(128), nullable=False)+ name: Mapped[str] = mapped_column(String(255), nullable=False)+ tier: Mapped[str] = mapped_column(String(8), nullable=False)+ bucket_start: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)+ samples: Mapped[int] = mapped_column(Integer, nullable=False)+ # Check-ins in which the container was not running; the rest of ``samples`` it was.+ down_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ avg_cpu: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_cpu: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_cpu: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_memory_mb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_memory_mb: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_memory_mb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ avg_memory_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ n_memory_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ max_memory_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ max_memory_limit_mb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)+ max_restarts: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)diff --git a/src/backend/rollups.py b/src/backend/rollups.pynew file mode 100644index 0000000..f09e534--- /dev/null+++ b/src/backend/rollups.py@@ -0,0 +1,215 @@+"""Summarised history: hourly, daily, weekly and monthly rollups of the metrics tables.++Charts and statistics over weeks, months and years can't read every per-minute row, and keeping those rows for a+year isn't realistic. A background job therefore turns them into one summary row per bucket (average and maximum+per metric). Each tier is built from the finer one *when its period closes*:++ check-in samples -> hour -> day -> week (Monday to Sunday) -> month++All buckets are aligned to UTC. Rules the job keeps:++* Only complete buckets are written, a few minutes after they end so that check-ins in flight are not missed.+* A bucket is written by deleting whatever was there and inserting it again in one transaction, so running the job+ twice, or again after a failure, never duplicates a row.+* A watermark per tier (``rollup_done_<tier>`` in ``app_settings``) records how far each tier got. A stretch with no+ data advances it too, so a long gap can't make the job stand still. After downtime the job catches up, a limited+ number of buckets per run.+* A period in which a server sent nothing has no row for it. A gap stays a gap; nothing is filled in.+* Averages are exact. Each row stores, per averaged metric, how many check-ins reported it (``n_<metric>``), and a+ coarser tier weights by that, so a day with few check-ins, or a container that was stopped (and so reported no+ CPU) for half of it, counts for what it really measured. The maximum is kept so a short spike stays visible in+ a yearly view. A metric that was never reported stays None.++Deleting old rows is a separate matter (see ``purge_old_metrics`` in main.py); the purge only ever removes+check-in samples that have already been summarised.+"""+from __future__ import annotations++from datetime import datetime, timedelta, timezone++from sqlalchemy import Float, case, cast, delete, func, select++from .models import AppSetting, ContainerMetric, ContainerRollup, HostMetric, HostRollup, MountMetric, MountRollup++HOUR, DAY, WEEK, MONTH = "hour", "day", "week", "month"+TIERS = (HOUR, DAY, WEEK, MONTH)+FINER = {DAY: HOUR, WEEK: DAY, MONTH: DAY}+GRACE = timedelta(minutes=5)+# Per run, so catching up after a long outage (or the first run on a big database) works in chunks.+MAX_BUCKETS_PER_RUN = {HOUR: 400, DAY: 60, WEEK: 30, MONTH: 12}+WATERMARK_PREFIX = "rollup_done_"++# (rollup column, aggregate, source column on the raw table). "avg" is an average of the raw values, or a weighted+# average of the finer tier's averages (weighted by n_<metric>); "max" is a maximum either way.+HOST_COLUMNS = (+ ("avg_cpu", "avg", "cpu_utilization_pct"), ("max_cpu", "max", "cpu_utilization_pct"),+ ("avg_load", "avg", "load_1m"), ("max_load", "max", "load_1m"),+ ("avg_ram", "avg", "ram_pct"), ("max_ram", "max", "ram_pct"),+ ("avg_swap", "avg", "swap_pct"), ("max_swap", "max", "swap_pct"),+ ("avg_disk", "avg", "disk_usage_pct"), ("max_disk", "max", "disk_usage_pct"),+ ("avg_inodes", "avg", "disk_inodes_pct"), ("max_inodes", "max", "disk_inodes_pct"),+ ("avg_docker_ms", "avg", "docker_latency_ms"), ("max_docker_ms", "max", "docker_latency_ms"),+ ("avg_collect_ms", "avg", "collect_duration_ms"), ("max_collect_ms", "max", "collect_duration_ms"),+)+MOUNT_COLUMNS = (+ ("avg_used_pct", "avg", "used_pct"), ("max_used_pct", "max", "used_pct"),+ ("avg_used_gb", "avg", "used_gb"), ("max_total_gb", "max", "total_gb"),+ ("avg_inodes_pct", "avg", "inodes_pct"), ("max_inodes_pct", "max", "inodes_pct"),+)+CONTAINER_COLUMNS = (+ ("avg_cpu", "avg", "cpu_pct"), ("max_cpu", "max", "cpu_pct"),+ ("avg_memory_mb", "avg", "memory_used_mb"), ("max_memory_mb", "max", "memory_used_mb"),+ ("avg_memory_pct", "avg", "memory_pct"), ("max_memory_pct", "max", "memory_pct"),+ ("max_memory_limit_mb", "max", "memory_limit_mb"), ("max_restarts", "max", "restart_count"),+)+# What each kind of rollup groups by and builds from.+KINDS = (+ (HostRollup, HostMetric, ("server_id",), HOST_COLUMNS),+ (MountRollup, MountMetric, ("server_id", "mount"), MOUNT_COLUMNS),+ (ContainerRollup, ContainerMetric, ("server_id", "container_id"), CONTAINER_COLUMNS),+)+++def _utc(value: datetime) -> datetime:+ """SQLite hands back naive datetimes; everything stored here is UTC."""+ return value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value.astimezone(timezone.utc)+++def floor(tier: str, moment: datetime) -> datetime:+ """The start of the bucket that contains ``moment``."""+ moment = _utc(moment)+ if tier == HOUR:+ return moment.replace(minute=0, second=0, microsecond=0)+ day = moment.replace(hour=0, minute=0, second=0, microsecond=0)+ if tier == DAY:+ return day+ if tier == WEEK:+ return day - timedelta(days=day.weekday()) # Monday+ if tier == MONTH:+ return day.replace(day=1)+ raise ValueError(f"Unknown tier: {tier}")+++def following(tier: str, start: datetime) -> datetime:+ """The start of the bucket after the one beginning at ``start``."""+ if tier == HOUR:+ return start + timedelta(hours=1)+ if tier == DAY:+ return start + timedelta(days=1)+ if tier == WEEK:+ return start + timedelta(days=7)+ if tier == MONTH:+ return start.replace(year=start.year + 1, month=1) if start.month == 12 else start.replace(month=start.month + 1)+ raise ValueError(f"Unknown tier: {tier}")+++def _read_watermark(db, tier: str) -> datetime | None:+ setting = db.get(AppSetting, WATERMARK_PREFIX + tier)+ try:+ return _utc(datetime.fromisoformat(setting.value)) if setting else None+ except ValueError:+ return None+++def _write_watermark(db, tier: str, value: datetime) -> None:+ setting = db.get(AppSetting, WATERMARK_PREFIX + tier)+ if setting:+ setting.value = value.isoformat()+ else:+ db.add(AppSetting(key=WATERMARK_PREFIX + tier, value=value.isoformat()))+++def hour_watermark(db) -> datetime | None:+ """Check-in samples before this moment have been summarised and may be purged."""+ return _read_watermark(db, HOUR)+++def _aggregates(source, columns, from_raw: bool):+ expressions = []+ for name, kind, raw in columns:+ if kind == "max":+ expressions.append(func.max(getattr(source, raw if from_raw else name)).label(name))+ continue+ count_name = "n_" + name[len("avg_"):]+ if from_raw:+ expressions.append(func.avg(getattr(source, raw)).label(name))+ expressions.append(func.count(getattr(source, raw)).label(count_name))+ else:+ value, count = getattr(source, name), getattr(source, count_name)+ weighted = func.sum(case((value.is_not(None), value * count), else_=0))+ expressions.append((cast(weighted, Float) / func.nullif(func.sum(case((value.is_not(None), count), else_=0)), 0)).label(name))+ expressions.append(func.sum(count).label(count_name))+ return expressions+++def _build_bucket(db, tier: str, start: datetime, end: datetime) -> int:+ """Replace every rollup row of ``tier`` that begins at ``start`` and return how many were written."""+ written = 0+ for rollup, raw, keys, columns in KINDS:+ db.execute(delete(rollup).where(rollup.tier == tier, rollup.bucket_start == start))+ from_raw = tier == HOUR+ source = raw if from_raw else rollup+ key_columns = [getattr(source, key) for key in keys]+ extra = []+ if rollup is ContainerRollup:+ extra.append(func.max(source.name).label("name"))+ if from_raw:+ samples = func.count().label("samples")+ if rollup is ContainerRollup:+ extra.append(func.coalesce(func.sum(case((ContainerMetric.status != "running", 1), else_=0)), 0).label("down_samples"))+ window = (source.time >= start, source.time < end)+ else:+ samples = func.sum(source.samples).label("samples")+ if rollup is ContainerRollup:+ extra.append(func.coalesce(func.sum(source.down_samples), 0).label("down_samples"))+ window = (source.tier == FINER[tier], source.bucket_start >= start, source.bucket_start < end)+ query = select(*key_columns, samples, *extra, *_aggregates(source, columns, from_raw)).where(*window).group_by(*key_columns)+ for row in db.execute(query):+ values = row._asdict()+ db.add(rollup(tier=tier, bucket_start=start, **values))+ written += 1+ return written+++def _first_source_time(db, tier: str) -> datetime | None:+ """The earliest row the tier can be built from, in any of the tables it summarises."""+ if tier == HOUR:+ found = [db.execute(select(func.min(raw.time))).scalar() for _, raw, _, _ in KINDS]+ else:+ found = [db.execute(select(func.min(rollup.bucket_start)).where(rollup.tier == FINER[tier])).scalar() for rollup, _, _, _ in KINDS]+ found = [_utc(value) for value in found if value]+ return min(found) if found else None+++def _run_tier(db, tier: str, now: datetime) -> int:+ cursor = _read_watermark(db, tier)+ if cursor is None:+ first = _first_source_time(db, tier)+ if first is None:+ return 0+ cursor = floor(tier, first)+ # The finer tier must have got past a bucket before that bucket can be built from it.+ ready_until = now - GRACE+ if tier != HOUR:+ finer_done = _read_watermark(db, FINER[tier])+ if finer_done is None:+ return 0+ ready_until = min(ready_until, finer_done)+ built = 0+ while built < MAX_BUCKETS_PER_RUN[tier]:+ end = following(tier, cursor)+ if end > ready_until:+ break+ _build_bucket(db, tier, cursor, end)+ # The rows and the watermark are saved together, so a failure never leaves one without the other.+ _write_watermark(db, tier, end)+ db.commit()+ cursor = end+ built += 1+ return built+++def run_rollups(db, now: datetime | None = None) -> dict[str, int]:+ """Build every bucket that has completed since the last run, finest tier first. Returns buckets built per tier."""+ now = _utc(now or datetime.now(timezone.utc))+ return {tier: _run_tier(db, tier, now) for tier in TIERS}diff --git a/src/backend/stats.py b/src/backend/stats.pynew file mode 100644index 0000000..6b17eca--- /dev/null+++ b/src/backend/stats.py@@ -0,0 +1,133 @@+"""Fleet-wide statistics for the Statistics page: incidents over time and how reliably servers reported.++Only aggregated numbers leave this module, never raw rows. Incidents are counted by the time they opened, in+buckets that suit the range (hours up to a day, days up to a month, weeks up to a year, months beyond), and the+amount of data read is capped.+"""+from __future__ import annotations++from datetime import datetime, timedelta, timezone++from sqlalchemy import func, select++from . import history, rollups+from .alerts import CRITICAL, INFO, SEVERITIES, WARNING+from .models import HostMetric, IncidentAlert, Server++MAX_INCIDENTS_READ = 20000+# The bucket an incident count is grouped into, by range.+INCIDENT_BUCKET = {"1h": None, "6h": rollups.HOUR, "24h": rollups.HOUR, "7d": rollups.DAY, "30d": rollups.DAY, "90d": rollups.WEEK, "1y": rollups.WEEK, "all": rollups.MONTH}+KIND_LABELS = {+ "host:disk": "Disk space", "host:inodes": "Inodes", "host:memory": "Memory", "host:swap": "Swap", "host:load": "Load",+ "system:reboot": "Reboots", "system:oom": "Out-of-memory kills", "system:kernel": "Kernel faults", "system:ntp": "Clock not synced",+ "system:docker_unreachable": "Docker unreachable", "system:docker_slow": "Docker slow", "system:collect_slow": "Collector too slow",+ "system:reboot_required": "Reboot needed",+}+++def incident_kind(rule_key: str | None) -> str:+ """A readable group for an incident's rule, so the page never has to show raw rule keys."""+ if not rule_key:+ return "Other"+ if rule_key == "server_offline":+ return "Server offline"+ parts = rule_key.split(":")+ if len(parts) >= 2 and f"{parts[0]}:{parts[1]}" in KIND_LABELS:+ return KIND_LABELS[f"{parts[0]}:{parts[1]}"]+ return {"trend": "Possible memory leaks", "network": "Network", "container": "Containers"}.get(parts[0], "Other")+++def _utc(moment: datetime) -> datetime:+ return rollups._utc(moment)+++def window(db, range_name: str, now: datetime) -> datetime:+ """When the range starts: a fixed distance back, or the first incident for all time."""+ length = history.RANGES[range_name]+ if length is not None:+ return now - length+ first = db.execute(select(func.min(IncidentAlert.created_at))).scalar()+ return _utc(first) if first else now+++def incident_stats(db, range_name: str, now: datetime | None = None) -> dict:+ now = _utc(now or datetime.now(timezone.utc))+ start = window(db, range_name, now)+ bucket = INCIDENT_BUCKET[range_name]+ rows = db.execute(+ select(IncidentAlert.created_at, IncidentAlert.resolved_at, IncidentAlert.severity, IncidentAlert.rule_key, IncidentAlert.status)+ .where(IncidentAlert.created_at >= start).order_by(IncidentAlert.created_at).limit(MAX_INCIDENTS_READ + 1)+ ).all()+ truncated = len(rows) > MAX_INCIDENTS_READ+ rows = rows[:MAX_INCIDENTS_READ]+ columns: dict[datetime, dict[str, int]] = {}+ kinds: dict[str, int] = {}+ minutes: list[float] = []+ for created, resolved, severity, rule_key, status in rows:+ created = _utc(created)+ key = rollups.floor(bucket, created) if bucket else rollups.floor(rollups.HOUR, created)+ column = columns.setdefault(key, {level: 0 for level in SEVERITIES})+ column[severity if severity in column else INFO] += 1+ kinds[incident_kind(rule_key)] = kinds.get(incident_kind(rule_key), 0) + 1+ if resolved is not None and status == "resolved":+ minutes.append(max((_utc(resolved) - created).total_seconds() / 60, 0))+ open_now = {level: count for level, count in db.execute(select(IncidentAlert.severity, func.count()).where(IncidentAlert.status == "active").group_by(IncidentAlert.severity))}+ return {+ "range": range_name, "start": start.isoformat(), "end": now.isoformat(), "bucket": bucket or "hour",+ "opened": len(rows), "resolved": len(minutes), "truncated": truncated,+ "mean_minutes_to_resolve": (sum(minutes) / len(minutes)) if minutes else None,+ "open_now": {level.lower(): open_now.get(level, 0) for level in (CRITICAL, WARNING, INFO)},+ "by_bucket": [{"time": moment.isoformat(), **{level.lower(): counts[level] for level in (CRITICAL, WARNING, INFO)}} for moment, counts in sorted(columns.items())],+ "by_kind": [{"kind": kind, "count": count} for kind, count in sorted(kinds.items(), key=lambda item: (-item[1], item[0]))],+ }+++def availability(db, range_name: str, now: datetime | None = None) -> list[dict]:+ """Per server: the share of the check-ins it should have made in the range that actually arrived.++ Approximate: it assumes the check interval it has now held for the whole range, and the range starts when the+ server's first data does, so a new server isn't blamed for the time before it existed.+ """+ now = _utc(now or datetime.now(timezone.utc))+ result = []+ for server in db.execute(select(Server).order_by(Server.hostname)).scalars():+ series = history.history(db, history.HOST, server.id, None, range_name, (), server.check_interval_seconds, now=now)+ received = sum(point["samples"] for point in series["points"])+ span = (now - datetime.fromisoformat(series["start"])).total_seconds()+ expected = max(span / server.check_interval_seconds, 1)+ result.append({"server_id": server.id, "hostname": server.hostname, "checkins": received, "expected": round(expected),+ "percent": round(min(100.0, 100.0 * received / expected), 2) if series["data_since"] else None})+ return result+++HEATMAP_DAYS = 30+HEATMAP_METRICS = ("cpu", "ram")+MAX_OFFSET_MINUTES = 14 * 60+++def hour_weekday_heatmap(db, server: Server, metric: str, offset_minutes: int = 0, now: datetime | None = None) -> dict:+ """The average of a metric for each hour of the day and weekday over the last 30 days, for a time zone.++ ``offset_minutes`` is the viewer's offset from UTC (a fixed one: daylight saving changes inside the 30 days+ are ignored). Row 0 is Monday. Hours nobody reported stay None.+ """+ now = _utc(now or datetime.now(timezone.utc))+ items, _ = history._tiered_items(db, history.HOST, (metric,), server.id, None, rollups.HOUR, now - timedelta(days=HEATMAP_DAYS), now)+ totals = [[[0.0, 0] for _ in range(24)] for _ in range(7)]+ for start, agg in items:+ local = start + timedelta(minutes=offset_minutes)+ total, count, _maximum = agg.values.get(metric, (0.0, 0, None))+ cell = totals[local.weekday()][local.hour]+ cell[0] += total+ cell[1] += count+ since = history.data_since(db, history.HOST, server.id, None)+ return {+ "server_id": server.id, "metric": metric, "days": HEATMAP_DAYS, "offset_minutes": offset_minutes,+ "data_since": since.isoformat() if since else None,+ "cells": [[(cell[0] / cell[1]) if cell[1] else None for cell in row] for row in totals],+ "samples": [[cell[1] for cell in row] for row in totals],+ }+++def fleet_stats(db, range_name: str, now: datetime | None = None) -> dict:+ return {"incidents": incident_stats(db, range_name, now), "availability": availability(db, range_name, now)}diff --git a/src/frontend/src/App.jsx b/src/frontend/src/App.jsxindex bbd9018..99f9f2d 100644--- a/src/frontend/src/App.jsx+++ b/src/frontend/src/App.jsx@@ -1,5 +1,6 @@import { BrowserRouter, Link, NavLink, Navigate, Route, Routes, useLocation, useNavigate } from 'react-router-dom'import { useEffect, useRef, useState } from 'react'+import StatisticsContent from './StatisticsPage.jsx'const AUTH_KEY = 'homie-auth-user'const USERS_KEY = 'homie-auth-users'@@ -46,6 +47,7 @@ function parseAgentMessage(text) {const navigation = [{ label: 'Dashboard', to: '/', icon: '▣' },+ { label: 'Statistics', to: '/statistics', icon: '▤' },{ label: 'Agent', to: '/agent', icon: '◷' },{ label: 'Account', to: '/account', icon: '◉' },{ label: 'Admin', to: '/admin', icon: '⚙', adminOnly: true },@@ -2544,6 +2546,16 @@ function App() {</RequireAuth>}/>+ <Route+ path="/statistics"+ element={+ <RequireAuth user={user}>+ <AppShell user={user} onLogout={handleLogout}>+ <StatisticsContent request={apiRequest} />+ </AppShell>+ </RequireAuth>+ }+ /><Routepath="/account"element={diff --git a/src/frontend/src/ContainerStatistics.jsx b/src/frontend/src/ContainerStatistics.jsxnew file mode 100644index 0000000..e4fa5e7--- /dev/null+++ b/src/frontend/src/ContainerStatistics.jsx@@ -0,0 +1,148 @@+// The containers of one server as small cards; selecting one shows its statistics over the chosen range.+import { useEffect, useMemo, useState } from 'react'+import { TimeSeriesChart } from './charts.jsx'+import { describeBucket, seriesFor, summarise } from './chartMath.js'+import { problemRank, sortContainers } from './containerLogic.js'++const LIVE_RANGES = new Set(['1h', '6h', '24h'])+const REFRESH_SECONDS = 30+const METRICS = 'cpu,memory_mb,memory_limit_mb,restarts'+const SORTS = [+ { id: 'problems', label: 'Problems first' },+ { id: 'name', label: 'Name' },+ { id: 'memory', label: 'Memory' },+ { id: 'cpu', label: 'CPU' },+]++function statusColour(container) {+ const rank = problemRank(container)+ if (rank === 0) return container.status === 'restarting' ? 'bg-red-400' : 'bg-amber-400'+ return rank === 1 ? 'bg-red-400' : 'bg-emerald-400'+}++function ContainerCard({ container, selected, onSelect }) {+ const memoryPct = container.memory_pct+ return (+ <button+ type="button"+ aria-pressed={selected}+ onClick={onSelect}+ className={`min-w-0 rounded-2xl border p-3 text-left transition ${selected ? 'border-white/40 bg-white/10' : 'border-[#1f2326] bg-[#0d1013]/60 hover:border-white/20'}`}+ >+ <div className="flex items-center gap-2">+ <span aria-hidden="true" className={`h-2.5 w-2.5 shrink-0 rounded-full ${statusColour(container)}`} />+ <span className="truncate text-sm font-medium text-white">{container.name}</span>+ </div>+ <div className="mt-1 truncate text-xs text-[#7f8589]">+ {container.status}{container.healthcheck_status ? ` · ${container.healthcheck_status}` : ''}{container.restart_count > 0 ? ` · ${container.restart_count} restarts` : ''}+ </div>+ <div className="mt-2 flex items-baseline justify-between text-xs text-[#9aa0a6]">+ <span>CPU <span className="text-sm text-white">{container.cpu_pct == null ? '–' : `${container.cpu_pct}%`}</span></span>+ <span>Memory <span className="text-sm text-white">{container.memory_used_mb == null ? '–' : `${Math.round(container.memory_used_mb)} MB`}</span></span>+ </div>+ <div className="mt-2 h-1.5 overflow-hidden rounded-full bg-white/10" title={memoryPct == null ? 'No memory limit' : `${memoryPct}% of its memory limit`}>+ {memoryPct != null && <div className={`h-full rounded-full ${memoryPct >= 90 ? 'bg-red-400' : memoryPct >= 75 ? 'bg-amber-400' : 'bg-emerald-400'}`} style={{ width: `${Math.min(Math.max(memoryPct, 0), 100)}%` }} />}+ </div>+ </button>+ )+}++function Figure({ label, value }) {+ return (+ <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3">+ <div className="text-xs uppercase tracking-[0.18em] text-[#9aa0a6]">{label}</div>+ <div className="mt-1 text-xl font-semibold text-white">{value}</div>+ </div>+ )+}++function ContainerDetail({ request, serverId, container, range, onClose }) {+ const [data, setData] = useState(null)+ const [error, setError] = useState('')++ useEffect(() => {+ const controller = new AbortController()+ const load = () => request(`/monitoring/servers/${serverId}/containers/${encodeURIComponent(container.container_id)}/history?range=${range}&metrics=${METRICS}`, { method: 'GET', signal: controller.signal })+ .then((result) => { setData(result); setError('') })+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load this container.') })+ load()+ if (!LIVE_RANGES.has(range)) return () => controller.abort()+ const timer = setInterval(() => { if (!document.hidden) load() }, REFRESH_SECONDS * 1000)+ return () => { controller.abort(); clearInterval(timer) }+ }, [request, serverId, container.container_id, range])++ const current = data && data.container_id === container.container_id ? data : null+ const series = useMemo(() => ({ cpu: seriesFor(current?.points, 'cpu'), memory: seriesFor(current?.points, 'memory_mb') }), [current])+ const limit = useMemo(() => Math.max(0, ...(current?.points || []).map((point) => point.memory_limit_mb?.max ?? 0)) || null, [current])+ const figures = useMemo(() => {+ const points = current?.points || []+ const samples = points.reduce((total, point) => total + point.samples, 0)+ const down = points.reduce((total, point) => total + (point.down || 0), 0)+ const restarts = points.map((point) => point.restarts?.max).filter((value) => value != null)+ return { samples, notRunning: samples ? (100 * down) / samples : null, restarts: restarts.length ? restarts[restarts.length - 1] - restarts[0] : null }+ }, [current])+ const cpu = summarise(series.cpu)+ const memory = summarise(series.memory)+ const common = { bucketSeconds: current?.bucket_seconds ?? 60, start: current ? new Date(current.start).getTime() : undefined, end: current ? new Date(current.end).getTime() : undefined, utc: current ? ['day', 'week', 'month'].includes(current.source) : false }++ return (+ <section aria-label={`Statistics for ${container.name}`} className="space-y-4 rounded-3xl border border-white/10 bg-[#0d1013]/40 p-4">+ <div className="flex items-start justify-between gap-3">+ <div className="min-w-0">+ <h3 className="truncate text-lg font-semibold text-white">{container.name}</h3>+ <div className="truncate text-xs text-[#7f8589]">{container.image || 'unknown image'}</div>+ </div>+ <button type="button" onClick={onClose} className="shrink-0 rounded-xl border border-white/10 px-3 py-1.5 text-xs text-[#9aa0a6] hover:text-white">Close</button>+ </div>+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}+ {current && (+ <>+ <div className="grid grid-cols-2 gap-3 lg:grid-cols-4">+ <Figure label="Average CPU" value={cpu.average == null ? '–' : `${cpu.average.toFixed(1)}%`} />+ <Figure label="Peak memory" value={memory.maximum == null ? '–' : `${Math.round(memory.maximum)} MB`} />+ <Figure label="Not running" value={figures.notRunning == null ? '–' : `${figures.notRunning.toFixed(1)}% of the time`} />+ <Figure label="Restarts in range" value={figures.restarts == null ? '–' : figures.restarts} />+ </div>+ <p className="text-xs text-[#7f8589]">{current.points.length ? `One point per ${describeBucket(current.bucket_seconds)}. A container only reports CPU and memory while it is running.` : 'Nothing recorded for this container in this range.'}</p>+ <div className="grid gap-4 xl:grid-cols-2">+ <TimeSeriesChart {...common} title="CPU usage" series={series.cpu} color="#34d399" />+ <TimeSeriesChart {...common} title="Memory used" series={series.memory} color="#a78bfa" unit="MB" decimals={0} limit={limit} />+ </div>+ </>+ )}+ </section>+ )+}++export default function ContainerStatistics({ request, serverId, containers, range, selectedId, onSelect }) {+ const [query, setQuery] = useState('')+ const [sort, setSort] = useState('problems')+ const shown = useMemo(() => sortContainers(containers.filter((container) => container.name.toLowerCase().includes(query.trim().toLowerCase())), sort), [containers, query, sort])+ const selected = containers.find((container) => container.container_id === selectedId) || null+ const problems = containers.filter((container) => problemRank(container) < 2).length++ return (+ <section aria-label="Containers" className="space-y-4">+ <div className="flex flex-wrap items-end justify-between gap-3">+ <div>+ <h2 className="text-xl font-semibold text-white">Containers</h2>+ <p className="text-xs text-[#7f8589]">{containers.length} on this server{problems ? ` · ${problems} need attention` : ''}. Select one for its statistics.</p>+ </div>+ {containers.length > 0 && (+ <div className="flex flex-wrap gap-2">+ <input type="search" aria-label="Search containers" placeholder="Search" value={query} onChange={(event) => setQuery(event.target.value)} className="min-h-11 w-40 rounded-xl border border-white/10 bg-[#111417] px-3 text-sm text-white sm:min-h-0 sm:py-2" />+ <select aria-label="Sort containers" value={sort} onChange={(event) => setSort(event.target.value)} className="min-h-11 rounded-xl border border-white/10 bg-[#111417] px-3 text-sm text-white sm:min-h-0 sm:py-2">+ {SORTS.map((option) => <option key={option.id} value={option.id}>{option.label}</option>)}+ </select>+ </div>+ )}+ </div>+ {containers.length === 0 && <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-6 text-sm text-[#9aa0a6]">This server has no containers (or its collector has no Docker access).</div>}+ {containers.length > 0 && shown.length === 0 && <div className="text-sm text-[#7f8589]">No container matches “{query}”.</div>}+ <div className="grid grid-cols-1 gap-3 sm:grid-cols-2 xl:grid-cols-3">+ {shown.map((container) => <ContainerCard key={container.container_id} container={container} selected={container.container_id === selectedId} onSelect={() => onSelect(container.container_id === selectedId ? '' : container.container_id)} />)}+ </div>+ {selected && <ContainerDetail request={request} serverId={serverId} container={selected} range={range} onClose={() => onSelect('')} />}+ </section>+ )+}diff --git a/src/frontend/src/FleetStatistics.jsx b/src/frontend/src/FleetStatistics.jsxnew file mode 100644index 0000000..283060f--- /dev/null+++ b/src/frontend/src/FleetStatistics.jsx@@ -0,0 +1,95 @@+// The "All servers" view of the Statistics page: how the whole fleet is doing at a glance.+import { useEffect, useMemo, useState } from 'react'+import { BarList, ColumnChart, Donut } from './distributions.jsx'+import { availabilityRows, bucketLabel, containerStateSlices, formatDuration, incidentColumns, isOnline, serversByDisk, topContainers } from './fleetLogic.js'++const LIVE_RANGES = new Set(['1h', '6h', '24h'])+const REFRESH_SECONDS = 30+const SEVERITIES = [+ { key: 'critical', label: 'Critical', color: '#f87171' },+ { key: 'warning', label: 'Warning', color: '#fbbf24' },+ { key: 'info', label: 'Info', color: '#7dd3fc' },+]++function Figure({ label, value, detail }) {+ return (+ <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3">+ <div className="text-xs uppercase tracking-[0.18em] text-[#9aa0a6]">{label}</div>+ <div className="mt-1 text-2xl font-semibold text-white">{value}</div>+ {detail && <div className="text-xs text-[#7f8589]">{detail}</div>}+ </div>+ )+}++export default function FleetStatistics({ request, servers, range }) {+ const [data, setData] = useState(null)+ const [error, setError] = useState('')++ useEffect(() => {+ const controller = new AbortController()+ const load = () => request(`/stats/fleet?range=${range}`, { method: 'GET', signal: controller.signal })+ .then((result) => { setData(result); setError('') })+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load the fleet statistics.') })+ load()+ if (!LIVE_RANGES.has(range)) return () => controller.abort()+ const timer = setInterval(() => { if (!document.hidden) load() }, REFRESH_SECONDS * 1000)+ return () => { controller.abort(); clearInterval(timer) }+ }, [request, range])++ const incidents = data?.incidents+ const current = data && data.incidents.range === range ? data : null+ const online = servers.filter((server) => isOnline(server)).length+ const containerSlices = useMemo(() => containerStateSlices(servers), [servers])+ const containerTotal = containerSlices.reduce((sum, slice) => sum + slice.value, 0)+ const running = containerSlices.find((slice) => slice.label === 'Running')?.value ?? 0+ const columns = useMemo(() => (current ? incidentColumns(current.incidents.by_bucket, current.incidents.bucket, current.incidents.start, current.incidents.end) : []), [current])+ const categories = columns.map((column) => ({ label: bucketLabel(column.time, current.incidents.bucket), values: column.values }))+ const openNow = incidents ? Object.values(incidents.open_now).reduce((sum, count) => sum + count, 0) : 0++ return (+ <div className="space-y-5">+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}+ <div className="grid grid-cols-2 gap-3 lg:grid-cols-4">+ <Figure label="Servers online" value={`${online} of ${servers.length}`} />+ <Figure label="Containers running" value={`${running} of ${containerTotal}`} />+ <Figure label="Open incidents" value={incidents ? openNow : '–'} />+ <Figure label="Mean time to resolve" value={current ? formatDuration(current.incidents.mean_minutes_to_resolve) : '–'} detail={current ? `${current.incidents.resolved} resolved in this range` : undefined} />+ </div>++ <div className="grid gap-4 lg:grid-cols-2">+ <Donut title="Containers by state" subtitle="Right now, across all servers" slices={containerSlices} centreLabel="containers" empty="No containers are reported yet." />+ <Donut title="Open incidents by severity" subtitle="Right now" slices={SEVERITIES.map((severity) => ({ label: severity.label, color: severity.color, value: incidents?.open_now[severity.key] ?? 0 }))} centreLabel="open" empty="Nothing is open. All quiet." />+ </div>++ {current && (+ <ColumnChart+ title="Incidents opened"+ subtitle={`${current.incidents.opened} in this range, per ${current.incidents.bucket}${current.incidents.truncated ? ' (only the first part could be counted)' : ''}`}+ categories={categories}+ series={SEVERITIES}+ labelEvery={Math.max(1, Math.ceil(categories.length / 8))}+ empty="No incidents opened in this range."+ />+ )}++ <div className="grid gap-4 lg:grid-cols-2">+ {current && <BarList title="What the incidents were about" subtitle="Opened in this range" rows={current.incidents.by_kind.slice(0, 10).map((item) => ({ label: item.kind, value: item.count }))} color="#a78bfa" empty="No incidents opened in this range." />}+ <BarList title="Servers by disk usage" subtitle="Root disk, right now" rows={serversByDisk(servers)} max={100} unit="%" decimals={0} />+ <BarList title="Containers using the most memory" subtitle="Right now" rows={topContainers(servers, 'memory_used_mb')} unit=" MB" color="#a78bfa" />+ <BarList title="Containers using the most CPU" subtitle="Right now" rows={topContainers(servers, 'cpu_pct')} unit="%" decimals={1} color="#34d399" />+ </div>++ {current && (+ <BarList+ title="Availability"+ subtitle="Share of the expected check-ins that arrived. Approximate: it assumes each server's current check interval."+ rows={availabilityRows(current.availability)}+ max={100}+ unit="%"+ decimals={2}+ empty="No server has reported in this range."+ />+ )}+ </div>+ )+}diff --git a/src/frontend/src/ServerInsights.jsx b/src/frontend/src/ServerInsights.jsxnew file mode 100644index 0000000..2fe9230--- /dev/null+++ b/src/frontend/src/ServerInsights.jsx@@ -0,0 +1,100 @@+// The extra views of one server: how its load is distributed, when it is busy, how its disks are filling up.+import { useEffect, useMemo, useState } from 'react'+import { TimeSeriesChart } from './charts.jsx'+import { daysUntilFull, formatForecast, histogramBins, seriesFor } from './chartMath.js'+import { ColumnChart, Heatmap } from './distributions.jsx'++const LIVE_RANGES = new Set(['1h', '6h', '24h'])+const REFRESH_SECONDS = 30++// How much of the time a metric spent in each 10% band: weighted by the check-ins behind every point.+function shareOfTime(series) {+ const bins = histogramBins(series.filter((point) => point.avg !== null).map((point) => ({ value: point.avg, weight: point.samples })))+ const total = bins.reduce((sum, bin) => sum + bin.count, 0)+ return bins.map((bin) => ({ label: `${bin.from}–${bin.to}`, values: { share: total ? Math.round((1000 * bin.count) / total) / 10 : 0 } }))+}++export function Distributions({ cpu, memory }) {+ const share = [{ key: 'share', label: 'Share of the time', color: '#7dd3fc' }]+ const memoryShare = [{ key: 'share', label: 'Share of the time', color: '#a78bfa' }]+ return (+ <div className="grid gap-4 xl:grid-cols-2">+ <ColumnChart title="CPU usage distribution" subtitle="Share of the time the average CPU was in each band (%)" categories={shareOfTime(cpu)} series={share} unit="%" empty="Not enough data for this range." />+ <ColumnChart title="Memory usage distribution" subtitle="Share of the time the memory in use was in each band (%)" categories={shareOfTime(memory)} series={memoryShare} unit="%" empty="Not enough data for this range." />+ </div>+ )+}++export function BusyHours({ request, serverId }) {+ const [data, setData] = useState(null)+ const [metric, setMetric] = useState('cpu')+ const [error, setError] = useState('')+ const offset = -new Date().getTimezoneOffset()++ useEffect(() => {+ const controller = new AbortController()+ request(`/stats/servers/${serverId}/heatmap?metric=${metric}&offset=${offset}`, { method: 'GET', signal: controller.signal })+ .then((result) => { setData(result); setError('') })+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load the busy hours.') })+ return () => controller.abort()+ }, [request, serverId, metric, offset])++ const current = data && data.server_id === serverId && data.metric === metric ? data : null+ return (+ <div className="space-y-2">+ <div className="flex items-center justify-between gap-3">+ <h2 className="text-xl font-semibold text-white">When it is busy</h2>+ <div role="group" aria-label="Metric" className="flex gap-1.5">+ {[['cpu', 'CPU'], ['ram', 'Memory']].map(([id, label]) => (+ <button key={id} type="button" aria-pressed={metric === id} onClick={() => setMetric(id)} className={`min-h-9 rounded-xl border px-3 text-sm ${metric === id ? 'border-white/30 bg-white/10 text-white' : 'border-white/10 text-[#9aa0a6] hover:text-white'}`}>{label}</button>+ ))}+ </div>+ </div>+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}+ {current && <Heatmap title={`Average ${metric === 'cpu' ? 'CPU' : 'memory'} by hour of the day`} subtitle="The last 30 days, in your time zone, whatever range is selected above." grid={current.cells} color={metric === 'cpu' ? '#7dd3fc' : '#a78bfa'} empty="Not enough history yet. This fills in as days pass." />}+ </div>+ )+}++function MountChart({ request, serverId, mount, range }) {+ const [data, setData] = useState(null)+ const [error, setError] = useState('')++ useEffect(() => {+ const controller = new AbortController()+ const load = () => request(`/monitoring/servers/${serverId}/mounts/history?mount=${encodeURIComponent(mount.mount)}&range=${range}&metrics=used_pct,used_gb,total_gb`, { method: 'GET', signal: controller.signal })+ .then((result) => { setData(result); setError('') })+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load this filesystem.') })+ load()+ if (!LIVE_RANGES.has(range)) return () => controller.abort()+ const timer = setInterval(() => { if (!document.hidden) load() }, REFRESH_SECONDS * 1000)+ return () => { controller.abort(); clearInterval(timer) }+ }, [request, serverId, mount.mount, range])++ const current = data && data.mount === mount.mount ? data : null+ const series = useMemo(() => seriesFor(current?.points, 'used_pct'), [current])+ const forecast = useMemo(() => daysUntilFull(series.filter((point) => point.avg !== null).map((point) => ({ t: point.t, value: point.avg }))), [series])+ const common = { bucketSeconds: current?.bucket_seconds ?? 60, start: current ? new Date(current.start).getTime() : undefined, end: current ? new Date(current.end).getTime() : undefined, utc: current ? ['day', 'week', 'month'].includes(current.source) : false }+ return (+ <div className="min-w-0 space-y-2">+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}+ <TimeSeriesChart {...common} title={`Disk usage · ${mount.mount}`} series={series} color="#f472b6" domain={[0, 100]} />+ <p className="px-1 text-xs text-[#9aa0a6]">+ {mount.used_gb != null && mount.total_gb != null ? `${mount.used_gb.toFixed(1)} of ${mount.total_gb.toFixed(1)} GB used now. ` : ''}+ {current ? formatForecast(forecast) : ''}+ </p>+ </div>+ )+}++export function Storage({ request, serverId, mounts, range }) {+ if (!mounts.length) return null+ return (+ <div className="space-y-3">+ <h2 className="text-xl font-semibold text-white">Storage</h2>+ <div className="grid gap-4 xl:grid-cols-2">+ {mounts.map((mount) => <MountChart key={mount.mount} request={request} serverId={serverId} mount={mount} range={range} />)}+ </div>+ </div>+ )+}diff --git a/src/frontend/src/StatisticsPage.jsx b/src/frontend/src/StatisticsPage.jsxnew file mode 100644index 0000000..531c669--- /dev/null+++ b/src/frontend/src/StatisticsPage.jsx@@ -0,0 +1,211 @@+// The Statistics page: pick a server and a time range and see how it behaved, from the last hour to all time.+// It only needs a way to call the API, so it has no import from App.jsx.+import { useCallback, useEffect, useMemo, useState } from 'react'+import { useSearchParams } from 'react-router-dom'+import { TimeSeriesChart } from './charts.jsx'+import ContainerStatistics from './ContainerStatistics.jsx'+import FleetStatistics from './FleetStatistics.jsx'+import { BusyHours, Distributions, Storage } from './ServerInsights.jsx'+import { describeBucket, formatDateTime, seriesFor, summarise } from './chartMath.js'++const RANGES = [+ { id: '1h', label: '1 hour' },+ { id: '6h', label: '6 hours' },+ { id: '24h', label: '24 hours' },+ { id: '7d', label: '7 days' },+ { id: '30d', label: '30 days' },+ { id: '90d', label: '90 days' },+ { id: '1y', label: '1 year' },+ { id: 'all', label: 'All time' },+]+const RANGE_IDS = new Set(RANGES.map((range) => range.id))+const DEFAULT_RANGE = '24h'+const RANGE_STORAGE_KEY = 'homelab.statisticsRange'+// Only short ranges change from one minute to the next.+const LIVE_RANGES = new Set(['1h', '6h', '24h'])+const REFRESH_SECONDS = 30+const METRICS = 'cpu,ram,swap,load,disk,docker_ms,collect_ms'+const SOURCES = {+ detailed: 'check-ins grouped together',+ hour: 'hourly summaries',+ day: 'daily summaries',+ week: 'weekly summaries (Monday to Sunday, UTC)',+ month: 'monthly summaries (UTC)',+}++function rememberedRange() {+ try {+ const stored = window.localStorage.getItem(RANGE_STORAGE_KEY)+ return RANGE_IDS.has(stored) ? stored : DEFAULT_RANGE+ } catch {+ return DEFAULT_RANGE+ }+}++function SummaryCard({ label, value, detail }) {+ return (+ <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3">+ <div className="text-xs uppercase tracking-[0.18em] text-[#9aa0a6]">{label}</div>+ <div className="mt-1 text-2xl font-semibold text-white">{value}</div>+ {detail && <div className="text-xs text-[#7f8589]">{detail}</div>}+ </div>+ )+}++const percent = (value) => (value === null || value === undefined ? '–' : `${value.toFixed(1)}%`)++export default function StatisticsContent({ request }) {+ const [searchParams, setSearchParams] = useSearchParams()+ const [servers, setServers] = useState(null)+ const [data, setData] = useState(null)+ const [error, setError] = useState('')+ const [loading, setLoading] = useState(false)++ const range = RANGE_IDS.has(searchParams.get('range')) ? searchParams.get('range') : rememberedRange()+ const requestedServer = Number(searchParams.get('server'))+ const wantsAll = searchParams.get('server') === 'all'+ const serverId = useMemo(() => {+ if (!servers?.length || wantsAll) return null+ return servers.find((server) => server.id === requestedServer)?.id ?? (servers.find((server) => server.is_dashboard_host) || servers[0]).id+ }, [servers, requestedServer, wantsAll])++ const choose = useCallback((changes) => {+ const next = new URLSearchParams(searchParams)+ for (const [key, value] of Object.entries(changes)) {+ if (value === '' || value === null) next.delete(key)+ else next.set(key, String(value))+ }+ setSearchParams(next, { replace: true })+ }, [searchParams, setSearchParams])++ useEffect(() => {+ let cancelled = false+ const load = () => request('/monitoring/overview', { method: 'GET' })+ .then((overview) => { if (!cancelled) setServers((overview.servers || []).map((server) => ({ ...server, is_dashboard_host: Boolean(server.is_dashboard_host), containers: server.containers || [] }))) })+ .catch((requestError) => { if (!cancelled) { setServers((current) => current || []); setError(requestError.message || 'Could not load the servers.') } })+ load()+ // The containers on the cards change, so keep them current, but not while the tab is hidden.+ const timer = setInterval(() => { if (!document.hidden) load() }, REFRESH_SECONDS * 1000)+ return () => { cancelled = true; clearInterval(timer) }+ }, [request])++ useEffect(() => {+ if (serverId === null) return undefined+ const controller = new AbortController()+ const load = (showSpinner) => {+ if (showSpinner) setLoading(true)+ request(`/monitoring/servers/${serverId}/history?range=${range}&metrics=${METRICS}`, { method: 'GET', signal: controller.signal })+ .then((result) => { setData(result); setError('') })+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load the statistics.') })+ .finally(() => { if (!controller.signal.aborted) setLoading(false) })+ }+ load(true)+ if (!LIVE_RANGES.has(range)) return () => controller.abort()+ // Refresh the short ranges, but not while the tab is hidden.+ const timer = setInterval(() => { if (!document.hidden) load(false) }, REFRESH_SECONDS * 1000)+ return () => { controller.abort(); clearInterval(timer) }+ }, [request, serverId, range])++ const series = useMemo(() => Object.fromEntries(['cpu', 'ram', 'swap', 'load', 'disk', 'docker_ms', 'collect_ms'].map((metric) => [metric, seriesFor(data?.points, metric)])), [data])+ const cpu = summarise(series.cpu)+ const memory = summarise(series.ram)+ const start = data ? new Date(data.start).getTime() : undefined+ const end = data ? new Date(data.end).getTime() : undefined+ const utc = data ? ['day', 'week', 'month'].includes(data.source) : false+ const common = { bucketSeconds: data?.bucket_seconds ?? 60, start, end, utc }+ const ready = data && data.server_id === serverId++ return (+ <div className="space-y-5">+ <div>+ <div className="text-xs uppercase tracking-[0.22em] text-[#9aa0a6]">History</div>+ <h1 className="mt-1 text-3xl font-semibold text-white">Statistics</h1>+ <p className="mt-2 text-sm text-[#9aa0a6]">How a server behaved over time, from the last hour to everything that has been recorded.</p>+ </div>++ <div className="flex flex-wrap items-end gap-4">+ <label className="flex flex-col gap-1 text-xs text-[#9aa0a6]">+ Server+ <select+ value={wantsAll ? 'all' : serverId ?? ''}+ onChange={(event) => choose({ server: event.target.value, container: '' })}+ disabled={!servers?.length}+ className="min-h-11 rounded-xl border border-white/10 bg-[#111417] px-3 py-2 text-sm text-white sm:min-h-0"+ >+ {servers?.length > 0 && <option value="all">All servers</option>}+ {(servers || []).map((server) => <option key={server.id} value={server.id}>{server.hostname}{server.is_dashboard_host ? ' (this machine)' : ''}</option>)}+ </select>+ </label>+ <div role="group" aria-label="Time range" className="flex flex-wrap gap-1.5">+ {RANGES.map((option) => (+ <button+ key={option.id}+ type="button"+ aria-pressed={range === option.id}+ onClick={() => {+ try { window.localStorage.setItem(RANGE_STORAGE_KEY, option.id) } catch { /* the choice just isn't remembered */ }+ choose({ range: option.id })+ }}+ className={`min-h-11 rounded-xl border px-3 text-sm transition sm:min-h-9 ${range === option.id ? 'border-white/30 bg-white/10 text-white' : 'border-white/10 text-[#9aa0a6] hover:text-white'}`}+ >+ {option.label}+ </button>+ ))}+ </div>+ </div>++ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}+ {servers !== null && servers.length === 0 && !error && <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-6 text-sm text-[#9aa0a6]">No servers are reporting yet. Add one from the dashboard and its history will appear here.</div>}++ {servers?.length > 0 && wantsAll && <FleetStatistics request={request} servers={servers} range={range} />}++ {ready && (+ <div className={loading ? 'space-y-5 opacity-60 transition-opacity' : 'space-y-5 transition-opacity'}>+ <div className="grid grid-cols-2 gap-3 lg:grid-cols-4">+ <SummaryCard label="Average CPU" value={percent(cpu.average)} />+ <SummaryCard label="Peak CPU" value={percent(cpu.maximum)} />+ <SummaryCard label="Average memory" value={percent(memory.average)} />+ <SummaryCard label="Peak memory" value={percent(memory.maximum)} />+ </div>+ <p className="text-xs text-[#7f8589]">+ {data.points.length+ ? `${data.points.length} ${data.points.length === 1 ? 'point' : 'points'}, one per ${describeBucket(data.bucket_seconds)} (${SOURCES[data.source] || data.source}). The line is the average and the lighter band reaches up to the highest value. `+ : 'Nothing has been recorded for this server in this range. '}+ {data.data_since && `History starts ${formatDateTime(new Date(data.data_since).getTime(), utc)}. `}+ {utc ? 'Dates are UTC.' : 'Times are in your time zone.'}+ </p>+ <div className="grid gap-4 xl:grid-cols-2">+ <TimeSeriesChart {...common} title="CPU usage" series={series.cpu} color="#7dd3fc" domain={[0, 100]} />+ <TimeSeriesChart {...common} title="Memory used" series={series.ram} color="#a78bfa" domain={[0, 100]} />+ <TimeSeriesChart {...common} title="Swap used" series={series.swap} color="#fbbf24" domain={[0, 100]} />+ <TimeSeriesChart {...common} title="Load (1 minute average)" series={series.load} color="#34d399" unit="" decimals={2} />+ <TimeSeriesChart {...common} title="Root disk used" series={series.disk} color="#f472b6" domain={[0, 100]} />+ </div>+ <Distributions cpu={series.cpu} memory={series.ram} />+ <BusyHours request={request} serverId={serverId} />+ <Storage request={request} serverId={serverId} mounts={servers.find((server) => server.id === serverId)?.mounts || []} range={range} />+ {(series.docker_ms.length > 0 || series.collect_ms.length > 0) && (+ <div className="space-y-3">+ <h2 className="text-xl font-semibold text-white">Reliability</h2>+ <div className="grid gap-4 xl:grid-cols-2">+ <TimeSeriesChart {...common} title="Docker response time (listing containers)" series={series.docker_ms} color="#fb923c" unit="ms" decimals={0} />+ <TimeSeriesChart {...common} title="Time the collector takes per check-in" series={series.collect_ms} color="#94a3b8" unit="ms" decimals={0} />+ </div>+ </div>+ )}+ </div>+ )}+ {servers?.length > 0 && serverId !== null && !wantsAll && (+ <ContainerStatistics+ request={request}+ serverId={serverId}+ containers={servers.find((server) => server.id === serverId)?.containers || []}+ range={range}+ selectedId={searchParams.get('container') || ''}+ onSelect={(containerId) => choose({ container: containerId })}+ />+ )}+ {servers === null && !error && <div className="text-sm text-[#7f8589]">Loading…</div>}+ </div>+ )+}diff --git a/src/frontend/src/chartMath.js b/src/frontend/src/chartMath.jsnew file mode 100644index 0000000..95446ff--- /dev/null+++ b/src/frontend/src/chartMath.js@@ -0,0 +1,163 @@+// Pure helpers for the charts: ticks, labels, gaps and summary figures. No React in here.++const pad = (value) => String(value).padStart(2, '0')++export function niceTicks(low, high, wanted = 5) {+ if (!Number.isFinite(low) || !Number.isFinite(high)) return [0, 1]+ if (high <= low) high = low + 1+ const rough = (high - low) / Math.max(wanted, 1)+ const magnitude = 10 ** Math.floor(Math.log10(rough))+ const step = [1, 2, 2.5, 5, 10].map((factor) => factor * magnitude).find((candidate) => candidate >= rough) || 10 * magnitude+ const ticks = []+ // The last tick is at or above the top of the data, so nothing is clipped against the edge.+ const last = Math.ceil(high / step - 1e-9) * step+ for (let value = Math.ceil(low / step) * step; value <= last + step * 1e-9; value += step) ticks.push(Number(value.toFixed(10)))+ return ticks+}++// Day, week and month points are UTC-aligned, so they are labelled in UTC; shorter ranges use the viewer's clock.+export function formatTime(milliseconds, spanSeconds, utc) {+ const date = new Date(milliseconds)+ const get = utc+ ? { year: date.getUTCFullYear(), month: date.getUTCMonth(), day: date.getUTCDate(), hour: date.getUTCHours(), minute: date.getUTCMinutes() }+ : { year: date.getFullYear(), month: date.getMonth(), day: date.getDate(), hour: date.getHours(), minute: date.getMinutes() }+ const month = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'][get.month]+ if (spanSeconds <= 18 * 3600) return `${pad(get.hour)}:${pad(get.minute)}`+ if (spanSeconds <= 14 * 86400) return `${get.day} ${month} ${pad(get.hour)}:${pad(get.minute)}`+ if (spanSeconds <= 400 * 86400) return `${get.day} ${month}`+ return `${month} ${get.year}`+}++export function formatDateTime(milliseconds, utc) {+ const date = new Date(milliseconds)+ const pick = (local, universal) => (utc ? universal : local)+ const month = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'][pick(date.getMonth(), date.getUTCMonth())]+ const day = `${pick(date.getDate(), date.getUTCDate())} ${month} ${pick(date.getFullYear(), date.getUTCFullYear())}`+ return utc ? day : `${day} ${pad(date.getHours())}:${pad(date.getMinutes())}`+}++export function describeBucket(seconds) {+ if (seconds >= 28 * 86400) return 'month'+ if (seconds >= 7 * 86400) return 'week'+ if (seconds >= 86400) return seconds === 86400 ? 'day' : `${Math.round(seconds / 86400)} days`+ if (seconds >= 3600) return seconds === 3600 ? 'hour' : `${Math.round(seconds / 3600)} hours`+ if (seconds >= 60) return seconds === 60 ? 'minute' : `${Math.round(seconds / 60)} minutes`+ return `${seconds} seconds`+}++// Points become runs of neighbours; a hole bigger than a bucket and a half starts a new run, so a gap stays a gap.+export function splitRuns(points, bucketSeconds) {+ const runs = []+ let current = []+ let previous = null+ for (const point of points) {+ if (previous !== null && point.t - previous > bucketSeconds * 1500) {+ runs.push(current)+ current = []+ }+ current.push(point)+ previous = point.t+ }+ if (current.length) runs.push(current)+ return runs+}++// Turns the API's points into {t, avg, max, samples} for one metric, skipping points where it is unknown.+export function seriesFor(points, metric) {+ return (points || [])+ .map((point) => ({ t: new Date(point.time).getTime(), avg: point[metric]?.avg ?? null, max: point[metric]?.max ?? null, samples: point.samples }))+ .filter((point) => Number.isFinite(point.t) && (point.avg !== null || point.max !== null))+}++// Weighted average and highest maximum of a series, for the summary figures.+export function summarise(series) {+ let total = 0+ let weight = 0+ let maximum = null+ for (const point of series) {+ if (point.avg !== null) { total += point.avg * point.samples; weight += point.samples }+ if (point.max !== null && (maximum === null || point.max > maximum)) maximum = point.max+ }+ return { average: weight ? total / weight : null, maximum }+}+++// Counts of values per bin, between ``low`` and ``high``; each value may carry a weight (for example check-ins).+// A value equal to ``high`` goes in the last bin, and values outside the range are left out.+export function histogramBins(values, { low = 0, high = 100, bins = 10 } = {}) {+ const width = (high - low) / bins+ const counts = Array.from({ length: bins }, (_, index) => ({ from: low + index * width, to: low + (index + 1) * width, count: 0 }))+ for (const entry of values) {+ const value = typeof entry === 'number' ? entry : entry.value+ const weight = typeof entry === 'number' ? 1 : entry.weight ?? 1+ if (!Number.isFinite(value) || value < low || value > high) continue+ counts[Math.min(Math.floor((value - low) / width), bins - 1)].count += weight+ }+ return counts+}++// The average of a metric for each hour of the day (0-23) and weekday (0 = Monday ... 6 = Sunday), in the viewer's+// time zone or in UTC. Cells nobody reported stay null. Points are {t, avg, samples}.+export function hourWeekdayGrid(points, utc = false) {+ const sums = Array.from({ length: 7 }, () => Array.from({ length: 24 }, () => ({ total: 0, weight: 0 })))+ for (const point of points) {+ if (point.avg === null || point.avg === undefined) continue+ const date = new Date(point.t)+ const weekday = ((utc ? date.getUTCDay() : date.getDay()) + 6) % 7+ const cell = sums[weekday][utc ? date.getUTCHours() : date.getHours()]+ cell.total += point.avg * point.samples+ cell.weight += point.samples+ }+ return sums.map((row) => row.map((cell) => (cell.weight ? cell.total / cell.weight : null)))+}++// Slices as fractions of their total, for a donut; zero and invalid values are dropped.+export function donutSlices(slices) {+ const valid = slices.filter((slice) => Number.isFinite(slice.value) && slice.value > 0)+ const total = valid.reduce((sum, slice) => sum + slice.value, 0)+ let start = 0+ return valid.map((slice) => {+ const fraction = slice.value / total+ const result = { ...slice, fraction, start }+ start += fraction+ return result+ })+}++// Days until a straight-line trend through (time, value) points reaches ``limit``, or null when the trend is flat,+// falling, too short or too noisy to say. Times are milliseconds.+export function daysUntilFull(points, limit = 100, { minPoints = 6, minSpanDays = 1, minFit = 0.5 } = {}) {+ const usable = points.filter((point) => Number.isFinite(point.t) && Number.isFinite(point.value))+ if (usable.length < minPoints) return null+ const spanDays = (usable[usable.length - 1].t - usable[0].t) / 86400000+ if (spanDays < minSpanDays) return null+ const n = usable.length+ const meanT = usable.reduce((sum, point) => sum + point.t, 0) / n+ const meanV = usable.reduce((sum, point) => sum + point.value, 0) / n+ let covariance = 0+ let varianceT = 0+ let varianceV = 0+ for (const point of usable) {+ covariance += (point.t - meanT) * (point.value - meanV)+ varianceT += (point.t - meanT) ** 2+ varianceV += (point.value - meanV) ** 2+ }+ if (varianceT === 0 || varianceV === 0) return null+ const slope = covariance / varianceT // per millisecond+ const fit = (covariance * covariance) / (varianceT * varianceV) // r squared+ if (slope <= 0 || fit < minFit) return null+ const current = usable[usable.length - 1].value+ if (current >= limit) return 0+ return (limit - current) / slope / 86400000+}++// "Full in about 3 weeks" for a forecast in days, or an honest "no clear trend" when there is none.+export function formatForecast(days) {+ if (days === null || days === undefined || !Number.isFinite(days)) return 'No clear growth trend in this range, so no estimate.'+ if (days <= 0) return 'Full now.'+ if (days < 1) return 'Full in less than a day at this rate.'+ if (days < 60) return `Full in about ${Math.round(days)} ${Math.round(days) === 1 ? 'day' : 'days'} at this rate.`+ if (days < 730) return `Full in about ${Math.round(days / 30)} months at this rate.`+ if (days < 3650) return `Full in about ${(days / 365).toFixed(1)} years at this rate.`+ return 'Not full within ten years at this rate.'+}diff --git a/src/frontend/src/charts.jsx b/src/frontend/src/charts.jsxnew file mode 100644index 0000000..b395353--- /dev/null+++ b/src/frontend/src/charts.jsx@@ -0,0 +1,143 @@+// Chart components drawn with plain SVG: no charting library, so there is nothing extra to trust or update, and+// every label from the server is rendered as text by React, never as HTML.+import { useEffect, useMemo, useRef, useState } from 'react'+import { formatDateTime, formatTime, niceTicks, splitRuns, summarise } from './chartMath.js'++const MARGIN = { top: 10, right: 12, bottom: 24, left: 40 }++// The width of an element in pixels, kept up to date, so SVG text stays crisp at any screen size.+function useElementWidth(ref, fallback = 640) {+ const [width, setWidth] = useState(fallback)+ useEffect(() => {+ const element = ref.current+ if (!element) return undefined+ const update = () => setWidth(Math.max(Math.round(element.getBoundingClientRect().width), 200))+ update()+ if (typeof ResizeObserver === 'undefined') return undefined+ const observer = new ResizeObserver(update)+ observer.observe(element)+ return () => observer.disconnect()+ }, [ref])+ return width+}++const FIXED_DOMAIN_FALLBACK = [0, 1]++// An average line with a lighter band up to the maximum, over time. Hover or touch shows the exact figures; the+// whole chart is also available as a table.+export function TimeSeriesChart({ title, series, bucketSeconds, start, end, unit = '%', color = '#7dd3fc', domain = null, height = 200, utc = false, decimals = 1, limit = null }) {+ const wrapper = useRef(null)+ const width = useElementWidth(wrapper)+ const [hover, setHover] = useState(null)+ const [asTable, setAsTable] = useState(false)++ const layout = useMemo(() => {+ const tStart = start ?? (series[0]?.t ?? 0)+ const tEnd = Math.max(end ?? (series[series.length - 1]?.t ?? 1), tStart + 1)+ const values = series.flatMap((point) => [point.avg, point.max]).filter((value) => value !== null)+ if (limit !== null && !domain) values.push(limit / 1.1) // keep the limit line inside the chart+ const [fixedLow, fixedHigh] = domain || FIXED_DOMAIN_FALLBACK+ const yTicks = domain+ ? niceTicks(fixedLow, fixedHigh, 4)+ : niceTicks(0, values.length ? Math.max(...values) * 1.1 : 1, 4)+ const yMax = yTicks[yTicks.length - 1] || fixedHigh+ const plotWidth = Math.max(width - MARGIN.left - MARGIN.right, 50)+ const plotHeight = height - MARGIN.top - MARGIN.bottom+ const x = (t) => MARGIN.left + ((t - tStart) / (tEnd - tStart)) * plotWidth+ const y = (value) => MARGIN.top + plotHeight - (Math.min(Math.max(value, 0), yMax) / yMax) * plotHeight+ const spanSeconds = (tEnd - tStart) / 1000+ const tickCount = Math.max(2, Math.min(7, Math.floor(plotWidth / 110)))+ const xTicks = Array.from({ length: tickCount }, (_, index) => tStart + ((tEnd - tStart) * index) / (tickCount - 1))+ return { tStart, tEnd, yTicks, yMax, plotWidth, plotHeight, x, y, spanSeconds, xTicks }+ }, [series, start, end, width, height, domain, limit])++ const runs = useMemo(() => splitRuns(series, bucketSeconds), [series, bucketSeconds])+ const format = (value) => (value === null || value === undefined ? '–' : `${value.toFixed(decimals)}${unit === '%' ? '%' : ` ${unit}`.trimEnd()}`)++ if (!series.length) {+ return (+ <figure className="min-w-0 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 p-4">+ <figcaption className="text-sm font-medium text-white">{title}</figcaption>+ <div className="mt-3 flex items-center justify-center rounded-xl border border-dashed border-neutral-800 text-xs text-[#7f8589]" style={{ height }}>No data for this range yet</div>+ </figure>+ )+ }++ const move = (event) => {+ const box = wrapper.current?.getBoundingClientRect()+ if (!box) return+ const px = event.clientX - box.left+ let best = null+ for (const point of series) {+ const distance = Math.abs(layout.x(point.t) - px)+ if (best === null || distance < best.distance) best = { point, distance }+ }+ setHover(best && best.distance < 60 ? best.point : null)+ }+ const label = `${title}: average ${format(summarise(series).average)}, highest ${format(summarise(series).maximum)}`++ return (+ <figure className="min-w-0 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 p-4">+ <div className="flex items-center justify-between gap-3">+ <figcaption className="text-sm font-medium text-white">{title}</figcaption>+ <button type="button" onClick={() => setAsTable((value) => !value)} className="rounded-lg border border-white/10 px-2 py-1 text-[11px] text-[#9aa0a6] hover:text-white">{asTable ? 'View as chart' : 'View as table'}</button>+ </div>+ {asTable ? (+ <div className="mt-3 max-h-64 overflow-auto rounded-xl border border-[#1f2326]">+ <table className="w-full text-left text-xs text-[#bcbcbc]">+ <thead className="sticky top-0 bg-[#111417] text-[#9aa0a6]"><tr><th className="px-3 py-2 font-medium">Time</th><th className="px-3 py-2 font-medium">Average</th><th className="px-3 py-2 font-medium">Highest</th><th className="px-3 py-2 font-medium">Check-ins</th></tr></thead>+ <tbody>{series.map((point) => <tr key={point.t} className="border-t border-[#1f2326]"><td className="px-3 py-1.5">{formatDateTime(point.t, utc)}</td><td className="px-3 py-1.5">{format(point.avg)}</td><td className="px-3 py-1.5">{format(point.max)}</td><td className="px-3 py-1.5">{point.samples}</td></tr>)}</tbody>+ </table>+ </div>+ ) : (+ <div ref={wrapper} className="relative mt-2 w-full min-w-0 touch-pan-y select-none overflow-hidden" onPointerMove={move} onPointerLeave={() => setHover(null)} onPointerDown={move}>+ <svg width={width} height={height} role="img" aria-label={label} className="block">+ {layout.yTicks.map((tick) => (+ <g key={tick}>+ <line x1={MARGIN.left} x2={width - MARGIN.right} y1={layout.y(tick)} y2={layout.y(tick)} stroke="#ffffff" strokeOpacity="0.07" />+ <text x={MARGIN.left - 6} y={layout.y(tick) + 4} textAnchor="end" fontSize="10" fill="#7f8589">{tick}</text>+ </g>+ ))}+ {layout.xTicks.map((tick, index) => (+ <text key={tick} x={layout.x(tick)} y={height - 6} textAnchor={index === 0 ? 'start' : index === layout.xTicks.length - 1 ? 'end' : 'middle'} fontSize="10" fill="#7f8589">{formatTime(tick, layout.spanSeconds, utc)}</text>+ ))}+ {runs.map((run) => {+ const withMax = run.filter((point) => point.max !== null && point.avg !== null)+ const band = withMax.length > 1+ ? `${withMax.map((point) => `${layout.x(point.t)},${layout.y(point.max)}`).join(' ')} ${[...withMax].reverse().map((point) => `${layout.x(point.t)},${layout.y(point.avg)}`).join(' ')}`+ : null+ const line = run.filter((point) => point.avg !== null)+ return (+ <g key={run[0].t}>+ {band && <polygon points={band} fill={color} fillOpacity="0.22" />}+ {line.length > 1 && <polyline points={line.map((point) => `${layout.x(point.t)},${layout.y(point.avg)}`).join(' ')} fill="none" stroke={color} strokeWidth="2" strokeLinejoin="round" strokeLinecap="round" />}+ {line.length === 1 && <circle cx={layout.x(line[0].t)} cy={layout.y(line[0].avg)} r="3" fill={color} />}+ </g>+ )+ })}+ {limit !== null && (+ <g pointerEvents="none">+ <line x1={MARGIN.left} x2={width - MARGIN.right} y1={layout.y(limit)} y2={layout.y(limit)} stroke="#f87171" strokeOpacity="0.8" strokeDasharray="5 4" />+ <text x={width - MARGIN.right - 2} y={layout.y(limit) < MARGIN.top + 14 ? layout.y(limit) + 12 : layout.y(limit) - 4} textAnchor="end" fontSize="10" fill="#f87171">limit {limit.toFixed(0)} {unit}</text>+ </g>+ )}+ {hover && (+ <g pointerEvents="none">+ <line x1={layout.x(hover.t)} x2={layout.x(hover.t)} y1={MARGIN.top} y2={height - MARGIN.bottom} stroke="#ffffff" strokeOpacity="0.35" />+ {hover.avg !== null && <circle cx={layout.x(hover.t)} cy={layout.y(hover.avg)} r="4" fill={color} stroke="#0d1013" strokeWidth="2" />}+ </g>+ )}+ </svg>+ {hover && (+ <div role="status" className="pointer-events-none absolute z-10 rounded-xl border border-white/10 bg-[#111417]/95 px-3 py-2 text-xs text-[#e6e7ea] shadow-lg" style={{ top: 0, left: Math.min(Math.max(layout.x(hover.t) - 70, 0), Math.max(width - 150, 0)) }}>+ <div className="text-[#9aa0a6]">{formatDateTime(hover.t, utc)}</div>+ <div>Average <span className="font-medium text-white">{format(hover.avg)}</span></div>+ <div>Highest <span className="font-medium text-white">{format(hover.max)}</span></div>+ <div className="text-[#7f8589]">{hover.samples} check-ins</div>+ </div>+ )}+ </div>+ )}+ </figure>+ )+}diff --git a/src/frontend/src/containerLogic.js b/src/frontend/src/containerLogic.jsnew file mode 100644index 0000000..f32adc1--- /dev/null+++ b/src/frontend/src/containerLogic.js@@ -0,0 +1,15 @@+// Pure helpers for the container cards: which need attention, and how to order them.++// 0 = needs attention, 1 = running but unhealthy, 2 = fine.+export function problemRank(container) {+ if (['exited', 'dead', 'restarting', 'paused', 'created'].includes(container.status)) return 0+ return container.healthcheck_status === 'unhealthy' ? 1 : 2+}++export function sortContainers(containers, sort) {+ const byName = (a, b) => a.name.localeCompare(b.name)+ const descending = (key) => (a, b) => (b[key] ?? -1) - (a[key] ?? -1) || byName(a, b)+ const order = { name: byName, memory: descending('memory_used_mb'), cpu: descending('cpu_pct'), problems: (a, b) => problemRank(a) - problemRank(b) || byName(a, b) }+ return [...containers].sort(order[sort] || byName)+}+diff --git a/src/frontend/src/distributions.jsx b/src/frontend/src/distributions.jsxnew file mode 100644index 0000000..2b58552--- /dev/null+++ b/src/frontend/src/distributions.jsx@@ -0,0 +1,195 @@+// More chart types in plain SVG and HTML: stacked columns (also used for histograms), donut, ranked bars and a+// heatmap. Everything that comes from the server (names, labels) is rendered as text by React, never as HTML.+import { useEffect, useRef, useState } from 'react'+import { donutSlices } from './chartMath.js'++// The frame every chart sits in: a title, the "view as table" switch and the table itself.+function ChartFrame({ title, subtitle, table, children, empty }) {+ const [asTable, setAsTable] = useState(false)+ return (+ <figure className="min-w-0 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 p-4">+ <div className="flex items-start justify-between gap-3">+ <div className="min-w-0">+ <figcaption className="text-sm font-medium text-white">{title}</figcaption>+ {subtitle && <div className="text-xs text-[#7f8589]">{subtitle}</div>}+ </div>+ {!empty && table && <button type="button" onClick={() => setAsTable((value) => !value)} className="shrink-0 rounded-lg border border-white/10 px-2 py-1 text-[11px] text-[#9aa0a6] hover:text-white">{asTable ? 'View as chart' : 'View as table'}</button>}+ </div>+ {empty ? (+ <div className="mt-3 flex min-h-32 items-center justify-center rounded-xl border border-dashed border-neutral-800 px-3 text-center text-xs text-[#7f8589]">{empty}</div>+ ) : asTable && table ? (+ <div className="mt-3 max-h-64 overflow-auto rounded-xl border border-[#1f2326]">+ <table className="w-full text-left text-xs text-[#bcbcbc]">+ <thead className="sticky top-0 bg-[#111417] text-[#9aa0a6]"><tr>{table.head.map((cell) => <th key={cell} className="px-3 py-2 font-medium">{cell}</th>)}</tr></thead>+ <tbody>{table.rows.map((row, index) => <tr key={index} className="border-t border-[#1f2326]">{row.map((cell, cellIndex) => <td key={cellIndex} className="px-3 py-1.5">{cell}</td>)}</tr>)}</tbody>+ </table>+ </div>+ ) : <div className="mt-3">{children}</div>}+ </figure>+ )+}++function Legend({ items }) {+ return (+ <ul className="mt-2 flex flex-wrap gap-x-4 gap-y-1 text-xs text-[#bcbcbc]">+ {items.map((item) => <li key={item.label} className="flex items-center gap-1.5"><span aria-hidden="true" className="h-2.5 w-2.5 rounded-sm" style={{ background: item.color }} />{item.label}{item.value !== undefined && <span className="text-[#7f8589]">{item.value}</span>}</li>)}+ </ul>+ )+}++function useWidth(ref, fallback = 480) {+ const [width, setWidth] = useState(fallback)+ useEffect(() => {+ const element = ref.current+ if (!element) return undefined+ const update = () => setWidth(Math.max(Math.round(element.getBoundingClientRect().width), 160))+ update()+ if (typeof ResizeObserver === 'undefined') return undefined+ const observer = new ResizeObserver(update)+ observer.observe(element)+ return () => observer.disconnect()+ }, [ref])+ return width+}++// Columns over categories (days, bins, ...), optionally stacked. categories: [{label, values: {key: number}}].+export function ColumnChart({ title, subtitle, categories, series, height = 190, unit = '', empty = 'Nothing to show for this range yet', labelEvery = 1 }) {+ const wrapper = useRef(null)+ const width = useWidth(wrapper)+ const [hover, setHover] = useState(null)+ const margin = { top: 8, right: 8, bottom: 26, left: 34 }+ const totals = categories.map((category) => series.reduce((sum, item) => sum + (category.values[item.key] || 0), 0))+ const top = Math.max(1, ...totals)+ const niceTop = (() => { const magnitude = 10 ** Math.floor(Math.log10(top)); return [1, 2, 5, 10].map((f) => f * magnitude).find((value) => value >= top) })()+ const plotWidth = Math.max(width - margin.left - margin.right, 40)+ const plotHeight = height - margin.top - margin.bottom+ const slot = plotWidth / Math.max(categories.length, 1)+ const barWidth = Math.max(Math.min(slot * 0.72, 40), 1)+ const y = (value) => margin.top + plotHeight - (value / niceTop) * plotHeight+ const isEmpty = !categories.length || totals.every((total) => total === 0)+ const table = { head: ['', ...series.map((item) => item.label)], rows: categories.map((category) => [category.label, ...series.map((item) => category.values[item.key] || 0)]) }++ return (+ <ChartFrame title={title} subtitle={subtitle} table={table} empty={isEmpty ? empty : null}>+ <div ref={wrapper} className="relative w-full min-w-0 overflow-hidden" onPointerLeave={() => setHover(null)}>+ <svg width={width} height={height} role="img" aria-label={`${title}: ${categories.length} columns, highest ${top}${unit}`} className="block">+ {[0, niceTop / 2, niceTop].map((tick) => (+ <g key={tick}>+ <line x1={margin.left} x2={width - margin.right} y1={y(tick)} y2={y(tick)} stroke="#ffffff" strokeOpacity="0.07" />+ <text x={margin.left - 6} y={y(tick) + 4} textAnchor="end" fontSize="10" fill="#7f8589">{Number(tick.toFixed(2))}</text>+ </g>+ ))}+ {categories.map((category, index) => {+ const x = margin.left + slot * index + (slot - barWidth) / 2+ let base = 0+ return (+ <g key={category.label} onPointerEnter={() => setHover({ index, x: x + barWidth / 2 })} onPointerDown={() => setHover({ index, x: x + barWidth / 2 })}>+ <rect x={margin.left + slot * index} y={margin.top} width={slot} height={plotHeight} fill="transparent" />+ {series.map((item) => {+ const value = category.values[item.key] || 0+ if (!value) return null+ const rect = <rect key={item.key} x={x} y={y(base + value)} width={barWidth} height={Math.max(y(base) - y(base + value), 1)} fill={item.color} rx="2" opacity={hover && hover.index !== index ? 0.55 : 1} />+ base += value+ return rect+ })}+ {index % labelEvery === 0 && <text x={x + barWidth / 2} y={height - 8} textAnchor="middle" fontSize="10" fill="#7f8589">{category.label}</text>}+ </g>+ )+ })}+ </svg>+ {hover && (+ <div role="status" className="pointer-events-none absolute z-10 rounded-xl border border-white/10 bg-[#111417]/95 px-3 py-2 text-xs text-[#e6e7ea] shadow-lg" style={{ top: 0, left: Math.min(Math.max(hover.x - 60, 0), Math.max(width - 140, 0)) }}>+ <div className="text-[#9aa0a6]">{categories[hover.index].label}</div>+ {series.map((item) => <div key={item.key}><span aria-hidden="true" className="mr-1.5 inline-block h-2 w-2 rounded-sm" style={{ background: item.color }} />{item.label} <span className="font-medium text-white">{categories[hover.index].values[item.key] || 0}{unit}</span></div>)}+ </div>+ )}+ </div>+ {series.length > 1 && <Legend items={series} />}+ </ChartFrame>+ )+}++// A donut with the total in the middle and a legend that also says the numbers, so colour is never the only cue.+export function Donut({ title, subtitle, slices, centreLabel = 'total', empty = 'Nothing to show yet' }) {+ const parts = donutSlices(slices)+ const total = parts.reduce((sum, slice) => sum + slice.value, 0)+ const radius = 54+ const circumference = 2 * Math.PI * radius+ const table = { head: ['', 'Count', 'Share'], rows: parts.map((slice) => [slice.label, slice.value, `${(slice.fraction * 100).toFixed(1)}%`]) }+ return (+ <ChartFrame title={title} subtitle={subtitle} table={table} empty={total === 0 ? empty : null}>+ <div className="flex flex-wrap items-center gap-5">+ <svg width="140" height="140" viewBox="0 0 140 140" role="img" aria-label={`${title}: ${parts.map((slice) => `${slice.label} ${slice.value}`).join(', ')}`} className="shrink-0">+ <g transform="rotate(-90 70 70)">+ {parts.map((slice) => (+ <circle key={slice.label} cx="70" cy="70" r={radius} fill="none" stroke={slice.color} strokeWidth="18" strokeDasharray={`${Math.max(slice.fraction * circumference - (parts.length > 1 ? 1.5 : 0), 0.5)} ${circumference}`} strokeDashoffset={-slice.start * circumference} />+ ))}+ </g>+ <text x="70" y="68" textAnchor="middle" fontSize="24" fontWeight="600" fill="#ffffff">{total}</text>+ <text x="70" y="86" textAnchor="middle" fontSize="10" fill="#7f8589">{centreLabel}</text>+ </svg>+ <Legend items={parts.map((slice) => ({ label: slice.label, color: slice.color, value: `${slice.value}` }))} />+ </div>+ </ChartFrame>+ )+}++// Ranked horizontal bars (top containers, servers by disk usage ...). rows: [{label, value, detail?}].+export function BarList({ title, subtitle, rows, max = null, unit = '', color = '#7dd3fc', decimals = 0, empty = 'Nothing to show yet' }) {+ const scale = max ?? Math.max(1, ...rows.map((row) => row.value))+ const table = { head: ['', `Value${unit ? ` (${unit.trim()})` : ''}`], rows: rows.map((row) => [row.label, row.value.toFixed(decimals)]) }+ return (+ <ChartFrame title={title} subtitle={subtitle} table={table} empty={rows.length === 0 ? empty : null}>+ <ul className="space-y-2">+ {rows.map((row) => (+ <li key={row.label}>+ <div className="flex items-baseline justify-between gap-3 text-xs">+ <span className="min-w-0 truncate text-[#cfcfcf]">{row.label}</span>+ <span className="shrink-0 text-white">{row.value.toFixed(decimals)}{unit}{row.detail ? <span className="text-[#7f8589]"> {row.detail}</span> : null}</span>+ </div>+ <div className="mt-1 h-2 overflow-hidden rounded-full bg-white/10"><div className="h-full rounded-full" style={{ width: `${Math.min(Math.max((row.value / scale) * 100, 0), 100)}%`, background: row.color || color }} /></div>+ </li>+ ))}+ </ul>+ </ChartFrame>+ )+}++const DAY_NAMES = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']++// Average by hour of day (columns) and weekday (rows). grid: 7 rows of 24 numbers or null.+export function Heatmap({ title, subtitle, grid, unit = '%', color = '#7dd3fc', empty = 'Not enough history yet' }) {+ const [hover, setHover] = useState(null)+ const values = grid.flat().filter((value) => value !== null)+ const top = Math.max(1, ...values)+ const table = { head: ['', ...Array.from({ length: 24 }, (_, hour) => `${hour}`)], rows: grid.map((row, day) => [DAY_NAMES[day], ...row.map((value) => (value === null ? '–' : value.toFixed(0)))]) }+ return (+ <ChartFrame title={title} subtitle={subtitle} table={table} empty={values.length === 0 ? empty : null}>+ <div className="relative">+ <div className="overflow-x-auto">+ <div className="min-w-[420px]">+ {grid.map((row, day) => (+ <div key={DAY_NAMES[day]} className="flex items-center gap-1 py-px">+ <span className="w-8 shrink-0 text-[10px] text-[#7f8589]">{DAY_NAMES[day]}</span>+ {row.map((value, hour) => (+ <button+ key={hour}+ type="button"+ aria-label={`${DAY_NAMES[day]} ${hour}:00, ${value === null ? 'no data' : `${value.toFixed(0)}${unit}`}`}+ onPointerEnter={() => setHover({ day, hour, value })}+ onFocus={() => setHover({ day, hour, value })}+ onPointerLeave={() => setHover(null)}+ className="h-5 flex-1 rounded-[3px]"+ style={{ background: value === null ? 'rgba(255,255,255,0.04)' : color, opacity: value === null ? 1 : 0.15 + 0.85 * (value / top) }}+ />+ ))}+ </div>+ ))}+ <div className="mt-1 flex gap-1 pl-9 text-[10px] text-[#7f8589]">{[0, 6, 12, 18, 23].map((hour) => <span key={hour} style={{ position: 'relative', left: `${(hour / 24) * 100 - (hour === 23 ? 3 : 0)}%`, width: 0 }}>{hour}h</span>)}</div>+ </div>+ </div>+ <div className="mt-2 text-xs text-[#9aa0a6]" role="status">{hover ? `${DAY_NAMES[hover.day]} ${hover.hour}:00 · ${hover.value === null ? 'no data' : `${hover.value.toFixed(1)}${unit} on average`}` : `Darker is busier, up to ${top.toFixed(0)}${unit}.`}</div>+ </div>+ </ChartFrame>+ )+}diff --git a/src/frontend/src/fleetLogic.js b/src/frontend/src/fleetLogic.jsnew file mode 100644index 0000000..1642876--- /dev/null+++ b/src/frontend/src/fleetLogic.js@@ -0,0 +1,98 @@+// Pure helpers for the all-servers view: what to count, rank and label. No React in here.++const MONTHS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']+const pad = (value) => String(value).padStart(2, '0')++export const CONTAINER_STATES = [+ { key: 'running', label: 'Running', color: '#34d399' },+ { key: 'unhealthy', label: 'Unhealthy', color: '#f87171' },+ { key: 'restarting', label: 'Restarting', color: '#fbbf24' },+ { key: 'stopped', label: 'Stopped', color: '#94a3b8' },+]++export function containerState(container) {+ if (container.status === 'running') return container.healthcheck_status === 'unhealthy' ? 'unhealthy' : 'running'+ return container.status === 'restarting' ? 'restarting' : 'stopped'+}++export function containerStateSlices(servers) {+ const counts = { running: 0, unhealthy: 0, restarting: 0, stopped: 0 }+ for (const server of servers) for (const container of server.containers || []) counts[containerState(container)] += 1+ return CONTAINER_STATES.map((state) => ({ label: state.label, color: state.color, value: counts[state.key] }))+}++// The containers using the most of something right now, across all servers: [{label: 'server / name', value}].+export function topContainers(servers, key, count = 10) {+ const rows = []+ for (const server of servers) {+ for (const container of server.containers || []) {+ const value = container[key]+ if (typeof value === 'number' && Number.isFinite(value)) rows.push({ label: `${server.hostname} / ${container.name}`, value })+ }+ }+ return rows.sort((a, b) => b.value - a.value || a.label.localeCompare(b.label)).slice(0, count)+}++// Servers by how full their root disk is, coloured the way the dashboard colours it.+export function serversByDisk(servers, warning = 85, critical = 95) {+ return servers+ .filter((server) => typeof server.host?.disk_usage_pct === 'number')+ .map((server) => ({ label: server.hostname, value: server.host.disk_usage_pct, color: server.host.disk_usage_pct >= critical ? '#f87171' : server.host.disk_usage_pct >= warning ? '#fbbf24' : '#7dd3fc' }))+ .sort((a, b) => b.value - a.value || a.label.localeCompare(b.label))+}++export function isOnline(server, now = Date.now()) {+ if (!server.last_seen_at) return false+ return now - new Date(server.last_seen_at).getTime() < (server.check_interval_seconds || 60) * 3 * 1000+}++export function availabilityRows(availability) {+ return availability+ .filter((item) => item.percent !== null && item.percent !== undefined)+ .map((item) => ({ label: item.hostname, value: item.percent, color: item.percent >= 99 ? '#34d399' : item.percent >= 95 ? '#fbbf24' : '#f87171', detail: `${item.checkins} of about ${item.expected} check-ins` }))+ .sort((a, b) => a.value - b.value || a.label.localeCompare(b.label))+}++export function formatDuration(minutes) {+ if (minutes === null || minutes === undefined || !Number.isFinite(minutes)) return '–'+ if (minutes < 1) return 'under a minute'+ if (minutes < 120) return `${Math.round(minutes)} min`+ if (minutes < 48 * 60) return `${(minutes / 60).toFixed(1)} h`+ return `${(minutes / 1440).toFixed(1)} days`+}++function startOfBucket(bucket, time) {+ const date = new Date(time)+ if (bucket === 'hour') return Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate(), date.getUTCHours())+ const day = Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate())+ if (bucket === 'day') return day+ if (bucket === 'week') return day - ((date.getUTCDay() + 6) % 7) * 86400000+ return Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), 1)+}++function nextBucket(bucket, start) {+ if (bucket === 'hour') return start + 3600000+ if (bucket === 'day') return start + 86400000+ if (bucket === 'week') return start + 7 * 86400000+ const date = new Date(start)+ return Date.UTC(date.getUTCFullYear(), date.getUTCMonth() + 1, 1)+}++export function bucketLabel(time, bucket, utcHours = false) {+ const date = new Date(time)+ if (bucket === 'hour') return `${pad(utcHours ? date.getUTCHours() : date.getHours())}:00`+ if (bucket === 'month') return `${MONTHS[date.getUTCMonth()]} ${date.getUTCFullYear()}`+ return `${date.getUTCDate()} ${MONTHS[date.getUTCMonth()]}`+}++// One column for every bucket from ``start`` to ``end``, with zeros where nothing happened, so a quiet day shows as+// quiet instead of missing. At most ``limit`` (the newest) buckets are returned.+export function incidentColumns(byBucket, bucket, start, end, limit = 120) {+ const known = new Map(byBucket.map((entry) => [startOfBucket(bucket, new Date(entry.time).getTime()), entry]))+ const columns = []+ for (let moment = startOfBucket(bucket, new Date(start).getTime()); moment <= new Date(end).getTime(); moment = nextBucket(bucket, moment)) {+ const entry = known.get(moment)+ columns.push({ time: moment, values: { critical: entry?.critical ?? 0, warning: entry?.warning ?? 0, info: entry?.info ?? 0 } })+ }+ return columns.slice(-limit)+}diff --git a/src/frontend/tests/chartMath.test.js b/src/frontend/tests/chartMath.test.jsnew file mode 100644index 0000000..c5e1b07--- /dev/null+++ b/src/frontend/tests/chartMath.test.js@@ -0,0 +1,93 @@+// Tests for the pure chart helpers. Run with `node --test` in this folder (Node 20 or newer, no packages needed).+import assert from 'node:assert/strict'+import { test } from 'node:test'+import { daysUntilFull, donutSlices, formatTime, histogramBins, hourWeekdayGrid, niceTicks, seriesFor, splitRuns, summarise } from '../src/chartMath.js'++const DAY = 86400000++test('histogram: values land in the right bin, weights count, and out-of-range values are left out', () => {+ const bins = histogramBins([0, 5, 10, 99, 100, 100.1, -1, NaN, { value: 55, weight: 3 }])+ assert.equal(bins.length, 10)+ assert.deepEqual([bins[0].count, bins[1].count, bins[5].count, bins[9].count], [2, 1, 3, 2]) // 100 belongs in the last bin+ assert.equal(histogramBins([1], { low: 0, high: 1, bins: 4 })[3].count, 1)+ assert.equal(histogramBins([]).every((bin) => bin.count === 0), true)+})++test('hour-by-weekday grid: weighted average per cell, Monday first, empty cells stay null', () => {+ const monday = Date.UTC(2026, 2, 2, 10, 0)+ const grid = hourWeekdayGrid([+ { t: monday, avg: 40, samples: 1 },+ { t: monday + 60000, avg: 80, samples: 3 },+ { t: Date.UTC(2026, 2, 8, 23, 30), avg: 5, samples: 1 }, // Sunday 23:30+ { t: monday, avg: null, samples: 9 }, // unknown is not zero+ ], true)+ assert.equal(grid.length, 7)+ assert.equal(grid[0].length, 24)+ assert.equal(grid[0][10], 70)+ assert.equal(grid[6][23], 5)+ assert.equal(grid[3][3], null)+})++test('donut: invalid and zero slices are dropped and fractions add up', () => {+ const slices = donutSlices([{ label: 'a', value: 3 }, { label: 'b', value: 0 }, { label: 'c', value: 1 }, { label: 'x', value: NaN }, { label: 'n', value: -2 }])+ assert.deepEqual(slices.map((slice) => [slice.label, slice.fraction, slice.start]), [['a', 0.75, 0], ['c', 0.25, 0.75]])+ assert.deepEqual(donutSlices([]), [])+})++test('days until full: a clear rising trend gives an estimate, anything unclear gives none', () => {+ const rising = Array.from({ length: 10 }, (_, index) => ({ t: index * DAY, value: 50 + index * 2 })) // 68 now, +2 a day+ assert.ok(Math.abs(daysUntilFull(rising) - 16) < 1e-6)+ assert.equal(daysUntilFull(rising.slice(0, 3)), null) // too few points+ assert.equal(daysUntilFull(rising.map((point) => ({ ...point, value: 50 }))), null) // flat+ assert.equal(daysUntilFull(rising.map((point) => ({ ...point, value: 90 - point.value / 10 }))), null) // falling+ assert.equal(daysUntilFull(rising.map((point) => ({ t: point.t / 100, value: point.value }))), null) // under a day of data+ assert.equal(daysUntilFull(rising.map((point, index) => ({ ...point, value: 50 + (index % 2) * 40 }))), null) // too noisy+ assert.equal(daysUntilFull(rising.map((point) => ({ ...point, value: 95 + (point.t / DAY) * 0.6 }))), 0) // already full+ assert.equal(daysUntilFull([]), null)+})++test('axis ticks reach the top of the data, and bad input does not break them', () => {+ assert.deepEqual(niceTicks(0, 100, 4), [0, 25, 50, 75, 100])+ assert.deepEqual(niceTicks(0, 1024, 4), [0, 500, 1000, 1500])+ assert.deepEqual(niceTicks(NaN, 1), [0, 1])+ assert.ok(niceTicks(5, 5).length >= 2)+})++test('a hole of more than a bucket and a half splits a series, so a gap stays a gap', () => {+ assert.equal(splitRuns([{ t: 0 }, { t: 60000 }, { t: 400000 }], 60).length, 2)+ assert.equal(splitRuns([{ t: 0 }, { t: 60000 }, { t: 120000 }], 60).length, 1)+ assert.deepEqual(splitRuns([], 60), [])+})++test('series and summary: unknown values are skipped, the average is weighted and the maximum is kept', () => {+ const points = [+ { time: '2026-03-02T10:00:00+00:00', samples: 1, cpu: { avg: 100, max: 100 } },+ { time: '2026-03-02T11:00:00+00:00', samples: 3, cpu: { avg: 0, max: 40 } },+ { time: '2026-03-02T12:00:00+00:00', samples: 5, cpu: null },+ ]+ const series = seriesFor(points, 'cpu')+ assert.equal(series.length, 2)+ assert.deepEqual(summarise(series), { average: 25, maximum: 100 })+ assert.deepEqual(summarise([]), { average: null, maximum: null })+})++test('time labels follow the span of the chart', () => {+ const moment = Date.UTC(2026, 2, 5, 9, 7)+ assert.equal(formatTime(moment, 3600, true), '09:07')+ assert.equal(formatTime(moment, 3 * 86400, true), '5 Mar 09:07')+ assert.equal(formatTime(moment, 90 * 86400, true), '5 Mar')+ assert.equal(formatTime(moment, 800 * 86400, true), 'Mar 2026')+})++test('forecast wording: honest about missing trends and sensible at every scale', async () => {+ const { formatForecast } = await import('../src/chartMath.js')+ assert.match(formatForecast(null), /No clear growth trend/)+ assert.match(formatForecast(NaN), /No clear growth trend/)+ assert.equal(formatForecast(0), 'Full now.')+ assert.match(formatForecast(0.4), /less than a day/)+ assert.equal(formatForecast(1), 'Full in about 1 day at this rate.')+ assert.equal(formatForecast(16.2), 'Full in about 16 days at this rate.')+ assert.equal(formatForecast(120), 'Full in about 4 months at this rate.')+ assert.equal(formatForecast(1000), 'Full in about 2.7 years at this rate.')+ assert.match(formatForecast(9999), /Not full within ten years/)+})diff --git a/src/frontend/tests/fleetLogic.test.js b/src/frontend/tests/fleetLogic.test.jsnew file mode 100644index 0000000..bd505dd--- /dev/null+++ b/src/frontend/tests/fleetLogic.test.js@@ -0,0 +1,69 @@+// Tests for the all-servers helpers. Run with `node --test` in this folder.+import assert from 'node:assert/strict'+import { test } from 'node:test'+import { availabilityRows, bucketLabel, containerState, containerStateSlices, formatDuration, incidentColumns, isOnline, serversByDisk, topContainers } from '../src/fleetLogic.js'++const servers = [+ { hostname: 'nas', host: { disk_usage_pct: 91 }, containers: [+ { name: 'web', status: 'running', healthcheck_status: 'healthy', memory_used_mb: 200, cpu_pct: 5 },+ { name: 'db', status: 'running', healthcheck_status: 'unhealthy', memory_used_mb: 900, cpu_pct: null },+ { name: 'job', status: 'exited', memory_used_mb: null, cpu_pct: null },+ ] },+ { hostname: 'web01', host: { disk_usage_pct: 97 }, containers: [+ { name: 'proxy', status: 'restarting', memory_used_mb: 50, cpu_pct: 40 },+ { name: 'cache', status: 'running', memory_used_mb: 500, cpu_pct: 12 },+ ] },+ { hostname: 'empty', host: null, containers: undefined },+]++test('every container falls into exactly one state', () => {+ assert.equal(containerState({ status: 'running' }), 'running')+ assert.equal(containerState({ status: 'running', healthcheck_status: 'unhealthy' }), 'unhealthy')+ assert.equal(containerState({ status: 'restarting' }), 'restarting')+ for (const status of ['exited', 'dead', 'created', 'paused', 'weird']) assert.equal(containerState({ status }), 'stopped')+ const slices = containerStateSlices(servers)+ assert.deepEqual(slices.map((slice) => [slice.label, slice.value]), [['Running', 2], ['Unhealthy', 1], ['Restarting', 1], ['Stopped', 1]])+ assert.deepEqual(containerStateSlices([]).map((slice) => slice.value), [0, 0, 0, 0])+})++test('top containers rank across servers, skip unknown values and honour the count', () => {+ assert.deepEqual(topContainers(servers, 'memory_used_mb', 3), [{ label: 'nas / db', value: 900 }, { label: 'web01 / cache', value: 500 }, { label: 'nas / web', value: 200 }])+ assert.deepEqual(topContainers(servers, 'cpu_pct').map((row) => row.label), ['web01 / proxy', 'web01 / cache', 'nas / web'])+ assert.deepEqual(topContainers([], 'cpu_pct'), [])+})++test('servers are ranked by disk usage and coloured by the thresholds; servers without data are left out', () => {+ assert.deepEqual(serversByDisk(servers).map((row) => [row.label, row.value, row.color]), [['web01', 97, '#f87171'], ['nas', 91, '#fbbf24']])+ assert.equal(serversByDisk([{ hostname: 'x', host: { disk_usage_pct: 40 } }])[0].color, '#7dd3fc')+})++test('a server is online while it keeps reporting', () => {+ const now = Date.UTC(2026, 2, 5, 12, 0)+ assert.equal(isOnline({ last_seen_at: new Date(now - 100000).toISOString(), check_interval_seconds: 60 }, now), true)+ assert.equal(isOnline({ last_seen_at: new Date(now - 200000).toISOString(), check_interval_seconds: 60 }, now), false)+ assert.equal(isOnline({ last_seen_at: null }, now), false)+})++test('availability rows skip unknown servers, put the worst first and colour by how bad it is', () => {+ const rows = availabilityRows([{ hostname: 'a', percent: 100, checkins: 10, expected: 10 }, { hostname: 'b', percent: 96.5, checkins: 9, expected: 10 }, { hostname: 'c', percent: null, checkins: 0, expected: 5 }, { hostname: 'd', percent: 80, checkins: 8, expected: 10 }])+ assert.deepEqual(rows.map((row) => [row.label, row.color]), [['d', '#f87171'], ['b', '#fbbf24'], ['a', '#34d399']])+})++test('durations read naturally', () => {+ assert.equal(formatDuration(null), '–')+ assert.equal(formatDuration(0.2), 'under a minute')+ assert.equal(formatDuration(45), '45 min')+ assert.equal(formatDuration(300), '5.0 h')+ assert.equal(formatDuration(60 * 24 * 5), '5.0 days')+ assert.equal(formatDuration(NaN), '–')+})++test('incident columns fill quiet buckets with zeros, in order, and keep only the newest ones', () => {+ const days = incidentColumns([{ time: '2026-03-10T00:00:00+00:00', critical: 1, warning: 2, info: 0 }], 'day', '2026-03-08T10:00:00+00:00', '2026-03-11T12:00:00+00:00')+ assert.deepEqual(days.map((column) => [bucketLabel(column.time, 'day'), column.values.critical + column.values.warning]), [['8 Mar', 0], ['9 Mar', 0], ['10 Mar', 3], ['11 Mar', 0]])+ const weeks = incidentColumns([], 'week', '2026-03-04T00:00:00+00:00', '2026-03-20T00:00:00+00:00')+ assert.deepEqual(weeks.map((column) => bucketLabel(column.time, 'week')), ['2 Mar', '9 Mar', '16 Mar']) // Mondays+ const months = incidentColumns([], 'month', '2026-11-15T00:00:00+00:00', '2027-02-02T00:00:00+00:00')+ assert.deepEqual(months.map((column) => bucketLabel(column.time, 'month')), ['Nov 2026', 'Dec 2026', 'Jan 2027', 'Feb 2027'])+ assert.equal(incidentColumns([], 'hour', '2026-03-01T00:00:00+00:00', '2026-03-20T00:00:00+00:00', 5).length, 5)+})diff --git a/tests/test_history.py b/tests/test_history.pynew file mode 100644index 0000000..0cf9766--- /dev/null+++ b/tests/test_history.py@@ -0,0 +1,330 @@+import os+from datetime import datetime, timedelta, timezone++os.environ.setdefault("DATABASE_URL", "sqlite://")+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")++import pytest+from fastapi.testclient import TestClient+from sqlalchemy import create_engine+from sqlalchemy.orm import sessionmaker+from sqlalchemy.pool import StaticPool++from src.backend import history, main, rollups+from src.backend.models import AppSetting, Base, ContainerMetric, HostMetric, HostRollup, MountMetric, Server, User++NOW = datetime(2026, 3, 11, 12, 30, tzinfo=timezone.utc) # a Wednesday+++def db():+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ session.add(User(id=1, name="Owner", email="[email protected]", password_hash="x", role="admin"))+ session.add_all(Server(id=n, owner_id=1, hostname=f"srv{n}", check_interval_seconds=60) for n in (1, 2))+ session.commit()+ session.engine = engine+ return session+++def seed(db, start, count, step=timedelta(minutes=1), server=1, cpu=lambda i: float(i % 100), **extra):+ db.add_all(HostMetric(server_id=server, time=start + step * i, cpu_utilization_pct=cpu(i), ram_pct=50.0, **extra) for i in range(count))+ db.commit()+++def ask(db, range_name, metrics=("cpu",), server=1, spec=history.HOST, key=None, now=NOW):+ return history.history(db, spec, server, key, range_name, metrics, 60, now=now)+++def times(result):+ return [datetime.fromisoformat(point["time"]) for point in result["points"]]+++# --- short ranges come from the check-in samples -------------------------------------------------++def test_the_last_hour_is_one_point_per_check_in_with_its_average_and_maximum(db):+ seed(db, NOW - timedelta(minutes=90), 91)++ result = ask(db, "1h", ("cpu", "ram"))++ assert result["source"] == "detailed" and result["bucket_seconds"] == 60 and result["range"] == "1h"+ assert len(result["points"]) == 61 # both ends of the hour are included+ point = result["points"][-1]+ assert point["samples"] == 1 and point["cpu"]["avg"] == point["cpu"]["max"] and point["ram"] == {"avg": 50.0, "max": 50.0}+ assert times(result) == sorted(times(result)) and all(moment.tzinfo is not None for moment in times(result))+++def test_a_week_of_samples_is_grouped_into_at_most_five_hundred_points_without_losing_anything(db):+ seed(db, NOW - timedelta(days=7), 7 * 24 * 60 + 1)++ result = ask(db, "7d")++ assert result["source"] == "detailed" and result["bucket_seconds"] == 1800 and len(result["points"]) <= history.MAX_POINTS+ assert sum(point["samples"] for point in result["points"]) == 7 * 24 * 60 + 1+ first = result["points"][0]+ assert first["cpu"]["max"] >= first["cpu"]["avg"]+ raw = [row[0] for row in db.query(HostMetric.cpu_utilization_pct).filter(HostMetric.time >= NOW - timedelta(days=7))]+ total = sum(point["cpu"]["avg"] * point["samples"] for point in result["points"])+ assert total / len(raw) == pytest.approx(sum(raw) / len(raw))+ assert max(point["cpu"]["max"] for point in result["points"]) == max(raw)+++def test_grouping_keeps_the_exact_average_and_maximum_of_each_group(db):+ seed(db, NOW - timedelta(hours=23), 1) # makes the data span a whole day+ start = NOW - timedelta(hours=5)+ for minute, cpu in ((0, 10.0), (1, 20.0), (2, 90.0), (6, 40.0)): # the first three share a five-minute group+ seed(db, start + timedelta(minutes=minute), 1, cpu=lambda i, cpu=cpu: cpu)++ result = ask(db, "24h")++ grouped = [point for point in result["points"] if point["samples"] > 1]+ assert result["bucket_seconds"] == 300 and len(grouped) == 1+ assert grouped[0]["cpu"] == {"avg": pytest.approx(40.0), "max": 90.0} and grouped[0]["samples"] == 3+++def test_a_gap_stays_a_gap(db):+ seed(db, NOW - timedelta(hours=20), 10)+ seed(db, NOW - timedelta(hours=2), 10)++ result = ask(db, "24h")++ moments = times(result)+ assert max(b - a for a, b in zip(moments, moments[1:])) > timedelta(hours=15) # nothing invented in between+ assert sum(point["samples"] for point in result["points"]) == 20+++def test_a_metric_that_was_never_reported_is_null_not_zero(db):+ seed(db, NOW - timedelta(hours=1), 5)++ point = ask(db, "1h", ("cpu", "swap", "docker_ms"))["points"][-1]++ assert point["swap"] is None and point["docker_ms"] is None and point["cpu"] is not None+++def test_a_server_without_data_gives_an_empty_series(db):+ for name in history.RANGES:+ result = ask(db, name, server=2)+ assert result["points"] == [] and result["data_since"] is None+++# --- long ranges come from the summaries, topped up with the newest samples -----------------------++def test_thirty_days_use_hourly_summaries_and_match_the_samples(db):+ seed(db, NOW - timedelta(days=10), 10 * 24 * 6, step=timedelta(minutes=10))+ rollups.run_rollups(db, NOW)++ result = ask(db, "30d")++ assert result["source"] == "hour" and result["bucket_seconds"] == 3600+ assert sum(point["samples"] for point in result["points"]) == 10 * 24 * 6+ first = result["points"][0]+ raw = [row[0] for row in db.query(HostMetric.cpu_utilization_pct).filter(HostMetric.time >= datetime.fromisoformat(first["time"]), HostMetric.time < datetime.fromisoformat(first["time"]) + timedelta(hours=1))]+ assert first["cpu"] == {"avg": pytest.approx(sum(raw) / len(raw)), "max": max(raw)} and first["samples"] == len(raw)+++def test_the_hour_that_has_not_been_summarised_yet_is_still_in_the_answer(db):+ seed(db, NOW - timedelta(days=3), 3 * 24 * 60)+ rollups.run_rollups(db, NOW)+ seed(db, NOW - timedelta(minutes=20), 20, cpu=lambda i: 1000.0) # after the last closed hour++ result = ask(db, "30d")++ last = result["points"][-1]+ assert datetime.fromisoformat(last["time"]) == rollups.floor(rollups.HOUR, NOW) and last["cpu"]["max"] == 1000.0++[email protected]("range_name", ["30d", "90d", "1y", "all"])+def test_the_answer_does_not_depend_on_how_far_the_rollup_job_has_got(db, monkeypatch, range_name):+ seed(db, NOW - timedelta(days=25), 25 * 24 * 4, step=timedelta(minutes=15), cpu=lambda i: float((i * 7) % 101))+ answers = [ask(db, range_name)] # no summaries at all yet+ monkeypatch.setitem(rollups.MAX_BUCKETS_PER_RUN, rollups.HOUR, 200)+ monkeypatch.setitem(rollups.MAX_BUCKETS_PER_RUN, rollups.DAY, 5)+ for _ in range(3):+ rollups.run_rollups(db, NOW)+ answers.append(ask(db, range_name)) # partly caught up+ monkeypatch.undo()+ for _ in range(20):+ if not any(rollups.run_rollups(db, NOW).values()):+ break+ answers.append(ask(db, range_name)) # fully caught up++ assert len({repr(answer["points"]) for answer in answers}) == 1+ assert sum(point["samples"] for point in answers[0]["points"]) > 0+++def test_every_sample_is_counted_exactly_once_in_the_all_time_view(db):+ seed(db, NOW - timedelta(days=75), 75 * 24 * 2, step=timedelta(minutes=30))+ rollups.run_rollups(db, NOW)+ for _ in range(10):+ rollups.run_rollups(db, NOW)++ result = ask(db, "all")++ assert result["source"] == "month" and result["start"] == result["data_since"]+ assert sum(point["samples"] for point in result["points"]) == 75 * 24 * 2+ assert [datetime.fromisoformat(point["time"]).day for point in result["points"]] == [1, 1, 1, 1] # Dec, Jan, Feb, Mar (partial)+++def test_weekly_points_start_on_mondays_and_the_current_week_is_partial(db):+ seed(db, NOW - timedelta(days=30), 30 * 24, step=timedelta(hours=1))+ for _ in range(10):+ rollups.run_rollups(db, NOW)++ result = ask(db, "1y")++ assert result["source"] == "week" and result["bucket_seconds"] == 7 * 86400+ assert all(moment.weekday() == 0 and moment.hour == 0 for moment in times(result))+ assert datetime.fromisoformat(result["points"][-1]["time"]) == rollups.floor(rollups.WEEK, NOW)+ assert sum(point["samples"] for point in result["points"]) == 30 * 24+++def test_a_long_series_is_thinned_to_the_point_limit(db):+ start = rollups.floor(rollups.HOUR, NOW - timedelta(days=30))+ db.add_all(HostRollup(server_id=1, tier="hour", bucket_start=start + timedelta(hours=i), samples=60, avg_cpu=float(i % 50), n_cpu=60, max_cpu=float(i % 50) + 5) for i in range(720))+ db.add(AppSetting(key="rollup_done_hour", value=(start + timedelta(hours=720)).isoformat()))+ db.commit()++ result = ask(db, "30d", now=start + timedelta(hours=720))++ assert len(result["points"]) <= history.MAX_POINTS and result["bucket_seconds"] == 7200+ assert sum(point["samples"] for point in result["points"]) == 720 * 60+ assert result["points"][0]["cpu"] == {"avg": pytest.approx(0.5), "max": 6.0} # hours 0 and 1: avg (0 + 1) / 2, max of 5 and 6+++# --- when the history starts ----------------------------------------------------------------------++def test_history_starts_at_the_first_check_in_not_at_a_bucket_boundary(db):+ first = datetime(2026, 3, 9, 9, 17, tzinfo=timezone.utc)+ seed(db, first, 3 * 24 * 60)+ rollups.run_rollups(db, NOW) # hour and day summaries now start earlier, at 09:00 and midnight++ assert history.data_since(db, history.HOST, 1, None) == first+ assert datetime.fromisoformat(ask(db, "all")["data_since"]) == first+++def test_when_old_samples_are_gone_the_oldest_summary_that_reaches_back_decides(db):+ first = datetime(2026, 3, 9, 9, 17, tzinfo=timezone.utc)+ seed(db, first, 3 * 24 * 60)+ rollups.run_rollups(db, NOW)++ db.query(HostMetric).filter(HostMetric.time < datetime(2026, 3, 10, tzinfo=timezone.utc)).delete()+ db.commit()+ assert history.data_since(db, history.HOST, 1, None) == datetime(2026, 3, 9, 9, tzinfo=timezone.utc) # hourly rows still reach 09:00++ db.query(HostRollup).filter(HostRollup.tier == "hour").delete()+ db.commit()+ assert history.data_since(db, history.HOST, 1, None) == datetime(2026, 3, 9, tzinfo=timezone.utc) # only the daily row remains+++def test_a_series_with_no_data_has_no_start(db):+ assert history.data_since(db, history.HOST, 2, None) is None+++# --- containers and mounts ---------------------------------------------------------------------++def test_a_container_series_has_cpu_memory_limit_and_down_counts(db):+ for minute in range(10):+ db.add(ContainerMetric(server_id=1, time=NOW - timedelta(minutes=minute), container_id="abc", name="web", status="running" if minute < 7 else "exited",+ cpu_pct=float(minute * 10), memory_used_mb=100.0 + minute, memory_limit_mb=512.0, restart_count=minute // 5))+ db.add(ContainerMetric(server_id=1, time=NOW, container_id="other", name="db", status="running", cpu_pct=99.0))+ db.commit()++ result = history.history(db, history.CONTAINER, 1, "abc", "1h", ("cpu", "memory_mb", "memory_limit_mb", "restarts"), 60, now=NOW)++ assert sum(point["samples"] for point in result["points"]) == 10 and sum(point["down"] for point in result["points"]) == 3+ top = max(result["points"], key=lambda point: point["cpu"]["max"])+ assert top["cpu"]["max"] == 90.0 and top["memory_limit_mb"] == {"max": 512.0} and top["restarts"] == {"max": 1} # limit and restarts have no average+++def test_a_mount_series_is_picked_by_path(db):+ for mount, used in (("/", 10.0), ("/data", 90.0)):+ db.add(MountMetric(server_id=1, time=NOW - timedelta(minutes=1), mount=mount, used_pct=used, used_gb=used, total_gb=100.0))+ db.commit()++ result = history.history(db, history.MOUNT, 1, "/data", "1h", ("used_pct", "used_gb", "total_gb"), 60, now=NOW)++ point = result["points"][0]+ assert point["used_pct"] == {"avg": 90.0, "max": 90.0} and point["used_gb"] == {"avg": 90.0} and point["total_gb"] == {"max": 100.0}++[email protected]("range_name", ["30d", "all"])+def test_container_summaries_agree_with_the_samples(db, range_name):+ for i in range(3 * 24 * 12):+ db.add(ContainerMetric(server_id=1, time=NOW - timedelta(days=3) + timedelta(minutes=5 * i), container_id="abc", name="web", status="running", cpu_pct=float(i % 40), memory_used_mb=200.0))+ db.commit()+ before = history.history(db, history.CONTAINER, 1, "abc", range_name, ("cpu", "memory_mb"), 60, now=NOW)+ for _ in range(10):+ rollups.run_rollups(db, NOW)++ after = history.history(db, history.CONTAINER, 1, "abc", range_name, ("cpu", "memory_mb"), 60, now=NOW)++ assert after["points"] == before["points"] and sum(point["samples"] for point in after["points"]) == 3 * 24 * 12+++# --- the endpoints -------------------------------------------------------------------------------++def client(db, monkeypatch):+ main.app.dependency_overrides[main.get_db] = lambda: db+ monkeypatch.setattr(main, "_current_user", lambda session_id, db: {"id": 1, "role": "user"})+ yield TestClient(main.app)+ main.app.dependency_overrides.clear()+++def test_the_endpoint_returns_the_series_for_a_range(db, client):+ seed(db, datetime.now(timezone.utc) - timedelta(minutes=30), 30)++ response = client.get("/monitoring/servers/1/history?range=1h&metrics=cpu,ram")++ body = response.json()+ assert response.status_code == 200 and body["server_id"] == 1 and body["range"] == "1h" and body["metrics"] == ["cpu", "ram"]+ assert len(body["points"]) >= 29 and set(body["points"][0]) == {"time", "samples", "cpu", "ram"}+++def test_without_a_range_the_original_answer_is_unchanged(db, client):+ seed(db, datetime.now(timezone.utc) - timedelta(minutes=10), 5)++ body = client.get("/monitoring/servers/1/history?minutes=30").json()++ assert set(body) == {"server_id", "samples"} and set(body["samples"][0]) == {"time", "cpu", "ram", "disk"}++[email protected]("query", ["range=2h", "range=", "range=ALL", "range=1h&metrics=", "range=1h&metrics=cpu,,ram", "range=1h&metrics=password",+ "range=1h&metrics=cpu;drop", "range=1h&metrics=" + "cpu," * 100, "range=1h&metrics=cpu,ram_pct"])+def test_unknown_ranges_and_metrics_are_refused(client, query):+ assert client.get(f"/monitoring/servers/1/history?{query}").status_code == 422+++def test_unknown_servers_mounts_and_containers_are_not_found(db, client):+ seed(db, datetime.now(timezone.utc) - timedelta(minutes=5), 3)+ assert client.get("/monitoring/servers/99/history?range=1h").status_code == 404+ assert client.get("/monitoring/servers/1/mounts/history?mount=/data").status_code == 404+ assert client.get("/monitoring/servers/1/mounts/history?mount=data").status_code == 404 # not an absolute path+ assert client.get("/monitoring/servers/1/mounts/history").status_code == 422+ assert client.get("/monitoring/servers/1/containers/nope/history?range=7d").status_code == 404+ assert client.get("/monitoring/servers/99/containers/nope/history").status_code == 404+ assert client.get("/monitoring/servers/1/containers/" + "a" * 129 + "/history").status_code == 422+++def test_the_container_and_mount_endpoints_return_their_series(db, client):+ when = datetime.now(timezone.utc) - timedelta(minutes=2)+ db.add(ContainerMetric(server_id=1, time=when, container_id="abc", name="web", status="running", cpu_pct=12.0, memory_used_mb=100.0))+ db.add(MountMetric(server_id=1, time=when, mount="/data", used_pct=80.0))+ db.commit()++ container = client.get("/monitoring/servers/1/containers/abc/history?range=1h").json()+ mount = client.get("/monitoring/servers/1/mounts/history?mount=/data&range=1h&metrics=used_pct").json()++ assert container["name"] == "web" and container["container_id"] == "abc" and container["points"][0]["cpu"]["avg"] == 12.0+ assert mount["mount"] == "/data" and mount["points"][0]["used_pct"]["avg"] == 80.0+++def test_signing_in_is_required(db, monkeypatch):+ main.app.dependency_overrides[main.get_db] = lambda: db+ try:+ client = TestClient(main.app)+ for path in ("/monitoring/servers/1/history?range=1h", "/monitoring/servers/1/mounts/history?mount=/", "/monitoring/servers/1/containers/abc/history"):+ assert client.get(path).status_code == 401+ finally:+ main.app.dependency_overrides.clear()diff --git a/tests/test_indexes.py b/tests/test_indexes.pynew file mode 100644index 0000000..caace89--- /dev/null+++ b/tests/test_indexes.py@@ -0,0 +1,37 @@+import os++os.environ.setdefault("DATABASE_URL", "sqlite://")+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")++import pytest+from sqlalchemy import create_engine, inspect++from src.backend.models import Base++METRIC_TABLES = ("host_metrics", "mount_metrics", "container_metrics", "system_events")++[email protected](scope="module")+def indexes():+ engine = create_engine("sqlite://")+ Base.metadata.create_all(engine)+ return {table: {index["name"]: index["column_names"] for index in inspect(engine).get_indexes(table)} for table in METRIC_TABLES}+++def test_the_metrics_tables_carry_exactly_the_indexes_they_need(indexes):+ assert indexes["host_metrics"] == {"ix_host_metrics_time": ["time"], "ix_host_metrics_server_time": ["server_id", "time"]}+ assert indexes["mount_metrics"] == {"ix_mount_metrics_time": ["time"], "ix_mount_metrics_server_time": ["server_id", "time"]}+ assert indexes["container_metrics"] == {"ix_container_metrics_time": ["time"], "ix_container_metrics_server_container_time": ["server_id", "container_id", "time"]}+ assert indexes["system_events"] == {"ix_system_events_time": ["time"], "ix_system_events_server_time": ["server_id", "time"], "ix_system_events_event_key": ["event_key"]}++[email protected]("table", METRIC_TABLES)+def test_deleting_a_server_and_purging_old_rows_can_both_use_an_index(indexes, table):+ columns = list(indexes[table].values())+ assert any(index[0] == "server_id" for index in columns), "deleting a server cascades through server_id"+ assert ["time"] in columns, "the hourly purge deletes by time alone"++[email protected]("table", METRIC_TABLES)+def test_a_primary_key_is_never_indexed_a_second_time(indexes, table):+ assert not any(index == ["id"] for index in indexes[table].values())diff --git a/tests/test_rollups.py b/tests/test_rollups.pynew file mode 100644index 0000000..ea682df--- /dev/null+++ b/tests/test_rollups.py@@ -0,0 +1,437 @@+import os+from datetime import datetime, timedelta, timezone++os.environ.setdefault("DATABASE_URL", "sqlite://")+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")++import pytest+from sqlalchemy import create_engine, func+from sqlalchemy.orm import sessionmaker++from src.backend import main, rollups+from src.backend.models import (+ AppSetting, Base, ContainerMetric, ContainerRollup, HostMetric, HostRollup, MountMetric, MountRollup, Server, User,+)+++def T(year, month, day, hour=0, minute=0):+ return datetime(year, month, day, hour, minute, tzinfo=timezone.utc)+++def db():+ engine = create_engine("sqlite://")+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ session.add(User(id=1, name="Owner", email="[email protected]", password_hash="x", role="admin"))+ session.add_all(Server(id=n, owner_id=1, hostname=f"srv{n}", check_interval_seconds=60) for n in (1, 2))+ session.commit()+ return session+++def ago(days, hour=10):+ """A moment a few days back, for the tests that use the real clock (maintenance and server deletion)."""+ return (datetime.now(timezone.utc) - timedelta(days=days)).replace(hour=hour, minute=0, second=0, microsecond=0)+++def host(db, when, server=1, **fields):+ db.add(HostMetric(server_id=server, time=when, **fields))+ db.commit()+++def catch_up(db, now):+ """Run the job until it has nothing left to do (a long history takes several chunks)."""+ for _ in range(100):+ if not any(rollups.run_rollups(db, now).values()):+ return+ raise AssertionError("the job never finished")+++def hosts(db, tier, server=1):+ rows = db.query(HostRollup).filter_by(tier=tier, server_id=server).order_by(HostRollup.bucket_start).all()+ return [(rollups._utc(row.bucket_start), row) for row in rows]+++# --- bucket arithmetic ---------------------------------------------------------------------++def test_buckets_start_on_utc_boundaries_and_weeks_on_monday():+ assert rollups.floor(rollups.HOUR, T(2026, 3, 4, 17, 59)) == T(2026, 3, 4, 17)+ assert rollups.floor(rollups.DAY, T(2026, 3, 4, 17, 59)) == T(2026, 3, 4)+ sunday = T(2026, 3, 8, 23, 59)+ assert sunday.weekday() == 6+ assert rollups.floor(rollups.WEEK, sunday) == T(2026, 3, 2) and T(2026, 3, 2).weekday() == 0+ assert rollups.floor(rollups.WEEK, T(2026, 3, 9)) == T(2026, 3, 9) # the next Monday starts a new week+ assert rollups.floor(rollups.MONTH, T(2026, 3, 31, 23)) == T(2026, 3, 1)+++def test_a_time_with_another_offset_is_converted_to_utc_first():+ plus_two = datetime(2026, 3, 5, 1, 30, tzinfo=timezone(timedelta(hours=2))) # 23:30 UTC the day before+ assert rollups.floor(rollups.DAY, plus_two) == T(2026, 3, 4)+ assert rollups.floor(rollups.DAY, datetime(2026, 3, 4, 12, 0)) == T(2026, 3, 4) # naive counts as UTC+++def test_the_following_bucket_handles_month_ends_and_leap_years():+ assert rollups.following(rollups.MONTH, T(2026, 12, 1)) == T(2027, 1, 1)+ assert rollups.following(rollups.MONTH, T(2028, 2, 1)) == T(2028, 3, 1)+ assert rollups.following(rollups.WEEK, T(2026, 12, 28)) == T(2027, 1, 4)+ assert rollups.following(rollups.HOUR, T(2026, 3, 8, 23)) == T(2026, 3, 9)+++# --- hourly summaries ----------------------------------------------------------------------++def test_an_hour_is_summarised_with_average_maximum_and_count(db):+ for minute, cpu in ((0, 10.0), (20, 20.0), (40, 60.0)):+ host(db, T(2026, 3, 4, 10, minute), cpu_utilization_pct=cpu, ram_pct=50.0)++ assert rollups.run_rollups(db, T(2026, 3, 4, 11, 10))[rollups.HOUR] == 1++ [(start, row)] = hosts(db, rollups.HOUR)+ assert start == T(2026, 3, 4, 10)+ assert (row.samples, row.avg_cpu, row.max_cpu, row.avg_ram, row.max_ram) == (3, 30.0, 60.0, 50.0, 50.0)+ assert row.avg_swap is None and row.max_swap is None # never reported stays unknown, not zero+++def test_a_metric_missing_from_some_samples_is_averaged_over_the_ones_that_have_it(db):+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0)+ host(db, T(2026, 3, 4, 10, 1))+ host(db, T(2026, 3, 4, 10, 2), cpu_utilization_pct=30.0)++ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))++ [(_, row)] = hosts(db, rollups.HOUR)+ assert (row.samples, row.avg_cpu, row.max_cpu) == (3, 20.0, 30.0)+++def test_a_history_of_nothing_but_unknowns_still_counts_check_ins_and_does_not_break(db):+ for minute in range(5):+ host(db, T(2026, 3, 4, 10, minute))+ db.add(MountMetric(server_id=1, time=T(2026, 3, 4, 10, 0), mount="/"))+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, 10, 0), container_id="c", name="web", status="running"))+ db.commit()++ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))++ [(_, row)] = hosts(db, rollups.HOUR)+ assert row.samples == 5 and row.avg_cpu is None and row.max_cpu is None+ assert db.query(MountRollup).one().avg_used_pct is None+ container = db.query(ContainerRollup).one()+ assert (container.samples, container.down_samples, container.avg_cpu, container.max_restarts) == (1, 0, None, 0) # restarts default to 0+++def test_only_completed_buckets_are_written_after_a_short_grace_period(db):+ host(db, T(2026, 3, 4, 10, 30), cpu_utilization_pct=10.0)++ assert rollups.run_rollups(db, T(2026, 3, 4, 10, 59))[rollups.HOUR] == 0+ assert rollups.run_rollups(db, T(2026, 3, 4, 11, 4))[rollups.HOUR] == 0 # ended, but check-ins may still be arriving+ assert hosts(db, rollups.HOUR) == []+ assert rollups.run_rollups(db, T(2026, 3, 4, 11, 6))[rollups.HOUR] == 1+ assert len(hosts(db, rollups.HOUR)) == 1+++def test_running_twice_changes_nothing(db):+ for minute in range(0, 60, 10):+ host(db, T(2026, 3, 4, 10, minute), cpu_utilization_pct=float(minute))+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, 10, 0), container_id="c", name="web", status="running", cpu_pct=5.0))+ db.commit()+ now = T(2026, 3, 6, 0, 30)++ rollups.run_rollups(db, now)+ first = [(r.tier, r.bucket_start, r.samples, r.avg_cpu) for r in db.query(HostRollup).order_by(HostRollup.id)]+ again = rollups.run_rollups(db, now)++ assert again == {"hour": 0, "day": 0, "week": 0, "month": 0}+ assert [(r.tier, r.bucket_start, r.samples, r.avg_cpu) for r in db.query(HostRollup).order_by(HostRollup.id)] == first+++def test_rebuilding_a_bucket_replaces_it_instead_of_duplicating_it(db):+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0)+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))+ db.query(AppSetting).filter(AppSetting.key == "rollup_done_hour").delete() # forget how far it got+ host(db, T(2026, 3, 4, 10, 30), cpu_utilization_pct=30.0)++ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))++ [(_, row)] = hosts(db, rollups.HOUR)+ assert (row.samples, row.avg_cpu) == (2, 20.0)+++def test_servers_mounts_and_containers_are_summarised_separately(db):+ host(db, T(2026, 3, 4, 10, 0), server=1, cpu_utilization_pct=10.0)+ host(db, T(2026, 3, 4, 10, 0), server=2, cpu_utilization_pct=90.0)+ for mount, used in (("/", 40.0), ("/data", 80.0)):+ db.add(MountMetric(server_id=1, time=T(2026, 3, 4, 10, 0), mount=mount, used_pct=used, used_gb=used, total_gb=100.0))+ for name, cpu in (("web", 5.0), ("db", 50.0)):+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, 10, 0), container_id=f"id-{name}", name=name, status="running", cpu_pct=cpu, memory_used_mb=100.0, memory_limit_mb=512.0, restart_count=2))+ db.commit()++ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))++ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR, 1)] == [10.0] and [row.avg_cpu for _, row in hosts(db, rollups.HOUR, 2)] == [90.0]+ mounts = {row.mount: (row.avg_used_pct, row.max_total_gb) for row in db.query(MountRollup)}+ assert mounts == {"/": (40.0, 100.0), "/data": (80.0, 100.0)}+ containers = {row.name: (row.container_id, row.avg_cpu, row.max_memory_limit_mb, row.max_restarts) for row in db.query(ContainerRollup)}+ assert containers == {"web": ("id-web", 5.0, 512.0, 2), "db": ("id-db", 50.0, 512.0, 2)}+++def test_down_samples_count_the_check_ins_where_a_container_was_not_running(db):+ for minute, status in ((0, "running"), (1, "exited"), (2, "exited"), (3, "restarting")):+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, 10, minute), container_id="c", name="web", status=status))+ db.commit()++ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))++ row = db.query(ContainerRollup).one()+ assert (row.samples, row.down_samples) == (4, 3)+++# --- daily, weekly and monthly summaries -----------------------------------------------------++def test_a_day_is_the_samples_weighted_average_of_its_hours_and_keeps_the_maximum(db):+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=100.0)+ for minute in range(3):+ host(db, T(2026, 3, 4, 11, minute), cpu_utilization_pct=0.0)++ rollups.run_rollups(db, T(2026, 3, 5, 0, 10))++ [(start, day)] = hosts(db, rollups.DAY)+ assert start == T(2026, 3, 4)+ assert (day.samples, day.avg_cpu, day.max_cpu) == (4, 25.0, 100.0) # 1 sample of 100 and 3 of 0, not (100 + 0) / 2+++def test_each_average_remembers_how_many_check_ins_reported_it(db):+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0, ram_pct=40.0)+ host(db, T(2026, 3, 4, 10, 1), cpu_utilization_pct=20.0)+ host(db, T(2026, 3, 4, 10, 2))++ rollups.run_rollups(db, T(2026, 3, 5, 0, 10))++ for tier in (rollups.HOUR, rollups.DAY):+ [(_, row)] = hosts(db, tier)+ assert (row.samples, row.n_cpu, row.n_ram, row.n_swap) == (3, 2, 1, 0)+++def test_a_container_that_was_stopped_part_of_the_time_is_averaged_over_the_time_it_ran(db):+ for hour, status, cpu in ((10, "running", 90.0), (11, "exited", None), (12, "exited", None), (13, "exited", None)):+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, hour), container_id="c", name="web", status=status, cpu_pct=cpu))+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 5, 10), container_id="c", name="web", status="running", cpu_pct=10.0))+ db.commit()++ catch_up(db, T(2026, 3, 9, 0, 10))++ day = db.query(ContainerRollup).filter_by(tier="day", bucket_start=T(2026, 3, 4)).one()+ week = db.query(ContainerRollup).filter_by(tier="week").one()+ assert (day.samples, day.down_samples, day.avg_cpu, day.n_cpu) == (4, 3, 90.0, 1)+ assert (week.samples, week.down_samples, week.avg_cpu, week.n_cpu) == (5, 3, 50.0, 2) # (90 + 10) / 2, not (90*4 + 10) / 5+++def test_hours_that_never_reported_a_metric_do_not_pull_a_day_toward_zero(db):+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=80.0)+ host(db, T(2026, 3, 4, 11, 0), docker_latency_ms=50)+ host(db, T(2026, 3, 4, 11, 1), docker_latency_ms=70)++ rollups.run_rollups(db, T(2026, 3, 5, 0, 10))++ [(_, day)] = hosts(db, rollups.DAY)+ assert (day.avg_cpu, day.max_cpu, day.avg_docker_ms, day.max_docker_ms, day.samples) == (80.0, 80.0, 60.0, 70, 3)+++def test_a_week_runs_monday_to_sunday_and_is_written_only_when_sunday_has_ended(db):+ assert T(2026, 3, 2).weekday() == 0+ for day in range(2, 9): # Monday 2 March to Sunday 8 March+ host(db, T(2026, 3, day, 12, 0), cpu_utilization_pct=float(day))+ host(db, T(2026, 3, 9, 12, 0), cpu_utilization_pct=1000.0) # the following Monday belongs to the next week++ rollups.run_rollups(db, T(2026, 3, 8, 23, 59))+ assert hosts(db, rollups.WEEK) == [] # still Sunday++ rollups.run_rollups(db, T(2026, 3, 9, 0, 10))+ [(start, week)] = hosts(db, rollups.WEEK)+ assert start == T(2026, 3, 2)+ assert (week.samples, week.avg_cpu, week.max_cpu) == (7, 5.0, 8.0) # days 2..8 average 5, the Monday after is excluded+++def test_a_month_is_built_from_its_days_including_a_leap_february(db):+ for day in range(1, 30):+ host(db, T(2028, 2, day, 12, 0), cpu_utilization_pct=10.0)++ catch_up(db, T(2028, 3, 1, 0, 10))++ [(start, month)] = hosts(db, rollups.MONTH)+ assert start == T(2028, 2, 1) and (month.samples, month.avg_cpu) == (29, 10.0)+++def test_data_either_side_of_new_year_lands_in_different_months_and_weeks(db):+ host(db, T(2026, 12, 31, 12, 0), cpu_utilization_pct=10.0) # a Thursday+ host(db, T(2027, 1, 1, 12, 0), cpu_utilization_pct=20.0)++ catch_up(db, T(2027, 2, 1, 1, 0))++ assert [(start, row.avg_cpu) for start, row in hosts(db, rollups.MONTH)] == [(T(2026, 12, 1), 10.0), (T(2027, 1, 1), 20.0)]+ assert [(start, row.avg_cpu) for start, row in hosts(db, rollups.WEEK)] == [(T(2026, 12, 28), 15.0)] # one ISO week spans both+++def test_every_tier_is_built_for_mounts_and_containers_too(db):+ for day in (2, 3):+ db.add(MountMetric(server_id=1, time=T(2026, 3, day, 12), mount="/", used_pct=40.0 + day, used_gb=10.0, total_gb=100.0))+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, day, 12), container_id="c", name="web", status="running", cpu_pct=float(day), memory_used_mb=100.0))+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 3, 13), container_id="c", name="web", status="exited"))+ db.commit()++ catch_up(db, T(2026, 4, 2))++ for model in (MountRollup, ContainerRollup):+ assert {row.tier for row in db.query(model)} == {"hour", "day", "week", "month"}+ week = db.query(ContainerRollup).filter_by(tier="week").one()+ assert (week.samples, week.down_samples, week.avg_cpu, week.max_cpu, week.avg_memory_mb) == (3, 1, 2.5, 3.0, 100.0)+ assert db.query(MountRollup).filter_by(tier="week").one().max_used_pct == 43.0+++# --- gaps, catching up and the watermark -----------------------------------------------------++def test_a_gap_stays_a_gap(db):+ host(db, T(2026, 3, 4, 10, 0), server=1, cpu_utilization_pct=10.0)+ host(db, T(2026, 3, 4, 10, 0), server=2, cpu_utilization_pct=20.0)+ host(db, T(2026, 3, 4, 14, 0), server=1, cpu_utilization_pct=30.0) # server 2 was silent at 14:00 and 11:00-13:00+ host(db, T(2026, 3, 6, 9, 0), server=1, cpu_utilization_pct=40.0) # nothing at all on the 5th++ rollups.run_rollups(db, T(2026, 3, 7, 0, 10))++ assert [start for start, _ in hosts(db, rollups.HOUR, 1)] == [T(2026, 3, 4, 10), T(2026, 3, 4, 14), T(2026, 3, 6, 9)]+ assert [start for start, _ in hosts(db, rollups.HOUR, 2)] == [T(2026, 3, 4, 10)]+ assert [start for start, _ in hosts(db, rollups.DAY, 1)] == [T(2026, 3, 4), T(2026, 3, 6)]+++def test_the_first_run_summarises_the_history_that_already_exists(db):+ for day in range(2, 11): # Monday 2 March to Tuesday 10 March+ for hour in range(0, 24, 6):+ host(db, T(2026, 3, day, hour, 0), cpu_utilization_pct=float(day))++ result = rollups.run_rollups(db, T(2026, 3, 11, 0, 10))++ assert result[rollups.HOUR] == 216 and result[rollups.DAY] == 9 and result[rollups.WEEK] == 1 # only 2-8 March is a complete week+ assert len(hosts(db, rollups.HOUR)) == 36 and len(hosts(db, rollups.DAY)) == 9+++def test_new_samples_are_picked_up_by_the_next_run(db):+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0)+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))+ host(db, T(2026, 3, 4, 12, 0), cpu_utilization_pct=20.0)++ assert rollups.run_rollups(db, T(2026, 3, 4, 13, 10))[rollups.HOUR] == 2 # the empty 11:00 hour and the 12:00 one+ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR)] == [10.0, 20.0]+++def test_after_downtime_the_job_catches_up_a_chunk_at_a_time(db, monkeypatch):+ monkeypatch.setitem(rollups.MAX_BUCKETS_PER_RUN, rollups.HOUR, 5)+ for hour in range(12):+ host(db, T(2026, 3, 4, hour, 0), cpu_utilization_pct=float(hour))+ now = T(2026, 3, 4, 12, 10)++ built = [rollups.run_rollups(db, now)[rollups.HOUR] for _ in range(4)]++ assert built == [5, 5, 2, 0]+ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR)] == [float(hour) for hour in range(12)]+++def test_a_long_empty_stretch_cannot_stall_the_job(db, monkeypatch):+ monkeypatch.setitem(rollups.MAX_BUCKETS_PER_RUN, rollups.HOUR, 5)+ host(db, T(2026, 3, 4, 0, 0), cpu_utilization_pct=1.0)+ host(db, T(2026, 3, 5, 6, 0), cpu_utilization_pct=2.0) # 30 hours later, with nothing between++ for _ in range(10):+ rollups.run_rollups(db, T(2026, 3, 5, 8, 0))++ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR)] == [1.0, 2.0]+++def test_the_watermark_is_stored_per_tier_and_a_damaged_one_is_ignored(db):+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0)+ rollups.run_rollups(db, T(2026, 3, 6, 0, 10))++ assert db.get(AppSetting, "rollup_done_hour").value.startswith("2026-03-06T00:00:00")+ assert db.get(AppSetting, "rollup_done_day").value.startswith("2026-03-06T00:00:00")++ db.get(AppSetting, "rollup_done_hour").value = "not a date"+ db.commit()+ rollups.run_rollups(db, T(2026, 3, 6, 0, 10)) # starts over from the first sample, harmlessly+ assert len(hosts(db, rollups.HOUR)) == 1 and len(hosts(db, rollups.DAY)) == 1+++# --- purging only what has been summarised ---------------------------------------------------++def _set_retention(db, days):+ db.add(AppSetting(key=main.METRICS_RETENTION_KEY, value=str(days)))+ db.commit()+++def test_without_any_summaries_the_purge_behaves_as_before(db):+ _set_retention(db, 1)+ host(db, T(2026, 3, 1, 10, 0), cpu_utilization_pct=1.0)+ host(db, T(2026, 3, 5, 10, 0), cpu_utilization_pct=2.0)++ main.purge_old_metrics(db, now=T(2026, 3, 5, 12, 0))++ assert db.query(HostMetric).count() == 1+++def test_samples_that_were_not_summarised_yet_are_never_purged(db):+ _set_retention(db, 1)+ for day in range(1, 5):+ host(db, T(2026, 3, day, 10, 0), cpu_utilization_pct=float(day))+ db.add(AppSetting(key="rollup_done_hour", value=T(2026, 3, 2, 12).isoformat())) # only the first day and a half are summarised+ db.commit()++ main.purge_old_metrics(db, now=T(2026, 3, 5, 12, 0)) # retention alone would delete everything before 4 March++ # 1 and 2 March are summarised and old enough; 3 March is old enough but not summarised yet, so it stays+ assert [rollups._utc(row.time).day for row in db.query(HostMetric).order_by(HostMetric.time)] == [3, 4]+++def test_maintenance_summarises_before_it_purges(db):+ _set_retention(db, 1)+ for days_back, cpu in ((4, 1.0), (3, 2.0)):+ host(db, ago(days_back), cpu_utilization_pct=cpu)+ db.add(ContainerMetric(server_id=1, time=ago(4), container_id="c", name="web", status="running"))+ host(db, datetime.now(timezone.utc) - timedelta(minutes=1), cpu_utilization_pct=99.0) # recent: kept+ db.commit()++ assert main.metrics_maintenance(db, purge_due=True) is True++ assert [row.cpu_utilization_pct for row in db.query(HostMetric)] == [99.0] and db.query(ContainerMetric).count() == 0+ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR)] == [1.0, 2.0] and db.query(ContainerRollup).filter_by(tier="hour").count() == 1+++def test_a_failing_rollup_job_stops_the_purge(db, monkeypatch):+ _set_retention(db, 1)+ host(db, ago(4), cpu_utilization_pct=1.0)++ def broken(*args, **kwargs):+ raise RuntimeError("database trouble")++ monkeypatch.setattr(rollups, "run_rollups", broken)++ assert main.metrics_maintenance(db, purge_due=True) is False+ assert db.query(HostMetric).count() == 1+++def test_between_purges_maintenance_only_summarises(db):+ host(db, ago(2), cpu_utilization_pct=1.0)++ assert main.metrics_maintenance(db, purge_due=False) is False+ assert len(hosts(db, rollups.HOUR)) == 1 and db.query(HostMetric).count() == 1+++def test_deleting_a_server_deletes_its_summaries_but_not_another_servers(db, monkeypatch):+ monkeypatch.setattr(main, "_require_admin", lambda session_id, db: {"id": 1})+ for server in (1, 2):+ host(db, ago(2), server=server, cpu_utilization_pct=10.0)+ db.add(MountMetric(server_id=server, time=ago(2), mount="/", used_pct=1.0))+ db.add(ContainerMetric(server_id=server, time=ago(2), container_id="c", name="web", status="running"))+ db.commit()+ rollups.run_rollups(db)++ main.delete_monitored_server(1, session_id="s", db=db)++ for model in (HostRollup, MountRollup, ContainerRollup):+ assert db.query(model).filter_by(server_id=1).count() == 0+ assert db.query(model).filter_by(server_id=2).count() > 0diff --git a/tests/test_stats.py b/tests/test_stats.pynew file mode 100644index 0000000..5aa7fca--- /dev/null+++ b/tests/test_stats.py@@ -0,0 +1,302 @@+import json+import os+from datetime import datetime, timedelta, timezone++os.environ.setdefault("DATABASE_URL", "sqlite://")+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")++import pytest+from fastapi.testclient import TestClient+from sqlalchemy import create_engine+from sqlalchemy.orm import sessionmaker+from sqlalchemy.pool import StaticPool++from src.backend import main, stats+from src.backend.models import Base, HostMetric, IncidentAlert, Server, User+from tests.fakes import FakeRedis++NOW = datetime(2026, 3, 11, 12, 30, tzinfo=timezone.utc) # a Wednesday+++def db():+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)+ Base.metadata.create_all(engine)+ session = sessionmaker(bind=engine)()+ session.add(User(id=1, name="Owner", email="[email protected]", password_hash="x", role="admin"))+ session.add_all(Server(id=n, owner_id=1, hostname=f"srv{n}", check_interval_seconds=60) for n in (1, 2))+ session.commit()+ return session+++def incident(db, opened, severity="WARNING", rule="host:disk:/", resolved=None, server=1):+ db.add(IncidentAlert(server_id=server, severity=severity, title="t", description="d", rule_key=rule, created_at=opened,+ status="resolved" if resolved else "active", resolved_at=resolved))+ db.commit()+++# --- grouping incidents by what they are about ------------------------------------------------------+[email protected]("key, label", [+ ("host:disk:/", "Disk space"), ("host:disk:/media/data", "Disk space"), ("host:memory", "Memory"), ("host:inodes:/", "Inodes"),+ ("system:oom", "Out-of-memory kills"), ("system:docker_slow", "Docker slow"), ("system:collect_slow", "Collector too slow"),+ ("server_offline", "Server offline"), ("trend:memory_leak:abc", "Possible memory leaks"), ("network:listener:tcp:22", "Network"),+ ("container:abc:restart_loop", "Containers"), (None, "Other"), ("", "Other"), ("something:else", "Other"), ("host", "Other"), (":" * 50, "Other"),+])+def test_rule_keys_become_readable_groups(key, label):+ assert stats.incident_kind(key) == label+++def test_long_or_odd_rule_keys_never_leak_through():+ assert stats.incident_kind("x" * 5000) == "Other" and stats.incident_kind("host:unknown_metric") == "Other"+++# --- incident statistics ------------------------------------------------------------------------++def test_incidents_are_counted_per_day_and_severity_for_a_week(db):+ incident(db, NOW - timedelta(days=1, hours=3), "CRITICAL")+ incident(db, NOW - timedelta(days=1, hours=1), "WARNING")+ incident(db, NOW - timedelta(days=1, hours=2), "WARNING", rule="system:oom")+ incident(db, NOW - timedelta(days=3), "INFO", rule="system:reboot")++ result = stats.incident_stats(db, "7d", NOW)++ assert result["bucket"] == "day" and result["opened"] == 4+ assert result["by_bucket"] == [+ {"time": "2026-03-08T00:00:00+00:00", "critical": 0, "warning": 0, "info": 1},+ {"time": "2026-03-10T00:00:00+00:00", "critical": 1, "warning": 2, "info": 0},+ ]+ assert result["by_kind"][0] == {"kind": "Disk space", "count": 2}+ assert sorted(item["kind"] for item in result["by_kind"]) == ["Disk space", "Out-of-memory kills", "Reboots"]+++def test_the_bucket_size_follows_the_range(db):+ incident(db, NOW - timedelta(minutes=30))+ assert [stats.incident_stats(db, name, NOW)["bucket"] for name in ("1h", "6h", "24h", "7d", "30d", "90d", "1y", "all")] == ["hour", "hour", "hour", "day", "day", "week", "week", "month"]+ assert [stats.incident_stats(db, "1y", NOW)["by_bucket"][0]["time"]][0] == "2026-03-09T00:00:00+00:00" # weeks start on Monday+ assert stats.incident_stats(db, "all", NOW)["by_bucket"][0]["time"] == "2026-03-01T00:00:00+00:00"+++def test_incidents_older_than_the_range_are_left_out(db):+ incident(db, NOW - timedelta(days=40))+ incident(db, NOW - timedelta(days=2))++ assert stats.incident_stats(db, "7d", NOW)["opened"] == 1+ assert stats.incident_stats(db, "90d", NOW)["opened"] == 2+ assert stats.incident_stats(db, "all", NOW)["opened"] == 2+++def test_mean_time_to_resolve_uses_only_resolved_incidents(db):+ incident(db, NOW - timedelta(hours=5), resolved=NOW - timedelta(hours=4)) # 60 minutes+ incident(db, NOW - timedelta(hours=5), resolved=NOW - timedelta(hours=4, minutes=20)) # 40 minutes+ incident(db, NOW - timedelta(hours=2)) # still open++ result = stats.incident_stats(db, "24h", NOW)++ assert (result["opened"], result["resolved"], result["mean_minutes_to_resolve"]) == (3, 2, 50.0)+++def test_open_incidents_are_counted_by_severity_whatever_the_range(db):+ incident(db, NOW - timedelta(days=200), "CRITICAL")+ incident(db, NOW - timedelta(days=1), "WARNING")+ incident(db, NOW - timedelta(days=1), "WARNING", resolved=NOW)++ result = stats.incident_stats(db, "24h", NOW)++ assert result["open_now"] == {"critical": 1, "warning": 1, "info": 0} # the 200-day-old one still counts as open+ assert result["opened"] == 2 # but only two opened within the day+++def test_an_install_without_incidents_gives_empty_answers(db):+ for name in ("1h", "30d", "all"):+ result = stats.incident_stats(db, name, NOW)+ assert result["opened"] == 0 and result["by_bucket"] == [] and result["by_kind"] == [] and result["mean_minutes_to_resolve"] is None+ assert result["open_now"] == {"critical": 0, "warning": 0, "info": 0}+++def test_an_unknown_severity_is_counted_as_info_not_dropped(db):+ incident(db, NOW - timedelta(hours=1), "SURPRISE")+ assert stats.incident_stats(db, "24h", NOW)["by_bucket"][0]["info"] == 1+++def test_the_amount_of_data_read_is_capped_and_says_so(db, monkeypatch):+ monkeypatch.setattr(stats, "MAX_INCIDENTS_READ", 3)+ for hours in range(6):+ incident(db, NOW - timedelta(hours=hours))++ result = stats.incident_stats(db, "24h", NOW)++ assert result["truncated"] is True and result["opened"] == 3+++# --- availability ------------------------------------------------------------------------------++def checkins(db, server, start, count, step=timedelta(minutes=1)):+ db.add_all(HostMetric(server_id=server, time=start + step * i, cpu_utilization_pct=1.0) for i in range(count))+ db.commit()+++def test_a_server_that_always_reported_is_fully_available(db):+ checkins(db, 1, NOW - timedelta(hours=6), 361)++ [first, second] = stats.availability(db, "6h", NOW)++ assert first["server_id"] == 1 and first["percent"] == 100.0 and first["checkins"] == 361+ assert second["server_id"] == 2 and second["percent"] is None and second["checkins"] == 0 # never reported: unknown, not 0%+++def test_a_server_that_was_silent_for_half_the_time_is_half_available(db):+ checkins(db, 1, NOW - timedelta(hours=6), 181)+ checkins(db, 1, NOW - timedelta(minutes=1), 1) # reported at the start and now, nothing in between++ assert stats.availability(db, "6h", NOW)[0]["percent"] == pytest.approx(51.0, abs=2)+++def test_a_new_server_is_not_blamed_for_the_time_before_it_existed(db):+ checkins(db, 1, NOW - timedelta(hours=1), 61)++ assert stats.availability(db, "30d", NOW)[0]["percent"] == 100.0+++def test_availability_is_exact_even_after_the_history_has_been_summarised(db):+ from src.backend import rollups+ checkins(db, 1, NOW - timedelta(days=3, hours=-9, minutes=-17), 3 * 24 * 60 - 9 * 60 - 17) # starts at 09:17 on the first day+ rollups.run_rollups(db, NOW)++ for name in ("7d", "30d", "all"):+ assert stats.availability(db, name, NOW)[0]["percent"] == 100.0+++def test_availability_never_exceeds_one_hundred_percent(db):+ checkins(db, 1, NOW - timedelta(hours=1), 600, step=timedelta(seconds=6)) # far more often than the 60 s interval++ assert stats.availability(db, "6h", NOW)[0]["percent"] == 100.0+++def test_availability_works_for_every_range_without_data(db):+ for name in ("1h", "6h", "24h", "7d", "30d", "90d", "1y", "all"):+ assert [item["percent"] for item in stats.availability(db, name, NOW)] == [None, None]+++# --- the endpoint ---------------------------------------------------------------------------------++def client(db, monkeypatch):+ main.app.dependency_overrides[main.get_db] = lambda: db+ monkeypatch.setattr(main, "_current_user", lambda session_id, db: {"id": 1, "role": "user"})+ monkeypatch.setattr(main, "redis_client", FakeRedis())+ yield TestClient(main.app)+ main.app.dependency_overrides.clear()+++def test_the_endpoint_returns_incidents_and_availability(db, client):+ incident(db, datetime.now(timezone.utc) - timedelta(hours=2), "CRITICAL")++ body = client.get("/stats/fleet?range=24h").json()++ assert set(body) == {"incidents", "availability"} and body["incidents"]["opened"] == 1 and body["incidents"]["open_now"]["critical"] == 1+ assert [item["hostname"] for item in body["availability"]] == ["srv1", "srv2"]+++def test_answers_are_cached_for_a_minute_per_range_only(db, client, monkeypatch):+ calls = []+ real = stats.fleet_stats+ monkeypatch.setattr(stats, "fleet_stats", lambda db, range_name: calls.append(range_name) or real(db, range_name))++ client.get("/stats/fleet?range=7d"); client.get("/stats/fleet?range=7d"); client.get("/stats/fleet?range=30d")++ assert calls == ["7d", "30d"]+ assert sorted(key for key in main.redis_client.keys("*")) == ["homelab:stats:fleet:30d", "homelab:stats:fleet:7d"]+++def test_a_broken_cache_does_not_break_the_page(db, client, monkeypatch):+ class Broken:+ def get(self, *args, **kwargs): raise ConnectionError("redis is down")+ def set(self, *args, **kwargs): raise ConnectionError("redis is down")++ monkeypatch.setattr(main, "redis_client", Broken())++ assert client.get("/stats/fleet?range=24h").status_code == 200++[email protected]("query", ["range=2h", "range=", "range=ALL", "range=1h;drop", "range=" + "a" * 5000])+def test_unknown_ranges_are_refused_and_never_become_cache_keys(db, client, query):+ assert client.get(f"/stats/fleet?{query}").status_code == 422+ assert main.redis_client.keys("*") == []+++def test_signing_in_is_required(db, monkeypatch):+ main.app.dependency_overrides[main.get_db] = lambda: db+ monkeypatch.setattr(main, "redis_client", FakeRedis())+ try:+ assert TestClient(main.app).get("/stats/fleet?range=24h").status_code == 401+ finally:+ main.app.dependency_overrides.clear()+++# --- when a server is busy: hour of day by weekday ----------------------------------------------------++def server_row(db, server_id=1):+ return db.get(Server, server_id)+++def test_the_heatmap_puts_each_hour_in_its_weekday_and_hour_cell(db):+ monday_ten = datetime(2026, 3, 9, 10, 0, tzinfo=timezone.utc) # a Monday+ db.add_all(HostMetric(server_id=1, time=monday_ten + timedelta(minutes=minute), cpu_utilization_pct=value) for minute, value in ((0, 20.0), (1, 60.0)))+ db.add(HostMetric(server_id=1, time=datetime(2026, 3, 8, 23, 30, tzinfo=timezone.utc), cpu_utilization_pct=5.0)) # Sunday 23:30+ db.commit()++ result = stats.hour_weekday_heatmap(db, server_row(db), "cpu", 0, NOW)++ assert result["cells"][0][10] == 40.0 and result["samples"][0][10] == 2+ assert result["cells"][6][23] == 5.0+ assert result["cells"][2][2] is None and len(result["cells"]) == 7 and all(len(row) == 24 for row in result["cells"])+++def test_the_heatmap_follows_the_viewers_time_zone(db):+ when = datetime(2026, 3, 8, 23, 30, tzinfo=timezone.utc) # Sunday 23:30 UTC+ db.add(HostMetric(server_id=1, time=when, cpu_utilization_pct=5.0))+ db.commit()++ east = stats.hour_weekday_heatmap(db, server_row(db), "cpu", 120, NOW) # UTC+2: already Monday 01:30+ west = stats.hour_weekday_heatmap(db, server_row(db), "cpu", -300, NOW) # UTC-5: Sunday 18:30++ assert east["cells"][0][1] == 5.0 and west["cells"][6][18] == 5.0+++def test_the_heatmap_ignores_samples_older_than_thirty_days_and_unknowns(db):+ db.add(HostMetric(server_id=1, time=NOW - timedelta(days=40), cpu_utilization_pct=99.0))+ db.add(HostMetric(server_id=1, time=NOW - timedelta(days=2), ram_pct=50.0)) # no CPU reading+ db.commit()++ assert all(cell is None for row in stats.hour_weekday_heatmap(db, server_row(db), "cpu", 0, NOW)["cells"] for cell in row)+++def test_the_heatmap_is_the_same_before_and_after_summarising(db):+ from src.backend import rollups+ db.add_all(HostMetric(server_id=1, time=NOW - timedelta(days=10) + timedelta(minutes=10 * i), cpu_utilization_pct=float(i % 90)) for i in range(10 * 24 * 6))+ db.commit()+ before = stats.hour_weekday_heatmap(db, server_row(db), "cpu", 60, NOW)+ for _ in range(10):+ rollups.run_rollups(db, NOW)++ assert stats.hour_weekday_heatmap(db, server_row(db), "cpu", 60, NOW) == before+++def test_the_heatmap_endpoint_validates_everything_and_caches_per_server_metric_and_offset(db, client):+ assert client.get("/stats/servers/1/heatmap?metric=cpu&offset=60").status_code == 200+ assert client.get("/stats/servers/1/heatmap?metric=cpu&offset=60").status_code == 200+ assert main.redis_client.keys("*") == ["homelab:stats:heatmap:1:cpu:60"]+ for query in ("metric=disk", "metric=cpu;x", "offset=900", "offset=-900", "offset=abc", "offset=1.5"):+ assert client.get(f"/stats/servers/1/heatmap?{query}").status_code == 422+ assert client.get("/stats/servers/999/heatmap").status_code == 404+ assert main.redis_client.keys("*") == ["homelab:stats:heatmap:1:cpu:60"] # refused requests left nothing behind+++def test_the_heatmap_endpoint_needs_a_session(db, monkeypatch):+ main.app.dependency_overrides[main.get_db] = lambda: db+ monkeypatch.setattr(main, "redis_client", FakeRedis())+ try:+ assert TestClient(main.app).get("/stats/servers/1/heatmap").status_code == 401+ finally:+ main.app.dependency_overrides.clear()