Skip to content

Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.

feat: Statistics page: fleet view, container cards, histograms, heatmap, storage forecast

karloks20052fa8a13e722e
diff --git a/README.md b/README.md
index b0fc2c1..e722451 100644
--- a/README.md
+++ b/README.md
@@ -419,6 +419,42 @@ lists always start closed.
<ui - Fleet section with every card collapsed to one line (status dot, name, This machine and incident badges, CPU/RAM/fullest disk) and the Collapse all / Expand all button>
+## Statistics
+
+**Statistics** in the navigation shows how one server behaved over time. Pick a server and a
+range (1 hour, 6 hours, 24 hours, 7 days, 30 days, 90 days, 1 year or all time) and you get
+CPU, memory, swap, load and root disk charts. Each chart is the average as a line with a
+lighter band up to the highest value, so a short spike stays visible even in a yearly view.
+Hover or touch a point for the exact time, average, highest value and number of check-ins, or
+use **View as table**. Gaps (a server that was offline) are drawn as gaps. A note under the
+cards says what each point is (for example "one per hour, from hourly summaries") and when
+the history starts. The selection is kept in the address, so a view can be bookmarked, and
+the range is remembered per browser. The short ranges refresh every 30 seconds, except
+while the tab is hidden.
+
+Under the charts, **Containers** shows every container of that server as a small card (status
+dot, health, restarts, CPU, memory against its limit), with search and sorting (problems first
+by default). Select a card for that container's CPU and memory over the chosen range, with its
+memory limit drawn as a dashed line, how much of the time it was not running, and how many
+times it restarted in the range. A container only reports CPU and memory while it runs.
+
+Further down a server's page: how its CPU and memory use were **distributed** (the share of the
+time in each 10% band), **when it is busy** (average CPU or memory by hour of the day and weekday
+over the last 30 days, in your time zone), **storage** (a chart per filesystem with an estimate
+such as "full in about 3 days" when there is a clear growth trend, and none otherwise), and
+**reliability** (how long Docker takes to list containers, and how long the collector takes).
+
+Choose **All servers** for the whole fleet: containers by state and open incidents by severity
+(donuts), incidents opened over time by severity (stacked columns), what the incidents were
+about, servers by disk usage, the containers using the most memory and CPU right now, and each
+server's availability. Availability is the share of expected check-ins that arrived; it is
+approximate because it assumes each server's current check interval. Every chart can be shown
+as a table.
+
+Short ranges come from the stored check-ins, longer ones from the summaries described under
+[Database schema](#database-schema), so a year of history costs about as much to load as an
+hour. CSV/JSON export of the charts is planned (Milestone 8 on GitHub).
+
## Alerts and incidents
Rules run on every collector check-in. Each problem opens at most one incident
@@ -758,7 +794,8 @@ pip install -r requirements-dev.txt # the app's requirements plus pytest
pytest
```
-No Docker, PostgreSQL or Redis is needed; the suite takes about 15 seconds. See
+No Docker, PostgreSQL or Redis is needed; the suite takes about a minute and a half. The chart
+helpers have their own tests that need only Node 20 or newer: `cd src/frontend && node --test`. See
[TESTS.md](TESTS.md) for how the tests work, what they don't cover, and what every
test is for.
@@ -777,6 +814,21 @@ recovery codes (the encrypted two-factor secret lives on the `users` row), and p
thresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialises
the `users` table on a fresh volume.
+**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-in
+samples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or container
+per hour, day, week (Monday to Sunday) and month, all in UTC, with the average and maximum of each metric.
+Each tier is built from the finer one when its period ends, a period without check-ins gets no row, and the
+purge never deletes samples that have not been summarised yet. Nothing displays these yet; they are the
+foundation for the long-range graphs and statistics.
+
+The metrics tables carry only the indexes their queries use: `time` (for the hourly purge) and
+`(server_id, time)`, or `(server_id, container_id, time)` for containers. Older databases are
+migrated at start: the new indexes are created and the redundant single-column ones are dropped,
+which can be repeated safely. Creating an index briefly blocks writes to that table (a few
+seconds for a table of millions of rows); on a very large install, create the indexes yourself
+first with `CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (<columns>)` using the names
+in `src/backend/models.py`, and the start-up migration will skip them.
+
## Project layout
```text
@@ -804,6 +856,9 @@ src/
backend/ API: auth, monitoring, enrollment, rate limits, models
alerts.py alert engine: one incident per rule, auto-resolve
rules.py alert rules and default thresholds
+ rollups.py summarises check-ins into hourly/daily/weekly/monthly rollups
+ history.py range-aware history behind the Statistics page
+ stats.py fleet statistics: incidents over time, availability, busy-hours heatmap
totp.py one-time codes and recovery codes (standard library only)
secrets_box.py encrypts stored secrets with SECRETS_KEY; also prints a new key
reset_2fa.py command-line two-factor reset for a locked-out account
@@ -813,7 +868,9 @@ src/
system_helper.py optional kernel-log / wtmp helper (standard library only)
health_checks.py read-only checks available to the assistant
service.py, prompts.py Gemini assistant
- frontend/ React + Tailwind dashboard (Vite)
+ frontend/ React + Tailwind dashboard (Vite); charts.jsx and chartMath.js are the
+ charts (distributions.jsx: columns, donut, bars, heatmap), StatisticsPage.jsx,
+ FleetStatistics.jsx, ServerInsights.jsx and ContainerStatistics.jsx the page
tests/ pytest suite
```
@@ -832,7 +889,10 @@ All paths are relative to `/api` when going through the frontend/nginx.
| `GET /auth/sessions`, `POST /auth/sessions/revoke-others`, `DELETE /auth/sessions/{id}` | session | List and end your own sessions. |
| `GET /auth/config` | – | Whether registration is open and whether the first account still needs a setup token. |
| `GET /monitoring/overview` | session | Servers (latest metrics, filesystems, containers from the latest check-in, open incident count), pending servers (admins), active incidents and disk thresholds. |
-| `GET /monitoring/servers/{id}/history` | session | CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). |
+| `GET /monitoring/servers/{id}/history` | session | A server's metrics over time. With `?range=` (`1h`, `6h`, `24h`, `7d`, `30d`, `90d`, `1y`, `all`) it returns at most about 500 points from the right level of detail, each with average and maximum of the requested `metrics` (`cpu,ram,swap,load,disk,inodes,docker_ms,collect_ms`). Without `range`: CPU, memory and root disk samples (`?minutes=`, default 30, max 1440). |
+| `GET /stats/fleet` | session | Incidents opened per bucket and severity, by kind, open now, mean time to resolve, and per-server availability for `?range=`. Cached for a minute. |
+| `GET /stats/servers/{id}/heatmap` | session | Average `?metric=cpu\|ram` by hour of the day and weekday over the last 30 days, for `?offset=` minutes from UTC. |
+| `GET /monitoring/servers/{id}/mounts/history`, `GET /monitoring/servers/{id}/containers/{container_id}/history` | session | The same for one filesystem (`?mount=/data`) or one container. |
| `GET /monitoring/servers/{id}/containers/{container_id}` | session | One container's latest stats and history (`?limit=`, max 500). |
| `POST /monitoring/servers` | admin | Add a server manually; returns its collector token. |
| `PATCH /monitoring/servers/{id}/interval` | admin | Set check interval (15–86400 s). |
diff --git a/TESTS.md b/TESTS.md
index f7ee386..a6d3bbd 100644
--- a/TESTS.md
+++ b/TESTS.md
@@ -4,8 +4,8 @@ Vexion ships with an automated test suite. This page explains how to run it, wha
covers (and what it doesn't), and what every single test is for, so you can decide
whether a failure matters before you upgrade or change your setup.
-- **676 test cases** from 384 test functions (some functions run once per input).
-- **About 15 to 20 seconds** to run. No Docker, PostgreSQL, Redis, network or root access needed.
+- **798 test cases** from 469 test functions, plus 16 small tests of the chart helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).
+- **About a minute and a half** to run. No Docker, PostgreSQL, Redis, network or root access needed.
- **Safe to run next to a live install.** The tests use a throwaway in-memory database
and fakes; they never touch your running containers or your data. (Two collector
tests read this machine's `/proc` and root filesystem, read-only.)
@@ -86,7 +86,11 @@ Being clear about the limits matters, especially for a security-focused project:
- **The web interface** (React) has no automated tests. Build errors show up when you
run `docker compose up -d --build`; look at the dashboard after upgrading. This
includes the Account page and the two-factor prompts: the endpoints behind them are
- tested, the screens are not.
+ tested, the screens are not. The Account page, the sign-in code step and the Statistics
+ page (including container cards, the all-servers view, histograms, the heatmap and storage
+ charts) were checked by hand in a real browser (headless Chromium against a running stack,
+ desktop and phone width). Only the plain helper functions behind the charts have automated
+ tests.
- **Your deployment**: Compose files, Traefik labels, nginx configuration, your VPN,
firewall rules and the collector gate are not exercised by the suite. They were
checked by hand against a running stack when written, not by an automated test.
@@ -134,9 +138,13 @@ docker compose logs --tail 20 collector # "Sent host metrics and N contai
| [`test_docker_proxy.py`](#test_docker_proxypy) | 111 | The read-only Docker proxy: what it allows, refuses and hides. |
| [`test_enrollment.py`](#test_enrollmentpy) | 10 | Enrollment keys, collector credentials and offline-server detection. |
| [`test_health_checks.py`](#test_health_checkspy) | 4 | The assistant's fixed health checks and the critical-only webhook. |
+| [`test_history.py`](#test_historypy) | 36 | The history API behind long-range charts: which level of detail answers each range, point limits, gaps, and agreement with the raw samples. |
+| [`test_indexes.py`](#test_indexespy) | 9 | The database indexes on the metrics tables: only the ones queries use, none duplicated. |
| [`test_network_checks.py`](#test_network_checkspy) | 94 | The host's real network totals, listening-port inventory, optional internet check, firewall fingerprints, and the validation of all of it. |
| [`test_pending_collectors.py`](#test_pending_collectorspy) | 28 | Adding servers: approval, join tokens, strict mode, re-pairing safety. |
+| [`test_rollups.py`](#test_rollupspy) | 31 | Summarised history: hourly, daily, weekly and monthly rollups, catching up, gaps, and purging only what has been summarised. |
| [`test_rules.py`](#test_rulespy) | 20 | Alert thresholds: sustained breaches, per-mount disks, settings validation. |
+| [`test_stats.py`](#test_statspy) | 46 | Fleet statistics (incidents over time, availability) and the busy-hours heatmap: grouping, ranges, caching and validation. |
| [`test_system_checks.py`](#test_system_checkspy) | 17 | Reboot, OOM, kernel-fault, clock and pending-reboot detection, and the System row's event list. |
| [`test_system_helper.py`](#test_system_helperpy) | 35 | The optional system helper: kernel-log parsing, `wtmp`, crash detection, hostile input. |
@@ -504,6 +512,49 @@ tested hardest (91 cases). If any of these fail, do not run the proxy until it i
| `test_unknown_check_is_rejected` | Asking for any check that isn't one of the fixed ones raises an error. |
| `test_daily_checkup_alerts_only_for_critical_findings` | The daily check-up notifies the webhook only when something is critical. |
+### test_history.py
+
+The history endpoints that charts will use for any range from the last hour to all time. The key promise is
+that the answer is the same whether or not the background summarising has caught up.
+
+| Test | What it checks |
+|---|---|
+| `test_the_last_hour_is_one_point_per_check_in_with_its_average_and_maximum` | The last hour is one point per check-in (both ends included), each with average and maximum, oldest first, with UTC times. |
+| `test_a_week_of_samples_is_grouped_into_at_most_five_hundred_points_without_losing_anything` | A week is grouped into at most 500 points; the check-in count, the overall average and the highest maximum are unchanged by grouping. |
+| `test_grouping_keeps_the_exact_average_and_maximum_of_each_group` | Three check-ins grouped into one five-minute point give their exact average and maximum. |
+| `test_a_gap_stays_a_gap` | Two bursts of data hours apart have no points between them. |
+| `test_a_metric_that_was_never_reported_is_null_not_zero` | A metric the collector never sent comes back as null, not 0. |
+| `test_a_server_without_data_gives_an_empty_series` | A server with no data gives an empty series for every range, without errors. |
+| `test_thirty_days_use_hourly_summaries_and_match_the_samples` | 30 days is answered from hourly summaries whose figures equal the samples they summarise. |
+| `test_the_hour_that_has_not_been_summarised_yet_is_still_in_the_answer` | The newest, not yet summarised hour is built from the samples so the right edge is current. |
+| `test_the_answer_does_not_depend_on_how_far_the_rollup_job_has_got` *(4 inputs)* | For 30 days, 90 days, 1 year and all time, the answer is identical with no summaries, part-way through catching up, and fully caught up. |
+| `test_every_sample_is_counted_exactly_once_in_the_all_time_view` | All time is monthly and every check-in is counted exactly once. |
+| `test_weekly_points_start_on_mondays_and_the_current_week_is_partial` | Weekly points start on Mondays and the unfinished week is included. |
+| `test_a_long_series_is_thinned_to_the_point_limit` | 720 hourly rows are merged into at most 500 points without losing check-ins. |
+| `test_history_starts_at_the_first_check_in_not_at_a_bucket_boundary` | "History starts" is the first check-in, not the midnight or hour boundary of a summary, even after summarising. |
+| `test_when_old_samples_are_gone_the_oldest_summary_that_reaches_back_decides` | Once old samples are purged, the oldest hourly (then daily) summary that reaches further back gives the start. |
+| `test_a_series_with_no_data_has_no_start` | A series without data has no start. |
+| `test_a_container_series_has_cpu_memory_limit_and_down_counts` | A container series has CPU, memory, its limit and restarts (limit and restarts only with a maximum), and counts the check-ins when it wasn't running. |
+| `test_a_mount_series_is_picked_by_path` | A filesystem is chosen by its path, and other mounts are excluded. |
+| `test_container_summaries_agree_with_the_samples` *(2 inputs)* | Container history is identical before and after summarising. |
+| `test_the_endpoint_returns_the_series_for_a_range` | The server endpoint returns the requested metrics for a range. |
+| `test_without_a_range_the_original_answer_is_unchanged` | Without a range, the dashboard's original answer is unchanged. |
+| `test_unknown_ranges_and_metrics_are_refused` *(9 inputs)* | Unknown ranges, empty, doubled, oversized or odd metric lists are refused (422). |
+| `test_unknown_servers_mounts_and_containers_are_not_found` | Unknown servers, mounts and containers give 404; a relative mount path, a missing mount or an oversized id are refused. |
+| `test_the_container_and_mount_endpoints_return_their_series` | The container and mount endpoints return their series with names. |
+| `test_signing_in_is_required` | All the history endpoints refuse requests without a session. |
+
+### test_indexes.py
+
+The indexes on the metrics tables. The live migration is checked by hand against PostgreSQL
+(these tests use SQLite and check what the models declare).
+
+| Test | What it checks |
+|---|---|
+| `test_the_metrics_tables_carry_exactly_the_indexes_they_need` | Each metrics table has only the planned indexes: `time` for the purge, `(server_id, time)` (or `(server_id, container_id, time)` for containers) for reads. |
+| `test_deleting_a_server_and_purging_old_rows_can_both_use_an_index` *(4 inputs)* | Every metrics table keeps an index starting with `server_id` (deleting a server cascades through it) and one on `time` alone (the hourly purge). |
+| `test_a_primary_key_is_never_indexed_a_second_time` *(4 inputs)* | No table gets a redundant index on its own `id`. |
+
### test_network_checks.py
Network checks: the host's real traffic totals, which ports it listens on, whether it can
@@ -648,6 +699,46 @@ from machines or people you didn't invite.
| `test_a_bad_token_is_refused_even_in_open_mode` | Even in open mode, a token that is presented but wrong is refused. |
| `test_install_info_reports_the_configured_addresses` | **Connect a server** receives the configured VPN address and image names, and whether strict mode is on. |
+### test_rollups.py
+
+The background job that summarises check-in samples into hourly, daily, weekly (Monday to Sunday) and
+monthly rows, all in UTC. If these fail, long-range history could be wrong, duplicated, or deleted before
+it was kept.
+
+| Test | What it checks |
+|---|---|
+| `test_buckets_start_on_utc_boundaries_and_weeks_on_monday` | Hours, days and months start on the boundary and a week starts on Monday (Sunday 23:59 belongs to the week before the next Monday). |
+| `test_a_time_with_another_offset_is_converted_to_utc_first` | A time with another offset (or none) is converted to UTC before choosing its bucket. |
+| `test_the_following_bucket_handles_month_ends_and_leap_years` | The next month after December is January, February in a leap year ends correctly, and weeks cross new year. |
+| `test_an_hour_is_summarised_with_average_maximum_and_count` | An hour's row has the correct check-in count, average and maximum, and a metric that was never reported stays unknown (not zero). |
+| `test_a_metric_missing_from_some_samples_is_averaged_over_the_ones_that_have_it` | A metric missing from some check-ins is averaged over those that have it. |
+| `test_a_history_of_nothing_but_unknowns_still_counts_check_ins_and_does_not_break` | Check-ins that report nothing still produce rows with unknown averages and don't break the job. |
+| `test_only_completed_buckets_are_written_after_a_short_grace_period` | An hour is written only after it ended plus a five-minute grace, so check-ins in flight are not missed. |
+| `test_running_twice_changes_nothing` | A second run with nothing new builds nothing and leaves every row as it was. |
+| `test_rebuilding_a_bucket_replaces_it_instead_of_duplicating_it` | Rebuilding a bucket (for example after losing the watermark) replaces its rows and includes late samples. |
+| `test_servers_mounts_and_containers_are_summarised_separately` | Each server, mount and container gets its own rows. |
+| `test_down_samples_count_the_check_ins_where_a_container_was_not_running` | A container's row records how many check-ins it was not running. |
+| `test_a_day_is_the_samples_weighted_average_of_its_hours_and_keeps_the_maximum` | A day is the weighted average of its hours (one reading of 100 and three of 0 average 25, not 50) and keeps the highest maximum. |
+| `test_each_average_remembers_how_many_check_ins_reported_it` | Every averaged metric records how many check-ins reported it. |
+| `test_a_container_that_was_stopped_part_of_the_time_is_averaged_over_the_time_it_ran` | A container stopped for part of the day isn't averaged as if it were idle: its average is over the time it ran. |
+| `test_hours_that_never_reported_a_metric_do_not_pull_a_day_toward_zero` | Hours without a value for a metric don't drag a day's average toward zero. |
+| `test_a_week_runs_monday_to_sunday_and_is_written_only_when_sunday_has_ended` | A week covers Monday to Sunday, excludes the next Monday, and is written only once Sunday is over. |
+| `test_a_month_is_built_from_its_days_including_a_leap_february` | A month is built from its days, including the 29 days of a leap February. |
+| `test_data_either_side_of_new_year_lands_in_different_months_and_weeks` | New Year's Eve and New Year's Day land in different months but in the same week. |
+| `test_every_tier_is_built_for_mounts_and_containers_too` | Filesystems and containers get all four tiers, not just servers. |
+| `test_a_gap_stays_a_gap` | Hours, days or servers with no data get no row; nothing is filled in. |
+| `test_the_first_run_summarises_the_history_that_already_exists` | On upgrade, existing samples are summarised, and only complete weeks are written. |
+| `test_new_samples_are_picked_up_by_the_next_run` | Later runs continue from where the last one stopped. |
+| `test_after_downtime_the_job_catches_up_a_chunk_at_a_time` | After downtime the job catches up in limited chunks until nothing is left. |
+| `test_a_long_empty_stretch_cannot_stall_the_job` | A long stretch with no data at all still moves the job forward. |
+| `test_the_watermark_is_stored_per_tier_and_a_damaged_one_is_ignored` | The progress marker is saved per tier, and a damaged one makes the job start over harmlessly. |
+| `test_without_any_summaries_the_purge_behaves_as_before` | With no summaries yet, retention alone decides what the purge deletes (unchanged behaviour). |
+| `test_samples_that_were_not_summarised_yet_are_never_purged` | The purge never deletes a sample newer than the last summarised hour, even when retention says it is old enough. |
+| `test_maintenance_summarises_before_it_purges` | Maintenance summarises first, then purges, so history is kept before it is deleted. |
+| `test_a_failing_rollup_job_stops_the_purge` | If summarising fails, nothing is purged that run. |
+| `test_between_purges_maintenance_only_summarises` | Between hourly purges, maintenance only summarises. |
+| `test_deleting_a_server_deletes_its_summaries_but_not_another_servers` | Deleting a server removes its summaries and leaves other servers' alone. |
+
### test_rules.py
The alert thresholds, and the settings that control them.
@@ -675,6 +766,40 @@ The alert thresholds, and the settings that control them.
| `test_corrupt_mount_thresholds_are_ignored` | Damaged per-mount settings are ignored. |
| `test_overview_shows_latest_mounts_and_disk_thresholds` | The dashboard shows the latest filesystems and the thresholds used to colour them. |
+### test_stats.py
+
+The all-servers statistics and the busy-hours heatmap behind the Statistics page.
+
+| Test | What it checks |
+|---|---|
+| `test_rule_keys_become_readable_groups` *(15 inputs)* | Alert rule keys become readable groups ("Disk space", "Server offline"); unknown, empty or odd keys become "Other". |
+| `test_long_or_odd_rule_keys_never_leak_through` | A huge or unknown rule key is never shown as is. |
+| `test_incidents_are_counted_per_day_and_severity_for_a_week` | Incidents opened are counted per day and severity, and per kind of problem. |
+| `test_the_bucket_size_follows_the_range` | Hours up to a day, days up to a month, weeks up to a year and months beyond; weeks start on Monday. |
+| `test_incidents_older_than_the_range_are_left_out` | Only incidents that opened inside the range are counted. |
+| `test_mean_time_to_resolve_uses_only_resolved_incidents` | The average time to resolve counts resolved incidents only. |
+| `test_open_incidents_are_counted_by_severity_whatever_the_range` | Incidents open right now are counted by severity, however old they are. |
+| `test_an_install_without_incidents_gives_empty_answers` | No incidents gives empty, well-formed answers for every range. |
+| `test_an_unknown_severity_is_counted_as_info_not_dropped` | An unexpected severity is counted as info instead of being lost. |
+| `test_the_amount_of_data_read_is_capped_and_says_so` | Reading is capped and the answer says when it was cut. |
+| `test_a_server_that_always_reported_is_fully_available` | A server that reported every time is 100% available; one that never reported is unknown, not 0%. |
+| `test_a_server_that_was_silent_for_half_the_time_is_half_available` | A long silence lowers availability accordingly. |
+| `test_a_new_server_is_not_blamed_for_the_time_before_it_existed` | A new server is judged from its first check-in. |
+| `test_availability_is_exact_even_after_the_history_has_been_summarised` | Availability stays correct once history is summarised (a first version counted from midnight). |
+| `test_availability_never_exceeds_one_hundred_percent` | Reporting more often than expected still caps at 100%. |
+| `test_availability_works_for_every_range_without_data` | Every range works for servers without data. |
+| `test_the_endpoint_returns_incidents_and_availability` | The endpoint returns both parts. |
+| `test_answers_are_cached_for_a_minute_per_range_only` | Answers are cached for a minute, one entry per range. |
+| `test_a_broken_cache_does_not_break_the_page` | If Redis is down the page still works. |
+| `test_unknown_ranges_are_refused_and_never_become_cache_keys` *(5 inputs)* | Odd ranges are refused (422) and leave nothing in the cache. |
+| `test_signing_in_is_required` | The fleet endpoint needs a session. |
+| `test_the_heatmap_puts_each_hour_in_its_weekday_and_hour_cell` | Each hour lands in the right weekday and hour cell, averaged. |
+| `test_the_heatmap_follows_the_viewers_time_zone` | The viewer's time zone moves cells across midnight correctly. |
+| `test_the_heatmap_ignores_samples_older_than_thirty_days_and_unknowns` | Samples older than 30 days and missing readings are left out. |
+| `test_the_heatmap_is_the_same_before_and_after_summarising` | The heatmap is identical before and after the history is summarised. |
+| `test_the_heatmap_endpoint_validates_everything_and_caches_per_server_metric_and_offset` | Unknown metric, offset out of range or unknown server are refused and cache nothing new. |
+| `test_the_heatmap_endpoint_needs_a_session` | The heatmap endpoint needs a session. |
+
### test_system_checks.py
Reboots, OOM kills, kernel faults, clock sync and pending reboots, using only what the
@@ -732,6 +857,24 @@ log lines, user names), so most of these tests feed it hostile input.
| `test_helper_arriving_after_the_reboot_was_noticed_updates_the_event_and_alert` | If the helper finishes after the reboot was first noticed, the event and incident are updated to the right answer. |
| `test_invalid_last_shutdown_values_are_dropped` | An unexpected value for "last shutdown" is discarded. |
+## Frontend helper tests
+
+The chart helpers (axis ticks, gaps, histograms, the hour-by-weekday grid, donut slices, the
+"full in about N days" estimate, container and fleet rankings) are plain JavaScript with no
+dependencies, so they have tests that run with Node's built-in runner:
+
+```bash
+cd src/frontend && node --test # Node 20 or newer; nothing to install
+```
+
+| File | Tests | What they protect |
+|---|---|---|
+| `tests/chartMath.test.js` | 9 | Histogram bins and weights, the hour-by-weekday grid, donut fractions, the disk-full estimate (clear trends only), axis ticks, gaps staying gaps, weighted summaries, time labels and forecast wording. |
+| `tests/fleetLogic.test.js` | 7 | Container states, top-N rankings, disk ranking and colours, online detection, availability rows, readable durations, and incident columns that fill quiet days with zeros. |
+
+The React components themselves (the page layout, tooltips, selection) have no automated tests;
+they were checked by hand in a real browser.
+
## Adding a test
1. Put it in the file for the area it belongs to, or a new `tests/test_<area>.py`
diff --git a/src/backend/history.py b/src/backend/history.py
new file mode 100644
index 0000000..ef19774
--- /dev/null
+++ b/src/backend/history.py
@@ -0,0 +1,242 @@
+"""Range-aware history for charts: the last hour to all time, from the right level of detail.
+
+A request names a range (``1h`` ... ``all``). Short ranges are read from the check-in samples, longer ones from the
+hourly, weekly or monthly summaries (rollups.py). Whatever the source, the answer is a series of at most about
+``MAX_POINTS`` points, each with the average and maximum of every requested metric, so a year costs about as much
+as an hour. The newest period, which the rollup job has not closed yet, is built from the finer data below it, so
+the right-hand edge is always current. Periods without data have no point: a gap stays a gap, and the response
+says how long one point's period is (``bucket_seconds``) so a chart can tell a gap from a neighbour.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime, timedelta, timezone
+from typing import Any
+
+from sqlalchemy import func, select
+
+from . import rollups
+from .models import ContainerMetric, ContainerRollup, HostMetric, HostRollup, MountMetric, MountRollup
+
+DETAILED = "detailed"
+RANGES: dict[str, timedelta | None] = {
+ "1h": timedelta(hours=1), "6h": timedelta(hours=6), "24h": timedelta(hours=24), "7d": timedelta(days=7),
+ "30d": timedelta(days=30), "90d": timedelta(days=90), "1y": timedelta(days=365), "all": None,
+}
+# Which level of detail answers each range.
+SOURCE = {"1h": DETAILED, "6h": DETAILED, "24h": DETAILED, "7d": DETAILED,
+ "30d": rollups.HOUR, "90d": rollups.WEEK, "1y": rollups.WEEK, "all": rollups.MONTH}
+MAX_POINTS = 500
+NATURAL_SECONDS = {rollups.HOUR: 3600, rollups.DAY: 86400, rollups.WEEK: 7 * 86400, rollups.MONTH: 31 * 86400}
+# Bucket sizes a chart can be given; the smallest that keeps a range within MAX_POINTS is used.
+NICE_STEPS = (15, 30, 60, 120, 300, 600, 900, 1800, 3600, 7200, 14400, 21600, 43200, 86400, 7 * 86400)
+# The tiers a partly finished period of each tier is assembled from, coarsest first.
+FINER_CHAIN = {rollups.HOUR: (), rollups.DAY: (rollups.HOUR,), rollups.WEEK: (rollups.DAY, rollups.HOUR), rollups.MONTH: (rollups.DAY, rollups.HOUR)}
+
+
+@dataclass(frozen=True)
+class Metric:
+ """How one charted value is read from a raw row and from a rollup row. Either side may be absent."""
+
+ raw: str
+ avg: str | None = None # rollup column holding the average (its count column is n_<name>)
+ max: str | None = None
+
+
+def _m(raw, name, *, avg=True, maximum=True):
+ return Metric(raw, f"avg_{name}" if avg else None, f"max_{name}" if maximum else None)
+
+
+HOST_METRICS = {
+ "cpu": _m("cpu_utilization_pct", "cpu"), "load": _m("load_1m", "load"), "ram": _m("ram_pct", "ram"), "swap": _m("swap_pct", "swap"),
+ "disk": _m("disk_usage_pct", "disk"), "inodes": _m("disk_inodes_pct", "inodes"),
+ "docker_ms": _m("docker_latency_ms", "docker_ms"), "collect_ms": _m("collect_duration_ms", "collect_ms"),
+}
+MOUNT_METRICS = {
+ "used_pct": _m("used_pct", "used_pct"), "used_gb": _m("used_gb", "used_gb", maximum=False), "total_gb": _m("total_gb", "total_gb", avg=False),
+ "inodes_pct": _m("inodes_pct", "inodes_pct"),
+}
+CONTAINER_METRICS = {
+ "cpu": _m("cpu_pct", "cpu"), "memory_mb": _m("memory_used_mb", "memory_mb"), "memory_pct": _m("memory_pct", "memory_pct"),
+ "memory_limit_mb": _m("memory_limit_mb", "memory_limit_mb", avg=False), "restarts": _m("restart_count", "restarts", avg=False),
+}
+DEFAULT_HOST_METRICS = ("cpu", "ram", "swap", "load", "disk")
+
+
+@dataclass(frozen=True)
+class Series:
+ raw: Any
+ rollup: Any
+ metrics: dict[str, Metric]
+ has_down: bool = False
+
+
+HOST = Series(HostMetric, HostRollup, HOST_METRICS)
+MOUNT = Series(MountMetric, MountRollup, MOUNT_METRICS)
+CONTAINER = Series(ContainerMetric, ContainerRollup, CONTAINER_METRICS, has_down=True)
+
+
+class Agg:
+ """Sum, count and maximum of each metric over some check-ins; adding two of them gives the exact combined figures."""
+
+ __slots__ = ("samples", "down", "values")
+
+ def __init__(self, samples: int = 0, down: int = 0):
+ self.samples, self.down, self.values = samples, down, {}
+
+ def add_value(self, name: str, total: float | None, count: int, maximum: float | None) -> None:
+ current = self.values.setdefault(name, [0.0, 0, None])
+ if total is not None and count:
+ current[0] += total
+ current[1] += count
+ if maximum is not None and (current[2] is None or maximum > current[2]):
+ current[2] = maximum
+
+ def merge(self, other: "Agg") -> None:
+ self.samples += other.samples
+ self.down += other.down
+ for name, (total, count, maximum) in other.values.items():
+ self.add_value(name, total, count, maximum)
+
+
+def _raw_agg(spec: Series, wanted, row) -> Agg:
+ agg = Agg(samples=1)
+ for index, name in enumerate(wanted):
+ value = row[index + 1]
+ if value is not None:
+ agg.add_value(name, float(value), 1, float(value))
+ if spec.has_down:
+ agg.down = 0 if row[-1] == "running" else 1
+ return agg
+
+
+def _rollup_agg(spec: Series, wanted, row) -> Agg:
+ agg = Agg(samples=row.samples, down=getattr(row, "down_samples", 0) if spec.has_down else 0)
+ for name in wanted:
+ metric = spec.metrics[name]
+ average = getattr(row, metric.avg) if metric.avg else None
+ count = getattr(row, "n_" + metric.avg[len("avg_"):]) if metric.avg else 0
+ maximum = getattr(row, metric.max) if metric.max else None
+ agg.add_value(name, average * count if average is not None and count else None, count if average is not None else 0, maximum)
+ return agg
+
+
+def _point(spec: Series, wanted, start: datetime, agg: Agg) -> dict:
+ point: dict[str, Any] = {"time": start.isoformat(), "samples": agg.samples}
+ if spec.has_down:
+ point["down"] = agg.down
+ for name in wanted:
+ total, count, maximum = agg.values.get(name, (0.0, 0, None))
+ metric = spec.metrics[name]
+ value: dict[str, float] = {}
+ if metric.avg and count:
+ value["avg"] = total / count
+ if metric.max and maximum is not None:
+ value["max"] = maximum
+ point[name] = value or None
+ return point
+
+
+def nice_step(seconds: float) -> int:
+ return next((step for step in NICE_STEPS if step >= seconds), NICE_STEPS[-1])
+
+
+def _where(spec: Series, kind: str, server_id: int, key: str | None):
+ """Filters that pick one server's series (and one mount or container) from a raw or rollup table."""
+ table = spec.raw if kind == "raw" else spec.rollup
+ conditions = [table.server_id == server_id]
+ if spec is MOUNT:
+ conditions.append(table.mount == key)
+ elif spec is CONTAINER:
+ conditions.append(table.container_id == key)
+ return conditions
+
+
+def _raw_items(db, spec: Series, wanted, server_id, key, since: datetime, until: datetime):
+ columns = [getattr(spec.raw, spec.metrics[name].raw) for name in wanted]
+ if spec.has_down:
+ columns.append(spec.raw.status)
+ query = select(spec.raw.time, *columns).where(*_where(spec, "raw", server_id, key), spec.raw.time >= since, spec.raw.time < until).order_by(spec.raw.time)
+ return [(rollups._utc(row[0]), _raw_agg(spec, wanted, row)) for row in db.execute(query)]
+
+
+def _rollup_items(db, spec: Series, wanted, server_id, key, tier: str, since: datetime, until: datetime):
+ if until <= since:
+ return []
+ query = select(spec.rollup).where(*_where(spec, "rollup", server_id, key), spec.rollup.tier == tier,
+ spec.rollup.bucket_start >= since, spec.rollup.bucket_start < until).order_by(spec.rollup.bucket_start)
+ return [(rollups._utc(row.bucket_start), _rollup_agg(spec, wanted, row)) for row in db.scalars(query)]
+
+
+def data_since(db, spec: Series, server_id: int, key: str | None) -> datetime | None:
+ """The earliest moment this series has data: the first sample if it is still stored, else the start of the
+ oldest summary that reaches further back than anything finer.
+
+ A summary's bucket starts on a boundary (midnight, Monday), which can be long before the first check-in, so a
+ coarser tier only counts when its oldest bucket ends before the finer data begins; otherwise the finer data
+ already says exactly when the history starts.
+ """
+ first = db.execute(select(func.min(spec.raw.time)).where(*_where(spec, "raw", server_id, key))).scalar()
+ since = rollups._utc(first) if first else None
+ for tier in (rollups.HOUR, rollups.DAY, rollups.WEEK, rollups.MONTH):
+ oldest = db.execute(select(func.min(spec.rollup.bucket_start)).where(*_where(spec, "rollup", server_id, key), spec.rollup.tier == tier)).scalar()
+ if oldest is None:
+ continue
+ oldest = rollups._utc(oldest)
+ if since is None or rollups.following(tier, oldest) <= since:
+ since = oldest
+ return since
+
+
+def _group(items, bucket_of):
+ grouped: dict[datetime, Agg] = {}
+ for start, agg in items:
+ grouped.setdefault(bucket_of(start), Agg()).merge(agg)
+ return grouped
+
+
+def _tiered_items(db, spec, wanted, server_id, key, tier: str, window_start: datetime, now: datetime):
+ """One Agg per bucket of ``tier``: closed buckets from the rollups, the newest ones from the finer data."""
+ first = rollups.floor(tier, window_start)
+ done = rollups._read_watermark(db, tier)
+ closed_until = min(done, now) if done and done > first else first
+ items = _rollup_items(db, spec, wanted, server_id, key, tier, first, closed_until)
+ grouped = {start: agg for start, agg in items}
+ tail: list = []
+ cursor = closed_until
+ for finer in FINER_CHAIN[tier]:
+ finer_done = rollups._read_watermark(db, finer)
+ if finer_done and finer_done > cursor:
+ tail += _rollup_items(db, spec, wanted, server_id, key, finer, cursor, min(finer_done, now))
+ cursor = min(finer_done, now)
+ tail += _raw_items(db, spec, wanted, server_id, key, cursor, now + timedelta(seconds=1))
+ for start, agg in _group(tail, lambda moment: rollups.floor(tier, moment)).items():
+ grouped.setdefault(start, Agg()).merge(agg)
+ return sorted(grouped.items()), NATURAL_SECONDS[tier]
+
+
+def history(db, spec: Series, server_id: int, key: str | None, range_name: str, wanted, interval_seconds: int, now: datetime | None = None) -> dict:
+ """The chart series for one server (and mount or container) over ``range_name``."""
+ now = rollups._utc(now or datetime.now(timezone.utc))
+ since = data_since(db, spec, server_id, key)
+ length = RANGES[range_name]
+ window_start = max(now - length, since) if length and since else (since or now - (length or timedelta(0)))
+ source = SOURCE[range_name]
+ if since is None:
+ items, natural = [], NATURAL_SECONDS.get(source, interval_seconds)
+ elif source == DETAILED:
+ items, natural = _raw_items(db, spec, wanted, server_id, key, window_start, now + timedelta(seconds=1)), interval_seconds
+ else:
+ items, natural = _tiered_items(db, spec, wanted, server_id, key, source, window_start, now)
+ span = max((now - window_start).total_seconds(), 1)
+ step = max(nice_step(max(span / MAX_POINTS, interval_seconds if source == DETAILED else 0)), natural if source != DETAILED else 0)
+ if source == DETAILED or step > natural:
+ epoch = lambda moment: datetime.fromtimestamp(int(moment.timestamp()) // step * step, tz=timezone.utc) # noqa: E731
+ items = sorted(_group(items, epoch).items())
+ else:
+ step = natural
+ return {
+ "range": range_name, "source": source, "bucket_seconds": step,
+ "start": window_start.isoformat(), "end": now.isoformat(), "data_since": since.isoformat() if since else None,
+ "metrics": list(wanted), "points": [_point(spec, wanted, start, agg) for start, agg in items],
+ }
diff --git a/src/backend/main.py b/src/backend/main.py
index 6aba15f..34c9fcf 100644
--- a/src/backend/main.py
+++ b/src/backend/main.py
@@ -10,25 +10,26 @@ import threading
import time
from pathlib import Path
from datetime import datetime, timedelta, timezone
-from typing import Annotated
+from typing import Annotated, Literal
from urllib.parse import urlsplit
import redis
from fastapi import Cookie, Depends, FastAPI, Header, HTTPException, Query, Request, Response, status
+from fastapi import Path as PathParam
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse, PlainTextResponse
from pydantic import BaseModel, EmailStr, Field
from sqlalchemy import case, func, inspect, text
from .alerts import CONTAINER_RULE_PREFIX, CRITICAL, SERVER_OFFLINE_RULE, SEVERITIES, active_rule_keys, raise_alert, resolve_alerts, sync_alerts
-from . import secrets_box, totp
+from . import history as history_service, rollups, secrets_box, stats, totp
from .auth import (
STAGE_ENROLLMENT, STAGE_FULL, count_pending_attempt, create_pending_two_factor, create_session, delete_pending_two_factor, delete_session, get_recent_logs,
get_session, get_user_from_session, hash_password, issue_recovery_codes, list_user_sessions, read_pending_two_factor, recovery_codes_left, record_audit_log,
register_user, revoke_other_sessions, revoke_user_session, validate_password, verify_credentials, verify_password, verify_second_factor,
)
from .db import SessionLocal, check_database, get_db
-from .models import AppSetting, AuditLog, Base, ChatMessage, ContainerMetric, Conversation, EnrollmentKey, FirewallState, HostMetric, IncidentAlert, JoinToken, ListenerState, MountMetric, PendingCollector, RecoveryCode, Server, SystemEvent, User
+from .models import AppSetting, AuditLog, Base, ChatMessage, ContainerMetric, ContainerRollup, Conversation, EnrollmentKey, FirewallState, HostMetric, HostRollup, IncidentAlert, JoinToken, ListenerState, MountMetric, MountRollup, PendingCollector, RecoveryCode, Server, SystemEvent, User
from .db import engine
from .config import (
@@ -229,6 +230,22 @@ def startup_event():
connection.execute(text("ALTER TABLE system_events ADD COLUMN IF NOT EXISTS event_key VARCHAR(128)"))
connection.execute(text("ALTER TABLE pending_collectors ADD COLUMN IF NOT EXISTS join_token_id INTEGER REFERENCES join_tokens(id) ON DELETE SET NULL"))
connection.execute(text("CREATE INDEX IF NOT EXISTS ix_system_events_event_key ON system_events (event_key)"))
+ # Composite indexes that match how the metrics tables are read, then the single-column ones they make
+ # redundant (and the pointless indexes on each table's own primary key). Safe to repeat. Creating an index
+ # briefly blocks writes to that table; on a very large table create them first by hand with CONCURRENTLY.
+ for table, name, columns in (
+ ("host_metrics", "ix_host_metrics_server_time", "server_id, time"),
+ ("mount_metrics", "ix_mount_metrics_server_time", "server_id, time"),
+ ("system_events", "ix_system_events_server_time", "server_id, time"),
+ ("container_metrics", "ix_container_metrics_server_container_time", "server_id, container_id, time"),
+ ):
+ connection.execute(text(f"CREATE INDEX IF NOT EXISTS {name} ON {table} ({columns})"))
+ for redundant in (
+ "ix_host_metrics_id", "ix_host_metrics_server_id", "ix_mount_metrics_id", "ix_mount_metrics_server_id",
+ "ix_system_events_id", "ix_system_events_server_id",
+ "ix_container_metrics_id", "ix_container_metrics_server_id", "ix_container_metrics_container_id",
+ ):
+ connection.execute(text(f"DROP INDEX IF EXISTS {redundant}"))
for column, column_type in (("totp_secret_enc", "TEXT"), ("totp_enabled", "BOOLEAN NOT NULL DEFAULT FALSE"), ("totp_last_step", "INTEGER"), ("password_changed_at", "TIMESTAMP WITH TIME ZONE")):
connection.execute(text(f"ALTER TABLE users ADD COLUMN IF NOT EXISTS {column} {column_type}"))
connection.execute(text("ALTER TABLE incident_alerts ADD COLUMN IF NOT EXISTS rule_key VARCHAR(255)"))
@@ -724,6 +741,10 @@ def _admin_settings(db) -> dict:
def purge_old_metrics(db, now: datetime | None = None) -> int:
"""Delete host and container samples older than the retention period."""
cutoff = (now or datetime.now(timezone.utc)) - timedelta(days=get_metrics_retention_days(db))
+ # Check-in samples that haven't been summarised yet are never deleted, whatever the retention says.
+ summarised_until = rollups.hour_watermark(db)
+ if summarised_until is not None:
+ cutoff = min(cutoff, summarised_until)
deleted = db.query(HostMetric).filter(HostMetric.time < cutoff).delete(synchronize_session=False)
deleted += db.query(ContainerMetric).filter(ContainerMetric.time < cutoff).delete(synchronize_session=False)
deleted += db.query(MountMetric).filter(MountMetric.time < cutoff).delete(synchronize_session=False)
@@ -914,17 +935,132 @@ def latest_containers(db, server_id: int, host_time: datetime | None) -> list[Co
HISTORY_MAX_MINUTES = 24 * 60
+RangeName = Literal["1h", "6h", "24h", "7d", "30d", "90d", "1y", "all"]
+
+
+def _wanted_metrics(raw: str | None, allowed: dict, default: tuple[str, ...]) -> tuple[str, ...]:
+ """The metrics a history request asked for, from a comma-separated list; only known names are accepted."""
+ if raw is None:
+ return default
+ names = tuple(dict.fromkeys(name.strip() for name in raw.split(",")))
+ if not names or len(raw) > 200 or any(name not in allowed for name in names):
+ raise HTTPException(status_code=422, detail=f"metrics must be a comma-separated list of: {', '.join(allowed)}.")
+ return names
+
+
+def _history_response(db, spec, server: Server, key: str | None, range_name: str, wanted: tuple[str, ...]) -> dict:
+ result = history_service.history(db, spec, server.id, key, range_name, wanted, server.check_interval_seconds)
+ return {"server_id": server.id, **result}
+
+
@app.get("/monitoring/servers/{server_id}/history")
-def server_history(server_id: int, minutes: int = 30, session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
- """CPU, memory and root disk usage of one server over the last ``minutes``, oldest first."""
+def server_history(
+ server_id: int, minutes: int = 30, range_name: Annotated[RangeName | None, Query(alias="range")] = None, metrics: str | None = None,
+ session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db),
+):
+ """A server's metrics over time.
+
+ With ``range`` (1h to all): at most about 500 points from the right level of detail, each with the average and
+ maximum of the requested ``metrics``. Without it: the original answer for the last ``minutes`` (CPU, memory and
+ root disk, oldest first), which the dashboard's live chart uses.
+ """
_current_user(session_id, db)
- if not db.query(Server).filter(Server.id == server_id).first():
+ server = db.query(Server).filter(Server.id == server_id).first()
+ if not server:
raise HTTPException(status_code=404, detail="Server not found.")
+ if range_name is not None:
+ return _history_response(db, history_service.HOST, server, None, range_name, _wanted_metrics(metrics, history_service.HOST_METRICS, history_service.DEFAULT_HOST_METRICS))
since = datetime.now(timezone.utc) - timedelta(minutes=max(1, min(minutes, HISTORY_MAX_MINUTES)))
rows = db.query(HostMetric.time, HostMetric.cpu_utilization_pct, HostMetric.ram_pct, HostMetric.disk_usage_pct).filter(HostMetric.server_id == server_id, HostMetric.time >= since).order_by(HostMetric.time).all()
return {"server_id": server_id, "samples": [{"time": row.time.isoformat(), "cpu": row.cpu_utilization_pct, "ram": row.ram_pct, "disk": row.disk_usage_pct} for row in rows]}
+STATS_CACHE_SECONDS = 60
+
+
[email protected]("/stats/fleet")
+def fleet_statistics(range_name: Annotated[RangeName, Query(alias="range")] = "24h", session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db)):
+ """Incidents over time and per-server availability for the Statistics page's all-servers view."""
+ _current_user(session_id, db)
+ # Keyed only by the validated range, so a request can't create an unbounded number of cache entries.
+ cache_key = f"homelab:stats:fleet:{range_name}"
+ try:
+ cached = redis_client.get(cache_key)
+ if cached:
+ return json.loads(cached)
+ except Exception: # a cache problem must never break the page
+ cached = None
+ result = stats.fleet_stats(db, range_name)
+ try:
+ redis_client.set(cache_key, json.dumps(result), ex=STATS_CACHE_SECONDS)
+ except Exception:
+ pass
+ return result
+
+
[email protected]("/stats/servers/{server_id}/heatmap")
+def server_heatmap(
+ server_id: int, metric: Literal["cpu", "ram"] = "cpu", offset: Annotated[int, Query(ge=-stats.MAX_OFFSET_MINUTES, le=stats.MAX_OFFSET_MINUTES)] = 0,
+ session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db),
+):
+ """When a server is busy: its average per hour of the day and weekday over the last 30 days."""
+ _current_user(session_id, db)
+ server = db.query(Server).filter(Server.id == server_id).first()
+ if not server:
+ raise HTTPException(status_code=404, detail="Server not found.")
+ # Only existing servers, known metrics and a bounded offset get this far, so the cache can't grow without limit.
+ cache_key = f"homelab:stats:heatmap:{server.id}:{metric}:{offset}"
+ try:
+ cached = redis_client.get(cache_key)
+ if cached:
+ return json.loads(cached)
+ except Exception:
+ pass
+ result = stats.hour_weekday_heatmap(db, server, metric, offset)
+ try:
+ redis_client.set(cache_key, json.dumps(result), ex=STATS_CACHE_SECONDS)
+ except Exception:
+ pass
+ return result
+
+
[email protected]("/monitoring/servers/{server_id}/mounts/history")
+def mount_history(
+ server_id: int, mount: Annotated[str, Query(min_length=1, max_length=255)], range_name: Annotated[RangeName, Query(alias="range")] = "24h", metrics: str | None = None,
+ session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db),
+):
+ """One filesystem's usage over time (``mount`` is its path, for example ``/data``)."""
+ _current_user(session_id, db)
+ server = db.query(Server).filter(Server.id == server_id).first()
+ if not server or not mount.startswith("/"):
+ raise HTTPException(status_code=404, detail="Server or mount not found.")
+ wanted = _wanted_metrics(metrics, history_service.MOUNT_METRICS, ("used_pct", "used_gb", "total_gb"))
+ result = _history_response(db, history_service.MOUNT, server, mount, range_name, wanted)
+ if result["data_since"] is None:
+ raise HTTPException(status_code=404, detail="Server or mount not found.")
+ return {**result, "mount": mount}
+
+
[email protected]("/monitoring/servers/{server_id}/containers/{container_id}/history")
+def container_history(
+ server_id: int, container_id: Annotated[str, PathParam(min_length=1, max_length=128)], range_name: Annotated[RangeName, Query(alias="range")] = "24h", metrics: str | None = None,
+ session_id: Annotated[str | None, Cookie(alias=SESSION_COOKIE_NAME)] = None, db=Depends(get_db),
+):
+ """One container's CPU and memory over time."""
+ _current_user(session_id, db)
+ server = db.query(Server).filter(Server.id == server_id).first()
+ if not server:
+ raise HTTPException(status_code=404, detail="Server or container not found.")
+ wanted = _wanted_metrics(metrics, history_service.CONTAINER_METRICS, ("cpu", "memory_mb", "memory_pct", "memory_limit_mb"))
+ result = _history_response(db, history_service.CONTAINER, server, container_id, range_name, wanted)
+ if result["data_since"] is None:
+ raise HTTPException(status_code=404, detail="Server or container not found.")
+ name = db.query(ContainerMetric.name).filter(ContainerMetric.server_id == server.id, ContainerMetric.container_id == container_id).order_by(ContainerMetric.time.desc()).limit(1).scalar()
+ if name is None:
+ name = db.query(ContainerRollup.name).filter(ContainerRollup.server_id == server.id, ContainerRollup.container_id == container_id).order_by(ContainerRollup.bucket_start.desc()).limit(1).scalar()
+ return {**result, "container_id": container_id, "name": name}
+
+
CONTAINER_DETAIL_FIELDS = (
"status", "healthcheck_status", "restart_count", "cpu_pct", "memory_used_mb", "memory_limit_mb", "memory_pct",
"net_rx_mb", "net_tx_mb", "block_read_mb", "block_write_mb", "pids",
@@ -1233,6 +1369,24 @@ def analyze_memory_trends(db, now: datetime | None = None) -> None:
db.commit()
+def metrics_maintenance(db, purge_due: bool) -> bool:
+ """Summarise newly completed periods, then (when due) purge and analyse. True when the purge ran.
+
+ Summaries come first and the purge is skipped if they fail, so history is never deleted before it was kept.
+ """
+ try:
+ rollups.run_rollups(db)
+ except Exception as exc:
+ db.rollback()
+ print(f"Rollups failed: {type(exc).__name__}: {exc}", flush=True)
+ return False
+ if not purge_due:
+ return False
+ purge_old_metrics(db)
+ analyze_memory_trends(db)
+ return True
+
+
def _offline_watch_loop() -> None:
last_metrics_purge: float | None = None
while True:
@@ -1241,9 +1395,8 @@ def _offline_watch_loop() -> None:
try:
mark_offline_servers(db)
purge_stale_pending(db)
- if last_metrics_purge is None or time.monotonic() - last_metrics_purge >= METRICS_PURGE_INTERVAL_SECONDS:
- purge_old_metrics(db)
- analyze_memory_trends(db)
+ purge_due = last_metrics_purge is None or time.monotonic() - last_metrics_purge >= METRICS_PURGE_INTERVAL_SECONDS
+ if metrics_maintenance(db, purge_due):
last_metrics_purge = time.monotonic()
except Exception as exc: # keep the watcher alive through transient DB errors
print(f"Offline watch failed: {type(exc).__name__}: {exc}", flush=True)
@@ -1270,6 +1423,8 @@ def delete_monitored_server(server_id: int, session_id: Annotated[str | None, Co
if not server:
raise HTTPException(status_code=404, detail="Server not found.")
+ for rollup in (HostRollup, MountRollup, ContainerRollup):
+ db.query(rollup).filter(rollup.server_id == server.id).delete(synchronize_session=False)
db.query(HostMetric).filter(HostMetric.server_id == server.id).delete(synchronize_session=False)
db.query(ContainerMetric).filter(ContainerMetric.server_id == server.id).delete(synchronize_session=False)
db.query(MountMetric).filter(MountMetric.server_id == server.id).delete(synchronize_session=False)
diff --git a/src/backend/models.py b/src/backend/models.py
index 0c150ea..50a44b0 100644
--- a/src/backend/models.py
+++ b/src/backend/models.py
@@ -1,7 +1,7 @@
from datetime import datetime, timezone
from typing import Optional
-from sqlalchemy import Boolean, DateTime, Float, ForeignKey, Integer, String, Text, UniqueConstraint
+from sqlalchemy import Boolean, DateTime, Float, ForeignKey, Index, Integer, String, Text, UniqueConstraint
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
@@ -172,8 +172,11 @@ class EnrollmentKey(Base):
class HostMetric(Base):
__tablename__ = "host_metrics"
- id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
- server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)
+ # Reads are "this server, newest first", so (server_id, time) serves them; time alone serves the purge.
+ __table_args__ = (Index("ix_host_metrics_server_time", "server_id", "time"),)
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True)
+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
time: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)
cpu_utilization_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
load_1m: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
@@ -213,8 +216,10 @@ class SystemEvent(Base):
__tablename__ = "system_events"
- id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
- server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)
+ __table_args__ = (Index("ix_system_events_server_time", "server_id", "time"),)
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True)
+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
time: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)
kind: Mapped[str] = mapped_column(String(32), nullable=False) # reboot | oom_kill | kernel_fault | kernel_error
# Identifies the occurrence so a snapshot that repeats it doesn't record it twice.
@@ -227,8 +232,10 @@ class MountMetric(Base):
__tablename__ = "mount_metrics"
- id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
- server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)
+ __table_args__ = (Index("ix_mount_metrics_server_time", "server_id", "time"),)
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True)
+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
time: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)
mount: Mapped[str] = mapped_column(String(255), nullable=False)
fs_type: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
@@ -271,9 +278,13 @@ class FirewallState(Base):
class ContainerMetric(Base):
__tablename__ = "container_metrics"
- id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
- server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), index=True, nullable=False)
- container_id: Mapped[str] = mapped_column(String(128), index=True, nullable=False)
+ # One container's history is "this server, this container, newest first". time alone serves the purge and the
+ # "containers at one check-in" lookup, where an exact timestamp already identifies the check-in.
+ __table_args__ = (Index("ix_container_metrics_server_container_time", "server_id", "container_id", "time"),)
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True)
+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
+ container_id: Mapped[str] = mapped_column(String(128), nullable=False)
name: Mapped[str] = mapped_column(String(255), nullable=False)
image: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
status: Mapped[str] = mapped_column(String(64), nullable=False)
@@ -311,3 +322,97 @@ class IncidentAlert(Base):
rule_key: Mapped[Optional[str]] = mapped_column(String(255), index=True, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True, nullable=False)
resolved_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
+
+
+class HostRollup(Base):
+ """A summary of one server's check-ins over an hour, day, week or month (UTC). See rollups.py.
+
+ Every metric has an average and a maximum. ``samples`` is how many check-ins went into the row; ``n_<metric>`` is
+ how many of them reported that metric, which is the weight when a coarser tier averages this one. A metric the
+ collector never reported is None.
+ """
+
+ __tablename__ = "host_rollups"
+ # The unique key serves one server's history; (tier, bucket_start) serves building and purging a tier.
+ __table_args__ = (UniqueConstraint("server_id", "tier", "bucket_start"), Index("ix_host_rollups_tier_bucket", "tier", "bucket_start"))
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True)
+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
+ tier: Mapped[str] = mapped_column(String(8), nullable=False) # hour | day | week | month
+ bucket_start: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+ samples: Mapped[int] = mapped_column(Integer, nullable=False)
+ avg_cpu: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_cpu: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_cpu: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_load: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_load: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_load: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_ram: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_ram: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_ram: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_swap: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_swap: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_swap: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_disk: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_disk: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_disk: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_inodes: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_inodes: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_inodes: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_docker_ms: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_docker_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_docker_ms: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_collect_ms: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_collect_ms: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_collect_ms: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+
+
+class MountRollup(Base):
+ """A summary of one filesystem's usage over an hour, day, week or month (UTC)."""
+
+ __tablename__ = "mount_rollups"
+ __table_args__ = (UniqueConstraint("server_id", "mount", "tier", "bucket_start"), Index("ix_mount_rollups_tier_bucket", "tier", "bucket_start"))
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True)
+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
+ mount: Mapped[str] = mapped_column(String(255), nullable=False)
+ tier: Mapped[str] = mapped_column(String(8), nullable=False)
+ bucket_start: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+ samples: Mapped[int] = mapped_column(Integer, nullable=False)
+ avg_used_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_used_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_used_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_used_gb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_used_gb: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_total_gb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_inodes_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_inodes_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_inodes_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+
+
+class ContainerRollup(Base):
+ """A summary of one container over an hour, day, week or month (UTC)."""
+
+ __tablename__ = "container_rollups"
+ __table_args__ = (UniqueConstraint("server_id", "container_id", "tier", "bucket_start"), Index("ix_container_rollups_tier_bucket", "tier", "bucket_start"))
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True)
+ server_id: Mapped[int] = mapped_column(ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
+ container_id: Mapped[str] = mapped_column(String(128), nullable=False)
+ name: Mapped[str] = mapped_column(String(255), nullable=False)
+ tier: Mapped[str] = mapped_column(String(8), nullable=False)
+ bucket_start: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+ samples: Mapped[int] = mapped_column(Integer, nullable=False)
+ # Check-ins in which the container was not running; the rest of ``samples`` it was.
+ down_samples: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ avg_cpu: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_cpu: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_cpu: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_memory_mb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_memory_mb: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_memory_mb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ avg_memory_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ n_memory_pct: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+ max_memory_pct: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ max_memory_limit_mb: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
+ max_restarts: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
diff --git a/src/backend/rollups.py b/src/backend/rollups.py
new file mode 100644
index 0000000..f09e534
--- /dev/null
+++ b/src/backend/rollups.py
@@ -0,0 +1,215 @@
+"""Summarised history: hourly, daily, weekly and monthly rollups of the metrics tables.
+
+Charts and statistics over weeks, months and years can't read every per-minute row, and keeping those rows for a
+year isn't realistic. A background job therefore turns them into one summary row per bucket (average and maximum
+per metric). Each tier is built from the finer one *when its period closes*:
+
+ check-in samples -> hour -> day -> week (Monday to Sunday) -> month
+
+All buckets are aligned to UTC. Rules the job keeps:
+
+* Only complete buckets are written, a few minutes after they end so that check-ins in flight are not missed.
+* A bucket is written by deleting whatever was there and inserting it again in one transaction, so running the job
+ twice, or again after a failure, never duplicates a row.
+* A watermark per tier (``rollup_done_<tier>`` in ``app_settings``) records how far each tier got. A stretch with no
+ data advances it too, so a long gap can't make the job stand still. After downtime the job catches up, a limited
+ number of buckets per run.
+* A period in which a server sent nothing has no row for it. A gap stays a gap; nothing is filled in.
+* Averages are exact. Each row stores, per averaged metric, how many check-ins reported it (``n_<metric>``), and a
+ coarser tier weights by that, so a day with few check-ins, or a container that was stopped (and so reported no
+ CPU) for half of it, counts for what it really measured. The maximum is kept so a short spike stays visible in
+ a yearly view. A metric that was never reported stays None.
+
+Deleting old rows is a separate matter (see ``purge_old_metrics`` in main.py); the purge only ever removes
+check-in samples that have already been summarised.
+"""
+from __future__ import annotations
+
+from datetime import datetime, timedelta, timezone
+
+from sqlalchemy import Float, case, cast, delete, func, select
+
+from .models import AppSetting, ContainerMetric, ContainerRollup, HostMetric, HostRollup, MountMetric, MountRollup
+
+HOUR, DAY, WEEK, MONTH = "hour", "day", "week", "month"
+TIERS = (HOUR, DAY, WEEK, MONTH)
+FINER = {DAY: HOUR, WEEK: DAY, MONTH: DAY}
+GRACE = timedelta(minutes=5)
+# Per run, so catching up after a long outage (or the first run on a big database) works in chunks.
+MAX_BUCKETS_PER_RUN = {HOUR: 400, DAY: 60, WEEK: 30, MONTH: 12}
+WATERMARK_PREFIX = "rollup_done_"
+
+# (rollup column, aggregate, source column on the raw table). "avg" is an average of the raw values, or a weighted
+# average of the finer tier's averages (weighted by n_<metric>); "max" is a maximum either way.
+HOST_COLUMNS = (
+ ("avg_cpu", "avg", "cpu_utilization_pct"), ("max_cpu", "max", "cpu_utilization_pct"),
+ ("avg_load", "avg", "load_1m"), ("max_load", "max", "load_1m"),
+ ("avg_ram", "avg", "ram_pct"), ("max_ram", "max", "ram_pct"),
+ ("avg_swap", "avg", "swap_pct"), ("max_swap", "max", "swap_pct"),
+ ("avg_disk", "avg", "disk_usage_pct"), ("max_disk", "max", "disk_usage_pct"),
+ ("avg_inodes", "avg", "disk_inodes_pct"), ("max_inodes", "max", "disk_inodes_pct"),
+ ("avg_docker_ms", "avg", "docker_latency_ms"), ("max_docker_ms", "max", "docker_latency_ms"),
+ ("avg_collect_ms", "avg", "collect_duration_ms"), ("max_collect_ms", "max", "collect_duration_ms"),
+)
+MOUNT_COLUMNS = (
+ ("avg_used_pct", "avg", "used_pct"), ("max_used_pct", "max", "used_pct"),
+ ("avg_used_gb", "avg", "used_gb"), ("max_total_gb", "max", "total_gb"),
+ ("avg_inodes_pct", "avg", "inodes_pct"), ("max_inodes_pct", "max", "inodes_pct"),
+)
+CONTAINER_COLUMNS = (
+ ("avg_cpu", "avg", "cpu_pct"), ("max_cpu", "max", "cpu_pct"),
+ ("avg_memory_mb", "avg", "memory_used_mb"), ("max_memory_mb", "max", "memory_used_mb"),
+ ("avg_memory_pct", "avg", "memory_pct"), ("max_memory_pct", "max", "memory_pct"),
+ ("max_memory_limit_mb", "max", "memory_limit_mb"), ("max_restarts", "max", "restart_count"),
+)
+# What each kind of rollup groups by and builds from.
+KINDS = (
+ (HostRollup, HostMetric, ("server_id",), HOST_COLUMNS),
+ (MountRollup, MountMetric, ("server_id", "mount"), MOUNT_COLUMNS),
+ (ContainerRollup, ContainerMetric, ("server_id", "container_id"), CONTAINER_COLUMNS),
+)
+
+
+def _utc(value: datetime) -> datetime:
+ """SQLite hands back naive datetimes; everything stored here is UTC."""
+ return value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value.astimezone(timezone.utc)
+
+
+def floor(tier: str, moment: datetime) -> datetime:
+ """The start of the bucket that contains ``moment``."""
+ moment = _utc(moment)
+ if tier == HOUR:
+ return moment.replace(minute=0, second=0, microsecond=0)
+ day = moment.replace(hour=0, minute=0, second=0, microsecond=0)
+ if tier == DAY:
+ return day
+ if tier == WEEK:
+ return day - timedelta(days=day.weekday()) # Monday
+ if tier == MONTH:
+ return day.replace(day=1)
+ raise ValueError(f"Unknown tier: {tier}")
+
+
+def following(tier: str, start: datetime) -> datetime:
+ """The start of the bucket after the one beginning at ``start``."""
+ if tier == HOUR:
+ return start + timedelta(hours=1)
+ if tier == DAY:
+ return start + timedelta(days=1)
+ if tier == WEEK:
+ return start + timedelta(days=7)
+ if tier == MONTH:
+ return start.replace(year=start.year + 1, month=1) if start.month == 12 else start.replace(month=start.month + 1)
+ raise ValueError(f"Unknown tier: {tier}")
+
+
+def _read_watermark(db, tier: str) -> datetime | None:
+ setting = db.get(AppSetting, WATERMARK_PREFIX + tier)
+ try:
+ return _utc(datetime.fromisoformat(setting.value)) if setting else None
+ except ValueError:
+ return None
+
+
+def _write_watermark(db, tier: str, value: datetime) -> None:
+ setting = db.get(AppSetting, WATERMARK_PREFIX + tier)
+ if setting:
+ setting.value = value.isoformat()
+ else:
+ db.add(AppSetting(key=WATERMARK_PREFIX + tier, value=value.isoformat()))
+
+
+def hour_watermark(db) -> datetime | None:
+ """Check-in samples before this moment have been summarised and may be purged."""
+ return _read_watermark(db, HOUR)
+
+
+def _aggregates(source, columns, from_raw: bool):
+ expressions = []
+ for name, kind, raw in columns:
+ if kind == "max":
+ expressions.append(func.max(getattr(source, raw if from_raw else name)).label(name))
+ continue
+ count_name = "n_" + name[len("avg_"):]
+ if from_raw:
+ expressions.append(func.avg(getattr(source, raw)).label(name))
+ expressions.append(func.count(getattr(source, raw)).label(count_name))
+ else:
+ value, count = getattr(source, name), getattr(source, count_name)
+ weighted = func.sum(case((value.is_not(None), value * count), else_=0))
+ expressions.append((cast(weighted, Float) / func.nullif(func.sum(case((value.is_not(None), count), else_=0)), 0)).label(name))
+ expressions.append(func.sum(count).label(count_name))
+ return expressions
+
+
+def _build_bucket(db, tier: str, start: datetime, end: datetime) -> int:
+ """Replace every rollup row of ``tier`` that begins at ``start`` and return how many were written."""
+ written = 0
+ for rollup, raw, keys, columns in KINDS:
+ db.execute(delete(rollup).where(rollup.tier == tier, rollup.bucket_start == start))
+ from_raw = tier == HOUR
+ source = raw if from_raw else rollup
+ key_columns = [getattr(source, key) for key in keys]
+ extra = []
+ if rollup is ContainerRollup:
+ extra.append(func.max(source.name).label("name"))
+ if from_raw:
+ samples = func.count().label("samples")
+ if rollup is ContainerRollup:
+ extra.append(func.coalesce(func.sum(case((ContainerMetric.status != "running", 1), else_=0)), 0).label("down_samples"))
+ window = (source.time >= start, source.time < end)
+ else:
+ samples = func.sum(source.samples).label("samples")
+ if rollup is ContainerRollup:
+ extra.append(func.coalesce(func.sum(source.down_samples), 0).label("down_samples"))
+ window = (source.tier == FINER[tier], source.bucket_start >= start, source.bucket_start < end)
+ query = select(*key_columns, samples, *extra, *_aggregates(source, columns, from_raw)).where(*window).group_by(*key_columns)
+ for row in db.execute(query):
+ values = row._asdict()
+ db.add(rollup(tier=tier, bucket_start=start, **values))
+ written += 1
+ return written
+
+
+def _first_source_time(db, tier: str) -> datetime | None:
+ """The earliest row the tier can be built from, in any of the tables it summarises."""
+ if tier == HOUR:
+ found = [db.execute(select(func.min(raw.time))).scalar() for _, raw, _, _ in KINDS]
+ else:
+ found = [db.execute(select(func.min(rollup.bucket_start)).where(rollup.tier == FINER[tier])).scalar() for rollup, _, _, _ in KINDS]
+ found = [_utc(value) for value in found if value]
+ return min(found) if found else None
+
+
+def _run_tier(db, tier: str, now: datetime) -> int:
+ cursor = _read_watermark(db, tier)
+ if cursor is None:
+ first = _first_source_time(db, tier)
+ if first is None:
+ return 0
+ cursor = floor(tier, first)
+ # The finer tier must have got past a bucket before that bucket can be built from it.
+ ready_until = now - GRACE
+ if tier != HOUR:
+ finer_done = _read_watermark(db, FINER[tier])
+ if finer_done is None:
+ return 0
+ ready_until = min(ready_until, finer_done)
+ built = 0
+ while built < MAX_BUCKETS_PER_RUN[tier]:
+ end = following(tier, cursor)
+ if end > ready_until:
+ break
+ _build_bucket(db, tier, cursor, end)
+ # The rows and the watermark are saved together, so a failure never leaves one without the other.
+ _write_watermark(db, tier, end)
+ db.commit()
+ cursor = end
+ built += 1
+ return built
+
+
+def run_rollups(db, now: datetime | None = None) -> dict[str, int]:
+ """Build every bucket that has completed since the last run, finest tier first. Returns buckets built per tier."""
+ now = _utc(now or datetime.now(timezone.utc))
+ return {tier: _run_tier(db, tier, now) for tier in TIERS}
diff --git a/src/backend/stats.py b/src/backend/stats.py
new file mode 100644
index 0000000..6b17eca
--- /dev/null
+++ b/src/backend/stats.py
@@ -0,0 +1,133 @@
+"""Fleet-wide statistics for the Statistics page: incidents over time and how reliably servers reported.
+
+Only aggregated numbers leave this module, never raw rows. Incidents are counted by the time they opened, in
+buckets that suit the range (hours up to a day, days up to a month, weeks up to a year, months beyond), and the
+amount of data read is capped.
+"""
+from __future__ import annotations
+
+from datetime import datetime, timedelta, timezone
+
+from sqlalchemy import func, select
+
+from . import history, rollups
+from .alerts import CRITICAL, INFO, SEVERITIES, WARNING
+from .models import HostMetric, IncidentAlert, Server
+
+MAX_INCIDENTS_READ = 20000
+# The bucket an incident count is grouped into, by range.
+INCIDENT_BUCKET = {"1h": None, "6h": rollups.HOUR, "24h": rollups.HOUR, "7d": rollups.DAY, "30d": rollups.DAY, "90d": rollups.WEEK, "1y": rollups.WEEK, "all": rollups.MONTH}
+KIND_LABELS = {
+ "host:disk": "Disk space", "host:inodes": "Inodes", "host:memory": "Memory", "host:swap": "Swap", "host:load": "Load",
+ "system:reboot": "Reboots", "system:oom": "Out-of-memory kills", "system:kernel": "Kernel faults", "system:ntp": "Clock not synced",
+ "system:docker_unreachable": "Docker unreachable", "system:docker_slow": "Docker slow", "system:collect_slow": "Collector too slow",
+ "system:reboot_required": "Reboot needed",
+}
+
+
+def incident_kind(rule_key: str | None) -> str:
+ """A readable group for an incident's rule, so the page never has to show raw rule keys."""
+ if not rule_key:
+ return "Other"
+ if rule_key == "server_offline":
+ return "Server offline"
+ parts = rule_key.split(":")
+ if len(parts) >= 2 and f"{parts[0]}:{parts[1]}" in KIND_LABELS:
+ return KIND_LABELS[f"{parts[0]}:{parts[1]}"]
+ return {"trend": "Possible memory leaks", "network": "Network", "container": "Containers"}.get(parts[0], "Other")
+
+
+def _utc(moment: datetime) -> datetime:
+ return rollups._utc(moment)
+
+
+def window(db, range_name: str, now: datetime) -> datetime:
+ """When the range starts: a fixed distance back, or the first incident for all time."""
+ length = history.RANGES[range_name]
+ if length is not None:
+ return now - length
+ first = db.execute(select(func.min(IncidentAlert.created_at))).scalar()
+ return _utc(first) if first else now
+
+
+def incident_stats(db, range_name: str, now: datetime | None = None) -> dict:
+ now = _utc(now or datetime.now(timezone.utc))
+ start = window(db, range_name, now)
+ bucket = INCIDENT_BUCKET[range_name]
+ rows = db.execute(
+ select(IncidentAlert.created_at, IncidentAlert.resolved_at, IncidentAlert.severity, IncidentAlert.rule_key, IncidentAlert.status)
+ .where(IncidentAlert.created_at >= start).order_by(IncidentAlert.created_at).limit(MAX_INCIDENTS_READ + 1)
+ ).all()
+ truncated = len(rows) > MAX_INCIDENTS_READ
+ rows = rows[:MAX_INCIDENTS_READ]
+ columns: dict[datetime, dict[str, int]] = {}
+ kinds: dict[str, int] = {}
+ minutes: list[float] = []
+ for created, resolved, severity, rule_key, status in rows:
+ created = _utc(created)
+ key = rollups.floor(bucket, created) if bucket else rollups.floor(rollups.HOUR, created)
+ column = columns.setdefault(key, {level: 0 for level in SEVERITIES})
+ column[severity if severity in column else INFO] += 1
+ kinds[incident_kind(rule_key)] = kinds.get(incident_kind(rule_key), 0) + 1
+ if resolved is not None and status == "resolved":
+ minutes.append(max((_utc(resolved) - created).total_seconds() / 60, 0))
+ open_now = {level: count for level, count in db.execute(select(IncidentAlert.severity, func.count()).where(IncidentAlert.status == "active").group_by(IncidentAlert.severity))}
+ return {
+ "range": range_name, "start": start.isoformat(), "end": now.isoformat(), "bucket": bucket or "hour",
+ "opened": len(rows), "resolved": len(minutes), "truncated": truncated,
+ "mean_minutes_to_resolve": (sum(minutes) / len(minutes)) if minutes else None,
+ "open_now": {level.lower(): open_now.get(level, 0) for level in (CRITICAL, WARNING, INFO)},
+ "by_bucket": [{"time": moment.isoformat(), **{level.lower(): counts[level] for level in (CRITICAL, WARNING, INFO)}} for moment, counts in sorted(columns.items())],
+ "by_kind": [{"kind": kind, "count": count} for kind, count in sorted(kinds.items(), key=lambda item: (-item[1], item[0]))],
+ }
+
+
+def availability(db, range_name: str, now: datetime | None = None) -> list[dict]:
+ """Per server: the share of the check-ins it should have made in the range that actually arrived.
+
+ Approximate: it assumes the check interval it has now held for the whole range, and the range starts when the
+ server's first data does, so a new server isn't blamed for the time before it existed.
+ """
+ now = _utc(now or datetime.now(timezone.utc))
+ result = []
+ for server in db.execute(select(Server).order_by(Server.hostname)).scalars():
+ series = history.history(db, history.HOST, server.id, None, range_name, (), server.check_interval_seconds, now=now)
+ received = sum(point["samples"] for point in series["points"])
+ span = (now - datetime.fromisoformat(series["start"])).total_seconds()
+ expected = max(span / server.check_interval_seconds, 1)
+ result.append({"server_id": server.id, "hostname": server.hostname, "checkins": received, "expected": round(expected),
+ "percent": round(min(100.0, 100.0 * received / expected), 2) if series["data_since"] else None})
+ return result
+
+
+HEATMAP_DAYS = 30
+HEATMAP_METRICS = ("cpu", "ram")
+MAX_OFFSET_MINUTES = 14 * 60
+
+
+def hour_weekday_heatmap(db, server: Server, metric: str, offset_minutes: int = 0, now: datetime | None = None) -> dict:
+ """The average of a metric for each hour of the day and weekday over the last 30 days, for a time zone.
+
+ ``offset_minutes`` is the viewer's offset from UTC (a fixed one: daylight saving changes inside the 30 days
+ are ignored). Row 0 is Monday. Hours nobody reported stay None.
+ """
+ now = _utc(now or datetime.now(timezone.utc))
+ items, _ = history._tiered_items(db, history.HOST, (metric,), server.id, None, rollups.HOUR, now - timedelta(days=HEATMAP_DAYS), now)
+ totals = [[[0.0, 0] for _ in range(24)] for _ in range(7)]
+ for start, agg in items:
+ local = start + timedelta(minutes=offset_minutes)
+ total, count, _maximum = agg.values.get(metric, (0.0, 0, None))
+ cell = totals[local.weekday()][local.hour]
+ cell[0] += total
+ cell[1] += count
+ since = history.data_since(db, history.HOST, server.id, None)
+ return {
+ "server_id": server.id, "metric": metric, "days": HEATMAP_DAYS, "offset_minutes": offset_minutes,
+ "data_since": since.isoformat() if since else None,
+ "cells": [[(cell[0] / cell[1]) if cell[1] else None for cell in row] for row in totals],
+ "samples": [[cell[1] for cell in row] for row in totals],
+ }
+
+
+def fleet_stats(db, range_name: str, now: datetime | None = None) -> dict:
+ return {"incidents": incident_stats(db, range_name, now), "availability": availability(db, range_name, now)}
diff --git a/src/frontend/src/App.jsx b/src/frontend/src/App.jsx
index bbd9018..99f9f2d 100644
--- a/src/frontend/src/App.jsx
+++ b/src/frontend/src/App.jsx
@@ -1,5 +1,6 @@
import { BrowserRouter, Link, NavLink, Navigate, Route, Routes, useLocation, useNavigate } from 'react-router-dom'
import { useEffect, useRef, useState } from 'react'
+import StatisticsContent from './StatisticsPage.jsx'
const AUTH_KEY = 'homie-auth-user'
const USERS_KEY = 'homie-auth-users'
@@ -46,6 +47,7 @@ function parseAgentMessage(text) {
const navigation = [
{ label: 'Dashboard', to: '/', icon: '▣' },
+ { label: 'Statistics', to: '/statistics', icon: '▤' },
{ label: 'Agent', to: '/agent', icon: '◷' },
{ label: 'Account', to: '/account', icon: '◉' },
{ label: 'Admin', to: '/admin', icon: '⚙', adminOnly: true },
@@ -2544,6 +2546,16 @@ function App() {
</RequireAuth>
}
/>
+ <Route
+ path="/statistics"
+ element={
+ <RequireAuth user={user}>
+ <AppShell user={user} onLogout={handleLogout}>
+ <StatisticsContent request={apiRequest} />
+ </AppShell>
+ </RequireAuth>
+ }
+ />
<Route
path="/account"
element={
diff --git a/src/frontend/src/ContainerStatistics.jsx b/src/frontend/src/ContainerStatistics.jsx
new file mode 100644
index 0000000..e4fa5e7
--- /dev/null
+++ b/src/frontend/src/ContainerStatistics.jsx
@@ -0,0 +1,148 @@
+// The containers of one server as small cards; selecting one shows its statistics over the chosen range.
+import { useEffect, useMemo, useState } from 'react'
+import { TimeSeriesChart } from './charts.jsx'
+import { describeBucket, seriesFor, summarise } from './chartMath.js'
+import { problemRank, sortContainers } from './containerLogic.js'
+
+const LIVE_RANGES = new Set(['1h', '6h', '24h'])
+const REFRESH_SECONDS = 30
+const METRICS = 'cpu,memory_mb,memory_limit_mb,restarts'
+const SORTS = [
+ { id: 'problems', label: 'Problems first' },
+ { id: 'name', label: 'Name' },
+ { id: 'memory', label: 'Memory' },
+ { id: 'cpu', label: 'CPU' },
+]
+
+function statusColour(container) {
+ const rank = problemRank(container)
+ if (rank === 0) return container.status === 'restarting' ? 'bg-red-400' : 'bg-amber-400'
+ return rank === 1 ? 'bg-red-400' : 'bg-emerald-400'
+}
+
+function ContainerCard({ container, selected, onSelect }) {
+ const memoryPct = container.memory_pct
+ return (
+ <button
+ type="button"
+ aria-pressed={selected}
+ onClick={onSelect}
+ className={`min-w-0 rounded-2xl border p-3 text-left transition ${selected ? 'border-white/40 bg-white/10' : 'border-[#1f2326] bg-[#0d1013]/60 hover:border-white/20'}`}
+ >
+ <div className="flex items-center gap-2">
+ <span aria-hidden="true" className={`h-2.5 w-2.5 shrink-0 rounded-full ${statusColour(container)}`} />
+ <span className="truncate text-sm font-medium text-white">{container.name}</span>
+ </div>
+ <div className="mt-1 truncate text-xs text-[#7f8589]">
+ {container.status}{container.healthcheck_status ? ` · ${container.healthcheck_status}` : ''}{container.restart_count > 0 ? ` · ${container.restart_count} restarts` : ''}
+ </div>
+ <div className="mt-2 flex items-baseline justify-between text-xs text-[#9aa0a6]">
+ <span>CPU <span className="text-sm text-white">{container.cpu_pct == null ? '–' : `${container.cpu_pct}%`}</span></span>
+ <span>Memory <span className="text-sm text-white">{container.memory_used_mb == null ? '–' : `${Math.round(container.memory_used_mb)} MB`}</span></span>
+ </div>
+ <div className="mt-2 h-1.5 overflow-hidden rounded-full bg-white/10" title={memoryPct == null ? 'No memory limit' : `${memoryPct}% of its memory limit`}>
+ {memoryPct != null && <div className={`h-full rounded-full ${memoryPct >= 90 ? 'bg-red-400' : memoryPct >= 75 ? 'bg-amber-400' : 'bg-emerald-400'}`} style={{ width: `${Math.min(Math.max(memoryPct, 0), 100)}%` }} />}
+ </div>
+ </button>
+ )
+}
+
+function Figure({ label, value }) {
+ return (
+ <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3">
+ <div className="text-xs uppercase tracking-[0.18em] text-[#9aa0a6]">{label}</div>
+ <div className="mt-1 text-xl font-semibold text-white">{value}</div>
+ </div>
+ )
+}
+
+function ContainerDetail({ request, serverId, container, range, onClose }) {
+ const [data, setData] = useState(null)
+ const [error, setError] = useState('')
+
+ useEffect(() => {
+ const controller = new AbortController()
+ const load = () => request(`/monitoring/servers/${serverId}/containers/${encodeURIComponent(container.container_id)}/history?range=${range}&metrics=${METRICS}`, { method: 'GET', signal: controller.signal })
+ .then((result) => { setData(result); setError('') })
+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load this container.') })
+ load()
+ if (!LIVE_RANGES.has(range)) return () => controller.abort()
+ const timer = setInterval(() => { if (!document.hidden) load() }, REFRESH_SECONDS * 1000)
+ return () => { controller.abort(); clearInterval(timer) }
+ }, [request, serverId, container.container_id, range])
+
+ const current = data && data.container_id === container.container_id ? data : null
+ const series = useMemo(() => ({ cpu: seriesFor(current?.points, 'cpu'), memory: seriesFor(current?.points, 'memory_mb') }), [current])
+ const limit = useMemo(() => Math.max(0, ...(current?.points || []).map((point) => point.memory_limit_mb?.max ?? 0)) || null, [current])
+ const figures = useMemo(() => {
+ const points = current?.points || []
+ const samples = points.reduce((total, point) => total + point.samples, 0)
+ const down = points.reduce((total, point) => total + (point.down || 0), 0)
+ const restarts = points.map((point) => point.restarts?.max).filter((value) => value != null)
+ return { samples, notRunning: samples ? (100 * down) / samples : null, restarts: restarts.length ? restarts[restarts.length - 1] - restarts[0] : null }
+ }, [current])
+ const cpu = summarise(series.cpu)
+ const memory = summarise(series.memory)
+ const common = { bucketSeconds: current?.bucket_seconds ?? 60, start: current ? new Date(current.start).getTime() : undefined, end: current ? new Date(current.end).getTime() : undefined, utc: current ? ['day', 'week', 'month'].includes(current.source) : false }
+
+ return (
+ <section aria-label={`Statistics for ${container.name}`} className="space-y-4 rounded-3xl border border-white/10 bg-[#0d1013]/40 p-4">
+ <div className="flex items-start justify-between gap-3">
+ <div className="min-w-0">
+ <h3 className="truncate text-lg font-semibold text-white">{container.name}</h3>
+ <div className="truncate text-xs text-[#7f8589]">{container.image || 'unknown image'}</div>
+ </div>
+ <button type="button" onClick={onClose} className="shrink-0 rounded-xl border border-white/10 px-3 py-1.5 text-xs text-[#9aa0a6] hover:text-white">Close</button>
+ </div>
+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}
+ {current && (
+ <>
+ <div className="grid grid-cols-2 gap-3 lg:grid-cols-4">
+ <Figure label="Average CPU" value={cpu.average == null ? '–' : `${cpu.average.toFixed(1)}%`} />
+ <Figure label="Peak memory" value={memory.maximum == null ? '–' : `${Math.round(memory.maximum)} MB`} />
+ <Figure label="Not running" value={figures.notRunning == null ? '–' : `${figures.notRunning.toFixed(1)}% of the time`} />
+ <Figure label="Restarts in range" value={figures.restarts == null ? '–' : figures.restarts} />
+ </div>
+ <p className="text-xs text-[#7f8589]">{current.points.length ? `One point per ${describeBucket(current.bucket_seconds)}. A container only reports CPU and memory while it is running.` : 'Nothing recorded for this container in this range.'}</p>
+ <div className="grid gap-4 xl:grid-cols-2">
+ <TimeSeriesChart {...common} title="CPU usage" series={series.cpu} color="#34d399" />
+ <TimeSeriesChart {...common} title="Memory used" series={series.memory} color="#a78bfa" unit="MB" decimals={0} limit={limit} />
+ </div>
+ </>
+ )}
+ </section>
+ )
+}
+
+export default function ContainerStatistics({ request, serverId, containers, range, selectedId, onSelect }) {
+ const [query, setQuery] = useState('')
+ const [sort, setSort] = useState('problems')
+ const shown = useMemo(() => sortContainers(containers.filter((container) => container.name.toLowerCase().includes(query.trim().toLowerCase())), sort), [containers, query, sort])
+ const selected = containers.find((container) => container.container_id === selectedId) || null
+ const problems = containers.filter((container) => problemRank(container) < 2).length
+
+ return (
+ <section aria-label="Containers" className="space-y-4">
+ <div className="flex flex-wrap items-end justify-between gap-3">
+ <div>
+ <h2 className="text-xl font-semibold text-white">Containers</h2>
+ <p className="text-xs text-[#7f8589]">{containers.length} on this server{problems ? ` · ${problems} need attention` : ''}. Select one for its statistics.</p>
+ </div>
+ {containers.length > 0 && (
+ <div className="flex flex-wrap gap-2">
+ <input type="search" aria-label="Search containers" placeholder="Search" value={query} onChange={(event) => setQuery(event.target.value)} className="min-h-11 w-40 rounded-xl border border-white/10 bg-[#111417] px-3 text-sm text-white sm:min-h-0 sm:py-2" />
+ <select aria-label="Sort containers" value={sort} onChange={(event) => setSort(event.target.value)} className="min-h-11 rounded-xl border border-white/10 bg-[#111417] px-3 text-sm text-white sm:min-h-0 sm:py-2">
+ {SORTS.map((option) => <option key={option.id} value={option.id}>{option.label}</option>)}
+ </select>
+ </div>
+ )}
+ </div>
+ {containers.length === 0 && <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-6 text-sm text-[#9aa0a6]">This server has no containers (or its collector has no Docker access).</div>}
+ {containers.length > 0 && shown.length === 0 && <div className="text-sm text-[#7f8589]">No container matches “{query}”.</div>}
+ <div className="grid grid-cols-1 gap-3 sm:grid-cols-2 xl:grid-cols-3">
+ {shown.map((container) => <ContainerCard key={container.container_id} container={container} selected={container.container_id === selectedId} onSelect={() => onSelect(container.container_id === selectedId ? '' : container.container_id)} />)}
+ </div>
+ {selected && <ContainerDetail request={request} serverId={serverId} container={selected} range={range} onClose={() => onSelect('')} />}
+ </section>
+ )
+}
diff --git a/src/frontend/src/FleetStatistics.jsx b/src/frontend/src/FleetStatistics.jsx
new file mode 100644
index 0000000..283060f
--- /dev/null
+++ b/src/frontend/src/FleetStatistics.jsx
@@ -0,0 +1,95 @@
+// The "All servers" view of the Statistics page: how the whole fleet is doing at a glance.
+import { useEffect, useMemo, useState } from 'react'
+import { BarList, ColumnChart, Donut } from './distributions.jsx'
+import { availabilityRows, bucketLabel, containerStateSlices, formatDuration, incidentColumns, isOnline, serversByDisk, topContainers } from './fleetLogic.js'
+
+const LIVE_RANGES = new Set(['1h', '6h', '24h'])
+const REFRESH_SECONDS = 30
+const SEVERITIES = [
+ { key: 'critical', label: 'Critical', color: '#f87171' },
+ { key: 'warning', label: 'Warning', color: '#fbbf24' },
+ { key: 'info', label: 'Info', color: '#7dd3fc' },
+]
+
+function Figure({ label, value, detail }) {
+ return (
+ <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3">
+ <div className="text-xs uppercase tracking-[0.18em] text-[#9aa0a6]">{label}</div>
+ <div className="mt-1 text-2xl font-semibold text-white">{value}</div>
+ {detail && <div className="text-xs text-[#7f8589]">{detail}</div>}
+ </div>
+ )
+}
+
+export default function FleetStatistics({ request, servers, range }) {
+ const [data, setData] = useState(null)
+ const [error, setError] = useState('')
+
+ useEffect(() => {
+ const controller = new AbortController()
+ const load = () => request(`/stats/fleet?range=${range}`, { method: 'GET', signal: controller.signal })
+ .then((result) => { setData(result); setError('') })
+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load the fleet statistics.') })
+ load()
+ if (!LIVE_RANGES.has(range)) return () => controller.abort()
+ const timer = setInterval(() => { if (!document.hidden) load() }, REFRESH_SECONDS * 1000)
+ return () => { controller.abort(); clearInterval(timer) }
+ }, [request, range])
+
+ const incidents = data?.incidents
+ const current = data && data.incidents.range === range ? data : null
+ const online = servers.filter((server) => isOnline(server)).length
+ const containerSlices = useMemo(() => containerStateSlices(servers), [servers])
+ const containerTotal = containerSlices.reduce((sum, slice) => sum + slice.value, 0)
+ const running = containerSlices.find((slice) => slice.label === 'Running')?.value ?? 0
+ const columns = useMemo(() => (current ? incidentColumns(current.incidents.by_bucket, current.incidents.bucket, current.incidents.start, current.incidents.end) : []), [current])
+ const categories = columns.map((column) => ({ label: bucketLabel(column.time, current.incidents.bucket), values: column.values }))
+ const openNow = incidents ? Object.values(incidents.open_now).reduce((sum, count) => sum + count, 0) : 0
+
+ return (
+ <div className="space-y-5">
+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}
+ <div className="grid grid-cols-2 gap-3 lg:grid-cols-4">
+ <Figure label="Servers online" value={`${online} of ${servers.length}`} />
+ <Figure label="Containers running" value={`${running} of ${containerTotal}`} />
+ <Figure label="Open incidents" value={incidents ? openNow : '–'} />
+ <Figure label="Mean time to resolve" value={current ? formatDuration(current.incidents.mean_minutes_to_resolve) : '–'} detail={current ? `${current.incidents.resolved} resolved in this range` : undefined} />
+ </div>
+
+ <div className="grid gap-4 lg:grid-cols-2">
+ <Donut title="Containers by state" subtitle="Right now, across all servers" slices={containerSlices} centreLabel="containers" empty="No containers are reported yet." />
+ <Donut title="Open incidents by severity" subtitle="Right now" slices={SEVERITIES.map((severity) => ({ label: severity.label, color: severity.color, value: incidents?.open_now[severity.key] ?? 0 }))} centreLabel="open" empty="Nothing is open. All quiet." />
+ </div>
+
+ {current && (
+ <ColumnChart
+ title="Incidents opened"
+ subtitle={`${current.incidents.opened} in this range, per ${current.incidents.bucket}${current.incidents.truncated ? ' (only the first part could be counted)' : ''}`}
+ categories={categories}
+ series={SEVERITIES}
+ labelEvery={Math.max(1, Math.ceil(categories.length / 8))}
+ empty="No incidents opened in this range."
+ />
+ )}
+
+ <div className="grid gap-4 lg:grid-cols-2">
+ {current && <BarList title="What the incidents were about" subtitle="Opened in this range" rows={current.incidents.by_kind.slice(0, 10).map((item) => ({ label: item.kind, value: item.count }))} color="#a78bfa" empty="No incidents opened in this range." />}
+ <BarList title="Servers by disk usage" subtitle="Root disk, right now" rows={serversByDisk(servers)} max={100} unit="%" decimals={0} />
+ <BarList title="Containers using the most memory" subtitle="Right now" rows={topContainers(servers, 'memory_used_mb')} unit=" MB" color="#a78bfa" />
+ <BarList title="Containers using the most CPU" subtitle="Right now" rows={topContainers(servers, 'cpu_pct')} unit="%" decimals={1} color="#34d399" />
+ </div>
+
+ {current && (
+ <BarList
+ title="Availability"
+ subtitle="Share of the expected check-ins that arrived. Approximate: it assumes each server's current check interval."
+ rows={availabilityRows(current.availability)}
+ max={100}
+ unit="%"
+ decimals={2}
+ empty="No server has reported in this range."
+ />
+ )}
+ </div>
+ )
+}
diff --git a/src/frontend/src/ServerInsights.jsx b/src/frontend/src/ServerInsights.jsx
new file mode 100644
index 0000000..2fe9230
--- /dev/null
+++ b/src/frontend/src/ServerInsights.jsx
@@ -0,0 +1,100 @@
+// The extra views of one server: how its load is distributed, when it is busy, how its disks are filling up.
+import { useEffect, useMemo, useState } from 'react'
+import { TimeSeriesChart } from './charts.jsx'
+import { daysUntilFull, formatForecast, histogramBins, seriesFor } from './chartMath.js'
+import { ColumnChart, Heatmap } from './distributions.jsx'
+
+const LIVE_RANGES = new Set(['1h', '6h', '24h'])
+const REFRESH_SECONDS = 30
+
+// How much of the time a metric spent in each 10% band: weighted by the check-ins behind every point.
+function shareOfTime(series) {
+ const bins = histogramBins(series.filter((point) => point.avg !== null).map((point) => ({ value: point.avg, weight: point.samples })))
+ const total = bins.reduce((sum, bin) => sum + bin.count, 0)
+ return bins.map((bin) => ({ label: `${bin.from}–${bin.to}`, values: { share: total ? Math.round((1000 * bin.count) / total) / 10 : 0 } }))
+}
+
+export function Distributions({ cpu, memory }) {
+ const share = [{ key: 'share', label: 'Share of the time', color: '#7dd3fc' }]
+ const memoryShare = [{ key: 'share', label: 'Share of the time', color: '#a78bfa' }]
+ return (
+ <div className="grid gap-4 xl:grid-cols-2">
+ <ColumnChart title="CPU usage distribution" subtitle="Share of the time the average CPU was in each band (%)" categories={shareOfTime(cpu)} series={share} unit="%" empty="Not enough data for this range." />
+ <ColumnChart title="Memory usage distribution" subtitle="Share of the time the memory in use was in each band (%)" categories={shareOfTime(memory)} series={memoryShare} unit="%" empty="Not enough data for this range." />
+ </div>
+ )
+}
+
+export function BusyHours({ request, serverId }) {
+ const [data, setData] = useState(null)
+ const [metric, setMetric] = useState('cpu')
+ const [error, setError] = useState('')
+ const offset = -new Date().getTimezoneOffset()
+
+ useEffect(() => {
+ const controller = new AbortController()
+ request(`/stats/servers/${serverId}/heatmap?metric=${metric}&offset=${offset}`, { method: 'GET', signal: controller.signal })
+ .then((result) => { setData(result); setError('') })
+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load the busy hours.') })
+ return () => controller.abort()
+ }, [request, serverId, metric, offset])
+
+ const current = data && data.server_id === serverId && data.metric === metric ? data : null
+ return (
+ <div className="space-y-2">
+ <div className="flex items-center justify-between gap-3">
+ <h2 className="text-xl font-semibold text-white">When it is busy</h2>
+ <div role="group" aria-label="Metric" className="flex gap-1.5">
+ {[['cpu', 'CPU'], ['ram', 'Memory']].map(([id, label]) => (
+ <button key={id} type="button" aria-pressed={metric === id} onClick={() => setMetric(id)} className={`min-h-9 rounded-xl border px-3 text-sm ${metric === id ? 'border-white/30 bg-white/10 text-white' : 'border-white/10 text-[#9aa0a6] hover:text-white'}`}>{label}</button>
+ ))}
+ </div>
+ </div>
+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}
+ {current && <Heatmap title={`Average ${metric === 'cpu' ? 'CPU' : 'memory'} by hour of the day`} subtitle="The last 30 days, in your time zone, whatever range is selected above." grid={current.cells} color={metric === 'cpu' ? '#7dd3fc' : '#a78bfa'} empty="Not enough history yet. This fills in as days pass." />}
+ </div>
+ )
+}
+
+function MountChart({ request, serverId, mount, range }) {
+ const [data, setData] = useState(null)
+ const [error, setError] = useState('')
+
+ useEffect(() => {
+ const controller = new AbortController()
+ const load = () => request(`/monitoring/servers/${serverId}/mounts/history?mount=${encodeURIComponent(mount.mount)}&range=${range}&metrics=used_pct,used_gb,total_gb`, { method: 'GET', signal: controller.signal })
+ .then((result) => { setData(result); setError('') })
+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load this filesystem.') })
+ load()
+ if (!LIVE_RANGES.has(range)) return () => controller.abort()
+ const timer = setInterval(() => { if (!document.hidden) load() }, REFRESH_SECONDS * 1000)
+ return () => { controller.abort(); clearInterval(timer) }
+ }, [request, serverId, mount.mount, range])
+
+ const current = data && data.mount === mount.mount ? data : null
+ const series = useMemo(() => seriesFor(current?.points, 'used_pct'), [current])
+ const forecast = useMemo(() => daysUntilFull(series.filter((point) => point.avg !== null).map((point) => ({ t: point.t, value: point.avg }))), [series])
+ const common = { bucketSeconds: current?.bucket_seconds ?? 60, start: current ? new Date(current.start).getTime() : undefined, end: current ? new Date(current.end).getTime() : undefined, utc: current ? ['day', 'week', 'month'].includes(current.source) : false }
+ return (
+ <div className="min-w-0 space-y-2">
+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}
+ <TimeSeriesChart {...common} title={`Disk usage · ${mount.mount}`} series={series} color="#f472b6" domain={[0, 100]} />
+ <p className="px-1 text-xs text-[#9aa0a6]">
+ {mount.used_gb != null && mount.total_gb != null ? `${mount.used_gb.toFixed(1)} of ${mount.total_gb.toFixed(1)} GB used now. ` : ''}
+ {current ? formatForecast(forecast) : ''}
+ </p>
+ </div>
+ )
+}
+
+export function Storage({ request, serverId, mounts, range }) {
+ if (!mounts.length) return null
+ return (
+ <div className="space-y-3">
+ <h2 className="text-xl font-semibold text-white">Storage</h2>
+ <div className="grid gap-4 xl:grid-cols-2">
+ {mounts.map((mount) => <MountChart key={mount.mount} request={request} serverId={serverId} mount={mount} range={range} />)}
+ </div>
+ </div>
+ )
+}
diff --git a/src/frontend/src/StatisticsPage.jsx b/src/frontend/src/StatisticsPage.jsx
new file mode 100644
index 0000000..531c669
--- /dev/null
+++ b/src/frontend/src/StatisticsPage.jsx
@@ -0,0 +1,211 @@
+// The Statistics page: pick a server and a time range and see how it behaved, from the last hour to all time.
+// It only needs a way to call the API, so it has no import from App.jsx.
+import { useCallback, useEffect, useMemo, useState } from 'react'
+import { useSearchParams } from 'react-router-dom'
+import { TimeSeriesChart } from './charts.jsx'
+import ContainerStatistics from './ContainerStatistics.jsx'
+import FleetStatistics from './FleetStatistics.jsx'
+import { BusyHours, Distributions, Storage } from './ServerInsights.jsx'
+import { describeBucket, formatDateTime, seriesFor, summarise } from './chartMath.js'
+
+const RANGES = [
+ { id: '1h', label: '1 hour' },
+ { id: '6h', label: '6 hours' },
+ { id: '24h', label: '24 hours' },
+ { id: '7d', label: '7 days' },
+ { id: '30d', label: '30 days' },
+ { id: '90d', label: '90 days' },
+ { id: '1y', label: '1 year' },
+ { id: 'all', label: 'All time' },
+]
+const RANGE_IDS = new Set(RANGES.map((range) => range.id))
+const DEFAULT_RANGE = '24h'
+const RANGE_STORAGE_KEY = 'homelab.statisticsRange'
+// Only short ranges change from one minute to the next.
+const LIVE_RANGES = new Set(['1h', '6h', '24h'])
+const REFRESH_SECONDS = 30
+const METRICS = 'cpu,ram,swap,load,disk,docker_ms,collect_ms'
+const SOURCES = {
+ detailed: 'check-ins grouped together',
+ hour: 'hourly summaries',
+ day: 'daily summaries',
+ week: 'weekly summaries (Monday to Sunday, UTC)',
+ month: 'monthly summaries (UTC)',
+}
+
+function rememberedRange() {
+ try {
+ const stored = window.localStorage.getItem(RANGE_STORAGE_KEY)
+ return RANGE_IDS.has(stored) ? stored : DEFAULT_RANGE
+ } catch {
+ return DEFAULT_RANGE
+ }
+}
+
+function SummaryCard({ label, value, detail }) {
+ return (
+ <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-3">
+ <div className="text-xs uppercase tracking-[0.18em] text-[#9aa0a6]">{label}</div>
+ <div className="mt-1 text-2xl font-semibold text-white">{value}</div>
+ {detail && <div className="text-xs text-[#7f8589]">{detail}</div>}
+ </div>
+ )
+}
+
+const percent = (value) => (value === null || value === undefined ? '–' : `${value.toFixed(1)}%`)
+
+export default function StatisticsContent({ request }) {
+ const [searchParams, setSearchParams] = useSearchParams()
+ const [servers, setServers] = useState(null)
+ const [data, setData] = useState(null)
+ const [error, setError] = useState('')
+ const [loading, setLoading] = useState(false)
+
+ const range = RANGE_IDS.has(searchParams.get('range')) ? searchParams.get('range') : rememberedRange()
+ const requestedServer = Number(searchParams.get('server'))
+ const wantsAll = searchParams.get('server') === 'all'
+ const serverId = useMemo(() => {
+ if (!servers?.length || wantsAll) return null
+ return servers.find((server) => server.id === requestedServer)?.id ?? (servers.find((server) => server.is_dashboard_host) || servers[0]).id
+ }, [servers, requestedServer, wantsAll])
+
+ const choose = useCallback((changes) => {
+ const next = new URLSearchParams(searchParams)
+ for (const [key, value] of Object.entries(changes)) {
+ if (value === '' || value === null) next.delete(key)
+ else next.set(key, String(value))
+ }
+ setSearchParams(next, { replace: true })
+ }, [searchParams, setSearchParams])
+
+ useEffect(() => {
+ let cancelled = false
+ const load = () => request('/monitoring/overview', { method: 'GET' })
+ .then((overview) => { if (!cancelled) setServers((overview.servers || []).map((server) => ({ ...server, is_dashboard_host: Boolean(server.is_dashboard_host), containers: server.containers || [] }))) })
+ .catch((requestError) => { if (!cancelled) { setServers((current) => current || []); setError(requestError.message || 'Could not load the servers.') } })
+ load()
+ // The containers on the cards change, so keep them current, but not while the tab is hidden.
+ const timer = setInterval(() => { if (!document.hidden) load() }, REFRESH_SECONDS * 1000)
+ return () => { cancelled = true; clearInterval(timer) }
+ }, [request])
+
+ useEffect(() => {
+ if (serverId === null) return undefined
+ const controller = new AbortController()
+ const load = (showSpinner) => {
+ if (showSpinner) setLoading(true)
+ request(`/monitoring/servers/${serverId}/history?range=${range}&metrics=${METRICS}`, { method: 'GET', signal: controller.signal })
+ .then((result) => { setData(result); setError('') })
+ .catch((requestError) => { if (requestError.name !== 'AbortError') setError(requestError.message || 'Could not load the statistics.') })
+ .finally(() => { if (!controller.signal.aborted) setLoading(false) })
+ }
+ load(true)
+ if (!LIVE_RANGES.has(range)) return () => controller.abort()
+ // Refresh the short ranges, but not while the tab is hidden.
+ const timer = setInterval(() => { if (!document.hidden) load(false) }, REFRESH_SECONDS * 1000)
+ return () => { controller.abort(); clearInterval(timer) }
+ }, [request, serverId, range])
+
+ const series = useMemo(() => Object.fromEntries(['cpu', 'ram', 'swap', 'load', 'disk', 'docker_ms', 'collect_ms'].map((metric) => [metric, seriesFor(data?.points, metric)])), [data])
+ const cpu = summarise(series.cpu)
+ const memory = summarise(series.ram)
+ const start = data ? new Date(data.start).getTime() : undefined
+ const end = data ? new Date(data.end).getTime() : undefined
+ const utc = data ? ['day', 'week', 'month'].includes(data.source) : false
+ const common = { bucketSeconds: data?.bucket_seconds ?? 60, start, end, utc }
+ const ready = data && data.server_id === serverId
+
+ return (
+ <div className="space-y-5">
+ <div>
+ <div className="text-xs uppercase tracking-[0.22em] text-[#9aa0a6]">History</div>
+ <h1 className="mt-1 text-3xl font-semibold text-white">Statistics</h1>
+ <p className="mt-2 text-sm text-[#9aa0a6]">How a server behaved over time, from the last hour to everything that has been recorded.</p>
+ </div>
+
+ <div className="flex flex-wrap items-end gap-4">
+ <label className="flex flex-col gap-1 text-xs text-[#9aa0a6]">
+ Server
+ <select
+ value={wantsAll ? 'all' : serverId ?? ''}
+ onChange={(event) => choose({ server: event.target.value, container: '' })}
+ disabled={!servers?.length}
+ className="min-h-11 rounded-xl border border-white/10 bg-[#111417] px-3 py-2 text-sm text-white sm:min-h-0"
+ >
+ {servers?.length > 0 && <option value="all">All servers</option>}
+ {(servers || []).map((server) => <option key={server.id} value={server.id}>{server.hostname}{server.is_dashboard_host ? ' (this machine)' : ''}</option>)}
+ </select>
+ </label>
+ <div role="group" aria-label="Time range" className="flex flex-wrap gap-1.5">
+ {RANGES.map((option) => (
+ <button
+ key={option.id}
+ type="button"
+ aria-pressed={range === option.id}
+ onClick={() => {
+ try { window.localStorage.setItem(RANGE_STORAGE_KEY, option.id) } catch { /* the choice just isn't remembered */ }
+ choose({ range: option.id })
+ }}
+ className={`min-h-11 rounded-xl border px-3 text-sm transition sm:min-h-9 ${range === option.id ? 'border-white/30 bg-white/10 text-white' : 'border-white/10 text-[#9aa0a6] hover:text-white'}`}
+ >
+ {option.label}
+ </button>
+ ))}
+ </div>
+ </div>
+
+ {error && <div role="alert" className="rounded-2xl border border-red-400/30 bg-red-500/10 px-4 py-3 text-sm text-red-200">{error}</div>}
+ {servers !== null && servers.length === 0 && !error && <div className="rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 px-4 py-6 text-sm text-[#9aa0a6]">No servers are reporting yet. Add one from the dashboard and its history will appear here.</div>}
+
+ {servers?.length > 0 && wantsAll && <FleetStatistics request={request} servers={servers} range={range} />}
+
+ {ready && (
+ <div className={loading ? 'space-y-5 opacity-60 transition-opacity' : 'space-y-5 transition-opacity'}>
+ <div className="grid grid-cols-2 gap-3 lg:grid-cols-4">
+ <SummaryCard label="Average CPU" value={percent(cpu.average)} />
+ <SummaryCard label="Peak CPU" value={percent(cpu.maximum)} />
+ <SummaryCard label="Average memory" value={percent(memory.average)} />
+ <SummaryCard label="Peak memory" value={percent(memory.maximum)} />
+ </div>
+ <p className="text-xs text-[#7f8589]">
+ {data.points.length
+ ? `${data.points.length} ${data.points.length === 1 ? 'point' : 'points'}, one per ${describeBucket(data.bucket_seconds)} (${SOURCES[data.source] || data.source}). The line is the average and the lighter band reaches up to the highest value. `
+ : 'Nothing has been recorded for this server in this range. '}
+ {data.data_since && `History starts ${formatDateTime(new Date(data.data_since).getTime(), utc)}. `}
+ {utc ? 'Dates are UTC.' : 'Times are in your time zone.'}
+ </p>
+ <div className="grid gap-4 xl:grid-cols-2">
+ <TimeSeriesChart {...common} title="CPU usage" series={series.cpu} color="#7dd3fc" domain={[0, 100]} />
+ <TimeSeriesChart {...common} title="Memory used" series={series.ram} color="#a78bfa" domain={[0, 100]} />
+ <TimeSeriesChart {...common} title="Swap used" series={series.swap} color="#fbbf24" domain={[0, 100]} />
+ <TimeSeriesChart {...common} title="Load (1 minute average)" series={series.load} color="#34d399" unit="" decimals={2} />
+ <TimeSeriesChart {...common} title="Root disk used" series={series.disk} color="#f472b6" domain={[0, 100]} />
+ </div>
+ <Distributions cpu={series.cpu} memory={series.ram} />
+ <BusyHours request={request} serverId={serverId} />
+ <Storage request={request} serverId={serverId} mounts={servers.find((server) => server.id === serverId)?.mounts || []} range={range} />
+ {(series.docker_ms.length > 0 || series.collect_ms.length > 0) && (
+ <div className="space-y-3">
+ <h2 className="text-xl font-semibold text-white">Reliability</h2>
+ <div className="grid gap-4 xl:grid-cols-2">
+ <TimeSeriesChart {...common} title="Docker response time (listing containers)" series={series.docker_ms} color="#fb923c" unit="ms" decimals={0} />
+ <TimeSeriesChart {...common} title="Time the collector takes per check-in" series={series.collect_ms} color="#94a3b8" unit="ms" decimals={0} />
+ </div>
+ </div>
+ )}
+ </div>
+ )}
+ {servers?.length > 0 && serverId !== null && !wantsAll && (
+ <ContainerStatistics
+ request={request}
+ serverId={serverId}
+ containers={servers.find((server) => server.id === serverId)?.containers || []}
+ range={range}
+ selectedId={searchParams.get('container') || ''}
+ onSelect={(containerId) => choose({ container: containerId })}
+ />
+ )}
+ {servers === null && !error && <div className="text-sm text-[#7f8589]">Loading…</div>}
+ </div>
+ )
+}
diff --git a/src/frontend/src/chartMath.js b/src/frontend/src/chartMath.js
new file mode 100644
index 0000000..95446ff
--- /dev/null
+++ b/src/frontend/src/chartMath.js
@@ -0,0 +1,163 @@
+// Pure helpers for the charts: ticks, labels, gaps and summary figures. No React in here.
+
+const pad = (value) => String(value).padStart(2, '0')
+
+export function niceTicks(low, high, wanted = 5) {
+ if (!Number.isFinite(low) || !Number.isFinite(high)) return [0, 1]
+ if (high <= low) high = low + 1
+ const rough = (high - low) / Math.max(wanted, 1)
+ const magnitude = 10 ** Math.floor(Math.log10(rough))
+ const step = [1, 2, 2.5, 5, 10].map((factor) => factor * magnitude).find((candidate) => candidate >= rough) || 10 * magnitude
+ const ticks = []
+ // The last tick is at or above the top of the data, so nothing is clipped against the edge.
+ const last = Math.ceil(high / step - 1e-9) * step
+ for (let value = Math.ceil(low / step) * step; value <= last + step * 1e-9; value += step) ticks.push(Number(value.toFixed(10)))
+ return ticks
+}
+
+// Day, week and month points are UTC-aligned, so they are labelled in UTC; shorter ranges use the viewer's clock.
+export function formatTime(milliseconds, spanSeconds, utc) {
+ const date = new Date(milliseconds)
+ const get = utc
+ ? { year: date.getUTCFullYear(), month: date.getUTCMonth(), day: date.getUTCDate(), hour: date.getUTCHours(), minute: date.getUTCMinutes() }
+ : { year: date.getFullYear(), month: date.getMonth(), day: date.getDate(), hour: date.getHours(), minute: date.getMinutes() }
+ const month = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'][get.month]
+ if (spanSeconds <= 18 * 3600) return `${pad(get.hour)}:${pad(get.minute)}`
+ if (spanSeconds <= 14 * 86400) return `${get.day} ${month} ${pad(get.hour)}:${pad(get.minute)}`
+ if (spanSeconds <= 400 * 86400) return `${get.day} ${month}`
+ return `${month} ${get.year}`
+}
+
+export function formatDateTime(milliseconds, utc) {
+ const date = new Date(milliseconds)
+ const pick = (local, universal) => (utc ? universal : local)
+ const month = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'][pick(date.getMonth(), date.getUTCMonth())]
+ const day = `${pick(date.getDate(), date.getUTCDate())} ${month} ${pick(date.getFullYear(), date.getUTCFullYear())}`
+ return utc ? day : `${day} ${pad(date.getHours())}:${pad(date.getMinutes())}`
+}
+
+export function describeBucket(seconds) {
+ if (seconds >= 28 * 86400) return 'month'
+ if (seconds >= 7 * 86400) return 'week'
+ if (seconds >= 86400) return seconds === 86400 ? 'day' : `${Math.round(seconds / 86400)} days`
+ if (seconds >= 3600) return seconds === 3600 ? 'hour' : `${Math.round(seconds / 3600)} hours`
+ if (seconds >= 60) return seconds === 60 ? 'minute' : `${Math.round(seconds / 60)} minutes`
+ return `${seconds} seconds`
+}
+
+// Points become runs of neighbours; a hole bigger than a bucket and a half starts a new run, so a gap stays a gap.
+export function splitRuns(points, bucketSeconds) {
+ const runs = []
+ let current = []
+ let previous = null
+ for (const point of points) {
+ if (previous !== null && point.t - previous > bucketSeconds * 1500) {
+ runs.push(current)
+ current = []
+ }
+ current.push(point)
+ previous = point.t
+ }
+ if (current.length) runs.push(current)
+ return runs
+}
+
+// Turns the API's points into {t, avg, max, samples} for one metric, skipping points where it is unknown.
+export function seriesFor(points, metric) {
+ return (points || [])
+ .map((point) => ({ t: new Date(point.time).getTime(), avg: point[metric]?.avg ?? null, max: point[metric]?.max ?? null, samples: point.samples }))
+ .filter((point) => Number.isFinite(point.t) && (point.avg !== null || point.max !== null))
+}
+
+// Weighted average and highest maximum of a series, for the summary figures.
+export function summarise(series) {
+ let total = 0
+ let weight = 0
+ let maximum = null
+ for (const point of series) {
+ if (point.avg !== null) { total += point.avg * point.samples; weight += point.samples }
+ if (point.max !== null && (maximum === null || point.max > maximum)) maximum = point.max
+ }
+ return { average: weight ? total / weight : null, maximum }
+}
+
+
+// Counts of values per bin, between ``low`` and ``high``; each value may carry a weight (for example check-ins).
+// A value equal to ``high`` goes in the last bin, and values outside the range are left out.
+export function histogramBins(values, { low = 0, high = 100, bins = 10 } = {}) {
+ const width = (high - low) / bins
+ const counts = Array.from({ length: bins }, (_, index) => ({ from: low + index * width, to: low + (index + 1) * width, count: 0 }))
+ for (const entry of values) {
+ const value = typeof entry === 'number' ? entry : entry.value
+ const weight = typeof entry === 'number' ? 1 : entry.weight ?? 1
+ if (!Number.isFinite(value) || value < low || value > high) continue
+ counts[Math.min(Math.floor((value - low) / width), bins - 1)].count += weight
+ }
+ return counts
+}
+
+// The average of a metric for each hour of the day (0-23) and weekday (0 = Monday ... 6 = Sunday), in the viewer's
+// time zone or in UTC. Cells nobody reported stay null. Points are {t, avg, samples}.
+export function hourWeekdayGrid(points, utc = false) {
+ const sums = Array.from({ length: 7 }, () => Array.from({ length: 24 }, () => ({ total: 0, weight: 0 })))
+ for (const point of points) {
+ if (point.avg === null || point.avg === undefined) continue
+ const date = new Date(point.t)
+ const weekday = ((utc ? date.getUTCDay() : date.getDay()) + 6) % 7
+ const cell = sums[weekday][utc ? date.getUTCHours() : date.getHours()]
+ cell.total += point.avg * point.samples
+ cell.weight += point.samples
+ }
+ return sums.map((row) => row.map((cell) => (cell.weight ? cell.total / cell.weight : null)))
+}
+
+// Slices as fractions of their total, for a donut; zero and invalid values are dropped.
+export function donutSlices(slices) {
+ const valid = slices.filter((slice) => Number.isFinite(slice.value) && slice.value > 0)
+ const total = valid.reduce((sum, slice) => sum + slice.value, 0)
+ let start = 0
+ return valid.map((slice) => {
+ const fraction = slice.value / total
+ const result = { ...slice, fraction, start }
+ start += fraction
+ return result
+ })
+}
+
+// Days until a straight-line trend through (time, value) points reaches ``limit``, or null when the trend is flat,
+// falling, too short or too noisy to say. Times are milliseconds.
+export function daysUntilFull(points, limit = 100, { minPoints = 6, minSpanDays = 1, minFit = 0.5 } = {}) {
+ const usable = points.filter((point) => Number.isFinite(point.t) && Number.isFinite(point.value))
+ if (usable.length < minPoints) return null
+ const spanDays = (usable[usable.length - 1].t - usable[0].t) / 86400000
+ if (spanDays < minSpanDays) return null
+ const n = usable.length
+ const meanT = usable.reduce((sum, point) => sum + point.t, 0) / n
+ const meanV = usable.reduce((sum, point) => sum + point.value, 0) / n
+ let covariance = 0
+ let varianceT = 0
+ let varianceV = 0
+ for (const point of usable) {
+ covariance += (point.t - meanT) * (point.value - meanV)
+ varianceT += (point.t - meanT) ** 2
+ varianceV += (point.value - meanV) ** 2
+ }
+ if (varianceT === 0 || varianceV === 0) return null
+ const slope = covariance / varianceT // per millisecond
+ const fit = (covariance * covariance) / (varianceT * varianceV) // r squared
+ if (slope <= 0 || fit < minFit) return null
+ const current = usable[usable.length - 1].value
+ if (current >= limit) return 0
+ return (limit - current) / slope / 86400000
+}
+
+// "Full in about 3 weeks" for a forecast in days, or an honest "no clear trend" when there is none.
+export function formatForecast(days) {
+ if (days === null || days === undefined || !Number.isFinite(days)) return 'No clear growth trend in this range, so no estimate.'
+ if (days <= 0) return 'Full now.'
+ if (days < 1) return 'Full in less than a day at this rate.'
+ if (days < 60) return `Full in about ${Math.round(days)} ${Math.round(days) === 1 ? 'day' : 'days'} at this rate.`
+ if (days < 730) return `Full in about ${Math.round(days / 30)} months at this rate.`
+ if (days < 3650) return `Full in about ${(days / 365).toFixed(1)} years at this rate.`
+ return 'Not full within ten years at this rate.'
+}
diff --git a/src/frontend/src/charts.jsx b/src/frontend/src/charts.jsx
new file mode 100644
index 0000000..b395353
--- /dev/null
+++ b/src/frontend/src/charts.jsx
@@ -0,0 +1,143 @@
+// Chart components drawn with plain SVG: no charting library, so there is nothing extra to trust or update, and
+// every label from the server is rendered as text by React, never as HTML.
+import { useEffect, useMemo, useRef, useState } from 'react'
+import { formatDateTime, formatTime, niceTicks, splitRuns, summarise } from './chartMath.js'
+
+const MARGIN = { top: 10, right: 12, bottom: 24, left: 40 }
+
+// The width of an element in pixels, kept up to date, so SVG text stays crisp at any screen size.
+function useElementWidth(ref, fallback = 640) {
+ const [width, setWidth] = useState(fallback)
+ useEffect(() => {
+ const element = ref.current
+ if (!element) return undefined
+ const update = () => setWidth(Math.max(Math.round(element.getBoundingClientRect().width), 200))
+ update()
+ if (typeof ResizeObserver === 'undefined') return undefined
+ const observer = new ResizeObserver(update)
+ observer.observe(element)
+ return () => observer.disconnect()
+ }, [ref])
+ return width
+}
+
+const FIXED_DOMAIN_FALLBACK = [0, 1]
+
+// An average line with a lighter band up to the maximum, over time. Hover or touch shows the exact figures; the
+// whole chart is also available as a table.
+export function TimeSeriesChart({ title, series, bucketSeconds, start, end, unit = '%', color = '#7dd3fc', domain = null, height = 200, utc = false, decimals = 1, limit = null }) {
+ const wrapper = useRef(null)
+ const width = useElementWidth(wrapper)
+ const [hover, setHover] = useState(null)
+ const [asTable, setAsTable] = useState(false)
+
+ const layout = useMemo(() => {
+ const tStart = start ?? (series[0]?.t ?? 0)
+ const tEnd = Math.max(end ?? (series[series.length - 1]?.t ?? 1), tStart + 1)
+ const values = series.flatMap((point) => [point.avg, point.max]).filter((value) => value !== null)
+ if (limit !== null && !domain) values.push(limit / 1.1) // keep the limit line inside the chart
+ const [fixedLow, fixedHigh] = domain || FIXED_DOMAIN_FALLBACK
+ const yTicks = domain
+ ? niceTicks(fixedLow, fixedHigh, 4)
+ : niceTicks(0, values.length ? Math.max(...values) * 1.1 : 1, 4)
+ const yMax = yTicks[yTicks.length - 1] || fixedHigh
+ const plotWidth = Math.max(width - MARGIN.left - MARGIN.right, 50)
+ const plotHeight = height - MARGIN.top - MARGIN.bottom
+ const x = (t) => MARGIN.left + ((t - tStart) / (tEnd - tStart)) * plotWidth
+ const y = (value) => MARGIN.top + plotHeight - (Math.min(Math.max(value, 0), yMax) / yMax) * plotHeight
+ const spanSeconds = (tEnd - tStart) / 1000
+ const tickCount = Math.max(2, Math.min(7, Math.floor(plotWidth / 110)))
+ const xTicks = Array.from({ length: tickCount }, (_, index) => tStart + ((tEnd - tStart) * index) / (tickCount - 1))
+ return { tStart, tEnd, yTicks, yMax, plotWidth, plotHeight, x, y, spanSeconds, xTicks }
+ }, [series, start, end, width, height, domain, limit])
+
+ const runs = useMemo(() => splitRuns(series, bucketSeconds), [series, bucketSeconds])
+ const format = (value) => (value === null || value === undefined ? '–' : `${value.toFixed(decimals)}${unit === '%' ? '%' : ` ${unit}`.trimEnd()}`)
+
+ if (!series.length) {
+ return (
+ <figure className="min-w-0 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 p-4">
+ <figcaption className="text-sm font-medium text-white">{title}</figcaption>
+ <div className="mt-3 flex items-center justify-center rounded-xl border border-dashed border-neutral-800 text-xs text-[#7f8589]" style={{ height }}>No data for this range yet</div>
+ </figure>
+ )
+ }
+
+ const move = (event) => {
+ const box = wrapper.current?.getBoundingClientRect()
+ if (!box) return
+ const px = event.clientX - box.left
+ let best = null
+ for (const point of series) {
+ const distance = Math.abs(layout.x(point.t) - px)
+ if (best === null || distance < best.distance) best = { point, distance }
+ }
+ setHover(best && best.distance < 60 ? best.point : null)
+ }
+ const label = `${title}: average ${format(summarise(series).average)}, highest ${format(summarise(series).maximum)}`
+
+ return (
+ <figure className="min-w-0 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 p-4">
+ <div className="flex items-center justify-between gap-3">
+ <figcaption className="text-sm font-medium text-white">{title}</figcaption>
+ <button type="button" onClick={() => setAsTable((value) => !value)} className="rounded-lg border border-white/10 px-2 py-1 text-[11px] text-[#9aa0a6] hover:text-white">{asTable ? 'View as chart' : 'View as table'}</button>
+ </div>
+ {asTable ? (
+ <div className="mt-3 max-h-64 overflow-auto rounded-xl border border-[#1f2326]">
+ <table className="w-full text-left text-xs text-[#bcbcbc]">
+ <thead className="sticky top-0 bg-[#111417] text-[#9aa0a6]"><tr><th className="px-3 py-2 font-medium">Time</th><th className="px-3 py-2 font-medium">Average</th><th className="px-3 py-2 font-medium">Highest</th><th className="px-3 py-2 font-medium">Check-ins</th></tr></thead>
+ <tbody>{series.map((point) => <tr key={point.t} className="border-t border-[#1f2326]"><td className="px-3 py-1.5">{formatDateTime(point.t, utc)}</td><td className="px-3 py-1.5">{format(point.avg)}</td><td className="px-3 py-1.5">{format(point.max)}</td><td className="px-3 py-1.5">{point.samples}</td></tr>)}</tbody>
+ </table>
+ </div>
+ ) : (
+ <div ref={wrapper} className="relative mt-2 w-full min-w-0 touch-pan-y select-none overflow-hidden" onPointerMove={move} onPointerLeave={() => setHover(null)} onPointerDown={move}>
+ <svg width={width} height={height} role="img" aria-label={label} className="block">
+ {layout.yTicks.map((tick) => (
+ <g key={tick}>
+ <line x1={MARGIN.left} x2={width - MARGIN.right} y1={layout.y(tick)} y2={layout.y(tick)} stroke="#ffffff" strokeOpacity="0.07" />
+ <text x={MARGIN.left - 6} y={layout.y(tick) + 4} textAnchor="end" fontSize="10" fill="#7f8589">{tick}</text>
+ </g>
+ ))}
+ {layout.xTicks.map((tick, index) => (
+ <text key={tick} x={layout.x(tick)} y={height - 6} textAnchor={index === 0 ? 'start' : index === layout.xTicks.length - 1 ? 'end' : 'middle'} fontSize="10" fill="#7f8589">{formatTime(tick, layout.spanSeconds, utc)}</text>
+ ))}
+ {runs.map((run) => {
+ const withMax = run.filter((point) => point.max !== null && point.avg !== null)
+ const band = withMax.length > 1
+ ? `${withMax.map((point) => `${layout.x(point.t)},${layout.y(point.max)}`).join(' ')} ${[...withMax].reverse().map((point) => `${layout.x(point.t)},${layout.y(point.avg)}`).join(' ')}`
+ : null
+ const line = run.filter((point) => point.avg !== null)
+ return (
+ <g key={run[0].t}>
+ {band && <polygon points={band} fill={color} fillOpacity="0.22" />}
+ {line.length > 1 && <polyline points={line.map((point) => `${layout.x(point.t)},${layout.y(point.avg)}`).join(' ')} fill="none" stroke={color} strokeWidth="2" strokeLinejoin="round" strokeLinecap="round" />}
+ {line.length === 1 && <circle cx={layout.x(line[0].t)} cy={layout.y(line[0].avg)} r="3" fill={color} />}
+ </g>
+ )
+ })}
+ {limit !== null && (
+ <g pointerEvents="none">
+ <line x1={MARGIN.left} x2={width - MARGIN.right} y1={layout.y(limit)} y2={layout.y(limit)} stroke="#f87171" strokeOpacity="0.8" strokeDasharray="5 4" />
+ <text x={width - MARGIN.right - 2} y={layout.y(limit) < MARGIN.top + 14 ? layout.y(limit) + 12 : layout.y(limit) - 4} textAnchor="end" fontSize="10" fill="#f87171">limit {limit.toFixed(0)} {unit}</text>
+ </g>
+ )}
+ {hover && (
+ <g pointerEvents="none">
+ <line x1={layout.x(hover.t)} x2={layout.x(hover.t)} y1={MARGIN.top} y2={height - MARGIN.bottom} stroke="#ffffff" strokeOpacity="0.35" />
+ {hover.avg !== null && <circle cx={layout.x(hover.t)} cy={layout.y(hover.avg)} r="4" fill={color} stroke="#0d1013" strokeWidth="2" />}
+ </g>
+ )}
+ </svg>
+ {hover && (
+ <div role="status" className="pointer-events-none absolute z-10 rounded-xl border border-white/10 bg-[#111417]/95 px-3 py-2 text-xs text-[#e6e7ea] shadow-lg" style={{ top: 0, left: Math.min(Math.max(layout.x(hover.t) - 70, 0), Math.max(width - 150, 0)) }}>
+ <div className="text-[#9aa0a6]">{formatDateTime(hover.t, utc)}</div>
+ <div>Average <span className="font-medium text-white">{format(hover.avg)}</span></div>
+ <div>Highest <span className="font-medium text-white">{format(hover.max)}</span></div>
+ <div className="text-[#7f8589]">{hover.samples} check-ins</div>
+ </div>
+ )}
+ </div>
+ )}
+ </figure>
+ )
+}
diff --git a/src/frontend/src/containerLogic.js b/src/frontend/src/containerLogic.js
new file mode 100644
index 0000000..f32adc1
--- /dev/null
+++ b/src/frontend/src/containerLogic.js
@@ -0,0 +1,15 @@
+// Pure helpers for the container cards: which need attention, and how to order them.
+
+// 0 = needs attention, 1 = running but unhealthy, 2 = fine.
+export function problemRank(container) {
+ if (['exited', 'dead', 'restarting', 'paused', 'created'].includes(container.status)) return 0
+ return container.healthcheck_status === 'unhealthy' ? 1 : 2
+}
+
+export function sortContainers(containers, sort) {
+ const byName = (a, b) => a.name.localeCompare(b.name)
+ const descending = (key) => (a, b) => (b[key] ?? -1) - (a[key] ?? -1) || byName(a, b)
+ const order = { name: byName, memory: descending('memory_used_mb'), cpu: descending('cpu_pct'), problems: (a, b) => problemRank(a) - problemRank(b) || byName(a, b) }
+ return [...containers].sort(order[sort] || byName)
+}
+
diff --git a/src/frontend/src/distributions.jsx b/src/frontend/src/distributions.jsx
new file mode 100644
index 0000000..2b58552
--- /dev/null
+++ b/src/frontend/src/distributions.jsx
@@ -0,0 +1,195 @@
+// More chart types in plain SVG and HTML: stacked columns (also used for histograms), donut, ranked bars and a
+// heatmap. Everything that comes from the server (names, labels) is rendered as text by React, never as HTML.
+import { useEffect, useRef, useState } from 'react'
+import { donutSlices } from './chartMath.js'
+
+// The frame every chart sits in: a title, the "view as table" switch and the table itself.
+function ChartFrame({ title, subtitle, table, children, empty }) {
+ const [asTable, setAsTable] = useState(false)
+ return (
+ <figure className="min-w-0 rounded-2xl border border-[#1f2326] bg-[#0d1013]/60 p-4">
+ <div className="flex items-start justify-between gap-3">
+ <div className="min-w-0">
+ <figcaption className="text-sm font-medium text-white">{title}</figcaption>
+ {subtitle && <div className="text-xs text-[#7f8589]">{subtitle}</div>}
+ </div>
+ {!empty && table && <button type="button" onClick={() => setAsTable((value) => !value)} className="shrink-0 rounded-lg border border-white/10 px-2 py-1 text-[11px] text-[#9aa0a6] hover:text-white">{asTable ? 'View as chart' : 'View as table'}</button>}
+ </div>
+ {empty ? (
+ <div className="mt-3 flex min-h-32 items-center justify-center rounded-xl border border-dashed border-neutral-800 px-3 text-center text-xs text-[#7f8589]">{empty}</div>
+ ) : asTable && table ? (
+ <div className="mt-3 max-h-64 overflow-auto rounded-xl border border-[#1f2326]">
+ <table className="w-full text-left text-xs text-[#bcbcbc]">
+ <thead className="sticky top-0 bg-[#111417] text-[#9aa0a6]"><tr>{table.head.map((cell) => <th key={cell} className="px-3 py-2 font-medium">{cell}</th>)}</tr></thead>
+ <tbody>{table.rows.map((row, index) => <tr key={index} className="border-t border-[#1f2326]">{row.map((cell, cellIndex) => <td key={cellIndex} className="px-3 py-1.5">{cell}</td>)}</tr>)}</tbody>
+ </table>
+ </div>
+ ) : <div className="mt-3">{children}</div>}
+ </figure>
+ )
+}
+
+function Legend({ items }) {
+ return (
+ <ul className="mt-2 flex flex-wrap gap-x-4 gap-y-1 text-xs text-[#bcbcbc]">
+ {items.map((item) => <li key={item.label} className="flex items-center gap-1.5"><span aria-hidden="true" className="h-2.5 w-2.5 rounded-sm" style={{ background: item.color }} />{item.label}{item.value !== undefined && <span className="text-[#7f8589]">{item.value}</span>}</li>)}
+ </ul>
+ )
+}
+
+function useWidth(ref, fallback = 480) {
+ const [width, setWidth] = useState(fallback)
+ useEffect(() => {
+ const element = ref.current
+ if (!element) return undefined
+ const update = () => setWidth(Math.max(Math.round(element.getBoundingClientRect().width), 160))
+ update()
+ if (typeof ResizeObserver === 'undefined') return undefined
+ const observer = new ResizeObserver(update)
+ observer.observe(element)
+ return () => observer.disconnect()
+ }, [ref])
+ return width
+}
+
+// Columns over categories (days, bins, ...), optionally stacked. categories: [{label, values: {key: number}}].
+export function ColumnChart({ title, subtitle, categories, series, height = 190, unit = '', empty = 'Nothing to show for this range yet', labelEvery = 1 }) {
+ const wrapper = useRef(null)
+ const width = useWidth(wrapper)
+ const [hover, setHover] = useState(null)
+ const margin = { top: 8, right: 8, bottom: 26, left: 34 }
+ const totals = categories.map((category) => series.reduce((sum, item) => sum + (category.values[item.key] || 0), 0))
+ const top = Math.max(1, ...totals)
+ const niceTop = (() => { const magnitude = 10 ** Math.floor(Math.log10(top)); return [1, 2, 5, 10].map((f) => f * magnitude).find((value) => value >= top) })()
+ const plotWidth = Math.max(width - margin.left - margin.right, 40)
+ const plotHeight = height - margin.top - margin.bottom
+ const slot = plotWidth / Math.max(categories.length, 1)
+ const barWidth = Math.max(Math.min(slot * 0.72, 40), 1)
+ const y = (value) => margin.top + plotHeight - (value / niceTop) * plotHeight
+ const isEmpty = !categories.length || totals.every((total) => total === 0)
+ const table = { head: ['', ...series.map((item) => item.label)], rows: categories.map((category) => [category.label, ...series.map((item) => category.values[item.key] || 0)]) }
+
+ return (
+ <ChartFrame title={title} subtitle={subtitle} table={table} empty={isEmpty ? empty : null}>
+ <div ref={wrapper} className="relative w-full min-w-0 overflow-hidden" onPointerLeave={() => setHover(null)}>
+ <svg width={width} height={height} role="img" aria-label={`${title}: ${categories.length} columns, highest ${top}${unit}`} className="block">
+ {[0, niceTop / 2, niceTop].map((tick) => (
+ <g key={tick}>
+ <line x1={margin.left} x2={width - margin.right} y1={y(tick)} y2={y(tick)} stroke="#ffffff" strokeOpacity="0.07" />
+ <text x={margin.left - 6} y={y(tick) + 4} textAnchor="end" fontSize="10" fill="#7f8589">{Number(tick.toFixed(2))}</text>
+ </g>
+ ))}
+ {categories.map((category, index) => {
+ const x = margin.left + slot * index + (slot - barWidth) / 2
+ let base = 0
+ return (
+ <g key={category.label} onPointerEnter={() => setHover({ index, x: x + barWidth / 2 })} onPointerDown={() => setHover({ index, x: x + barWidth / 2 })}>
+ <rect x={margin.left + slot * index} y={margin.top} width={slot} height={plotHeight} fill="transparent" />
+ {series.map((item) => {
+ const value = category.values[item.key] || 0
+ if (!value) return null
+ const rect = <rect key={item.key} x={x} y={y(base + value)} width={barWidth} height={Math.max(y(base) - y(base + value), 1)} fill={item.color} rx="2" opacity={hover && hover.index !== index ? 0.55 : 1} />
+ base += value
+ return rect
+ })}
+ {index % labelEvery === 0 && <text x={x + barWidth / 2} y={height - 8} textAnchor="middle" fontSize="10" fill="#7f8589">{category.label}</text>}
+ </g>
+ )
+ })}
+ </svg>
+ {hover && (
+ <div role="status" className="pointer-events-none absolute z-10 rounded-xl border border-white/10 bg-[#111417]/95 px-3 py-2 text-xs text-[#e6e7ea] shadow-lg" style={{ top: 0, left: Math.min(Math.max(hover.x - 60, 0), Math.max(width - 140, 0)) }}>
+ <div className="text-[#9aa0a6]">{categories[hover.index].label}</div>
+ {series.map((item) => <div key={item.key}><span aria-hidden="true" className="mr-1.5 inline-block h-2 w-2 rounded-sm" style={{ background: item.color }} />{item.label} <span className="font-medium text-white">{categories[hover.index].values[item.key] || 0}{unit}</span></div>)}
+ </div>
+ )}
+ </div>
+ {series.length > 1 && <Legend items={series} />}
+ </ChartFrame>
+ )
+}
+
+// A donut with the total in the middle and a legend that also says the numbers, so colour is never the only cue.
+export function Donut({ title, subtitle, slices, centreLabel = 'total', empty = 'Nothing to show yet' }) {
+ const parts = donutSlices(slices)
+ const total = parts.reduce((sum, slice) => sum + slice.value, 0)
+ const radius = 54
+ const circumference = 2 * Math.PI * radius
+ const table = { head: ['', 'Count', 'Share'], rows: parts.map((slice) => [slice.label, slice.value, `${(slice.fraction * 100).toFixed(1)}%`]) }
+ return (
+ <ChartFrame title={title} subtitle={subtitle} table={table} empty={total === 0 ? empty : null}>
+ <div className="flex flex-wrap items-center gap-5">
+ <svg width="140" height="140" viewBox="0 0 140 140" role="img" aria-label={`${title}: ${parts.map((slice) => `${slice.label} ${slice.value}`).join(', ')}`} className="shrink-0">
+ <g transform="rotate(-90 70 70)">
+ {parts.map((slice) => (
+ <circle key={slice.label} cx="70" cy="70" r={radius} fill="none" stroke={slice.color} strokeWidth="18" strokeDasharray={`${Math.max(slice.fraction * circumference - (parts.length > 1 ? 1.5 : 0), 0.5)} ${circumference}`} strokeDashoffset={-slice.start * circumference} />
+ ))}
+ </g>
+ <text x="70" y="68" textAnchor="middle" fontSize="24" fontWeight="600" fill="#ffffff">{total}</text>
+ <text x="70" y="86" textAnchor="middle" fontSize="10" fill="#7f8589">{centreLabel}</text>
+ </svg>
+ <Legend items={parts.map((slice) => ({ label: slice.label, color: slice.color, value: `${slice.value}` }))} />
+ </div>
+ </ChartFrame>
+ )
+}
+
+// Ranked horizontal bars (top containers, servers by disk usage ...). rows: [{label, value, detail?}].
+export function BarList({ title, subtitle, rows, max = null, unit = '', color = '#7dd3fc', decimals = 0, empty = 'Nothing to show yet' }) {
+ const scale = max ?? Math.max(1, ...rows.map((row) => row.value))
+ const table = { head: ['', `Value${unit ? ` (${unit.trim()})` : ''}`], rows: rows.map((row) => [row.label, row.value.toFixed(decimals)]) }
+ return (
+ <ChartFrame title={title} subtitle={subtitle} table={table} empty={rows.length === 0 ? empty : null}>
+ <ul className="space-y-2">
+ {rows.map((row) => (
+ <li key={row.label}>
+ <div className="flex items-baseline justify-between gap-3 text-xs">
+ <span className="min-w-0 truncate text-[#cfcfcf]">{row.label}</span>
+ <span className="shrink-0 text-white">{row.value.toFixed(decimals)}{unit}{row.detail ? <span className="text-[#7f8589]"> {row.detail}</span> : null}</span>
+ </div>
+ <div className="mt-1 h-2 overflow-hidden rounded-full bg-white/10"><div className="h-full rounded-full" style={{ width: `${Math.min(Math.max((row.value / scale) * 100, 0), 100)}%`, background: row.color || color }} /></div>
+ </li>
+ ))}
+ </ul>
+ </ChartFrame>
+ )
+}
+
+const DAY_NAMES = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
+
+// Average by hour of day (columns) and weekday (rows). grid: 7 rows of 24 numbers or null.
+export function Heatmap({ title, subtitle, grid, unit = '%', color = '#7dd3fc', empty = 'Not enough history yet' }) {
+ const [hover, setHover] = useState(null)
+ const values = grid.flat().filter((value) => value !== null)
+ const top = Math.max(1, ...values)
+ const table = { head: ['', ...Array.from({ length: 24 }, (_, hour) => `${hour}`)], rows: grid.map((row, day) => [DAY_NAMES[day], ...row.map((value) => (value === null ? '–' : value.toFixed(0)))]) }
+ return (
+ <ChartFrame title={title} subtitle={subtitle} table={table} empty={values.length === 0 ? empty : null}>
+ <div className="relative">
+ <div className="overflow-x-auto">
+ <div className="min-w-[420px]">
+ {grid.map((row, day) => (
+ <div key={DAY_NAMES[day]} className="flex items-center gap-1 py-px">
+ <span className="w-8 shrink-0 text-[10px] text-[#7f8589]">{DAY_NAMES[day]}</span>
+ {row.map((value, hour) => (
+ <button
+ key={hour}
+ type="button"
+ aria-label={`${DAY_NAMES[day]} ${hour}:00, ${value === null ? 'no data' : `${value.toFixed(0)}${unit}`}`}
+ onPointerEnter={() => setHover({ day, hour, value })}
+ onFocus={() => setHover({ day, hour, value })}
+ onPointerLeave={() => setHover(null)}
+ className="h-5 flex-1 rounded-[3px]"
+ style={{ background: value === null ? 'rgba(255,255,255,0.04)' : color, opacity: value === null ? 1 : 0.15 + 0.85 * (value / top) }}
+ />
+ ))}
+ </div>
+ ))}
+ <div className="mt-1 flex gap-1 pl-9 text-[10px] text-[#7f8589]">{[0, 6, 12, 18, 23].map((hour) => <span key={hour} style={{ position: 'relative', left: `${(hour / 24) * 100 - (hour === 23 ? 3 : 0)}%`, width: 0 }}>{hour}h</span>)}</div>
+ </div>
+ </div>
+ <div className="mt-2 text-xs text-[#9aa0a6]" role="status">{hover ? `${DAY_NAMES[hover.day]} ${hover.hour}:00 · ${hover.value === null ? 'no data' : `${hover.value.toFixed(1)}${unit} on average`}` : `Darker is busier, up to ${top.toFixed(0)}${unit}.`}</div>
+ </div>
+ </ChartFrame>
+ )
+}
diff --git a/src/frontend/src/fleetLogic.js b/src/frontend/src/fleetLogic.js
new file mode 100644
index 0000000..1642876
--- /dev/null
+++ b/src/frontend/src/fleetLogic.js
@@ -0,0 +1,98 @@
+// Pure helpers for the all-servers view: what to count, rank and label. No React in here.
+
+const MONTHS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
+const pad = (value) => String(value).padStart(2, '0')
+
+export const CONTAINER_STATES = [
+ { key: 'running', label: 'Running', color: '#34d399' },
+ { key: 'unhealthy', label: 'Unhealthy', color: '#f87171' },
+ { key: 'restarting', label: 'Restarting', color: '#fbbf24' },
+ { key: 'stopped', label: 'Stopped', color: '#94a3b8' },
+]
+
+export function containerState(container) {
+ if (container.status === 'running') return container.healthcheck_status === 'unhealthy' ? 'unhealthy' : 'running'
+ return container.status === 'restarting' ? 'restarting' : 'stopped'
+}
+
+export function containerStateSlices(servers) {
+ const counts = { running: 0, unhealthy: 0, restarting: 0, stopped: 0 }
+ for (const server of servers) for (const container of server.containers || []) counts[containerState(container)] += 1
+ return CONTAINER_STATES.map((state) => ({ label: state.label, color: state.color, value: counts[state.key] }))
+}
+
+// The containers using the most of something right now, across all servers: [{label: 'server / name', value}].
+export function topContainers(servers, key, count = 10) {
+ const rows = []
+ for (const server of servers) {
+ for (const container of server.containers || []) {
+ const value = container[key]
+ if (typeof value === 'number' && Number.isFinite(value)) rows.push({ label: `${server.hostname} / ${container.name}`, value })
+ }
+ }
+ return rows.sort((a, b) => b.value - a.value || a.label.localeCompare(b.label)).slice(0, count)
+}
+
+// Servers by how full their root disk is, coloured the way the dashboard colours it.
+export function serversByDisk(servers, warning = 85, critical = 95) {
+ return servers
+ .filter((server) => typeof server.host?.disk_usage_pct === 'number')
+ .map((server) => ({ label: server.hostname, value: server.host.disk_usage_pct, color: server.host.disk_usage_pct >= critical ? '#f87171' : server.host.disk_usage_pct >= warning ? '#fbbf24' : '#7dd3fc' }))
+ .sort((a, b) => b.value - a.value || a.label.localeCompare(b.label))
+}
+
+export function isOnline(server, now = Date.now()) {
+ if (!server.last_seen_at) return false
+ return now - new Date(server.last_seen_at).getTime() < (server.check_interval_seconds || 60) * 3 * 1000
+}
+
+export function availabilityRows(availability) {
+ return availability
+ .filter((item) => item.percent !== null && item.percent !== undefined)
+ .map((item) => ({ label: item.hostname, value: item.percent, color: item.percent >= 99 ? '#34d399' : item.percent >= 95 ? '#fbbf24' : '#f87171', detail: `${item.checkins} of about ${item.expected} check-ins` }))
+ .sort((a, b) => a.value - b.value || a.label.localeCompare(b.label))
+}
+
+export function formatDuration(minutes) {
+ if (minutes === null || minutes === undefined || !Number.isFinite(minutes)) return '–'
+ if (minutes < 1) return 'under a minute'
+ if (minutes < 120) return `${Math.round(minutes)} min`
+ if (minutes < 48 * 60) return `${(minutes / 60).toFixed(1)} h`
+ return `${(minutes / 1440).toFixed(1)} days`
+}
+
+function startOfBucket(bucket, time) {
+ const date = new Date(time)
+ if (bucket === 'hour') return Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate(), date.getUTCHours())
+ const day = Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate())
+ if (bucket === 'day') return day
+ if (bucket === 'week') return day - ((date.getUTCDay() + 6) % 7) * 86400000
+ return Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), 1)
+}
+
+function nextBucket(bucket, start) {
+ if (bucket === 'hour') return start + 3600000
+ if (bucket === 'day') return start + 86400000
+ if (bucket === 'week') return start + 7 * 86400000
+ const date = new Date(start)
+ return Date.UTC(date.getUTCFullYear(), date.getUTCMonth() + 1, 1)
+}
+
+export function bucketLabel(time, bucket, utcHours = false) {
+ const date = new Date(time)
+ if (bucket === 'hour') return `${pad(utcHours ? date.getUTCHours() : date.getHours())}:00`
+ if (bucket === 'month') return `${MONTHS[date.getUTCMonth()]} ${date.getUTCFullYear()}`
+ return `${date.getUTCDate()} ${MONTHS[date.getUTCMonth()]}`
+}
+
+// One column for every bucket from ``start`` to ``end``, with zeros where nothing happened, so a quiet day shows as
+// quiet instead of missing. At most ``limit`` (the newest) buckets are returned.
+export function incidentColumns(byBucket, bucket, start, end, limit = 120) {
+ const known = new Map(byBucket.map((entry) => [startOfBucket(bucket, new Date(entry.time).getTime()), entry]))
+ const columns = []
+ for (let moment = startOfBucket(bucket, new Date(start).getTime()); moment <= new Date(end).getTime(); moment = nextBucket(bucket, moment)) {
+ const entry = known.get(moment)
+ columns.push({ time: moment, values: { critical: entry?.critical ?? 0, warning: entry?.warning ?? 0, info: entry?.info ?? 0 } })
+ }
+ return columns.slice(-limit)
+}
diff --git a/src/frontend/tests/chartMath.test.js b/src/frontend/tests/chartMath.test.js
new file mode 100644
index 0000000..c5e1b07
--- /dev/null
+++ b/src/frontend/tests/chartMath.test.js
@@ -0,0 +1,93 @@
+// Tests for the pure chart helpers. Run with `node --test` in this folder (Node 20 or newer, no packages needed).
+import assert from 'node:assert/strict'
+import { test } from 'node:test'
+import { daysUntilFull, donutSlices, formatTime, histogramBins, hourWeekdayGrid, niceTicks, seriesFor, splitRuns, summarise } from '../src/chartMath.js'
+
+const DAY = 86400000
+
+test('histogram: values land in the right bin, weights count, and out-of-range values are left out', () => {
+ const bins = histogramBins([0, 5, 10, 99, 100, 100.1, -1, NaN, { value: 55, weight: 3 }])
+ assert.equal(bins.length, 10)
+ assert.deepEqual([bins[0].count, bins[1].count, bins[5].count, bins[9].count], [2, 1, 3, 2]) // 100 belongs in the last bin
+ assert.equal(histogramBins([1], { low: 0, high: 1, bins: 4 })[3].count, 1)
+ assert.equal(histogramBins([]).every((bin) => bin.count === 0), true)
+})
+
+test('hour-by-weekday grid: weighted average per cell, Monday first, empty cells stay null', () => {
+ const monday = Date.UTC(2026, 2, 2, 10, 0)
+ const grid = hourWeekdayGrid([
+ { t: monday, avg: 40, samples: 1 },
+ { t: monday + 60000, avg: 80, samples: 3 },
+ { t: Date.UTC(2026, 2, 8, 23, 30), avg: 5, samples: 1 }, // Sunday 23:30
+ { t: monday, avg: null, samples: 9 }, // unknown is not zero
+ ], true)
+ assert.equal(grid.length, 7)
+ assert.equal(grid[0].length, 24)
+ assert.equal(grid[0][10], 70)
+ assert.equal(grid[6][23], 5)
+ assert.equal(grid[3][3], null)
+})
+
+test('donut: invalid and zero slices are dropped and fractions add up', () => {
+ const slices = donutSlices([{ label: 'a', value: 3 }, { label: 'b', value: 0 }, { label: 'c', value: 1 }, { label: 'x', value: NaN }, { label: 'n', value: -2 }])
+ assert.deepEqual(slices.map((slice) => [slice.label, slice.fraction, slice.start]), [['a', 0.75, 0], ['c', 0.25, 0.75]])
+ assert.deepEqual(donutSlices([]), [])
+})
+
+test('days until full: a clear rising trend gives an estimate, anything unclear gives none', () => {
+ const rising = Array.from({ length: 10 }, (_, index) => ({ t: index * DAY, value: 50 + index * 2 })) // 68 now, +2 a day
+ assert.ok(Math.abs(daysUntilFull(rising) - 16) < 1e-6)
+ assert.equal(daysUntilFull(rising.slice(0, 3)), null) // too few points
+ assert.equal(daysUntilFull(rising.map((point) => ({ ...point, value: 50 }))), null) // flat
+ assert.equal(daysUntilFull(rising.map((point) => ({ ...point, value: 90 - point.value / 10 }))), null) // falling
+ assert.equal(daysUntilFull(rising.map((point) => ({ t: point.t / 100, value: point.value }))), null) // under a day of data
+ assert.equal(daysUntilFull(rising.map((point, index) => ({ ...point, value: 50 + (index % 2) * 40 }))), null) // too noisy
+ assert.equal(daysUntilFull(rising.map((point) => ({ ...point, value: 95 + (point.t / DAY) * 0.6 }))), 0) // already full
+ assert.equal(daysUntilFull([]), null)
+})
+
+test('axis ticks reach the top of the data, and bad input does not break them', () => {
+ assert.deepEqual(niceTicks(0, 100, 4), [0, 25, 50, 75, 100])
+ assert.deepEqual(niceTicks(0, 1024, 4), [0, 500, 1000, 1500])
+ assert.deepEqual(niceTicks(NaN, 1), [0, 1])
+ assert.ok(niceTicks(5, 5).length >= 2)
+})
+
+test('a hole of more than a bucket and a half splits a series, so a gap stays a gap', () => {
+ assert.equal(splitRuns([{ t: 0 }, { t: 60000 }, { t: 400000 }], 60).length, 2)
+ assert.equal(splitRuns([{ t: 0 }, { t: 60000 }, { t: 120000 }], 60).length, 1)
+ assert.deepEqual(splitRuns([], 60), [])
+})
+
+test('series and summary: unknown values are skipped, the average is weighted and the maximum is kept', () => {
+ const points = [
+ { time: '2026-03-02T10:00:00+00:00', samples: 1, cpu: { avg: 100, max: 100 } },
+ { time: '2026-03-02T11:00:00+00:00', samples: 3, cpu: { avg: 0, max: 40 } },
+ { time: '2026-03-02T12:00:00+00:00', samples: 5, cpu: null },
+ ]
+ const series = seriesFor(points, 'cpu')
+ assert.equal(series.length, 2)
+ assert.deepEqual(summarise(series), { average: 25, maximum: 100 })
+ assert.deepEqual(summarise([]), { average: null, maximum: null })
+})
+
+test('time labels follow the span of the chart', () => {
+ const moment = Date.UTC(2026, 2, 5, 9, 7)
+ assert.equal(formatTime(moment, 3600, true), '09:07')
+ assert.equal(formatTime(moment, 3 * 86400, true), '5 Mar 09:07')
+ assert.equal(formatTime(moment, 90 * 86400, true), '5 Mar')
+ assert.equal(formatTime(moment, 800 * 86400, true), 'Mar 2026')
+})
+
+test('forecast wording: honest about missing trends and sensible at every scale', async () => {
+ const { formatForecast } = await import('../src/chartMath.js')
+ assert.match(formatForecast(null), /No clear growth trend/)
+ assert.match(formatForecast(NaN), /No clear growth trend/)
+ assert.equal(formatForecast(0), 'Full now.')
+ assert.match(formatForecast(0.4), /less than a day/)
+ assert.equal(formatForecast(1), 'Full in about 1 day at this rate.')
+ assert.equal(formatForecast(16.2), 'Full in about 16 days at this rate.')
+ assert.equal(formatForecast(120), 'Full in about 4 months at this rate.')
+ assert.equal(formatForecast(1000), 'Full in about 2.7 years at this rate.')
+ assert.match(formatForecast(9999), /Not full within ten years/)
+})
diff --git a/src/frontend/tests/fleetLogic.test.js b/src/frontend/tests/fleetLogic.test.js
new file mode 100644
index 0000000..bd505dd
--- /dev/null
+++ b/src/frontend/tests/fleetLogic.test.js
@@ -0,0 +1,69 @@
+// Tests for the all-servers helpers. Run with `node --test` in this folder.
+import assert from 'node:assert/strict'
+import { test } from 'node:test'
+import { availabilityRows, bucketLabel, containerState, containerStateSlices, formatDuration, incidentColumns, isOnline, serversByDisk, topContainers } from '../src/fleetLogic.js'
+
+const servers = [
+ { hostname: 'nas', host: { disk_usage_pct: 91 }, containers: [
+ { name: 'web', status: 'running', healthcheck_status: 'healthy', memory_used_mb: 200, cpu_pct: 5 },
+ { name: 'db', status: 'running', healthcheck_status: 'unhealthy', memory_used_mb: 900, cpu_pct: null },
+ { name: 'job', status: 'exited', memory_used_mb: null, cpu_pct: null },
+ ] },
+ { hostname: 'web01', host: { disk_usage_pct: 97 }, containers: [
+ { name: 'proxy', status: 'restarting', memory_used_mb: 50, cpu_pct: 40 },
+ { name: 'cache', status: 'running', memory_used_mb: 500, cpu_pct: 12 },
+ ] },
+ { hostname: 'empty', host: null, containers: undefined },
+]
+
+test('every container falls into exactly one state', () => {
+ assert.equal(containerState({ status: 'running' }), 'running')
+ assert.equal(containerState({ status: 'running', healthcheck_status: 'unhealthy' }), 'unhealthy')
+ assert.equal(containerState({ status: 'restarting' }), 'restarting')
+ for (const status of ['exited', 'dead', 'created', 'paused', 'weird']) assert.equal(containerState({ status }), 'stopped')
+ const slices = containerStateSlices(servers)
+ assert.deepEqual(slices.map((slice) => [slice.label, slice.value]), [['Running', 2], ['Unhealthy', 1], ['Restarting', 1], ['Stopped', 1]])
+ assert.deepEqual(containerStateSlices([]).map((slice) => slice.value), [0, 0, 0, 0])
+})
+
+test('top containers rank across servers, skip unknown values and honour the count', () => {
+ assert.deepEqual(topContainers(servers, 'memory_used_mb', 3), [{ label: 'nas / db', value: 900 }, { label: 'web01 / cache', value: 500 }, { label: 'nas / web', value: 200 }])
+ assert.deepEqual(topContainers(servers, 'cpu_pct').map((row) => row.label), ['web01 / proxy', 'web01 / cache', 'nas / web'])
+ assert.deepEqual(topContainers([], 'cpu_pct'), [])
+})
+
+test('servers are ranked by disk usage and coloured by the thresholds; servers without data are left out', () => {
+ assert.deepEqual(serversByDisk(servers).map((row) => [row.label, row.value, row.color]), [['web01', 97, '#f87171'], ['nas', 91, '#fbbf24']])
+ assert.equal(serversByDisk([{ hostname: 'x', host: { disk_usage_pct: 40 } }])[0].color, '#7dd3fc')
+})
+
+test('a server is online while it keeps reporting', () => {
+ const now = Date.UTC(2026, 2, 5, 12, 0)
+ assert.equal(isOnline({ last_seen_at: new Date(now - 100000).toISOString(), check_interval_seconds: 60 }, now), true)
+ assert.equal(isOnline({ last_seen_at: new Date(now - 200000).toISOString(), check_interval_seconds: 60 }, now), false)
+ assert.equal(isOnline({ last_seen_at: null }, now), false)
+})
+
+test('availability rows skip unknown servers, put the worst first and colour by how bad it is', () => {
+ const rows = availabilityRows([{ hostname: 'a', percent: 100, checkins: 10, expected: 10 }, { hostname: 'b', percent: 96.5, checkins: 9, expected: 10 }, { hostname: 'c', percent: null, checkins: 0, expected: 5 }, { hostname: 'd', percent: 80, checkins: 8, expected: 10 }])
+ assert.deepEqual(rows.map((row) => [row.label, row.color]), [['d', '#f87171'], ['b', '#fbbf24'], ['a', '#34d399']])
+})
+
+test('durations read naturally', () => {
+ assert.equal(formatDuration(null), '–')
+ assert.equal(formatDuration(0.2), 'under a minute')
+ assert.equal(formatDuration(45), '45 min')
+ assert.equal(formatDuration(300), '5.0 h')
+ assert.equal(formatDuration(60 * 24 * 5), '5.0 days')
+ assert.equal(formatDuration(NaN), '–')
+})
+
+test('incident columns fill quiet buckets with zeros, in order, and keep only the newest ones', () => {
+ const days = incidentColumns([{ time: '2026-03-10T00:00:00+00:00', critical: 1, warning: 2, info: 0 }], 'day', '2026-03-08T10:00:00+00:00', '2026-03-11T12:00:00+00:00')
+ assert.deepEqual(days.map((column) => [bucketLabel(column.time, 'day'), column.values.critical + column.values.warning]), [['8 Mar', 0], ['9 Mar', 0], ['10 Mar', 3], ['11 Mar', 0]])
+ const weeks = incidentColumns([], 'week', '2026-03-04T00:00:00+00:00', '2026-03-20T00:00:00+00:00')
+ assert.deepEqual(weeks.map((column) => bucketLabel(column.time, 'week')), ['2 Mar', '9 Mar', '16 Mar']) // Mondays
+ const months = incidentColumns([], 'month', '2026-11-15T00:00:00+00:00', '2027-02-02T00:00:00+00:00')
+ assert.deepEqual(months.map((column) => bucketLabel(column.time, 'month')), ['Nov 2026', 'Dec 2026', 'Jan 2027', 'Feb 2027'])
+ assert.equal(incidentColumns([], 'hour', '2026-03-01T00:00:00+00:00', '2026-03-20T00:00:00+00:00', 5).length, 5)
+})
diff --git a/tests/test_history.py b/tests/test_history.py
new file mode 100644
index 0000000..0cf9766
--- /dev/null
+++ b/tests/test_history.py
@@ -0,0 +1,330 @@
+import os
+from datetime import datetime, timedelta, timezone
+
+os.environ.setdefault("DATABASE_URL", "sqlite://")
+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")
+
+import pytest
+from fastapi.testclient import TestClient
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+from sqlalchemy.pool import StaticPool
+
+from src.backend import history, main, rollups
+from src.backend.models import AppSetting, Base, ContainerMetric, HostMetric, HostRollup, MountMetric, Server, User
+
+NOW = datetime(2026, 3, 11, 12, 30, tzinfo=timezone.utc) # a Wednesday
+
+
+def db():
+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)
+ Base.metadata.create_all(engine)
+ session = sessionmaker(bind=engine)()
+ session.add(User(id=1, name="Owner", email="[email protected]", password_hash="x", role="admin"))
+ session.add_all(Server(id=n, owner_id=1, hostname=f"srv{n}", check_interval_seconds=60) for n in (1, 2))
+ session.commit()
+ session.engine = engine
+ return session
+
+
+def seed(db, start, count, step=timedelta(minutes=1), server=1, cpu=lambda i: float(i % 100), **extra):
+ db.add_all(HostMetric(server_id=server, time=start + step * i, cpu_utilization_pct=cpu(i), ram_pct=50.0, **extra) for i in range(count))
+ db.commit()
+
+
+def ask(db, range_name, metrics=("cpu",), server=1, spec=history.HOST, key=None, now=NOW):
+ return history.history(db, spec, server, key, range_name, metrics, 60, now=now)
+
+
+def times(result):
+ return [datetime.fromisoformat(point["time"]) for point in result["points"]]
+
+
+# --- short ranges come from the check-in samples -------------------------------------------------
+
+def test_the_last_hour_is_one_point_per_check_in_with_its_average_and_maximum(db):
+ seed(db, NOW - timedelta(minutes=90), 91)
+
+ result = ask(db, "1h", ("cpu", "ram"))
+
+ assert result["source"] == "detailed" and result["bucket_seconds"] == 60 and result["range"] == "1h"
+ assert len(result["points"]) == 61 # both ends of the hour are included
+ point = result["points"][-1]
+ assert point["samples"] == 1 and point["cpu"]["avg"] == point["cpu"]["max"] and point["ram"] == {"avg": 50.0, "max": 50.0}
+ assert times(result) == sorted(times(result)) and all(moment.tzinfo is not None for moment in times(result))
+
+
+def test_a_week_of_samples_is_grouped_into_at_most_five_hundred_points_without_losing_anything(db):
+ seed(db, NOW - timedelta(days=7), 7 * 24 * 60 + 1)
+
+ result = ask(db, "7d")
+
+ assert result["source"] == "detailed" and result["bucket_seconds"] == 1800 and len(result["points"]) <= history.MAX_POINTS
+ assert sum(point["samples"] for point in result["points"]) == 7 * 24 * 60 + 1
+ first = result["points"][0]
+ assert first["cpu"]["max"] >= first["cpu"]["avg"]
+ raw = [row[0] for row in db.query(HostMetric.cpu_utilization_pct).filter(HostMetric.time >= NOW - timedelta(days=7))]
+ total = sum(point["cpu"]["avg"] * point["samples"] for point in result["points"])
+ assert total / len(raw) == pytest.approx(sum(raw) / len(raw))
+ assert max(point["cpu"]["max"] for point in result["points"]) == max(raw)
+
+
+def test_grouping_keeps_the_exact_average_and_maximum_of_each_group(db):
+ seed(db, NOW - timedelta(hours=23), 1) # makes the data span a whole day
+ start = NOW - timedelta(hours=5)
+ for minute, cpu in ((0, 10.0), (1, 20.0), (2, 90.0), (6, 40.0)): # the first three share a five-minute group
+ seed(db, start + timedelta(minutes=minute), 1, cpu=lambda i, cpu=cpu: cpu)
+
+ result = ask(db, "24h")
+
+ grouped = [point for point in result["points"] if point["samples"] > 1]
+ assert result["bucket_seconds"] == 300 and len(grouped) == 1
+ assert grouped[0]["cpu"] == {"avg": pytest.approx(40.0), "max": 90.0} and grouped[0]["samples"] == 3
+
+
+def test_a_gap_stays_a_gap(db):
+ seed(db, NOW - timedelta(hours=20), 10)
+ seed(db, NOW - timedelta(hours=2), 10)
+
+ result = ask(db, "24h")
+
+ moments = times(result)
+ assert max(b - a for a, b in zip(moments, moments[1:])) > timedelta(hours=15) # nothing invented in between
+ assert sum(point["samples"] for point in result["points"]) == 20
+
+
+def test_a_metric_that_was_never_reported_is_null_not_zero(db):
+ seed(db, NOW - timedelta(hours=1), 5)
+
+ point = ask(db, "1h", ("cpu", "swap", "docker_ms"))["points"][-1]
+
+ assert point["swap"] is None and point["docker_ms"] is None and point["cpu"] is not None
+
+
+def test_a_server_without_data_gives_an_empty_series(db):
+ for name in history.RANGES:
+ result = ask(db, name, server=2)
+ assert result["points"] == [] and result["data_since"] is None
+
+
+# --- long ranges come from the summaries, topped up with the newest samples -----------------------
+
+def test_thirty_days_use_hourly_summaries_and_match_the_samples(db):
+ seed(db, NOW - timedelta(days=10), 10 * 24 * 6, step=timedelta(minutes=10))
+ rollups.run_rollups(db, NOW)
+
+ result = ask(db, "30d")
+
+ assert result["source"] == "hour" and result["bucket_seconds"] == 3600
+ assert sum(point["samples"] for point in result["points"]) == 10 * 24 * 6
+ first = result["points"][0]
+ raw = [row[0] for row in db.query(HostMetric.cpu_utilization_pct).filter(HostMetric.time >= datetime.fromisoformat(first["time"]), HostMetric.time < datetime.fromisoformat(first["time"]) + timedelta(hours=1))]
+ assert first["cpu"] == {"avg": pytest.approx(sum(raw) / len(raw)), "max": max(raw)} and first["samples"] == len(raw)
+
+
+def test_the_hour_that_has_not_been_summarised_yet_is_still_in_the_answer(db):
+ seed(db, NOW - timedelta(days=3), 3 * 24 * 60)
+ rollups.run_rollups(db, NOW)
+ seed(db, NOW - timedelta(minutes=20), 20, cpu=lambda i: 1000.0) # after the last closed hour
+
+ result = ask(db, "30d")
+
+ last = result["points"][-1]
+ assert datetime.fromisoformat(last["time"]) == rollups.floor(rollups.HOUR, NOW) and last["cpu"]["max"] == 1000.0
+
+
[email protected]("range_name", ["30d", "90d", "1y", "all"])
+def test_the_answer_does_not_depend_on_how_far_the_rollup_job_has_got(db, monkeypatch, range_name):
+ seed(db, NOW - timedelta(days=25), 25 * 24 * 4, step=timedelta(minutes=15), cpu=lambda i: float((i * 7) % 101))
+ answers = [ask(db, range_name)] # no summaries at all yet
+ monkeypatch.setitem(rollups.MAX_BUCKETS_PER_RUN, rollups.HOUR, 200)
+ monkeypatch.setitem(rollups.MAX_BUCKETS_PER_RUN, rollups.DAY, 5)
+ for _ in range(3):
+ rollups.run_rollups(db, NOW)
+ answers.append(ask(db, range_name)) # partly caught up
+ monkeypatch.undo()
+ for _ in range(20):
+ if not any(rollups.run_rollups(db, NOW).values()):
+ break
+ answers.append(ask(db, range_name)) # fully caught up
+
+ assert len({repr(answer["points"]) for answer in answers}) == 1
+ assert sum(point["samples"] for point in answers[0]["points"]) > 0
+
+
+def test_every_sample_is_counted_exactly_once_in_the_all_time_view(db):
+ seed(db, NOW - timedelta(days=75), 75 * 24 * 2, step=timedelta(minutes=30))
+ rollups.run_rollups(db, NOW)
+ for _ in range(10):
+ rollups.run_rollups(db, NOW)
+
+ result = ask(db, "all")
+
+ assert result["source"] == "month" and result["start"] == result["data_since"]
+ assert sum(point["samples"] for point in result["points"]) == 75 * 24 * 2
+ assert [datetime.fromisoformat(point["time"]).day for point in result["points"]] == [1, 1, 1, 1] # Dec, Jan, Feb, Mar (partial)
+
+
+def test_weekly_points_start_on_mondays_and_the_current_week_is_partial(db):
+ seed(db, NOW - timedelta(days=30), 30 * 24, step=timedelta(hours=1))
+ for _ in range(10):
+ rollups.run_rollups(db, NOW)
+
+ result = ask(db, "1y")
+
+ assert result["source"] == "week" and result["bucket_seconds"] == 7 * 86400
+ assert all(moment.weekday() == 0 and moment.hour == 0 for moment in times(result))
+ assert datetime.fromisoformat(result["points"][-1]["time"]) == rollups.floor(rollups.WEEK, NOW)
+ assert sum(point["samples"] for point in result["points"]) == 30 * 24
+
+
+def test_a_long_series_is_thinned_to_the_point_limit(db):
+ start = rollups.floor(rollups.HOUR, NOW - timedelta(days=30))
+ db.add_all(HostRollup(server_id=1, tier="hour", bucket_start=start + timedelta(hours=i), samples=60, avg_cpu=float(i % 50), n_cpu=60, max_cpu=float(i % 50) + 5) for i in range(720))
+ db.add(AppSetting(key="rollup_done_hour", value=(start + timedelta(hours=720)).isoformat()))
+ db.commit()
+
+ result = ask(db, "30d", now=start + timedelta(hours=720))
+
+ assert len(result["points"]) <= history.MAX_POINTS and result["bucket_seconds"] == 7200
+ assert sum(point["samples"] for point in result["points"]) == 720 * 60
+ assert result["points"][0]["cpu"] == {"avg": pytest.approx(0.5), "max": 6.0} # hours 0 and 1: avg (0 + 1) / 2, max of 5 and 6
+
+
+# --- when the history starts ----------------------------------------------------------------------
+
+def test_history_starts_at_the_first_check_in_not_at_a_bucket_boundary(db):
+ first = datetime(2026, 3, 9, 9, 17, tzinfo=timezone.utc)
+ seed(db, first, 3 * 24 * 60)
+ rollups.run_rollups(db, NOW) # hour and day summaries now start earlier, at 09:00 and midnight
+
+ assert history.data_since(db, history.HOST, 1, None) == first
+ assert datetime.fromisoformat(ask(db, "all")["data_since"]) == first
+
+
+def test_when_old_samples_are_gone_the_oldest_summary_that_reaches_back_decides(db):
+ first = datetime(2026, 3, 9, 9, 17, tzinfo=timezone.utc)
+ seed(db, first, 3 * 24 * 60)
+ rollups.run_rollups(db, NOW)
+
+ db.query(HostMetric).filter(HostMetric.time < datetime(2026, 3, 10, tzinfo=timezone.utc)).delete()
+ db.commit()
+ assert history.data_since(db, history.HOST, 1, None) == datetime(2026, 3, 9, 9, tzinfo=timezone.utc) # hourly rows still reach 09:00
+
+ db.query(HostRollup).filter(HostRollup.tier == "hour").delete()
+ db.commit()
+ assert history.data_since(db, history.HOST, 1, None) == datetime(2026, 3, 9, tzinfo=timezone.utc) # only the daily row remains
+
+
+def test_a_series_with_no_data_has_no_start(db):
+ assert history.data_since(db, history.HOST, 2, None) is None
+
+
+# --- containers and mounts ---------------------------------------------------------------------
+
+def test_a_container_series_has_cpu_memory_limit_and_down_counts(db):
+ for minute in range(10):
+ db.add(ContainerMetric(server_id=1, time=NOW - timedelta(minutes=minute), container_id="abc", name="web", status="running" if minute < 7 else "exited",
+ cpu_pct=float(minute * 10), memory_used_mb=100.0 + minute, memory_limit_mb=512.0, restart_count=minute // 5))
+ db.add(ContainerMetric(server_id=1, time=NOW, container_id="other", name="db", status="running", cpu_pct=99.0))
+ db.commit()
+
+ result = history.history(db, history.CONTAINER, 1, "abc", "1h", ("cpu", "memory_mb", "memory_limit_mb", "restarts"), 60, now=NOW)
+
+ assert sum(point["samples"] for point in result["points"]) == 10 and sum(point["down"] for point in result["points"]) == 3
+ top = max(result["points"], key=lambda point: point["cpu"]["max"])
+ assert top["cpu"]["max"] == 90.0 and top["memory_limit_mb"] == {"max": 512.0} and top["restarts"] == {"max": 1} # limit and restarts have no average
+
+
+def test_a_mount_series_is_picked_by_path(db):
+ for mount, used in (("/", 10.0), ("/data", 90.0)):
+ db.add(MountMetric(server_id=1, time=NOW - timedelta(minutes=1), mount=mount, used_pct=used, used_gb=used, total_gb=100.0))
+ db.commit()
+
+ result = history.history(db, history.MOUNT, 1, "/data", "1h", ("used_pct", "used_gb", "total_gb"), 60, now=NOW)
+
+ point = result["points"][0]
+ assert point["used_pct"] == {"avg": 90.0, "max": 90.0} and point["used_gb"] == {"avg": 90.0} and point["total_gb"] == {"max": 100.0}
+
+
[email protected]("range_name", ["30d", "all"])
+def test_container_summaries_agree_with_the_samples(db, range_name):
+ for i in range(3 * 24 * 12):
+ db.add(ContainerMetric(server_id=1, time=NOW - timedelta(days=3) + timedelta(minutes=5 * i), container_id="abc", name="web", status="running", cpu_pct=float(i % 40), memory_used_mb=200.0))
+ db.commit()
+ before = history.history(db, history.CONTAINER, 1, "abc", range_name, ("cpu", "memory_mb"), 60, now=NOW)
+ for _ in range(10):
+ rollups.run_rollups(db, NOW)
+
+ after = history.history(db, history.CONTAINER, 1, "abc", range_name, ("cpu", "memory_mb"), 60, now=NOW)
+
+ assert after["points"] == before["points"] and sum(point["samples"] for point in after["points"]) == 3 * 24 * 12
+
+
+# --- the endpoints -------------------------------------------------------------------------------
+
+def client(db, monkeypatch):
+ main.app.dependency_overrides[main.get_db] = lambda: db
+ monkeypatch.setattr(main, "_current_user", lambda session_id, db: {"id": 1, "role": "user"})
+ yield TestClient(main.app)
+ main.app.dependency_overrides.clear()
+
+
+def test_the_endpoint_returns_the_series_for_a_range(db, client):
+ seed(db, datetime.now(timezone.utc) - timedelta(minutes=30), 30)
+
+ response = client.get("/monitoring/servers/1/history?range=1h&metrics=cpu,ram")
+
+ body = response.json()
+ assert response.status_code == 200 and body["server_id"] == 1 and body["range"] == "1h" and body["metrics"] == ["cpu", "ram"]
+ assert len(body["points"]) >= 29 and set(body["points"][0]) == {"time", "samples", "cpu", "ram"}
+
+
+def test_without_a_range_the_original_answer_is_unchanged(db, client):
+ seed(db, datetime.now(timezone.utc) - timedelta(minutes=10), 5)
+
+ body = client.get("/monitoring/servers/1/history?minutes=30").json()
+
+ assert set(body) == {"server_id", "samples"} and set(body["samples"][0]) == {"time", "cpu", "ram", "disk"}
+
+
[email protected]("query", ["range=2h", "range=", "range=ALL", "range=1h&metrics=", "range=1h&metrics=cpu,,ram", "range=1h&metrics=password",
+ "range=1h&metrics=cpu;drop", "range=1h&metrics=" + "cpu," * 100, "range=1h&metrics=cpu,ram_pct"])
+def test_unknown_ranges_and_metrics_are_refused(client, query):
+ assert client.get(f"/monitoring/servers/1/history?{query}").status_code == 422
+
+
+def test_unknown_servers_mounts_and_containers_are_not_found(db, client):
+ seed(db, datetime.now(timezone.utc) - timedelta(minutes=5), 3)
+ assert client.get("/monitoring/servers/99/history?range=1h").status_code == 404
+ assert client.get("/monitoring/servers/1/mounts/history?mount=/data").status_code == 404
+ assert client.get("/monitoring/servers/1/mounts/history?mount=data").status_code == 404 # not an absolute path
+ assert client.get("/monitoring/servers/1/mounts/history").status_code == 422
+ assert client.get("/monitoring/servers/1/containers/nope/history?range=7d").status_code == 404
+ assert client.get("/monitoring/servers/99/containers/nope/history").status_code == 404
+ assert client.get("/monitoring/servers/1/containers/" + "a" * 129 + "/history").status_code == 422
+
+
+def test_the_container_and_mount_endpoints_return_their_series(db, client):
+ when = datetime.now(timezone.utc) - timedelta(minutes=2)
+ db.add(ContainerMetric(server_id=1, time=when, container_id="abc", name="web", status="running", cpu_pct=12.0, memory_used_mb=100.0))
+ db.add(MountMetric(server_id=1, time=when, mount="/data", used_pct=80.0))
+ db.commit()
+
+ container = client.get("/monitoring/servers/1/containers/abc/history?range=1h").json()
+ mount = client.get("/monitoring/servers/1/mounts/history?mount=/data&range=1h&metrics=used_pct").json()
+
+ assert container["name"] == "web" and container["container_id"] == "abc" and container["points"][0]["cpu"]["avg"] == 12.0
+ assert mount["mount"] == "/data" and mount["points"][0]["used_pct"]["avg"] == 80.0
+
+
+def test_signing_in_is_required(db, monkeypatch):
+ main.app.dependency_overrides[main.get_db] = lambda: db
+ try:
+ client = TestClient(main.app)
+ for path in ("/monitoring/servers/1/history?range=1h", "/monitoring/servers/1/mounts/history?mount=/", "/monitoring/servers/1/containers/abc/history"):
+ assert client.get(path).status_code == 401
+ finally:
+ main.app.dependency_overrides.clear()
diff --git a/tests/test_indexes.py b/tests/test_indexes.py
new file mode 100644
index 0000000..caace89
--- /dev/null
+++ b/tests/test_indexes.py
@@ -0,0 +1,37 @@
+import os
+
+os.environ.setdefault("DATABASE_URL", "sqlite://")
+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")
+
+import pytest
+from sqlalchemy import create_engine, inspect
+
+from src.backend.models import Base
+
+METRIC_TABLES = ("host_metrics", "mount_metrics", "container_metrics", "system_events")
+
+
[email protected](scope="module")
+def indexes():
+ engine = create_engine("sqlite://")
+ Base.metadata.create_all(engine)
+ return {table: {index["name"]: index["column_names"] for index in inspect(engine).get_indexes(table)} for table in METRIC_TABLES}
+
+
+def test_the_metrics_tables_carry_exactly_the_indexes_they_need(indexes):
+ assert indexes["host_metrics"] == {"ix_host_metrics_time": ["time"], "ix_host_metrics_server_time": ["server_id", "time"]}
+ assert indexes["mount_metrics"] == {"ix_mount_metrics_time": ["time"], "ix_mount_metrics_server_time": ["server_id", "time"]}
+ assert indexes["container_metrics"] == {"ix_container_metrics_time": ["time"], "ix_container_metrics_server_container_time": ["server_id", "container_id", "time"]}
+ assert indexes["system_events"] == {"ix_system_events_time": ["time"], "ix_system_events_server_time": ["server_id", "time"], "ix_system_events_event_key": ["event_key"]}
+
+
[email protected]("table", METRIC_TABLES)
+def test_deleting_a_server_and_purging_old_rows_can_both_use_an_index(indexes, table):
+ columns = list(indexes[table].values())
+ assert any(index[0] == "server_id" for index in columns), "deleting a server cascades through server_id"
+ assert ["time"] in columns, "the hourly purge deletes by time alone"
+
+
[email protected]("table", METRIC_TABLES)
+def test_a_primary_key_is_never_indexed_a_second_time(indexes, table):
+ assert not any(index == ["id"] for index in indexes[table].values())
diff --git a/tests/test_rollups.py b/tests/test_rollups.py
new file mode 100644
index 0000000..ea682df
--- /dev/null
+++ b/tests/test_rollups.py
@@ -0,0 +1,437 @@
+import os
+from datetime import datetime, timedelta, timezone
+
+os.environ.setdefault("DATABASE_URL", "sqlite://")
+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")
+
+import pytest
+from sqlalchemy import create_engine, func
+from sqlalchemy.orm import sessionmaker
+
+from src.backend import main, rollups
+from src.backend.models import (
+ AppSetting, Base, ContainerMetric, ContainerRollup, HostMetric, HostRollup, MountMetric, MountRollup, Server, User,
+)
+
+
+def T(year, month, day, hour=0, minute=0):
+ return datetime(year, month, day, hour, minute, tzinfo=timezone.utc)
+
+
+def db():
+ engine = create_engine("sqlite://")
+ Base.metadata.create_all(engine)
+ session = sessionmaker(bind=engine)()
+ session.add(User(id=1, name="Owner", email="[email protected]", password_hash="x", role="admin"))
+ session.add_all(Server(id=n, owner_id=1, hostname=f"srv{n}", check_interval_seconds=60) for n in (1, 2))
+ session.commit()
+ return session
+
+
+def ago(days, hour=10):
+ """A moment a few days back, for the tests that use the real clock (maintenance and server deletion)."""
+ return (datetime.now(timezone.utc) - timedelta(days=days)).replace(hour=hour, minute=0, second=0, microsecond=0)
+
+
+def host(db, when, server=1, **fields):
+ db.add(HostMetric(server_id=server, time=when, **fields))
+ db.commit()
+
+
+def catch_up(db, now):
+ """Run the job until it has nothing left to do (a long history takes several chunks)."""
+ for _ in range(100):
+ if not any(rollups.run_rollups(db, now).values()):
+ return
+ raise AssertionError("the job never finished")
+
+
+def hosts(db, tier, server=1):
+ rows = db.query(HostRollup).filter_by(tier=tier, server_id=server).order_by(HostRollup.bucket_start).all()
+ return [(rollups._utc(row.bucket_start), row) for row in rows]
+
+
+# --- bucket arithmetic ---------------------------------------------------------------------
+
+def test_buckets_start_on_utc_boundaries_and_weeks_on_monday():
+ assert rollups.floor(rollups.HOUR, T(2026, 3, 4, 17, 59)) == T(2026, 3, 4, 17)
+ assert rollups.floor(rollups.DAY, T(2026, 3, 4, 17, 59)) == T(2026, 3, 4)
+ sunday = T(2026, 3, 8, 23, 59)
+ assert sunday.weekday() == 6
+ assert rollups.floor(rollups.WEEK, sunday) == T(2026, 3, 2) and T(2026, 3, 2).weekday() == 0
+ assert rollups.floor(rollups.WEEK, T(2026, 3, 9)) == T(2026, 3, 9) # the next Monday starts a new week
+ assert rollups.floor(rollups.MONTH, T(2026, 3, 31, 23)) == T(2026, 3, 1)
+
+
+def test_a_time_with_another_offset_is_converted_to_utc_first():
+ plus_two = datetime(2026, 3, 5, 1, 30, tzinfo=timezone(timedelta(hours=2))) # 23:30 UTC the day before
+ assert rollups.floor(rollups.DAY, plus_two) == T(2026, 3, 4)
+ assert rollups.floor(rollups.DAY, datetime(2026, 3, 4, 12, 0)) == T(2026, 3, 4) # naive counts as UTC
+
+
+def test_the_following_bucket_handles_month_ends_and_leap_years():
+ assert rollups.following(rollups.MONTH, T(2026, 12, 1)) == T(2027, 1, 1)
+ assert rollups.following(rollups.MONTH, T(2028, 2, 1)) == T(2028, 3, 1)
+ assert rollups.following(rollups.WEEK, T(2026, 12, 28)) == T(2027, 1, 4)
+ assert rollups.following(rollups.HOUR, T(2026, 3, 8, 23)) == T(2026, 3, 9)
+
+
+# --- hourly summaries ----------------------------------------------------------------------
+
+def test_an_hour_is_summarised_with_average_maximum_and_count(db):
+ for minute, cpu in ((0, 10.0), (20, 20.0), (40, 60.0)):
+ host(db, T(2026, 3, 4, 10, minute), cpu_utilization_pct=cpu, ram_pct=50.0)
+
+ assert rollups.run_rollups(db, T(2026, 3, 4, 11, 10))[rollups.HOUR] == 1
+
+ [(start, row)] = hosts(db, rollups.HOUR)
+ assert start == T(2026, 3, 4, 10)
+ assert (row.samples, row.avg_cpu, row.max_cpu, row.avg_ram, row.max_ram) == (3, 30.0, 60.0, 50.0, 50.0)
+ assert row.avg_swap is None and row.max_swap is None # never reported stays unknown, not zero
+
+
+def test_a_metric_missing_from_some_samples_is_averaged_over_the_ones_that_have_it(db):
+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0)
+ host(db, T(2026, 3, 4, 10, 1))
+ host(db, T(2026, 3, 4, 10, 2), cpu_utilization_pct=30.0)
+
+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))
+
+ [(_, row)] = hosts(db, rollups.HOUR)
+ assert (row.samples, row.avg_cpu, row.max_cpu) == (3, 20.0, 30.0)
+
+
+def test_a_history_of_nothing_but_unknowns_still_counts_check_ins_and_does_not_break(db):
+ for minute in range(5):
+ host(db, T(2026, 3, 4, 10, minute))
+ db.add(MountMetric(server_id=1, time=T(2026, 3, 4, 10, 0), mount="/"))
+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, 10, 0), container_id="c", name="web", status="running"))
+ db.commit()
+
+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))
+
+ [(_, row)] = hosts(db, rollups.HOUR)
+ assert row.samples == 5 and row.avg_cpu is None and row.max_cpu is None
+ assert db.query(MountRollup).one().avg_used_pct is None
+ container = db.query(ContainerRollup).one()
+ assert (container.samples, container.down_samples, container.avg_cpu, container.max_restarts) == (1, 0, None, 0) # restarts default to 0
+
+
+def test_only_completed_buckets_are_written_after_a_short_grace_period(db):
+ host(db, T(2026, 3, 4, 10, 30), cpu_utilization_pct=10.0)
+
+ assert rollups.run_rollups(db, T(2026, 3, 4, 10, 59))[rollups.HOUR] == 0
+ assert rollups.run_rollups(db, T(2026, 3, 4, 11, 4))[rollups.HOUR] == 0 # ended, but check-ins may still be arriving
+ assert hosts(db, rollups.HOUR) == []
+ assert rollups.run_rollups(db, T(2026, 3, 4, 11, 6))[rollups.HOUR] == 1
+ assert len(hosts(db, rollups.HOUR)) == 1
+
+
+def test_running_twice_changes_nothing(db):
+ for minute in range(0, 60, 10):
+ host(db, T(2026, 3, 4, 10, minute), cpu_utilization_pct=float(minute))
+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, 10, 0), container_id="c", name="web", status="running", cpu_pct=5.0))
+ db.commit()
+ now = T(2026, 3, 6, 0, 30)
+
+ rollups.run_rollups(db, now)
+ first = [(r.tier, r.bucket_start, r.samples, r.avg_cpu) for r in db.query(HostRollup).order_by(HostRollup.id)]
+ again = rollups.run_rollups(db, now)
+
+ assert again == {"hour": 0, "day": 0, "week": 0, "month": 0}
+ assert [(r.tier, r.bucket_start, r.samples, r.avg_cpu) for r in db.query(HostRollup).order_by(HostRollup.id)] == first
+
+
+def test_rebuilding_a_bucket_replaces_it_instead_of_duplicating_it(db):
+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0)
+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))
+ db.query(AppSetting).filter(AppSetting.key == "rollup_done_hour").delete() # forget how far it got
+ host(db, T(2026, 3, 4, 10, 30), cpu_utilization_pct=30.0)
+
+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))
+
+ [(_, row)] = hosts(db, rollups.HOUR)
+ assert (row.samples, row.avg_cpu) == (2, 20.0)
+
+
+def test_servers_mounts_and_containers_are_summarised_separately(db):
+ host(db, T(2026, 3, 4, 10, 0), server=1, cpu_utilization_pct=10.0)
+ host(db, T(2026, 3, 4, 10, 0), server=2, cpu_utilization_pct=90.0)
+ for mount, used in (("/", 40.0), ("/data", 80.0)):
+ db.add(MountMetric(server_id=1, time=T(2026, 3, 4, 10, 0), mount=mount, used_pct=used, used_gb=used, total_gb=100.0))
+ for name, cpu in (("web", 5.0), ("db", 50.0)):
+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, 10, 0), container_id=f"id-{name}", name=name, status="running", cpu_pct=cpu, memory_used_mb=100.0, memory_limit_mb=512.0, restart_count=2))
+ db.commit()
+
+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))
+
+ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR, 1)] == [10.0] and [row.avg_cpu for _, row in hosts(db, rollups.HOUR, 2)] == [90.0]
+ mounts = {row.mount: (row.avg_used_pct, row.max_total_gb) for row in db.query(MountRollup)}
+ assert mounts == {"/": (40.0, 100.0), "/data": (80.0, 100.0)}
+ containers = {row.name: (row.container_id, row.avg_cpu, row.max_memory_limit_mb, row.max_restarts) for row in db.query(ContainerRollup)}
+ assert containers == {"web": ("id-web", 5.0, 512.0, 2), "db": ("id-db", 50.0, 512.0, 2)}
+
+
+def test_down_samples_count_the_check_ins_where_a_container_was_not_running(db):
+ for minute, status in ((0, "running"), (1, "exited"), (2, "exited"), (3, "restarting")):
+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, 10, minute), container_id="c", name="web", status=status))
+ db.commit()
+
+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))
+
+ row = db.query(ContainerRollup).one()
+ assert (row.samples, row.down_samples) == (4, 3)
+
+
+# --- daily, weekly and monthly summaries -----------------------------------------------------
+
+def test_a_day_is_the_samples_weighted_average_of_its_hours_and_keeps_the_maximum(db):
+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=100.0)
+ for minute in range(3):
+ host(db, T(2026, 3, 4, 11, minute), cpu_utilization_pct=0.0)
+
+ rollups.run_rollups(db, T(2026, 3, 5, 0, 10))
+
+ [(start, day)] = hosts(db, rollups.DAY)
+ assert start == T(2026, 3, 4)
+ assert (day.samples, day.avg_cpu, day.max_cpu) == (4, 25.0, 100.0) # 1 sample of 100 and 3 of 0, not (100 + 0) / 2
+
+
+def test_each_average_remembers_how_many_check_ins_reported_it(db):
+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0, ram_pct=40.0)
+ host(db, T(2026, 3, 4, 10, 1), cpu_utilization_pct=20.0)
+ host(db, T(2026, 3, 4, 10, 2))
+
+ rollups.run_rollups(db, T(2026, 3, 5, 0, 10))
+
+ for tier in (rollups.HOUR, rollups.DAY):
+ [(_, row)] = hosts(db, tier)
+ assert (row.samples, row.n_cpu, row.n_ram, row.n_swap) == (3, 2, 1, 0)
+
+
+def test_a_container_that_was_stopped_part_of_the_time_is_averaged_over_the_time_it_ran(db):
+ for hour, status, cpu in ((10, "running", 90.0), (11, "exited", None), (12, "exited", None), (13, "exited", None)):
+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 4, hour), container_id="c", name="web", status=status, cpu_pct=cpu))
+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 5, 10), container_id="c", name="web", status="running", cpu_pct=10.0))
+ db.commit()
+
+ catch_up(db, T(2026, 3, 9, 0, 10))
+
+ day = db.query(ContainerRollup).filter_by(tier="day", bucket_start=T(2026, 3, 4)).one()
+ week = db.query(ContainerRollup).filter_by(tier="week").one()
+ assert (day.samples, day.down_samples, day.avg_cpu, day.n_cpu) == (4, 3, 90.0, 1)
+ assert (week.samples, week.down_samples, week.avg_cpu, week.n_cpu) == (5, 3, 50.0, 2) # (90 + 10) / 2, not (90*4 + 10) / 5
+
+
+def test_hours_that_never_reported_a_metric_do_not_pull_a_day_toward_zero(db):
+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=80.0)
+ host(db, T(2026, 3, 4, 11, 0), docker_latency_ms=50)
+ host(db, T(2026, 3, 4, 11, 1), docker_latency_ms=70)
+
+ rollups.run_rollups(db, T(2026, 3, 5, 0, 10))
+
+ [(_, day)] = hosts(db, rollups.DAY)
+ assert (day.avg_cpu, day.max_cpu, day.avg_docker_ms, day.max_docker_ms, day.samples) == (80.0, 80.0, 60.0, 70, 3)
+
+
+def test_a_week_runs_monday_to_sunday_and_is_written_only_when_sunday_has_ended(db):
+ assert T(2026, 3, 2).weekday() == 0
+ for day in range(2, 9): # Monday 2 March to Sunday 8 March
+ host(db, T(2026, 3, day, 12, 0), cpu_utilization_pct=float(day))
+ host(db, T(2026, 3, 9, 12, 0), cpu_utilization_pct=1000.0) # the following Monday belongs to the next week
+
+ rollups.run_rollups(db, T(2026, 3, 8, 23, 59))
+ assert hosts(db, rollups.WEEK) == [] # still Sunday
+
+ rollups.run_rollups(db, T(2026, 3, 9, 0, 10))
+ [(start, week)] = hosts(db, rollups.WEEK)
+ assert start == T(2026, 3, 2)
+ assert (week.samples, week.avg_cpu, week.max_cpu) == (7, 5.0, 8.0) # days 2..8 average 5, the Monday after is excluded
+
+
+def test_a_month_is_built_from_its_days_including_a_leap_february(db):
+ for day in range(1, 30):
+ host(db, T(2028, 2, day, 12, 0), cpu_utilization_pct=10.0)
+
+ catch_up(db, T(2028, 3, 1, 0, 10))
+
+ [(start, month)] = hosts(db, rollups.MONTH)
+ assert start == T(2028, 2, 1) and (month.samples, month.avg_cpu) == (29, 10.0)
+
+
+def test_data_either_side_of_new_year_lands_in_different_months_and_weeks(db):
+ host(db, T(2026, 12, 31, 12, 0), cpu_utilization_pct=10.0) # a Thursday
+ host(db, T(2027, 1, 1, 12, 0), cpu_utilization_pct=20.0)
+
+ catch_up(db, T(2027, 2, 1, 1, 0))
+
+ assert [(start, row.avg_cpu) for start, row in hosts(db, rollups.MONTH)] == [(T(2026, 12, 1), 10.0), (T(2027, 1, 1), 20.0)]
+ assert [(start, row.avg_cpu) for start, row in hosts(db, rollups.WEEK)] == [(T(2026, 12, 28), 15.0)] # one ISO week spans both
+
+
+def test_every_tier_is_built_for_mounts_and_containers_too(db):
+ for day in (2, 3):
+ db.add(MountMetric(server_id=1, time=T(2026, 3, day, 12), mount="/", used_pct=40.0 + day, used_gb=10.0, total_gb=100.0))
+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, day, 12), container_id="c", name="web", status="running", cpu_pct=float(day), memory_used_mb=100.0))
+ db.add(ContainerMetric(server_id=1, time=T(2026, 3, 3, 13), container_id="c", name="web", status="exited"))
+ db.commit()
+
+ catch_up(db, T(2026, 4, 2))
+
+ for model in (MountRollup, ContainerRollup):
+ assert {row.tier for row in db.query(model)} == {"hour", "day", "week", "month"}
+ week = db.query(ContainerRollup).filter_by(tier="week").one()
+ assert (week.samples, week.down_samples, week.avg_cpu, week.max_cpu, week.avg_memory_mb) == (3, 1, 2.5, 3.0, 100.0)
+ assert db.query(MountRollup).filter_by(tier="week").one().max_used_pct == 43.0
+
+
+# --- gaps, catching up and the watermark -----------------------------------------------------
+
+def test_a_gap_stays_a_gap(db):
+ host(db, T(2026, 3, 4, 10, 0), server=1, cpu_utilization_pct=10.0)
+ host(db, T(2026, 3, 4, 10, 0), server=2, cpu_utilization_pct=20.0)
+ host(db, T(2026, 3, 4, 14, 0), server=1, cpu_utilization_pct=30.0) # server 2 was silent at 14:00 and 11:00-13:00
+ host(db, T(2026, 3, 6, 9, 0), server=1, cpu_utilization_pct=40.0) # nothing at all on the 5th
+
+ rollups.run_rollups(db, T(2026, 3, 7, 0, 10))
+
+ assert [start for start, _ in hosts(db, rollups.HOUR, 1)] == [T(2026, 3, 4, 10), T(2026, 3, 4, 14), T(2026, 3, 6, 9)]
+ assert [start for start, _ in hosts(db, rollups.HOUR, 2)] == [T(2026, 3, 4, 10)]
+ assert [start for start, _ in hosts(db, rollups.DAY, 1)] == [T(2026, 3, 4), T(2026, 3, 6)]
+
+
+def test_the_first_run_summarises_the_history_that_already_exists(db):
+ for day in range(2, 11): # Monday 2 March to Tuesday 10 March
+ for hour in range(0, 24, 6):
+ host(db, T(2026, 3, day, hour, 0), cpu_utilization_pct=float(day))
+
+ result = rollups.run_rollups(db, T(2026, 3, 11, 0, 10))
+
+ assert result[rollups.HOUR] == 216 and result[rollups.DAY] == 9 and result[rollups.WEEK] == 1 # only 2-8 March is a complete week
+ assert len(hosts(db, rollups.HOUR)) == 36 and len(hosts(db, rollups.DAY)) == 9
+
+
+def test_new_samples_are_picked_up_by_the_next_run(db):
+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0)
+ rollups.run_rollups(db, T(2026, 3, 4, 11, 10))
+ host(db, T(2026, 3, 4, 12, 0), cpu_utilization_pct=20.0)
+
+ assert rollups.run_rollups(db, T(2026, 3, 4, 13, 10))[rollups.HOUR] == 2 # the empty 11:00 hour and the 12:00 one
+ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR)] == [10.0, 20.0]
+
+
+def test_after_downtime_the_job_catches_up_a_chunk_at_a_time(db, monkeypatch):
+ monkeypatch.setitem(rollups.MAX_BUCKETS_PER_RUN, rollups.HOUR, 5)
+ for hour in range(12):
+ host(db, T(2026, 3, 4, hour, 0), cpu_utilization_pct=float(hour))
+ now = T(2026, 3, 4, 12, 10)
+
+ built = [rollups.run_rollups(db, now)[rollups.HOUR] for _ in range(4)]
+
+ assert built == [5, 5, 2, 0]
+ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR)] == [float(hour) for hour in range(12)]
+
+
+def test_a_long_empty_stretch_cannot_stall_the_job(db, monkeypatch):
+ monkeypatch.setitem(rollups.MAX_BUCKETS_PER_RUN, rollups.HOUR, 5)
+ host(db, T(2026, 3, 4, 0, 0), cpu_utilization_pct=1.0)
+ host(db, T(2026, 3, 5, 6, 0), cpu_utilization_pct=2.0) # 30 hours later, with nothing between
+
+ for _ in range(10):
+ rollups.run_rollups(db, T(2026, 3, 5, 8, 0))
+
+ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR)] == [1.0, 2.0]
+
+
+def test_the_watermark_is_stored_per_tier_and_a_damaged_one_is_ignored(db):
+ host(db, T(2026, 3, 4, 10, 0), cpu_utilization_pct=10.0)
+ rollups.run_rollups(db, T(2026, 3, 6, 0, 10))
+
+ assert db.get(AppSetting, "rollup_done_hour").value.startswith("2026-03-06T00:00:00")
+ assert db.get(AppSetting, "rollup_done_day").value.startswith("2026-03-06T00:00:00")
+
+ db.get(AppSetting, "rollup_done_hour").value = "not a date"
+ db.commit()
+ rollups.run_rollups(db, T(2026, 3, 6, 0, 10)) # starts over from the first sample, harmlessly
+ assert len(hosts(db, rollups.HOUR)) == 1 and len(hosts(db, rollups.DAY)) == 1
+
+
+# --- purging only what has been summarised ---------------------------------------------------
+
+def _set_retention(db, days):
+ db.add(AppSetting(key=main.METRICS_RETENTION_KEY, value=str(days)))
+ db.commit()
+
+
+def test_without_any_summaries_the_purge_behaves_as_before(db):
+ _set_retention(db, 1)
+ host(db, T(2026, 3, 1, 10, 0), cpu_utilization_pct=1.0)
+ host(db, T(2026, 3, 5, 10, 0), cpu_utilization_pct=2.0)
+
+ main.purge_old_metrics(db, now=T(2026, 3, 5, 12, 0))
+
+ assert db.query(HostMetric).count() == 1
+
+
+def test_samples_that_were_not_summarised_yet_are_never_purged(db):
+ _set_retention(db, 1)
+ for day in range(1, 5):
+ host(db, T(2026, 3, day, 10, 0), cpu_utilization_pct=float(day))
+ db.add(AppSetting(key="rollup_done_hour", value=T(2026, 3, 2, 12).isoformat())) # only the first day and a half are summarised
+ db.commit()
+
+ main.purge_old_metrics(db, now=T(2026, 3, 5, 12, 0)) # retention alone would delete everything before 4 March
+
+ # 1 and 2 March are summarised and old enough; 3 March is old enough but not summarised yet, so it stays
+ assert [rollups._utc(row.time).day for row in db.query(HostMetric).order_by(HostMetric.time)] == [3, 4]
+
+
+def test_maintenance_summarises_before_it_purges(db):
+ _set_retention(db, 1)
+ for days_back, cpu in ((4, 1.0), (3, 2.0)):
+ host(db, ago(days_back), cpu_utilization_pct=cpu)
+ db.add(ContainerMetric(server_id=1, time=ago(4), container_id="c", name="web", status="running"))
+ host(db, datetime.now(timezone.utc) - timedelta(minutes=1), cpu_utilization_pct=99.0) # recent: kept
+ db.commit()
+
+ assert main.metrics_maintenance(db, purge_due=True) is True
+
+ assert [row.cpu_utilization_pct for row in db.query(HostMetric)] == [99.0] and db.query(ContainerMetric).count() == 0
+ assert [row.avg_cpu for _, row in hosts(db, rollups.HOUR)] == [1.0, 2.0] and db.query(ContainerRollup).filter_by(tier="hour").count() == 1
+
+
+def test_a_failing_rollup_job_stops_the_purge(db, monkeypatch):
+ _set_retention(db, 1)
+ host(db, ago(4), cpu_utilization_pct=1.0)
+
+ def broken(*args, **kwargs):
+ raise RuntimeError("database trouble")
+
+ monkeypatch.setattr(rollups, "run_rollups", broken)
+
+ assert main.metrics_maintenance(db, purge_due=True) is False
+ assert db.query(HostMetric).count() == 1
+
+
+def test_between_purges_maintenance_only_summarises(db):
+ host(db, ago(2), cpu_utilization_pct=1.0)
+
+ assert main.metrics_maintenance(db, purge_due=False) is False
+ assert len(hosts(db, rollups.HOUR)) == 1 and db.query(HostMetric).count() == 1
+
+
+def test_deleting_a_server_deletes_its_summaries_but_not_another_servers(db, monkeypatch):
+ monkeypatch.setattr(main, "_require_admin", lambda session_id, db: {"id": 1})
+ for server in (1, 2):
+ host(db, ago(2), server=server, cpu_utilization_pct=10.0)
+ db.add(MountMetric(server_id=server, time=ago(2), mount="/", used_pct=1.0))
+ db.add(ContainerMetric(server_id=server, time=ago(2), container_id="c", name="web", status="running"))
+ db.commit()
+ rollups.run_rollups(db)
+
+ main.delete_monitored_server(1, session_id="s", db=db)
+
+ for model in (HostRollup, MountRollup, ContainerRollup):
+ assert db.query(model).filter_by(server_id=1).count() == 0
+ assert db.query(model).filter_by(server_id=2).count() > 0
diff --git a/tests/test_stats.py b/tests/test_stats.py
new file mode 100644
index 0000000..5aa7fca
--- /dev/null
+++ b/tests/test_stats.py
@@ -0,0 +1,302 @@
+import json
+import os
+from datetime import datetime, timedelta, timezone
+
+os.environ.setdefault("DATABASE_URL", "sqlite://")
+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")
+
+import pytest
+from fastapi.testclient import TestClient
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+from sqlalchemy.pool import StaticPool
+
+from src.backend import main, stats
+from src.backend.models import Base, HostMetric, IncidentAlert, Server, User
+from tests.fakes import FakeRedis
+
+NOW = datetime(2026, 3, 11, 12, 30, tzinfo=timezone.utc) # a Wednesday
+
+
+def db():
+ engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)
+ Base.metadata.create_all(engine)
+ session = sessionmaker(bind=engine)()
+ session.add(User(id=1, name="Owner", email="[email protected]", password_hash="x", role="admin"))
+ session.add_all(Server(id=n, owner_id=1, hostname=f"srv{n}", check_interval_seconds=60) for n in (1, 2))
+ session.commit()
+ return session
+
+
+def incident(db, opened, severity="WARNING", rule="host:disk:/", resolved=None, server=1):
+ db.add(IncidentAlert(server_id=server, severity=severity, title="t", description="d", rule_key=rule, created_at=opened,
+ status="resolved" if resolved else "active", resolved_at=resolved))
+ db.commit()
+
+
+# --- grouping incidents by what they are about ------------------------------------------------------
+
[email protected]("key, label", [
+ ("host:disk:/", "Disk space"), ("host:disk:/media/data", "Disk space"), ("host:memory", "Memory"), ("host:inodes:/", "Inodes"),
+ ("system:oom", "Out-of-memory kills"), ("system:docker_slow", "Docker slow"), ("system:collect_slow", "Collector too slow"),
+ ("server_offline", "Server offline"), ("trend:memory_leak:abc", "Possible memory leaks"), ("network:listener:tcp:22", "Network"),
+ ("container:abc:restart_loop", "Containers"), (None, "Other"), ("", "Other"), ("something:else", "Other"), ("host", "Other"), (":" * 50, "Other"),
+])
+def test_rule_keys_become_readable_groups(key, label):
+ assert stats.incident_kind(key) == label
+
+
+def test_long_or_odd_rule_keys_never_leak_through():
+ assert stats.incident_kind("x" * 5000) == "Other" and stats.incident_kind("host:unknown_metric") == "Other"
+
+
+# --- incident statistics ------------------------------------------------------------------------
+
+def test_incidents_are_counted_per_day_and_severity_for_a_week(db):
+ incident(db, NOW - timedelta(days=1, hours=3), "CRITICAL")
+ incident(db, NOW - timedelta(days=1, hours=1), "WARNING")
+ incident(db, NOW - timedelta(days=1, hours=2), "WARNING", rule="system:oom")
+ incident(db, NOW - timedelta(days=3), "INFO", rule="system:reboot")
+
+ result = stats.incident_stats(db, "7d", NOW)
+
+ assert result["bucket"] == "day" and result["opened"] == 4
+ assert result["by_bucket"] == [
+ {"time": "2026-03-08T00:00:00+00:00", "critical": 0, "warning": 0, "info": 1},
+ {"time": "2026-03-10T00:00:00+00:00", "critical": 1, "warning": 2, "info": 0},
+ ]
+ assert result["by_kind"][0] == {"kind": "Disk space", "count": 2}
+ assert sorted(item["kind"] for item in result["by_kind"]) == ["Disk space", "Out-of-memory kills", "Reboots"]
+
+
+def test_the_bucket_size_follows_the_range(db):
+ incident(db, NOW - timedelta(minutes=30))
+ assert [stats.incident_stats(db, name, NOW)["bucket"] for name in ("1h", "6h", "24h", "7d", "30d", "90d", "1y", "all")] == ["hour", "hour", "hour", "day", "day", "week", "week", "month"]
+ assert [stats.incident_stats(db, "1y", NOW)["by_bucket"][0]["time"]][0] == "2026-03-09T00:00:00+00:00" # weeks start on Monday
+ assert stats.incident_stats(db, "all", NOW)["by_bucket"][0]["time"] == "2026-03-01T00:00:00+00:00"
+
+
+def test_incidents_older_than_the_range_are_left_out(db):
+ incident(db, NOW - timedelta(days=40))
+ incident(db, NOW - timedelta(days=2))
+
+ assert stats.incident_stats(db, "7d", NOW)["opened"] == 1
+ assert stats.incident_stats(db, "90d", NOW)["opened"] == 2
+ assert stats.incident_stats(db, "all", NOW)["opened"] == 2
+
+
+def test_mean_time_to_resolve_uses_only_resolved_incidents(db):
+ incident(db, NOW - timedelta(hours=5), resolved=NOW - timedelta(hours=4)) # 60 minutes
+ incident(db, NOW - timedelta(hours=5), resolved=NOW - timedelta(hours=4, minutes=20)) # 40 minutes
+ incident(db, NOW - timedelta(hours=2)) # still open
+
+ result = stats.incident_stats(db, "24h", NOW)
+
+ assert (result["opened"], result["resolved"], result["mean_minutes_to_resolve"]) == (3, 2, 50.0)
+
+
+def test_open_incidents_are_counted_by_severity_whatever_the_range(db):
+ incident(db, NOW - timedelta(days=200), "CRITICAL")
+ incident(db, NOW - timedelta(days=1), "WARNING")
+ incident(db, NOW - timedelta(days=1), "WARNING", resolved=NOW)
+
+ result = stats.incident_stats(db, "24h", NOW)
+
+ assert result["open_now"] == {"critical": 1, "warning": 1, "info": 0} # the 200-day-old one still counts as open
+ assert result["opened"] == 2 # but only two opened within the day
+
+
+def test_an_install_without_incidents_gives_empty_answers(db):
+ for name in ("1h", "30d", "all"):
+ result = stats.incident_stats(db, name, NOW)
+ assert result["opened"] == 0 and result["by_bucket"] == [] and result["by_kind"] == [] and result["mean_minutes_to_resolve"] is None
+ assert result["open_now"] == {"critical": 0, "warning": 0, "info": 0}
+
+
+def test_an_unknown_severity_is_counted_as_info_not_dropped(db):
+ incident(db, NOW - timedelta(hours=1), "SURPRISE")
+ assert stats.incident_stats(db, "24h", NOW)["by_bucket"][0]["info"] == 1
+
+
+def test_the_amount_of_data_read_is_capped_and_says_so(db, monkeypatch):
+ monkeypatch.setattr(stats, "MAX_INCIDENTS_READ", 3)
+ for hours in range(6):
+ incident(db, NOW - timedelta(hours=hours))
+
+ result = stats.incident_stats(db, "24h", NOW)
+
+ assert result["truncated"] is True and result["opened"] == 3
+
+
+# --- availability ------------------------------------------------------------------------------
+
+def checkins(db, server, start, count, step=timedelta(minutes=1)):
+ db.add_all(HostMetric(server_id=server, time=start + step * i, cpu_utilization_pct=1.0) for i in range(count))
+ db.commit()
+
+
+def test_a_server_that_always_reported_is_fully_available(db):
+ checkins(db, 1, NOW - timedelta(hours=6), 361)
+
+ [first, second] = stats.availability(db, "6h", NOW)
+
+ assert first["server_id"] == 1 and first["percent"] == 100.0 and first["checkins"] == 361
+ assert second["server_id"] == 2 and second["percent"] is None and second["checkins"] == 0 # never reported: unknown, not 0%
+
+
+def test_a_server_that_was_silent_for_half_the_time_is_half_available(db):
+ checkins(db, 1, NOW - timedelta(hours=6), 181)
+ checkins(db, 1, NOW - timedelta(minutes=1), 1) # reported at the start and now, nothing in between
+
+ assert stats.availability(db, "6h", NOW)[0]["percent"] == pytest.approx(51.0, abs=2)
+
+
+def test_a_new_server_is_not_blamed_for_the_time_before_it_existed(db):
+ checkins(db, 1, NOW - timedelta(hours=1), 61)
+
+ assert stats.availability(db, "30d", NOW)[0]["percent"] == 100.0
+
+
+def test_availability_is_exact_even_after_the_history_has_been_summarised(db):
+ from src.backend import rollups
+ checkins(db, 1, NOW - timedelta(days=3, hours=-9, minutes=-17), 3 * 24 * 60 - 9 * 60 - 17) # starts at 09:17 on the first day
+ rollups.run_rollups(db, NOW)
+
+ for name in ("7d", "30d", "all"):
+ assert stats.availability(db, name, NOW)[0]["percent"] == 100.0
+
+
+def test_availability_never_exceeds_one_hundred_percent(db):
+ checkins(db, 1, NOW - timedelta(hours=1), 600, step=timedelta(seconds=6)) # far more often than the 60 s interval
+
+ assert stats.availability(db, "6h", NOW)[0]["percent"] == 100.0
+
+
+def test_availability_works_for_every_range_without_data(db):
+ for name in ("1h", "6h", "24h", "7d", "30d", "90d", "1y", "all"):
+ assert [item["percent"] for item in stats.availability(db, name, NOW)] == [None, None]
+
+
+# --- the endpoint ---------------------------------------------------------------------------------
+
+def client(db, monkeypatch):
+ main.app.dependency_overrides[main.get_db] = lambda: db
+ monkeypatch.setattr(main, "_current_user", lambda session_id, db: {"id": 1, "role": "user"})
+ monkeypatch.setattr(main, "redis_client", FakeRedis())
+ yield TestClient(main.app)
+ main.app.dependency_overrides.clear()
+
+
+def test_the_endpoint_returns_incidents_and_availability(db, client):
+ incident(db, datetime.now(timezone.utc) - timedelta(hours=2), "CRITICAL")
+
+ body = client.get("/stats/fleet?range=24h").json()
+
+ assert set(body) == {"incidents", "availability"} and body["incidents"]["opened"] == 1 and body["incidents"]["open_now"]["critical"] == 1
+ assert [item["hostname"] for item in body["availability"]] == ["srv1", "srv2"]
+
+
+def test_answers_are_cached_for_a_minute_per_range_only(db, client, monkeypatch):
+ calls = []
+ real = stats.fleet_stats
+ monkeypatch.setattr(stats, "fleet_stats", lambda db, range_name: calls.append(range_name) or real(db, range_name))
+
+ client.get("/stats/fleet?range=7d"); client.get("/stats/fleet?range=7d"); client.get("/stats/fleet?range=30d")
+
+ assert calls == ["7d", "30d"]
+ assert sorted(key for key in main.redis_client.keys("*")) == ["homelab:stats:fleet:30d", "homelab:stats:fleet:7d"]
+
+
+def test_a_broken_cache_does_not_break_the_page(db, client, monkeypatch):
+ class Broken:
+ def get(self, *args, **kwargs): raise ConnectionError("redis is down")
+ def set(self, *args, **kwargs): raise ConnectionError("redis is down")
+
+ monkeypatch.setattr(main, "redis_client", Broken())
+
+ assert client.get("/stats/fleet?range=24h").status_code == 200
+
+
[email protected]("query", ["range=2h", "range=", "range=ALL", "range=1h;drop", "range=" + "a" * 5000])
+def test_unknown_ranges_are_refused_and_never_become_cache_keys(db, client, query):
+ assert client.get(f"/stats/fleet?{query}").status_code == 422
+ assert main.redis_client.keys("*") == []
+
+
+def test_signing_in_is_required(db, monkeypatch):
+ main.app.dependency_overrides[main.get_db] = lambda: db
+ monkeypatch.setattr(main, "redis_client", FakeRedis())
+ try:
+ assert TestClient(main.app).get("/stats/fleet?range=24h").status_code == 401
+ finally:
+ main.app.dependency_overrides.clear()
+
+
+# --- when a server is busy: hour of day by weekday ----------------------------------------------------
+
+def server_row(db, server_id=1):
+ return db.get(Server, server_id)
+
+
+def test_the_heatmap_puts_each_hour_in_its_weekday_and_hour_cell(db):
+ monday_ten = datetime(2026, 3, 9, 10, 0, tzinfo=timezone.utc) # a Monday
+ db.add_all(HostMetric(server_id=1, time=monday_ten + timedelta(minutes=minute), cpu_utilization_pct=value) for minute, value in ((0, 20.0), (1, 60.0)))
+ db.add(HostMetric(server_id=1, time=datetime(2026, 3, 8, 23, 30, tzinfo=timezone.utc), cpu_utilization_pct=5.0)) # Sunday 23:30
+ db.commit()
+
+ result = stats.hour_weekday_heatmap(db, server_row(db), "cpu", 0, NOW)
+
+ assert result["cells"][0][10] == 40.0 and result["samples"][0][10] == 2
+ assert result["cells"][6][23] == 5.0
+ assert result["cells"][2][2] is None and len(result["cells"]) == 7 and all(len(row) == 24 for row in result["cells"])
+
+
+def test_the_heatmap_follows_the_viewers_time_zone(db):
+ when = datetime(2026, 3, 8, 23, 30, tzinfo=timezone.utc) # Sunday 23:30 UTC
+ db.add(HostMetric(server_id=1, time=when, cpu_utilization_pct=5.0))
+ db.commit()
+
+ east = stats.hour_weekday_heatmap(db, server_row(db), "cpu", 120, NOW) # UTC+2: already Monday 01:30
+ west = stats.hour_weekday_heatmap(db, server_row(db), "cpu", -300, NOW) # UTC-5: Sunday 18:30
+
+ assert east["cells"][0][1] == 5.0 and west["cells"][6][18] == 5.0
+
+
+def test_the_heatmap_ignores_samples_older_than_thirty_days_and_unknowns(db):
+ db.add(HostMetric(server_id=1, time=NOW - timedelta(days=40), cpu_utilization_pct=99.0))
+ db.add(HostMetric(server_id=1, time=NOW - timedelta(days=2), ram_pct=50.0)) # no CPU reading
+ db.commit()
+
+ assert all(cell is None for row in stats.hour_weekday_heatmap(db, server_row(db), "cpu", 0, NOW)["cells"] for cell in row)
+
+
+def test_the_heatmap_is_the_same_before_and_after_summarising(db):
+ from src.backend import rollups
+ db.add_all(HostMetric(server_id=1, time=NOW - timedelta(days=10) + timedelta(minutes=10 * i), cpu_utilization_pct=float(i % 90)) for i in range(10 * 24 * 6))
+ db.commit()
+ before = stats.hour_weekday_heatmap(db, server_row(db), "cpu", 60, NOW)
+ for _ in range(10):
+ rollups.run_rollups(db, NOW)
+
+ assert stats.hour_weekday_heatmap(db, server_row(db), "cpu", 60, NOW) == before
+
+
+def test_the_heatmap_endpoint_validates_everything_and_caches_per_server_metric_and_offset(db, client):
+ assert client.get("/stats/servers/1/heatmap?metric=cpu&offset=60").status_code == 200
+ assert client.get("/stats/servers/1/heatmap?metric=cpu&offset=60").status_code == 200
+ assert main.redis_client.keys("*") == ["homelab:stats:heatmap:1:cpu:60"]
+ for query in ("metric=disk", "metric=cpu;x", "offset=900", "offset=-900", "offset=abc", "offset=1.5"):
+ assert client.get(f"/stats/servers/1/heatmap?{query}").status_code == 422
+ assert client.get("/stats/servers/999/heatmap").status_code == 404
+ assert main.redis_client.keys("*") == ["homelab:stats:heatmap:1:cpu:60"] # refused requests left nothing behind
+
+
+def test_the_heatmap_endpoint_needs_a_session(db, monkeypatch):
+ main.app.dependency_overrides[main.get_db] = lambda: db
+ monkeypatch.setattr(main, "redis_client", FakeRedis())
+ try:
+ assert TestClient(main.app).get("/stats/servers/1/heatmap").status_code == 401
+ finally:
+ main.app.dependency_overrides.clear()